AI & Answer Engines
llms.txt And AI Crawler Configuration
Give AI systems a clean, deliberate route into your best content.
AI crawlers work with a limited context budget and parse pages far less reliably than a browser renders them. Providing a curated index and clean text endpoints means these systems consume the content you want them to, in a form they can actually use.
You probably need this if
- You have no explicit policy for AI crawlers in robots.txt
- Your key content is buried in heavy templates with poor text-to-markup ratio
- You want AI systems to prioritise documentation or service pages
- AI tools quote navigation or boilerplate instead of your actual content
What you get
- llms.txt file curating your most important content with descriptions
- Explicit robots.txt rules for each major AI crawler, decided deliberately
- Markdown or clean-text endpoints for key pages where valuable
- Token-efficient content structure on priority templates
- Verification that AI crawlers are fetching what you intended
How we approach it
-
Decide the policy
Make an explicit, documented decision on which AI crawlers are welcome, rather than inheriting whatever the previous robots.txt happened to say.
-
Curate the index
Build llms.txt as a hand-picked guide to your genuinely important content, since its value lies in curation rather than completeness.
-
Provide clean text
Offer Markdown or simplified endpoints for priority pages so parsing does not have to fight your templates.
-
Verify in the logs
Confirm from server logs that AI crawlers are actually fetching these resources, because an unread file achieves nothing.
llms.txt Implementation: common questions
Is llms.txt an official standard?
No. It is a community proposal, not an agreed standard, and adoption by AI vendors is inconsistent and unconfirmed. It is cheap to implement and low risk, but treat claims about guaranteed effects with scepticism because the evidence base is still thin.
Does llms.txt replace robots.txt?
No, they do different jobs. Robots.txt controls crawler access and is universally respected. llms.txt is a curated content index intended to help AI systems find your best material. You need robots.txt regardless.
Which AI crawlers should I allow?
GPTBot and OAI-SearchBot for ChatGPT, PerplexityBot for Perplexity, ClaudeBot for Claude, and Google-Extended for Gemini. Most businesses benefit from allowing all of them, since each block removes a potential citation route.
Related services
AI & Answer Engines
Answer Engine Optimisation (AEO)
Be the source AI assistants cite when someone asks about your category.
AI & Answer Engines
ChatGPT Visibility
Be recommended when someone asks ChatGPT for a business like yours.
Technical SEO
Technical SEO
Remove the technical barriers preventing content from ranking at all.
Find out what is actually holding you back
A short conversation and a look at your site is usually enough to tell you where the real problem is. No obligation, and no pressure to sign anything.