llms.txt & AI Instructions Pages: Make Your Site Readable to AI

The machine-readability layer of AI visibility comes down to four tiers: allow the right crawlers (load-bearing), publish an llms.txt file (cheap, promising), maintain a canonical AI-instructions page (underrated), and keep your schema markup (don't bet on it). Most of GEO is content work; this layer sits underneath it, costs an afternoon, and getting it wrong can silently zero out everything else.

Tier 1: Crawler access (errors here nullify everything)

Generative engines can only cite what they can fetch. Each provider documents its crawlers; your robots.txt and CDN decide their access:

• OpenAI (docs): GPTBot (training), OAI-SearchBot (powers ChatGPT search — blocking it removes you from those answers), ChatGPT-User (user-initiated fetches; robots.txt may not apply), OAI-AdsBot (ad pages).
• Perplexity (docs): PerplexityBot (index, not training) and Perplexity-User (live fetches; generally ignores robots.txt).
• Anthropic (docs): ClaudeBot (training), Claude-SearchBot, Claude-User — all respect robots.txt.
• Google (docs): Google-Extended controls Gemini training and grounding only. It does not control AI Overviews — those are built on Google's regular index, so you can't block them without leaving Google Search.

The pattern: training crawlers and user-triggered agents are separate, and the user-triggered ones fetch pages for live, cited answers. Blocking "the AI training bot" while accidentally blocking those is the classic silent failure — and blocking became widespread early: 35.7% of the top 1,000 websites blocked GPTBot by August 2024, per Originality.ai's tracking, and many of those blocks were never revisited.

Check three things today: your robots.txt, your CDN's "AI bots" setting (it can block at the firewall, invisibly to robots.txt), and that your pages render as real HTML — curl a key page and search the raw output for your copy. If the text only appears after JavaScript runs, assume retrieval sees an empty shell.

Tier 2: llms.txt (cheap, promising, unproven)

llms.txt is a proposed standard: a plain-markdown file at yourdomain.com/llms.txt giving language models a curated summary of your site — a sitemap for meaning rather than URLs.

Status, honestly: adoption among AI providers is uneven and mostly undocumented, and no published measurement shows llms.txt alone increases citations. Publish one anyway: it costs twenty minutes, the only failure mode is letting it go stale, and writing "the five facts an AI should know about us" is a useful forcing function that feeds Tier 3. Keep it short, factual, and boring — no adjectives an AI would have to take on faith. (Ours is public; copy the structure.)

Tier 3: An AI-instructions page (the underrated one)

When an AI describes your company, it synthesizes whatever retrieval surfaces: a stale review, a Reddit thread, a competitor's comparison. You can't stop it from describing you — you can only compete to be its best source about yourself.

An AI-instructions page (ours) is an "about page for machines," written to be that source:

• What you are, in one liftable sentence
• Key facts: pricing, scope, dates, contact — the details models get wrong when guessing
• What you are not: categories you get lumped into, similarly-named companies you get confused with (we share a near-name with an unrelated company; our disambiguation section exists because AI answers conflated us)
• Known limitations — our bet, admittedly unmeasured, is that a source stating its own limits reads as more credible

This extrapolates from the best-studied content finding — specific, verifiable, sourced pages win more of the answer — applied to the one query you should win 100% of the time: "what is [your company]?" Ask each engine that question right now; if the answer hedges or errs, this page is your fix.

Tier 4: Structured data (keep it, don't bet on it)

Schema markup has well-documented value in classic search; for generative engines, no rigorous study isolates its effect. Keep it because it's cheap, it feeds the search infrastructure AI retrieval runs through, and consistent machine-readable facts make you an easier entity to describe confidently. Just don't buy it as a GEO silver bullet.

The one-afternoon checklist

1. robots.txt + CDN: AI crawlers allowed, or knowingly blocked — 15 min
2. curl top pages; confirm content in raw HTML — 15 min
3. Publish llms.txt — 30 min
4. Draft /ai-instructions — 90 min
5. Verify Organization + Product schema — 30 min
6. Ask each engine "what is [your company]?", log it, re-test in a month — 10 min

None of this replaces citable content — that work comes first. But it's the cheapest insurance in the discipline: an afternoon making sure you haven't silently opted out.

Read next: How to get cited by ChatGPT · How to check what ChatGPT says about you