llms.txt, Schema & the Files AI Looks For

The three gates that come before your content

Across the AI visibility reviews I've run, the pattern that surprised me most was how many businesses had done the hard part and were still invisible. Good websites. Real traffic. Years of keyword work that was genuinely paying off in Google. And when you ask ChatGPT or Perplexity who's best at what they do, they don't come up at all.

The reason is almost boring. AI search reads a website differently than a traditional search crawler does, and before any of your content is even considered, the engine has to clear three gates in order. This is Signal 6 of the 8 Signals, the framework we score every site against — and it's the one that caps all the others.

Miss the first gate and nothing else you do matters. Miss the second and your content exists but can't be read. Miss the third and it can be read but not understood — a machine sees paragraphs where it needed to see a business, a location, a service, an answer.

The good news is that all three are mechanical. There's no writing involved, no interviewing, no strategy session. It's the one category on the scorecard where a competent developer can close the whole gap in a day.

Gate one: are AI crawlers even allowed in?

Your robots.txt file sits at the root of your domain and tells crawlers what they may and may not access. Most business owners have never opened theirs. A fair number of them are blocking AI crawlers right now, and nobody involved ever made that decision — it came with a plugin, a theme, a security setting, or a developer who added a blanket rule in 2023 when the training-data debate was loud.

Go look at yours. Type your domain followed by /robots.txt into a browser. It takes four seconds.

What you're looking for is any rule disallowing the user agents the AI companies publish. The names you'll encounter most often:

• OpenAI — GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI's crawler documentation)
• Anthropic — ClaudeBot, Claude-User, Claude-SearchBot (Anthropic's crawler documentation)
• Perplexity — PerplexityBot, Perplexity-User (Perplexity's bot documentation)
• Google — Google-Extended, separate from Googlebot (Google's common crawlers)
• Apple — Applebot-Extended
• Common Crawl — CCBot, which many models draw from indirectly

There's a distinction here that trips people up, and it's worth getting right. Some of these crawlers gather content that may inform model training. Others fetch a page live, in the moment, because a user asked a question and the assistant went to look. Those are different activities with different implications, and you may reasonably feel differently about them.

What you don't want is to block the second kind by accident while intending to block the first. If somebody asks Claude or ChatGPT for the best provider in your city and the assistant tries to check your site in real time and can't get in, you've been removed from the answer — not because your content lost, but because it was never read.

Vendor documentation for these user agents changes. Before you finalize your rules, check each company's current published list rather than trusting any blog post, including this one.

Gate two: can the page actually render?

Getting in isn't the same as being able to read. If your key content is injected by JavaScript after the page loads, some crawlers will see a nearly empty shell where your services, your proof, and your answers were supposed to be.

Traditional search engines have gotten reasonably good at rendering JavaScript. AI retrieval is less consistent about it, and the failure is silent — nothing errors, nothing warns you, the page simply reads as thinner than it is.

The check is simple. Open your page, view the raw page source, and search it for a sentence you know is on the page. If the sentence isn't in the source, a machine may not be seeing it either. Same for text baked into images: a graphic with your service list in it is, to a crawler, a picture.

Gate three: can a machine tell what it's looking at?

This is the gate that does the most work, and the one most sites handle worst.

Schema markup — specifically JSON-LD, a block of structured data in the page's code, built on the shared vocabulary at schema.org — tells a machine what your page is rather than making it infer. Without it, an engine reads a paragraph containing an address and has to guess whether that's your address, a client's address, or an example. With it, the page states plainly: this is an organization, here is its name, here is its location, here are its services, here is the author of this article, here are the questions this page answers.

For most service businesses, four types cover nearly everything:

• Organization or LocalBusiness — who you are, where you are, how to reach you
• Article — what a given post is, who wrote it, when it was published and last updated
• FAQPage — the questions on the page and their answers, which is what lets an engine lift one cleanly (more on that in the questions your buyers ask AI)
• BreadcrumbList — where the page sits in your site's structure

One rule matters more than the markup itself: schema must match what's visibly on the page. If your FAQ schema contains an answer a human reader can't find, that's not optimization, it's a discrepancy — and inconsistency is precisely what erodes the trust score you're trying to build. Mark up what's there. Don't mark up what you wish were there. Getting the underlying facts consistent in the first place is its own signal — see entity clarity.

Your sitemap is doing less than you think

An accurate XML sitemap still earns its place. It's cheap, it's automatic in most platforms, and it gives a crawler a clean list of what exists.

What it doesn't do is tell anyone what matters. A sitemap listing four hundred URLs treats your cornerstone page and a thank-you page as equals. It's an inventory, not a recommendation. Worth having, worth keeping accurate, not worth expecting much from.

llms.txt: what it is, and an honest read on it

An llms.txt file is a plain Markdown file at the root of your domain — yourdomain.com/llms.txt — that lists, in order of importance, the pages you'd want a language model to read when it's trying to understand your business. The convention was proposed by Jeremy Howard of Answer.AI in September 2024, and the format is deliberately simple: your name as an H1, a one-paragraph plain-language summary as a blockquote, then grouped sections of links with a short description on each.

# StoryWorks

> StoryWorks is a StoryBrand marketing agency in Waukee, Iowa, led by
> Tim Yates, a Certified StoryBrand Guide. We help established service
> businesses get clear, get found, and grow. Our core conviction: good
> marketing and AI search reward the same thing — clarity. We make a
> business the answer AI gives, not just a link it might list.

## Services
- [Services — How We Work](...): The full offering across three stages —
  Get Clear, Get Found, Grow — and why no service is the right service
  until the diagnosis is done.
- [Local SEO Services](...): How a local service business gets found on
  Google and in the map pack.

## Proof & Case Studies
- [Advantage Roofing — 19.2X return on marketing spend](...): How a roofing
  company that was ranking but not converting went from a $2.2M plateau
  to $3.3M after a message rework and website rebuild.

That's an excerpt from the live file on this site — go read the whole thing if you want a working model to copy.

That blockquote at the top is the part worth agonizing over. If a model reads the file at all, that summary is the sentence most likely to come back out when someone asks what your company does. Write it the way you'd want to be described in an answer — plainly, no superlatives, no "leading provider of."

The twenty-minute self-check

Run this on your own site before you hire anybody, including me.

• Open yourdomain.com/robots.txt and look for rules blocking the user agents listed above.
• View the page source on your most important page and search it for a sentence you know is there.
• Check whether any of your key content lives only inside an image.
• Look for a <script type="application/ld+json"> block in your source. If there isn't one, that's your biggest single gap in this category.
• Confirm anything in your schema also appears visibly on the page.
• Open yourdomain.com/sitemap.xml and spot-check that the URLs are current.
• Check whether yourdomain.com/llms.txt exists. If it doesn't, decide whether it's worth twenty minutes to you.

Anything you find here is fixable without touching your design, your copy, or your budget for a rebuild. That's why this is the category I tell people to start with — not because it's the most valuable, but because it's the only one where the gap between knowing and fixing is measured in hours.

What I don't know yet

This is new territory and I'd rather say what's still open than pretend otherwise.

I don't know how much weight each engine puts on schema versus plain readable content. I don't know whether llms.txt gets adopted or quietly fades. I don't know how durable any of this is — the engines change their retrieval behavior without announcing it, and a finding from three months ago may not hold today.

What I can tell you is what's held consistently across the reviews I've run: sites that AI engines can reach, render, and recognize show up in answers, and sites that fail one of those three gates generally don't — no matter how good their content is. That much has been reliable. I'll update this page as the rest becomes clearer.

Frequently asked questions

Related reading

• The 8 Signals — the full framework, all eight scored categories, and what your total means.
• Entity Clarity: How AI Decides Who You Are — Signal 2, and the reason schema and your visible content have to agree.
• How to Write Content AI Can Quote — Signal 4. Technical access gets an engine in; extractability decides whether it quotes you.
• The Questions Your Buyers Are Asking AI — Signal 5, and where your FAQPage schema gets its content.
• Advantage Roofing case study — what this looked like on a real client site.
• What Is a Total Online Presence? — why your website, your Google Business Profile, and AI search work as one system.

This is Signal 6 of 8. See the full framework →