Do I need an llms.txt file? What Google, OpenAI and Anthropic say

Quick answer: Probably not. Google says Google Search doesn't use llms.txt, so the file neither helps nor hurts you there. OpenAI and Anthropic document robots.txt controls for their crawlers, not llms.txt. If your site already has one, keep it accurate. If your site is new, spend the time on the checks below.

What is an llms.txt file?

It's a plain text file at the root of a website, like yoursite.com/llms.txt, meant to give AI tools a short summary of the site and links to its most useful pages. It's a community proposal, not a web standard.

What does Google say about llms.txt?

Google's guide to generative AI features in Search lists llms.txt among the things you can ignore. You don't need new machine-readable files, AI text files or special markup to appear in Google Search, including AI Overviews and AI Mode, because Google Search doesn't use them.

The same guide says it's fine to keep an llms.txt file for other services that use one. In Google Search, it won't harm or help your site.

What about OpenAI and Anthropic?

Both companies document their crawlers and how to control them with robots.txt. Neither says its search reads a site's llms.txt file.

• OpenAI's crawler overview describes OAI-SearchBot, which surfaces sites in ChatGPT search, and GPTBot, which relates to model training. Each one is set separately in robots.txt. OpenAI does publish an llms.txt for its own developer docs, but the crawler page doesn't mention reading yours.
• Anthropic's crawler help page lists ClaudeBot for training, Claude-SearchBot for search and Claude-User for pages people ask Claude to open. Blocking Claude-SearchBot or Claude-User can reduce how often your site shows up in Claude's answers.

Nobody can promise you a citation in ChatGPT or Claude. Answers vary from one run to the next, and be wary of anyone who sells a guarantee.

Should I add one anyway?

We don't recommend building one for a new site. The payoff is small, and it can feel like AI search is handled when the basics aren't. If a developer or a past agency already added one, leave it and make sure it still lists pages that exist.

What should I do instead?

These checks matter for Google and for AI search tools. Each takes a few minutes.

Check it: your pages are indexed. Open Google Search Console, choose Pages under Indexing and look at the indexed count. Then paste one important page into the URL inspection bar at the top and confirm it says the URL is on Google.

Check it: you aren't blocking the search bots. Visit yoursite.com/robots.txt in a browser. Look for "Disallow" lines that apply to Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot or Claude-User. Blocking them can keep you out of those products' answers.

Check it: your firewall isn't blocking them either. Some hosts and CDNs (content delivery networks, the services that speed up and protect your site) block bots by default. Ask your host or developer whether these bots are allowed through. OpenAI publishes its bots' IP addresses for this.

Decide on training bots on purpose. Google-Extended, GPTBot and ClaudeBot relate to AI training. Allowing or blocking them is your call, and OpenAI and Anthropic both treat their training and search bots as separate settings.

Check it: your answers are on the page. Pick your five most visited pages. Does each one answer the question in its title within the first few lines, in words a customer would use?

Measure what you can. Google's guide points to the Generative AI performance report in Search Console. In your analytics, look for visits with the source chatgpt.com. Treat both as rough signals, not rankings.

Where to start

If you're not sure whether your site is indexed, crawlable and clear, our free audit checks those basics. If you'd rather hand it off, see our services.

Sources

• Google Search Central: Optimizing your website for generative AI features
• OpenAI: Overview of OpenAI crawlers
• Anthropic: Does Anthropic crawl data from the web, and how can site owners block the crawler?