Should You Let AI Bots Crawl Your Website? robots.txt, llms.txt and AI Search Explained
By Apex Lab Team · · 5 min read
If you want customers to find your business through ChatGPT, Perplexity or Google's AI, you should allow the AI search crawlers. You can block the training crawlers separately if you don't want your content used to train AI models, and doing so won't remove you from AI search. The mistake many sites make is blocking every bot with "AI" in its name, which quietly hides them from AI answers too.
Key Takeaways
- Each major AI company runs separate bots for search, training and user-requested visits, and each can be allowed or blocked independently in robots.txt.
- Blocking OpenAI's GPTBot only opts you out of model training. ChatGPT search visibility is controlled by OAI-SearchBot.
- Google says blocking Google-Extended does not affect inclusion or ranking in Google Search.
- llms.txt is an optional proposal that some AI tools read. Google says it isn't needed for AI Overviews or AI Mode.
The three kinds of AI bots
Search crawlers fetch pages so an AI tool can find and cite them in answers. Blocking them means your site is unlikely to appear as a source.
Training crawlers collect content to train future AI models. Blocking them has no direct effect on whether you're cited in search answers today.
User-triggered fetchers visit a page because a person asked the AI to look at it, for example by pasting your link into ChatGPT. Because a human started the request, these may not follow robots.txt rules.
The main AI bots and what they do
| Bot | Company | Purpose | Block it if... |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces sites in ChatGPT search | You don't want to appear in ChatGPT search answers |
| GPTBot | OpenAI | Collects content for model training | You don't want your content used for training |
| ChatGPT-User | OpenAI | Visits pages when a user asks | Rarely; robots.txt may not apply |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity answers | You don't want to appear in Perplexity |
| Perplexity-User | Perplexity | Visits pages when a user asks | Rarely; generally ignores robots.txt |
| Googlebot | Google Search, including AI Overviews and AI Mode | Never, unless you want out of Google entirely | |
| Google-Extended | Controls use for Gemini training and grounding | You don't want your content used to train Gemini |
Sources: OpenAI crawler documentation, Perplexity crawler documentation, Google common crawlers.
Two details are worth knowing. OpenAI says its settings are independent: you can allow OAI-SearchBot and block GPTBot, and you'll still appear in ChatGPT search. Changes take about 24 hours to apply. And Google-Extended is not a separate crawler at all. It's a token in robots.txt that tells Google how it may use pages Googlebot has already crawled. Google states it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal."
Example robots.txt setups
Most businesses: be visible everywhere. If your goal is more enquiries, the simplest setup is to allow everything except private areas:
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xml
Visible in AI search, opted out of training. Allow the search bots and block the training ones:
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
Remember that robots.txt is a request, not a lock. Reputable AI companies follow it, but it isn't a security control. Anything that must stay private needs a login.
Check your hosting and firewall too
A correct robots.txt isn't enough if something else blocks the bots first. Some hosting providers, CDNs and security plugins now block AI crawlers by default, or challenge them with bot checks they can't pass. Google's guidance for AI features specifically mentions making sure crawling is allowed "by any CDN or hosting provider" (Google Search Central). If you use Cloudflare or a WordPress security plugin, check its bot settings.
What about llms.txt?
llms.txt is a proposal from Jeremy Howard, first published in September 2024 (llmstxt.org). It's a Markdown file at the root of your site (for example example.com/llms.txt) that gives AI tools a short summary of who you are and links to your most important pages. The format is simple: a heading with your name, a one-line summary, then sections of links.
Should you add one? It's cheap to do and some AI tools and coding assistants read it. But keep expectations realistic: Google says you don't need AI text files to appear in AI Overviews or AI Mode, and there's no public evidence that major AI search engines rank sites higher for having one. Treat it as a tidy summary for machines, not a ranking factor.
On our own site, we allow all major AI search and training crawlers in robots.txt and publish an llms.txt that lists our services, packages, case studies and FAQs, generated from the same data as the website so it never goes out of date.
A five-minute check for your site
- Open
yourdomain.com/robots.txtand look forDisallow: /under any of the bots in the table above. - Search the file for
User-agent: *followed byDisallow: /, which blocks everything. - Check your CDN or security plugin's bot settings for an "AI crawlers" or "AI scrapers" toggle.
- In Google Search Console, use URL Inspection on a key page to confirm Googlebot can fetch it.
- Ask ChatGPT and Perplexity about your business by name and see whether they can find your site.
Our free website audit also flags crawl blocks that stop search engines reading your pages. For a broader view of getting cited in AI answers, read What is AEO?
Frequently asked questions
Does blocking GPTBot remove my website from ChatGPT?
No. OpenAI says GPTBot only collects content for model training. Whether your site appears in ChatGPT search is controlled by a separate crawler, OAI-SearchBot, and each setting is independent.
Does blocking Google-Extended affect my Google rankings?
No. Google states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. It controls whether your content is used to train Gemini models and for grounding in some other Google AI products.
Do I need an llms.txt file?
It's optional. llms.txt is a proposed standard that some AI tools read, and it's quick to create. Google says no AI text files are needed to appear in AI Overviews or AI Mode, so treat it as a helpful summary rather than a ranking factor.
Can robots.txt stop AI tools from reading my content?
Reputable AI companies follow robots.txt for their crawlers, but it is a request, not a security measure. User-triggered fetchers may not follow it, so anything that must stay private should sit behind a login.