Somewhere in your site’s root directory sits a two-kilobyte text file almost nobody reads, and it might be quietly telling ChatGPT, Claude, and every other AI answer engine to stay away from your content. Not because you decided that. Because a security plugin, a CDN default, or a robots.txt template you copied years ago made the call for you.
Meanwhile, the way people find businesses is splitting in two. Google search still matters, but a growing slice of queries now get answered directly inside ChatGPT, Perplexity, and Google’s AI Overviews, with a citation link (or none at all) standing in for the ten blue links you used to compete for. If your site is invisible to the crawlers that feed those answers, you don’t lose a ranking. You lose the mention entirely.
This is a practical guide to exactly which bots are crawling your site, what each one does with your content, and how to configure robots.txt (and the WordPress settings that quietly override it) so you’re making a real choice instead of an accidental one.
What Are AI Crawlers, and Why Do They Matter Now?
The New Traffic Source: AI Answer Engines
A crawler is just software that requests your pages, one URL at a time, so a company can build a product out of what it finds. Googlebot has done this for two decades to build the search index. In the last few years, OpenAI, Anthropic, Google, and others have shipped a second wave of crawlers whose job is different: instead of (or in addition to) building a search index, they collect text to train AI models, or they fetch a page live so a chatbot can answer a user’s question with your content as the source.
That second use case, live retrieval for an AI answer, is the one that matters most for a small business today. When someone asks ChatGPT or Perplexity “who does WordPress development near me” or “what’s the best way to add schema markup to a site,” the assistant’s answer engine dispatches a crawler in real time, reads a handful of candidate pages, and synthesizes a response, sometimes with a citation link, sometimes without. If your robots.txt disallows that bot, you were never a candidate.
Training vs. Search: Two Very Different Bots
Every major AI lab now ships separate bots for separate jobs, and conflating them is the single most common mistake site owners make. A bot that scrapes your site to train a future model is doing something very different from a bot that fetches your page right now because a user asked a live question. You can, and often should, treat them differently: allow the one that can send you a citation and a click, and block the one that only feeds a training dataset you’ll never see credit for.
A blocked crawler doesn’t show up as an error in Google Search Console or a dip in your analytics. It shows up as nothing: a channel that quietly never happens. Most site owners who are accidentally blocking AI answer engines have no idea, because there’s no dashboard that tells you what you’re missing.
Meet the Bots: A Field Guide to Who’s Crawling Your Site
OpenAI’s Three Bots
OpenAI publishes documentation for three distinct crawlers, each controllable independently in robots.txt. OAI-SearchBot surfaces your pages inside ChatGPT’s search feature; disallowing it means you can’t appear in ChatGPT search answers, though you can still show up as a navigational link. GPTBot is the training crawler, used to collect content for OpenAI’s generative model training; disallowing it opts your future content out of that training set without affecting search visibility. ChatGPT-User fires when a live user asks ChatGPT a question that requires fetching a specific page, and OpenAI notes it isn’t used for automatic crawling and isn’t the one to manage for search visibility, that’s OAI-SearchBot’s job.
Anthropic’s Three Bots
Anthropic runs a parallel structure. ClaudeBot collects web content that may contribute to training Claude models. Claude-User fires when someone asks Claude a question and the assistant needs to fetch a specific page in response. Claude-SearchBot crawls to improve the quality and relevance of Claude’s search results. Anthropic’s own documentation states plainly that disabling Claude-User “may reduce your site’s visibility for user-directed web search,” the clearest signal from any lab that blocking the wrong bot has a real visibility cost.
Google’s Split Identity: Googlebot vs. Google-Extended
This is the pairing that trips up the most site owners. Googlebot is the crawler that has always powered classic Google Search, and you almost never want to block it. Google-Extended is a separate, newer control, announced by Google in 2023 (originally covering Bard, now Gemini, and Vertex AI), that lets you decide independently whether your content can be used to improve Google’s generative AI models, without touching your regular search ranking at all. Because it’s a distinct user-agent token, you can disallow Google-Extended while leaving Googlebot completely untouched, opting out of AI training without opting out of search.
The Bots That Don’t Ask Permission
Not every AI company plays by the same rules. Cloudflare’s threat research team documented Perplexity using undeclared, stealth crawlers that ignored explicit robots.txt disallow rules and rotated through unofficial IP ranges and shifting ASNs, along with spoofed user agents, to keep accessing sites that had blocked its declared bot. robots.txt is a voluntary honor system defined by the Robots Exclusion Protocol (formalized in 2022 as RFC 9309), and a well-behaved crawler respects it because the company operating it says it will, not because there’s a technical lock. For bots that don’t comply, robots.txt alone isn’t enough; a firewall-level tool like Cloudflare’s bot management earns its keep there.
How robots.txt Actually Works
The Syntax You Actually Need
A robots.txt file is a plain text file at the root of your domain, and its whole vocabulary fits in a few lines. Each rule targets a specific User-agent token and applies a Disallow (or Allow) path. To block a single bot from your entire site:
User-agent: GPTBot
Disallow: /To allow that same bot everywhere except a private directory, use Allow alongside a narrower Disallow. And because AI crawlers can be aggressive, most labs also support the non-standard but widely honored Crawl-delay directive to slow a bot down without blocking it entirely:
User-agent: ClaudeBot
Crawl-delay: 1Rules are matched by user-agent string, so you write one block per bot you want to treat differently, stacking as many User-agent/Disallow pairs as you need in the same file.
What robots.txt Can’t Do
robots.txt controls crawling, not indexing, and it definitely doesn’t control what a well-behaved AI has already learned from a page it crawled before you added the rule. It also can’t retroactively pull your content out of a model that already trained on it, and, as the Perplexity example shows, it can’t force a non-compliant bot to obey. Think of it as a clearly posted sign, not a locked gate; it’s necessary, but for the crawlers that ignore signs, you need a second layer.
OAI-SearchBot ChatGPT search results GPTBot OpenAI model training ChatGPT-User Live ChatGPT user fetch ClaudeBot Claude model training Claude-User Live Claude user fetch Claude-SearchBot Claude search quality Googlebot Classic Google Search Google-Extended Gemini / Vertex AI training
Step-by-Step: Auditing and Configuring AI Crawler Access on WordPress
Step 1: Read Your Current robots.txt
Visit yourdomain.com/robots.txt in a browser right now. WordPress generates a virtual one by default if you haven’t created a physical file, but plenty of SEO and security plugins overwrite it. Look for any line disallowing / under a bot token you don’t recognize, and check whether it’s more restrictive than you intended.
Step 2: Decide Bot-by-Bot, Not All-or-Nothing
Resist the urge to either block everything labeled “AI” or allow everything indiscriminately. A defensible starting position for most small businesses: allow the search-oriented bots (OAI-SearchBot, Claude-SearchBot, Claude-User, ChatGPT-User, Googlebot) so you’re eligible to be cited when someone asks an AI assistant a relevant question, and make an explicit, informed choice about the training-oriented bots (GPTBot, ClaudeBot, Google-Extended) based on how you feel about your published content contributing to a model you don’t control.
Step 3: Watch for Silent Blockers
The rule you never wrote is usually the one causing the damage. Three places AI crawler access gets blocked without anyone intending it:
- Security and firewall plugins that ship a “block bad bots” list which lumps GPTBot in with actual scraper malware
- CDN and edge providers (Cloudflare included) that now offer a one-click “Block AI bots” toggle in their dashboard, bundling training and search crawlers into a single switch
- Caching or “AI content protection” plugins installed for an unrelated reason that quietly append their own robots.txt rules on activation
After any plugin install or CDN setting change, re-check /robots.txt. It takes ten seconds and it’s the single most common cause of “why isn’t ChatGPT ever citing us” support tickets.
Step 4: Verify With Real Traffic
Confirming a bot is actually respecting your rules (and that it’s genuinely who it claims to be) means checking your server logs for the relevant user-agent strings and cross-referencing hits against the IP ranges each company publishes (OpenAI publishes openai.com/gptbot.json and openai.com/searchbot.json; Google documents its verification method in Search Central). Google’s own robots.txt report in Search Console is the simplest starting point if you only check one thing, since it flags syntax errors and shows which rule is blocking which URL for Googlebot specifically.
Beyond robots.txt: Getting Cited Correctly, Not Just Crawled
Structured Data Is How AI Understands What It’s Reading
Letting a bot through the door only helps if what it finds inside is easy to understand. An AI crawler that reaches your page still has to parse messy HTML to figure out what it actually is: an article, a product, a local business, an FAQ. Google’s documentation on structured data explains that schema.org markup gives search and AI systems an explicit, machine-readable answer instead of forcing them to infer one, which is why it’s one of the highest-leverage additions you can make to a site that wants to be cited accurately rather than paraphrased or misrepresented.
The llms.txt Experiment
A newer, unofficial proposal called llms.txt suggests publishing a plain-Markdown summary of your site at /llms.txt specifically for language models to read instead of parsing full HTML pages. It’s not a recognized web standard the way robots.txt is (no major AI lab has committed to consuming it universally as of this writing), so treat it as a low-cost experiment worth trying, not a must-do that should distract from clean structured data and a sane robots.txt.
Where amplifi.schema Fits In
Once you’ve decided which bots are welcome, the next problem is making sure what they read is accurate. This is the exact gap we built amplifi.schema to close: a free, open-source WordPress plugin that uses Claude to generate, validate, and deploy correct schema.org JSON-LD across an entire site, so the AI crawlers you’ve just allowed through the door have a clean, structured answer waiting for them instead of a guess.
amplifi.schema is MIT licensed and free to use, part of the combined amplifi.plugins suite. If your site is letting AI crawlers in but giving them nothing structured to work with, this closes that gap in minutes.
View on GitHubFrequently Asked Questions
Check your robots.txt today, it takes ten seconds, and it might be the reason ChatGPT has never once mentioned your business.
Built by amplifi.studio — see also amplifi.schema: AI-generated schema.org JSON-LD for your whole site.