A while back we picked up a site whose staging copy was turning up in Google alongside the real thing. The staging server carried a blanket Disallow: / in robots.txt, somebody called that handled, and moved on. Google had the staging URLs in its index anyway, showing them with that grey little “No information is available for this page” line underneath. Nobody had told Google not to list those URLs. They had only told it not to read them, which is a different instruction entirely.
That failure mode isn’t rare, and it isn’t really anybody’s fault. WordPress gives you one checkbox and a vague label. The web gives you two completely different mechanisms that sound identical when you say them out loud. And most of the advice you’ll find conflates them, which is how you end up confidently blocking a page in a way that guarantees Google can never read the instruction telling it to drop the page.
So here’s the whole picture: what robots.txt is actually for, what noindex is actually for, exactly what WordPress emits on your behalf, and the order you have to do things in when a page is already indexed and you need it gone.
Crawling and Indexing Are Two Different Things
Every mistake in this article comes from one collapsed distinction. Crawling is a search engine fetching your URL. Indexing is a search engine deciding to store that URL and show it to people. They’re separate steps, and you control them with separate tools that live in separate places.
robots.txt is a traffic-management file, not a privacy control
Google’s own documentation opens by ruling out the use case most people reach for it. The introduction to robots.txt states that the file “is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.” That’s the first paragraph on the page. The very next sentence tells you what to use instead: “To keep a web page out of Google, block indexing with noindex or password-protect the page.”
The file lives at the root of your domain, it’s plain text, and it addresses crawlers by user agent. It’s genuinely useful for what it’s designed to do — stopping a crawler from hammering a faceted-search URL space with a few million filter combinations, keeping bots out of admin endpoints, telling them not to bother with resource files that don’t affect how a page renders. Those are crawl-budget and server-load problems, and robots.txt is the right answer to them.
A disallowed URL can still show up in search results
This is the part that surprises people, and Google spells it out on that same page: “A page that’s disallowed in robots.txt can still be indexed if linked to from other sites.” The mechanism is straightforward once you see it. Google won’t crawl the page, so it never reads the content. But if other pages link to that URL, Google knows the URL exists, and it can index the address itself. The docs continue: “the URL address and, potentially, other publicly available information such as anchor text in links to the page can still appear in Google Search results.”
That’s the grey “No information is available for this page” result. It’s not a bug. It’s a URL Google has been told not to read but was never told not to list. Blocking crawling is what created the useless-looking result, because a crawler that can’t fetch the page also can’t fetch anything that would let it write a better one.
The two tools actively interfere with each other. Google’s noindex documentation carries an explicit warning: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.” Block a page in robots.txt and then add noindex to it, and you’ve built a page that just sits there in the index, because the crawler can’t reach the instruction telling it to leave.
What WordPress Actually Emits on Your Behalf
Before you change anything, it helps to know what’s already there. WordPress has shipped a real robots API since version 5.7, and it isn’t documented anywhere a normal site owner would look.
The wp_robots API, added in WordPress 5.7
Core hooks a function called wp_robots() onto wp_head at priority 1, so the robots meta tag is one of the first things in your <head>. It builds an associative array of directives, runs it through the wp_robots filter, and prints a single consolidated tag. If the array comes back empty, it prints nothing at all — which is why a normal published post on a public site has no robots meta tag beyond the image-preview directive.
You can read the whole thing in wp-includes/robots-template.php. Core registers four callbacks on that filter unconditionally, in wp-includes/default-filters.php:
wp_robots_noindex()— fires when theblog_publicoption is off. It hands off towp_robots_no_robots(), which sets noindex plus nofollow.wp_robots_noindex_embeds()— adds noindex to oEmbed iframe endpoints.wp_robots_noindex_search()— adds noindex to internal search results pages.wp_robots_max_image_preview_large()— addsmax-image-preview:largewhen the site is public.
That file defines seven functions in total. One of them, wp_robots_no_robots(), is the shared primitive the others delegate to. Another, wp_robots_sensitive_page(), sets both noindex and noarchive and isn’t registered in default-filters.php at all — core adds it inside login_header() in wp-login.php, so it covers the login and registration screens. It’s there for you to reuse on anything genuinely sensitive. There’s also a conditional registration in default-filters.php that applies wp_robots_no_robots() to comment-reply links (?replytocom=) and unapproved-comment previews.
What the “Discourage search engines” checkbox does now
Settings › Reading has one checkbox, and it flips a single option: blog_public. Its behavior changed meaningfully in WordPress 5.3, and the changelog on do_robots() says so directly — 5.3.0 removed the Disallow: / output “in favor of robots meta HTML tag via wp_robots_no_robots() filter callback.”
That change was correct, and for exactly the reason this article exists. The old behavior blocked crawling site-wide, which meant crawlers couldn’t read the noindex either — the precise trap Google warns about. Modern WordPress leaves robots.txt permissive and emits <meta name='robots' content='noindex, nofollow' /> on every page that runs wp_head() instead, so the instruction is actually readable.
Two things still worth knowing. It’s site-wide, so it’s a staging-and-development switch, not a way to hide one page. And like every robots directive, it’s a request that compliant crawlers honor — it isn’t access control. If a staging site genuinely must not be seen, put HTTP authentication in front of it. The password is what makes it private; the meta tag is advisory.
Your robots.txt probably isn’t a file
Unless somebody put one on disk, WordPress generates /robots.txt on the fly through do_robots(), which has been in core since 2.1. On a standard root install the default output is three lines:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.phpThat’s it. No sitemap line by default, no blanket blocks, nothing clever. Those admin paths are derived at runtime rather than hardcoded, so a subdirectory install produces something like Disallow: /blog/wp-admin/ instead. The robots_txt filter (since 3.0) is how plugins append to it — which is why installing an SEO plugin often makes a sitemap reference appear without you touching anything. If a real robots.txt exists in your web root, most server configurations will serve that file directly and WordPress never gets asked, so an editing plugin that seems to do nothing is usually being overruled by a leftover file on disk. Check by curling the URL and comparing it to what the admin screen claims.
There’s no “hide this page from search engines” box in the WordPress editor, and there never has been. Core’s only indexing switch is the site-wide one in Settings › Reading. Per-page noindex comes from an SEO plugin (Yoast, Rank Math, SEOPress, AIOSEO) or from your own code on the wp_robots filter. If you’ve been looking for that checkbox in the editor sidebar and not finding it, that’s why.
How to Actually Keep a Page Out of Google
Pick the tool that matches the outcome you want. There are really only three situations.
You want it gone from search: use noindex, and let the crawler in
Google’s block-indexing documentation is unusually strong here: when Googlebot crawls the page and extracts the tag or header, “Google will drop that page entirely from Google Search results, regardless of whether other sites link to it.” No hedging about links or authority. The tag wins.
The markup goes in the <head>:
<meta name="robots" content="noindex">Use noindex, nofollow if you also don’t want link signals passed onward, though on a page you’re removing that’s usually a detail rather than a decision. The non-negotiable part is that the URL must stay crawlable. Don’t disallow it. Don’t put it behind a login. Don’t return a 403 to Googlebot and wonder why nothing happens.
In WordPress, this is a short filter callback if you’d rather not add a plugin for it:
add_filter( 'wp_robots', function ( $robots ) {
if ( is_page( 'thank-you' ) ) {
$robots['noindex'] = true;
}
return $robots;
} );Put that in a small site-specific plugin rather than your theme’s functions.php. Theme code goes away when you switch themes, and losing your indexing rules during a redesign is a genuinely annoying way to find out.
It’s a PDF or another non-HTML file: use X-Robots-Tag
You can’t put a meta tag in a PDF, a CSV, or an image. Google supports the same directives as an HTTP response header called X-Robots-Tag, and per its robots meta tag documentation, headers can be combined or given as a comma-separated list, optionally scoped to a named user agent. A response header looks like this:
X-Robots-Tag: noindexSet it in your server config for a path or file type. Same crawlability rule applies — the crawler has to be allowed to make the request in order to see the header in the response.
That same doc lists a few directives worth knowing beyond noindex. max-snippet:[number] caps the characters used in your snippet. unavailable_after takes a date, after which the page stops being shown. indexifembedded lets Google index content that’s embedded in another page via an iframe despite a noindex rule, and only does anything when it’s paired with noindex. One rule to remember when you start combining them: “In the case of conflicting robots rules, the more restrictive rule applies.”
You want to stop the crawling itself: that’s robots.txt
There is a legitimate job here. If a crawler is generating real load against endless URL permutations, or you want to keep bots off resource files, robots.txt is the correct instrument — Google’s docs describe using it “to manage crawling traffic if you think your server will be overwhelmed by requests” and to avoid crawling “unimportant or similar pages on your site.”
Media files are the one case where robots.txt does prevent appearing in results: it can keep images, video, and audio out of Google Search. Even there the docs add a caveat — it “won’t prevent other pages or users from linking to your image, video, or audio file.”
One more thing not to try. Some older guides suggest a Noindex: line inside robots.txt. Google’s documentation is explicit: “Specifying the noindex rule in the robots.txt file is not supported by Google.” It does nothing.
Remove a page from search results → noindex meta tag, page stays crawlable.
Remove a PDF or image from results → X-Robots-Tag: noindex response header.
Reduce crawler load on junk URLs → robots.txt disallow.
Hide an entire staging site → HTTP authentication, with the Reading checkbox as backup.
Genuinely private content → a real login, never a robots directive.
Removing a Page That’s Already Indexed
Sequence matters more than anything else in this article. Doing the right two steps in the wrong order produces a page that stays in the index indefinitely.
Unblock first, block later. If the URL is currently disallowed in robots.txt, remove that rule before you do anything else. Then add noindex, and confirm it’s actually present in the HTML that a crawler receives — not in the browser after JavaScript runs, but in the raw response. A quick curl against the URL and a grep for robots settles it in seconds. Then wait, and set your expectations from the right number. Google has to re-crawl the page before it can act on the tag, and the block-indexing docs are blunt about the unassisted timeline: “Depending on the importance of the page on the internet, it may take months for Googlebot to revisit a page.” Requesting a crawl through the URL Inspection tool is the documented way to speed that up, and Google puts the wait after a request at “a few days to a few weeks” — though it is explicit that a crawl request is not a promise of action either way. Once the URL has dropped out of results, you can add a robots.txt disallow if crawl load was your original reason for wanting one, though for a single page it rarely is.
If the page is urgent — a leaked document, wrong pricing, someone’s personal information — use the Removals tool in Google Search Console alongside the noindex. It’s temporary by design and buys you roughly six months of suppression, which is time to let the permanent fix take effect. It is not the permanent fix.
Four mistakes that keep showing up
Blocking and noindexing the same URL is the big one, covered above. The other three are quieter.
Launching with the Reading checkbox still ticked happens constantly, because the staging setting rides along into production during a migration. Your whole site quietly carries a site-wide noindex and traffic never arrives. Make it a launch-day check.
Noindexing a page you’re also linking to from your main navigation sends a genuinely mixed signal, and more practically means you’re spending internal links on something you’ve asked to be invisible. If it’s not worth indexing, it probably doesn’t belong in the nav either.
And two plugins both writing robots directives will eventually disagree. Since the more restrictive rule wins, the surprising outcome is usually more suppression than you intended, not less. Pick one plugin to own indexing and turn the feature off in the other.
Frequently Asked Questions
If you’re not sure what your site is currently telling crawlers, start by reading your own robots.txt and grepping a few pages for their robots meta tag. Most of the surprises turn up in that first five minutes.
Built by amplifi.studio — see also The robots.txt Mistake That’s Making You Invisible to ChatGPT and Claude.