Separates crawl control from index control, which are different mechanisms that cancel each other out whenever somebody uses one to do the other's job.
Untested, so what follows describes the contents rather than the outcome.
Almost every mistake in this area comes from treating two mechanisms as if they were one. Blocking a crawl and removing a page from an index are separate jobs with separate tools, and using either one to do the other's job produces a specific, durable failure. The headline case is the one this skill leads with: a page disallowed in robots.txt cannot be read, so a noindex served on that page is never seen, so the page can stay indexed indefinitely, typically listed with no description. The instruction that follows from it is blunt and counterintuitive. To get a page out of the index, you must allow the crawl.
The second thing it knows is what the robots file's own status code does, which inverts most people's assumption. A 4xx response other than 429 means the file is treated as though it does not exist, so every restriction you wrote silently evaporates. A persistent 5xx is the case that suppresses crawling of the host, and RFC 9309 puts that in mandatory language. So a broken deploy that starts serving 500 on robots.txt is a crawling incident, and one that starts serving 404 is an exposure incident, and they look equally harmless in a browser.
The rest is the material that turns those two facts into an audit: single-group matching, so a rule in the wildcard group is invisible to a crawler with its own group; longest-match precedence between Allow and Disallow; the directive list with honest notes on which engine honours what; a five-way decision table covering canonical, noindex, redirect, 404 and 410, including what happens when a canonical points at a noindexed page; and the arithmetic of facet URLs.
Who this is not for. A site of thirty static pages with no filters, no staging host and no PDFs does not have a crawl control problem, and installing this will produce recommendations about machinery you do not have.
Measuring one skill honestly costs about twenty model sessions: five runs with it, five without, on real material, each output graded alone by a session that is not told the other arm exists, against a rubric written by somebody who never saw the skill. We have not spent that on this one yet, so it ships labelled rather than ships silently.
How it would be measured. Tier A. A fixture site of about forty URLs carrying a known set of planted conflicts: a disallowed URL also serving noindex, a canonical pointing at a noindexed target, a rule placed in the wildcard group that a named crawler group would ignore, a PDF that can only be controlled by header, a staging host protected by robots.txt alone, and two facet patterns generating combinatorial URLs. Scored on planted conflicts found, correct mechanism prescribed for each, and clean URLs wrongly flagged.
This one has an unusually clean objective spine, which is why it is queued rather than abandoned. Conflicts between directives are either present or not, and a fixture site can carry a known list of them: a disallowed URL that also serves noindex, a canonical aimed at a noindexed target, a rule sitting in the wildcard group that a named crawler group would ignore entirely, a PDF controllable only by header, and a staging host defended by robots.txt alone.
The reason it has not run is that the interesting half is precision rather than recall. A strong model already knows what robots.txt is and will describe it fluently. What is worth checking is whether it prescribes the right mechanism for each case and leaves the clean URLs alone, so the fixture needs plausible decoys as much as it needs planted faults, and building those is most of the work.
The rule that decides pass or fail was written down before any run was executed and it does not move afterwards. It is in the method note on the hub, along with the full results table including every skill that was tested and cut.
Stated plainly, because a skill that claims everything is useful for nothing.
name and description in the file's frontmatter, so you can also invoke it by name.The specification is free, short and definitive, and for the syntax questions it beats any summary including this one. If your question is "does this rule match this path", read RFC 9309 and stop.
A crawler configured with the right user agent answers the questions that reading files cannot: what is actually served, what a given bot can actually reach, and where the redirect chains are. Own one. It will not tell you that the disallow you added last month is the reason the page still shows in results with no description, because that is a reasoning failure rather than a crawl finding.
For a small site the honest alternative is to have no robots.txt restrictions at all beyond a sitemap line, protect anything private with authentication, and use noindex on the handful of pages that genuinely should not be listed. That configuration is close to unbreakable, costs nothing to maintain, and stops being adequate roughly when you add your first filtered listing page or your first staging host.
--- name: crawl-control-audit description: Audits how a site controls crawling and indexing across robots.txt, robots meta tags, X-Robots-Tag headers, canonicals and status codes. Finds the conflicts where one mechanism silently cancels another, most commonly a URL that is disallowed in robots.txt while also serving a noindex that can therefore never be read. Covers group matching and precedence rules, what each response code on the robots file itself means, faceted URL explosion, and staging environments exposed because they were defended by robots.txt rather than by authentication. This skill should be used when deciding how to remove something from search results, before a staging or faceted section ships, or when a URL is indexed and nobody can work out why. --- # Crawl control audit ## The claim this skill is built on There are two separate jobs here and one word for both of them in most people's heads. **Crawl control** decides whether a bot is allowed to fetch a URL. That lives in robots.txt. **Index control** decides whether a fetched URL may appear in results. That lives in a robots meta tag or an `X-Robots-Tag` response header. They are not two ways of saying the same thing, and they interact in one direction only. Index control is delivered inside the response, so it can only be obeyed if the fetch is allowed. Crawl control blocks the fetch, so it destroys index control. That asymmetry produces the single most common self-inflicted indexing problem there is. Somebody wants a page out of search results, so they add a `Disallow` line for it. The crawler stops fetching the page. It never sees the `noindex` that is sitting right there in the HTML. The URL stays in the index, usually rendered as a bare listing with no description, because there is no permitted content to describe. Then somebody adds a second Disallow, and a third, and the pages stay. **The rule that follows is the opposite of the instinct: to remove a URL from the index, you must allow it to be crawled.** Allow the fetch, serve `noindex`, wait for the recrawl, and only then, if you still want to save the crawl, consider disallowing it. ## What the robots file's own status code does This is the second counterintuitive piece, and it is where a deploy accident turns into a search incident. The behaviour below is as set out in RFC 9309, the Robots Exclusion Protocol, which became a standards-track RFC in September 2022, and as documented by major crawler operators. Verified against the published specification in August 2026. - **2xx.** The file is parsed and the rules apply. Normal case. - **3xx.** Redirects are followed, and the specification asks crawlers to follow at least five consecutive hops. Beyond that the file is treated as unavailable. Redirects expressed in HTML or JavaScript rather than in a status code are not followed at all. - **4xx, other than 429.** The file is treated as though it does not exist, which means **there are no restrictions**. This is the one people get backwards. A 403 or a 404 on robots.txt does not lock a crawler out. It opens the whole host. Your directives vanish and nothing anywhere reports it. - **5xx.** The file is unreachable, and the specification uses mandatory language: crawlers must assume a complete disallow. After a long outage, given as roughly thirty days in the specification, a crawler may fall back to treating it as unavailable, meaning no restrictions, or keep using a cached copy. So a persistently failing server is the case that actually suppresses crawling. - **429.** Too many requests. Handled in the same bucket as 5xx by Google's documented behaviour rather than in the 4xx bucket, so it is a genuine slow-down signal. Two practical consequences. First, published crawler guidance explicitly warns against using 401 or 403 responses to limit crawl rate, precisely because a 4xx other than 429 reads as "no restrictions" rather than "go away". The documented way to ask a crawler to back off is 503 or 429, and only as a temporary measure. Second, robots.txt is commonly cached for up to about 24 hours, so neither a fix nor a mistake takes effect instantly, and testing a change by watching your logs for five minutes proves nothing. ## Robots.txt syntax that actually decides the outcome Most of the file is uncontroversial. These are the parts that change the result. **Only one group applies.** Records are grouped by `User-agent`, and a crawler obeys the single most specific group that matches its product token. It does not merge that group with the wildcard group. So if your file contains a named group for a crawler and a `User-agent: *` group, everything in the wildcard group is invisible to that named crawler. A rule added to the wildcard group to block a directory will simply not apply to any bot that has its own group above it. This is the most common robots.txt bug that is not a typo. **Longest match wins.** Between an `Allow` and a `Disallow` that both match a path, the rule with the longer path pattern takes precedence, not the one that appears first. `Disallow: /reports/` plus `Allow: /reports/public/` permits the public subtree because the Allow is longer. Where two rules are equally specific, the less restrictive one wins. **Wildcards and the end anchor.** `*` matches any run of characters, `$` anchors the end of the URL. `Disallow: /*?` blocks any URL containing a query string. `Disallow: /*.pdf$` blocks paths ending in `.pdf` but not `/file.pdf?download=1`, which is exactly the sort of gap that leaks. **Empty means allow.** `Disallow:` with nothing after it permits everything. `Disallow: /` blocks everything. One character apart, opposite meanings, and both appear in real deploys by accident. **Paths are case-sensitive; user agent names are not.** `/Reports/` and `/reports/` are different rules. **Location is strict.** The file must sit at `/robots.txt` at the root of the host, and it governs exactly one scheme, host and port combination. A file on the main host says nothing about a subdomain, nothing about the same host on a different port, and, strictly, nothing about the other protocol. Every hostname that serves pages needs its own. **Size.** Crawlers are required to parse at least 500 kibibytes, and Google documents that as its limit, with everything after the cut ignored. A generated robots file that grows with the catalogue will eventually cross it, and the rules that fall off the end are the newest ones. **Comments** run from `#` to end of line. **`Sitemap:`** is independent of the groups and may appear anywhere in the file. **Unsupported directives.** Google stopped honouring `noindex`, `nofollow` and `crawl-delay` inside robots.txt on 1 September 2019. A file containing `Noindex:` lines is doing nothing at all for that crawler. Some other engines have historically honoured `Crawl-delay`, so check the specific engine rather than assuming either way. ## X-Robots-Tag, and why it is not a duplicate of the meta tag The meta robots tag only exists inside HTML. **`X-Robots-Tag` is the only way to control a non-HTML resource**: PDFs, images, spreadsheets, video files, plain text. If your site publishes PDFs that outrank their own landing pages, a header is the only lever you have short of deleting them. The two are additive, not competing. When a page carries both a meta tag and a header, the union of the directives applies and **the most restrictive value wins**. `index` in one place and `noindex` in the other resolves to `noindex`. Both can also be aimed at a named crawler rather than at everyone, and a directive aimed at a specific crawler takes precedence over the generic one for that crawler. Directives worth knowing: - **`noindex`.** Do not show this in results. Broadly supported. - **`nofollow`.** Do not follow links from this page. Note this is the page-level directive, which is a different thing from the `rel="nofollow"` link attribute; that attribute became a hint rather than a directive for Google's crawling and indexing from 1 March 2020. - **`none`.** Equivalent to `noindex, nofollow`. - **`noarchive`.** No cached copy link. - **`nosnippet`.** No text snippet at all. - **`max-snippet:[n]`**, **`max-image-preview:[none|standard|large]`**, **`max-video-preview:[n]`.** Granular preview limits. Google documents these; support elsewhere is patchy and you should verify per engine rather than assume. - **`notranslate`.** Do not offer a translation of this result. - **`noimageindex`.** Do not index images on this page. - **`unavailable_after: [date]`.** Stop showing this after a date, which is genuinely useful for expiring event and offer pages. Where you are unsure whether a given engine honours a given directive, say so in the report rather than asserting it. The directive set is stable for Google and materially different elsewhere, and the answer-engine crawlers are a moving target. ## Crawl budget, honestly Most advice about crawl budget is folklore, repeated on sites that will never encounter it. Crawl budget is a real constraint for two populations. First, very large sites: Google's own guidance is aimed at properties above roughly one million unique pages with content changing weekly, or above roughly ten thousand unique pages with content changing daily. Second, and more common, sites that generate URLs faster than anyone can crawl them, which is almost always a faceted listing, a calendar, an internal search results page that is linkable, or a session parameter appended to every link. That second category can hit the wall at a few thousand real pages, because the real page count is not the URL count. The genuine signals, in rough order of usefulness: the share of crawler requests in your server logs that land on parameterised or non-canonical URLs rather than on content; a "discovered, currently not indexed" bucket that grows month over month while your publishing rate is flat; average response time in the crawl stats report trending up; and new pages taking materially longer to be crawled than they did a quarter ago. One asymmetry matters here and it follows from the opening claim. Blocking in robots.txt does save crawl effort, because the fetch never happens. `noindex` does not, because the page must be fetched for the directive to be read. So for a genuine crawl budget problem, robots.txt is the correct tool and noindex is not. That is the one case where reaching for Disallow is right, and it is worth stating plainly, because the rest of this skill spends its time telling you the opposite. ## The five-way decision table These five are routinely substituted for each other. Each has one correct use. | Situation | Correct tool | Why | | --- | --- | --- | | Two URLs, same content, both should stay reachable | `rel=canonical` on the duplicate, pointing at the preferred URL | Consolidates signals without removing anything. It is a hint, not a command. | | The page should exist for users but never appear in results | `noindex`, crawl allowed | The only mechanism that actually removes a listing. | | The content has moved and the old URL should stop existing | 301 permanent redirect | Signals consolidate onto the target. Use 302 only when the move is genuinely temporary. | | The page is gone, and you are not certain it will stay gone | 404 | Honest, reversible, no signal transfer. | | The page is gone deliberately and permanently | 410 | Same outcome as 404, generally dropped a little faster. Use it when you are sure. | | The URL is worthless and you never want it fetched again | `Disallow` | Saves the crawl. Does not remove anything already indexed. | **The destructive combination to look for: a canonical pointing at a noindexed page.** Page A canonicalises to page B, and page B carries `noindex`. Page A has said "treat B as the real one" and B has said "do not list me". The likely outcome is that neither is indexed, and the noindex can propagate to the whole cluster. This appears constantly on paginated and filtered pages where someone applied noindex to the canonical target as a tidiness measure. The same applies to a page that carries both `noindex` and a self-referencing canonical pointed elsewhere: the two directives contradict each other and the resolution is not something you should be gambling on. The second destructive combination: a canonical or a redirect on a URL that is disallowed in robots.txt. Neither can be read, so neither happens. ## Parameter and facet explosion, with the arithmetic The reason filtered listings break crawling is combinatorial, and the numbers are worth doing before the feature ships rather than after. Take a listing page with five independent filter groups, each offering six values, each multi-selectable in the URL. Ignoring multi-select, each group has seven states, six values plus unselected, so one base category produces 7 to the power of 5, which is 16,807 URLs. Add four sort orders and ten pages of pagination and it is 672,280. From one category. A catalogue with two hundred categories has generated over a hundred million linkable URLs from a few thousand products. Even the modest case is bad: twelve independent binary filters is 4,096 combinations per category, and nobody thinks of twelve checkboxes as a large feature. **The answer is almost always to stop generating linkable URLs, not to block them afterwards.** Apply filters without changing the URL, or behind a fragment, or via a POST, so the combination never becomes a crawlable link. Then expose deliberately, as real indexable URLs, only the small set of combinations that people actually search for, and give those real pages with real content. Blocking afterwards is the weaker fallback for three reasons. Disallowed URLs can still be indexed if they are linked from anywhere, since the block prevents the fetch, not the listing. The rules become a long and fragile pattern list that nobody dares edit. And the internal links pointing at blocked URLs are still consuming crawl attention. ## Staging and duplicate hosts The classic failure: a staging or preview host is protected by robots.txt with a blanket `Disallow: /` and nothing else. Six months later the staging site is in search results, competing with production on identical content. The mechanism is exactly the opening claim. `Disallow: /` prevents fetching, not listing. Any external link, any pasted URL in a public issue tracker, any analytics or link tool that reports the hostname, is enough for the URL to be discovered. It gets indexed on the strength of the link alone, with no content, and because the crawl is blocked, no `noindex` you add later will ever be read. **Authentication is the only reliable answer.** HTTP basic auth, an IP allowlist, or a platform-level password. An unfetchable host cannot be indexed on the strength of a link, because the crawler gets a 401 on the page itself rather than a page it may not read. This also solves the second problem, which is that a public robots.txt on a staging host is a published list of the paths you consider sensitive. If a staging host is already indexed, do not add a Disallow. Allow the crawl, serve `noindex` on every URL or return 410, wait for the recrawl, and use the temporary removals tool in Search Console to hide it in the meantime. Then, once the URLs are gone, put authentication in front of it so it cannot happen again. The same reasoning applies to any duplicate host: a `www` and non-`www` pair both serving 200, an alternate domain kept from a rebrand, a CDN hostname serving the whole site. Pick one, redirect the others, and never handle a duplicate host with robots.txt. ## The decision procedure Start from the outcome you want, not from the file you were about to edit. 1. **Do you want it out of the index, out of the crawl, or both?** 2. **Out of the index.** Is the URL currently crawlable? If yes, serve `noindex` and leave the crawl alone. If it is disallowed, **remove the Disallow first**, then serve `noindex`, then wait for a recrawl before doing anything else. Only after it has dropped out should you consider blocking the crawl again, and usually you should not bother. 3. **Out of the crawl, and it is not indexed.** Disallow it. This is the legitimate use. 4. **Out of the crawl, and it is indexed.** You cannot have both immediately. Choose: removal first via noindex with the crawl allowed, then block later; or accept that a blocked URL may keep its listing. 5. **It is a non-HTML file.** There is no meta tag. Use `X-Robots-Tag`. If your platform cannot set headers on that path, the only remaining options are removing the file or gating it behind auth. 6. **It is a duplicate rather than something unwanted.** Canonical, not noindex, and never Disallow. 7. **You cannot tell whether the URL is currently indexed.** Do not act. Every branch above depends on that answer and the wrong branch is the failure this whole skill is about. Check with URL Inspection on a verified property, which is the only authoritative source; a `site:` query is indicative at best. If nobody has Search Console access, getting it is the next task, not the robots edit. If it genuinely cannot be determined, take the branch that is safe under both readings: allow the crawl and serve `noindex`, which is correct whether or not the URL was indexed, and costs only crawl effort. ## Worked example, compressed A documentation and marketing site, roughly 900 real pages, publishing PDFs, with a filtered resource library. Traffic is flat and several odd URLs keep appearing in results. **Finding 1, fatal and self-inflicted.** `/account/` is disallowed in robots.txt and every page under it also serves `<meta name="robots" content="noindex">`. Eleven of those URLs are in results as bare listings with no description. The noindex has never been read. Fix: delete the Disallow, keep the noindex, wait for recrawl, and leave the Disallow off permanently since these pages are behind a login and cost almost nothing to crawl. **Finding 2, silent and total.** The robots file returns 404 on the `www` hostname, which is the one that serves the site, and 200 on the bare domain, which redirects. Every rule in it is currently inert, because a 4xx means no restrictions. Nobody noticed because the file exists in the repository. **Finding 3, invisible by design.** A `Disallow: /internal-search/` line sits in the `User-agent: *` group, below a named group for a major crawler that was added to set a `Crawl-delay`. That crawler matches its own group only, so it ignores the internal search block entirely, and its own group contains one directive that the crawler does not support. Fix: delete the named group so the crawler falls through to the wildcard rules. **Finding 4, wrong mechanism.** Forty-two PDFs carry a meta robots tag in the HTML of their landing pages, in the belief that this controls the PDFs. It does not. The PDFs need `X-Robots-Tag: noindex` on the file responses. **Finding 5, destructive pair.** Filtered library pages canonicalise to `/library/`, and `/library/` carries `noindex` because somebody decided the unfiltered view was thin. The canonical target is excluded from the index, so the cluster has nowhere to consolidate to. **Finding 6, arithmetic.** The library has six filter groups averaging five values, generating on the order of 46,000 URLs, all linkable, against 340 real documents. Currently handled by four wildcard Disallow patterns that do not cover the sort parameter. **Verdict: two blocking fixes and one design change.** Blocking: remove the `/account/` Disallow, and fix the robots file 404 on the serving hostname. Design: stop generating linkable filter URLs and expose the eleven combinations with real search demand as real pages. The PDF headers and the canonical target are the following week's work. ## Failure modes **Disallowing something to deindex it.** The founding error. It preserves the listing forever and removes your ability to fix it. **Assuming a 4xx on robots.txt is safe.** It removes every restriction on the host, and a browser check shows a perfectly ordinary error page. **Editing the wildcard group when the crawler has its own group.** The edit is real, deployed, and completely inert for the bot you cared about. **Reading the first matching rule instead of the longest.** Produces confident, wrong answers about whether a path is blocked, in both directions. **Testing robots.txt on the wrong hostname.** The file is per host, per scheme and per port, and the file that governs your traffic is the one on the hostname that actually serves pages. **Canonicalising to a noindexed page.** Quietly excludes the whole cluster, and every individual directive involved looks reasonable in isolation. **Treating staging robots.txt as security.** It publishes the paths you want hidden and does not stop them being indexed. **Blocking a facet explosion instead of not creating it.** Leaves a fragile pattern list, still leaks through the parameters nobody enumerated, and does not remove the URLs already discovered. **Expecting an instant effect.** Robots files are cached for hours and recrawls take days to weeks, so a change judged after one afternoon will be judged wrong. ## What this skill does not do - It does not tell you what is in the index. It reasons about what the directives should cause. Confirming the state of a specific URL needs URL Inspection on a verified property. - It does not fetch anything. Real status codes, real headers and real redirect chains need a live request, and a review from source alone will name the checks it could not complete. - It does not analyse logs or crawl statistics, so its statements about crawl budget are inferences from site shape rather than measurements of crawler behaviour. - It does not cover engine-specific behaviour beyond the well-documented cases, and it will say where support is uncertain rather than inventing a compatibility claim. - It does not decide which pages deserve to be indexed. That is a content and inventory question, and overlapping pages competing with each other is a different audit. - It changes nothing. Every output is a file edit, a header rule or an infrastructure decision for somebody with deploy access, and on some hosted platforms the robots file is not yours to edit.
These skills all ask your assistant to check things against your actual codebase, your actual schema, your actual design system. Locul keeps that context current on its own, from the files you already have, on your machine. Mac and Windows, free to start.