A tiny text file can determine whether search crawlers reach product pages, articles, images, scripts and entire site sections. That makes robots.txt one of the highest-leverage—and easiest to mishandle—parts of technical SEO.
A good robots.txt audit is not a syntax glance. It answers three operational questions: Can crawlers fetch the file? Do the rules match the team’s intent? Have you tested the exact URLs and crawler groups that matter to the business?
Google defines robots.txt primarily as a way to manage crawler traffic. It explicitly warns that a disallowed web page can still appear in search results if Google discovers the URL through other links. The IETF’s Robots Exclusion Protocol, standardized in RFC 9309, also treats the file as a voluntary access instruction for compliant crawlers—not as security.
That distinction should shape every audit. Start with business intent, prove behavior with test cases, and keep security and indexing controls in their proper layers.
What a robots.txt audit should prove
By the end of an audit, you should be able to demonstrate all of the following:
- each relevant protocol-and-host combination has the intended file at its root;
- the live response is available, readable and served as plain text;
- production rules match a documented crawl policy;
- important pages and rendering resources are not accidentally blocked;
- low-value crawl spaces are restricted only where there is a defensible reason;
- user-agent groups and path patterns behave as expected;
- sitemap declarations are absolute and current;
- the team can roll back a harmful change;
- monitoring will reveal fetch errors or unexpected crawl changes.
If you cannot show the test evidence, the audit is not complete—even if the file “looks right.”
Before you edit: understand the boundaries
Robots.txt answers a crawl question: may a cooperating crawler request this path? It does not reliably answer whether a URL may be indexed, whether a user may view it, or which URL should be canonical.
Use the control that matches the objective:
- To prevent unauthorized access, require authentication or restrict access at the server or application layer.
- To request exclusion from search results for a crawlable page, use a supported noindex directive in the page or HTTP response, subject to the search engine’s rules.
- To consolidate duplicate URLs, use canonicalization and consistent internal linking where appropriate.
- To remove a URL urgently, follow the search engine’s removal and source-remediation process.
- To guide discovery of preferred URLs, publish accurate XML sitemaps and crawlable internal links.
Do not block a page in robots.txt and then expect Google to read a noindex tag on that same page. A crawler that cannot fetch the page cannot see the page-level directive.
The ROBOTS SAFE audit framework
The UrleBird’s ROBOTS SAFE framework turns a risky one-file check into a repeatable release process.
#### R — Reach every origin
Robots.txt scope is tied to an origin: the scheme, host and port. A file at https://example.com/robots.txt does not automatically govern https://shop.example.com/ or http://example.com/. Google’s Search Console documentation also notes that subdomains and protocols can have separate files.
Create an origin inventory from DNS, analytics, Search Console properties, CDN configuration and application documentation. Include:
- HTTPS and any still-accessible HTTP origin;
- www and non-www hosts;
- mobile, shop, help, documentation and image subdomains;
- international hosts;
- alternate ports that are publicly reachable;
- staging or preview hosts, while remembering they should normally be access-controlled rather than merely disallowed.
For each origin, record the expected robots.txt URL, owner and intended policy. A missing file can be acceptable because it generally means crawling is allowed, but it should be an intentional state.
#### O — Observe the live response
Fetch the file as an unauthenticated visitor. Use a private browser window and an HTTP inspection tool approved by your organization. Record:
- final URL after redirects;
- HTTP status;
- content type;
- character encoding;
- response body;
- cache headers;
- edge or CDN variation by geography, if relevant.
RFC 9309 specifies a lowercase /robots.txt path at the top level and UTF-8 plain text. Google documents a 500 KiB processing limit and may cache a file for about 24 hours, sometimes longer when refreshes fail. A successful browser display is useful, but it does not replace a response-header check.
Investigate HTML error pages returned with a 200 status, authentication challenges, redirect loops, WAF blocks and inconsistent responses between edge locations. The crawler must receive the rules, not the site’s branded error template.
#### B — Build an intent map
Read every Allow and Disallow rule, then translate it into plain-language intent. For example: “Block internal search-result paths from Googlebot” is an auditable statement; “Disallow: /search” without an owner or reason is not.
For each rule, document:
- crawler group;
- matching path pattern;
- business purpose;
- affected template or system;
- expected examples that should be blocked;
- expected exceptions that should remain allowed;
- owner and review date.
Remove obsolete rules only after confirming the underlying paths no longer need crawl control. Legacy directives can hide forgotten applications, while apparently harmless cleanup can reopen infinite URL spaces.
#### O — Open essential resources
Modern pages often rely on CSS, JavaScript, images and API responses for layout and content. Google advises against blocking resources when doing so prevents its systems from understanding the page.
Choose representative templates—home, category, product, article, account landing page and international variant—and list the resources required to render their main content. Test those resource URLs against the relevant crawler group.
Pay particular attention to broad rules such as Disallow: /assets/, Disallow: /static/ or file-extension patterns. A directory name that sounds nonessential can contain the scripts or styles used across the whole site.
#### T — Test user agents and paths
Do not test only with the wildcard group. Google explains that its crawlers choose the most specific matching user-agent group; a specific group is not automatically combined with the wildcard group. Duplicate groups for the same specific crawler may be combined.
Build a test matrix with at least:
- each critical crawler group used in the file;
- a canonical URL from every important template;
- intentionally blocked examples for every restrictive rule;
- exceptions created by Allow rules;
- case variants where the application treats case differently;
- URLs with parameters;
- percent-encoded paths and non-ASCII characters when present;
- root paths, trailing slashes and file-like paths;
- CSS, JavaScript and primary image resources.
Google supports * as a wildcard and $ as an end-of-string marker in path values. Matching is case-sensitive for path values. Test the real URL, not a simplified approximation.
Use Search Console’s robots.txt report for live files and URL Inspection for individual Google access questions. Development teams can also test locally with Google’s open-source robots.txt parser. Keep the expected result next to the observed result so future reviewers can repeat the test.
#### S — Synchronize sitemap signals
Google supports Sitemap fields in robots.txt, and the URL must be fully qualified. Check that each declaration:
- uses the intended HTTPS URL;
- returns successfully;
- points to a sitemap or sitemap index, not an HTML page;
- contains canonical, indexable URLs appropriate to that host;
- does not rely on a path that the file itself blocks from crawling;
- matches the current production environment.
A sitemap is not an Allow rule, and listing a blocked URL in a sitemap does not override a Disallow rule. Conflicting signals create ambiguity and waste troubleshooting time.
#### SAFE — Stage, approve, follow and evaluate
Stage the exact file, not a manually recreated sample. Run automated and manual test cases before release. Require approval from the SEO owner and the technical owner for broad or user-agent-specific changes. Store the prior file for rollback.
After deployment, follow the live response through the CDN, then evaluate Search Console’s robots.txt status, URL Inspection samples, crawl statistics and server logs. Google says it usually discovers updated files automatically; reserve the report’s recrawl request for a fixed critical error or an urgent unblock, rather than routine repeated submissions.
A copyable robots.txt audit checklist
Use this checklist for quarterly reviews, migrations, redesigns and incident response.
Inventory and ownership:
- List all crawlable origins and robots.txt URLs.
- Name a business owner and technical owner for each file.
- Record whether a missing file is intentional.
- Confirm staging and private systems use real access controls.
Transport and format:
- Fetch the file while signed out.
- Record the status, redirect chain and final URL.
- Confirm lowercase /robots.txt at the origin root.
- Confirm UTF-8 plain text and readable line breaks.
- Check size and remove redundant rules before limits become relevant.
- Verify CDN and WAF behavior from representative locations.
Rules and scope:
- Map every user-agent group to an owner and purpose.
- Confirm each path starts as intended and respects case.
- Test wildcard and end-marker behavior.
- Check duplicate specific groups and group precedence.
- Flag unsupported directives; Google, for example, does not support crawl-delay in robots.txt.
- Confirm comments cannot accidentally alter a rule after a # character.
Critical access:
- Test the homepage and top organic landing pages.
- Test product, category, article and localized templates.
- Test canonical URLs and their rendering resources.
- Test URLs newly unblocked during migrations.
- Check image and media subdomains separately.
Signals and release:
- Validate every Sitemap URL.
- Compare the candidate file with production.
- Run the test matrix against the candidate.
- Save approval and rollback evidence.
- Deploy during a monitored window.
- Re-fetch the public file and repeat critical tests.
- Review Search Console and logs after release.
Common mistakes and safer fixes
Mistake: using Disallow: / as a temporary staging safeguard.
Safer fix: require authentication or restrict the host at the network or application layer. Robots.txt publishes the paths you are trying to protect and compliant behavior is voluntary.
Mistake: blocking parameter patterns without samples.
Safer fix: export real URLs from logs or analytics, test both valuable and low-value parameter combinations, and document the match boundary.
Mistake: assuming the wildcard group applies in addition to a specific Googlebot group.
Safer fix: test the selected group under the search engine’s documented precedence rules and include required rules in the appropriate specific group.
Mistake: blocking CSS or JavaScript to save crawl activity.
Safer fix: first identify whether those assets are needed to render indexable content. Restrict only resources that are demonstrably unnecessary and monitor the result.
Mistake: using robots.txt for deindexing.
Safer fix: select an indexing or access-control method suited to the actual goal, and make sure crawlers can fetch any page-level directive they must process.
Mistake: editing directly in production.
Safer fix: version the file, stage the candidate, run a URL test suite, obtain approval and prepare a one-step rollback.
Test matrix and release score
Use a binary test for each URL-and-user-agent pair: expected Allow or Disallow versus observed result.
Critical pass rate = passed critical cases ÷ total critical cases × 100.
Every critical URL should match its expected result before release. Do not average away a blocked revenue page with dozens of passing low-risk tests. Report critical and noncritical suites separately.
Add a change-risk score to decide the approval level:
Risk score = likelihood of unintended matching (1–5) × business impact (1–5).
A narrow change to a retired test path may score low. A wildcard rule affecting every product path should receive senior SEO and engineering review, a rollback owner and immediate post-release monitoring. This score prioritizes review; it does not predict rankings.
Monitoring after deployment
Check the live file immediately after release and again after caches could have refreshed. Then watch for:
- new robots.txt fetch errors or parsing warnings;
- unexpected drops in crawl requests for important directories;
- sharp changes in rendered-resource access;
- “blocked by robots.txt” signals for canonical pages;
- differences between intended rules and crawler activity in server logs;
- unplanned file changes from deployments, plugins or CDN configuration.
Keep a lightweight changelog with the deployment time, diff, approvers, test results and rollback reference. If organic performance changes, that evidence helps separate a robots release from unrelated site or search-system changes.
Robots.txt audits do not guarantee crawling, indexing or rankings. They reduce preventable ambiguity and give teams a verifiable explanation of what they asked compliant crawlers to do.
Frequently asked questions
Can robots.txt keep a page out of Google?
Not reliably. Google may still index a disallowed URL discovered through links, although it cannot crawl the page’s content. Use an appropriate indexing or access-control method for the real objective.
Where must robots.txt be located?
At /robots.txt on the root of the relevant origin. Protocols and subdomains can have different files and scope.
Does Google support crawl-delay?
Google’s published robots.txt specification says it supports user-agent, allow, disallow and sitemap fields; crawl-delay is not supported there.
Should robots.txt contain a sitemap URL?
It can. Use a fully qualified sitemap URL and verify that the sitemap itself is healthy. A Sitemap field does not override crawl restrictions.
How often should robots.txt be audited?
Review it after migrations, platform changes, new subdomains, CDN or WAF changes, and any crawl incident. A scheduled quarterly check is a useful baseline for active sites, but release-triggered checks matter more than the calendar.
Do I need to request a robots.txt recrawl after every change?
Usually no. Google says it recrawls robots.txt files often. Consider the Search Console recrawl option after fixing a critical fetch error or urgently unblocking important URLs.
