How to use this robots.txt tester

Load a robots.txt file, list the URLs you care about, pick a crawler. The robots.txt tester shows each URL as allowed or blocked, names the exact line that made the decision, and tells you which group of rules the crawler followed. That last part is where most robots.txt mistakes hide, and it is the part most testers leave out.

What this robots.txt checker shows you

Most robots checker tools stop at a plain yes or no, and that is not enough to fix a problem. Every result in this robots.txt checker carries three pieces of information, and together they explain why a crawler behaves the way it does.

Allowed or blocked, and the line that decided it

Crawlers pick one winning rule per URL. The tool reports that rule with its line number, for example Line 11: Disallow: /*.pdf$. If no rule matched, the URL is allowed by default and the result says so.

Which group the crawler actually followed

A robots.txt file is split into groups, each starting with one or more User-agent lines. A crawler follows exactly one group: the one that names it most specifically. Only if no group names it does it fall back to User-agent: *. The results tell you which group was used, including the less obvious cases, such as Googlebot-Image falling back to the Googlebot group, Applebot following Googlebot when it has no group of its own, and AdSense's Mediapartners-Google ignoring the * group completely.

Check robots.txt on a live site: the HTTP status

When you fetch a live file, the status code matters as much as the text. Google's handling is specific, and the tool applies it to the results automatically.

The server error row surprises people. A robots.txt that times out does not mean "no rules". For Google it means "do not crawl anything yet", and a flaky server can quietly slow down crawling of the whole site.

How Google decides if a URL is blocked by robots.txt

The rules come from the Robots Exclusion Protocol, published as RFC 9309 in September 2022, plus a few leniencies Google documents. The matching logic in this tool is a line by line port of Google's open-source robots.txt parser, and it gives the same answer as that parser on every URL matching check in the test suite Google publishes with it. In short:

1. The crawler finds the group that names it. Letters, hyphens and underscores count, case does not, so googlebot and Googlebot/2.1 both mean Googlebot.
2. If several groups name the same crawler, their rules are merged into one.
3. If no group names it, it uses the * group. If there is no * group either, everything is allowed.
4. Within the chosen group, the rule with the longest matching path wins.
5. If an Allow and a Disallow match with equal length, Allow wins.

The most specific group wins, and the * group is ignored

This is the mistake the sample file above demonstrates. Once a site adds a User-agent: Googlebot group, Googlebot stops reading the * group entirely. Every shared rule, such as keeping /wp-admin/ or internal search out, has to be repeated inside the Googlebot group. The validator below the tool flags this whenever named groups and a * group exist side by side.

The longest rule wins, and ties go to Allow

Wildcards: * and $

* matches any sequence of characters, including none. $ marks the end of the URL, but only when it is the last character of the rule. A rule always matches from the start of the path, so Disallow: /fish blocks /fish, /fish.html and /fishheads/, while Disallow: /*.pdf$ blocks /files/list.pdf but not /files/list.pdf?download=1.

Case, encoding, file size and typos

Paths are case sensitive: /Private/ and /private/ are different rules. Non-ASCII characters are compared in percent-encoded form, so a rule for /café/ matches /caf%C3%A9/. Google reads only the first 500 KiB of the file. It forgives a handful of misspellings, including Disalow and User agent, and a missing colon. Other crawlers are not guaranteed to, which is why the validator still flags them.

Test robots.txt for AI crawlers such as GPTBot and ClaudeBot

Step 3 of the tool runs one URL against every crawler in the list at once, grouped into Google, other search engines, AI crawlers and SEO tools. It is the quickest way to answer a question search engine testers were never built for: which AI systems can read this page?

Most AI companies now run more than one bot, and they are not interchangeable. Blocking a training crawler keeps your content out of future models. Blocking a search crawler removes you from that assistant's answers and citations.

Two details the matrix points out for you. Google-Extended and Applebot-Extended are control tokens: they never visit your site, they only tell Google and Apple how pages fetched by their main crawlers may be used. And user-triggered fetchers such as ChatGPT-User and Perplexity-User may not follow robots.txt at all, according to their own documentation, because a person asked for the page.

Blocking AI training while staying in AI search

A common choice is to refuse training but keep the search and answer bots, so the site can still be cited. Paste this into the editor above and switch between GPTBot and OAI-SearchBot to see the difference:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
Disallow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Googlebot blocked by robots.txt? How to find the cause

If Search Console reports a page as blocked, fetch your live file here, add the exact URL from the report, keep Googlebot selected and read the deciding line. It is almost always one of these:

• A short rule matching more than intended, such as Disallow: /p also blocking /products/ and /pricing/.
• A staging file shipped to production with Disallow: / still in it.
• A wildcard like Disallow: /*? catching tracking or pagination parameters on pages you want indexed.
• A rule written for the wrong host. www, the bare domain, each subdomain and http versus https each need their own file.
• A CDN or firewall serving a different robots.txt, or a 5xx error, to Google than to your browser.

"Blocked by robots.txt" and "Indexed, though blocked by robots.txt"

These two Search Console statuses are easy to mix up. Blocked by robots.txt means Google did not crawl the URL and did not index it. Indexed, though blocked by robots.txt means Google found links pointing to the URL and indexed the address anyway, usually with no description, because it was not allowed to read the page. robots.txt controls crawling, not indexing. To remove a page from results, allow crawling and add a noindex robots meta tag, so Google can actually see it.

Using the tool as a Googlebot checker

Pick Googlebot to see what Google Search can crawl, which is what most people want from a Googlebot tester. Pick Google-InspectionTool to see what the URL Inspection tool and Rich Results Test can reach; it follows the Googlebot group when it has none of its own. If you run AdSense, check Mediapartners-Google too. It ignores the * group, so a site wide Disallow: / does not stop it, but a group written for it will, and that costs ad relevance.

Robots.txt validator: the errors and warnings it flags

Below the tester, a robots.txt validator reads every line and reports problems with their line numbers. Click a line number to jump to it in the editor. It checks for:

• Errors: rules above the first User-agent line, noindex in robots.txt (unsupported by Google since September 2019), relative sitemap URLs, and files over 500 KiB.
• Warnings: misspelled field names, missing colons, paths that do not start with / or *, Disallow: / for all crawlers, rules that block CSS or JavaScript, and unknown fields.
• Notes: Crawl-delay (ignored by Google, followed by Bing), Yandex-only fields, user-agent values with versions attached, groups that are merged, and named groups that make crawlers skip the * rules.

What happened to the Google Search Console robots.txt tester

Google retired the old robots.txt tester in Search Console in late 2023 and replaced it with the robots.txt report. The report is useful: it lists the robots.txt files Google found for your top hosts, when each was last crawled, any fetch errors or warnings, and it lets you request a recrawl after you change the file.

What it no longer does is let you type a URL and see whether a given rule blocks it, or try an edit before you publish. That gap is what this page covers, and it works for any site, not only the ones you have verified in Search Console. Use both: test and fix here, upload the file, then request a recrawl in the report so Google picks up the change sooner than its usual 24 hour cache.

Robots.txt examples you can paste and test

Each of these works as written. Paste one into the editor to check it against your own URLs before using it.

Allow every crawler everything

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Block the whole site, for example on staging

User-agent: *
Disallow: /

Staging sites are better protected with a password, because a blocked URL can still be indexed from links.

Block a folder but allow one file inside it

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Block internal search and filter parameters

User-agent: *
Disallow: /*?s=
Disallow: /*?filter=
Disallow: /*&sort=

This is one of the few robots.txt changes that saves crawl budget on large sites, because faceted URLs multiply fast.

Block Googlebot only

User-agent: Googlebot
Disallow: /

User-agent: *
Disallow:

What robots.txt cannot do

• It does not remove pages from Google. Use a noindex meta tag or an X-Robots-Tag header, and leave the URL crawlable so the tag can be seen.
• It is not security. The file is public and lists the paths you would rather hide. Protect private areas with a login.
• It is a request, not a barrier. Reputable crawlers follow it. Scrapers and some user-triggered fetchers do not.
• It covers one host. https://example.com/robots.txt says nothing about https://shop.example.com or http://example.com.
• It does not describe your content. That is the job of structured data. If you are working on how pages appear in results, the schema markup generator builds the JSON-LD for that.

Other free SEO tools on Calcxi

These sit in the same SEO category as this robots.txt tester.

Robots.txt tester FAQ

How the live fetch works, and its limits

Browsers cannot read another site's robots.txt directly, so the Fetch button in this robots tester asks a small Calcxi server to download it. The server only ever requests /robots.txt, identifies itself as CalcxiRobotsTester, follows at most five redirects like Google does, stops reading at 500 KiB and stores nothing. Private and internal network addresses are refused.

Some sites show different content to Googlebot than to other visitors, and a firewall may block the fetch entirely. If the result looks wrong, open your robots.txt in a browser, paste the text into the editor and test it there. Pasted text and test URLs never leave your browser. This tool is a guide to how crawlers read the file; the final word on what Google did with your site is always Search Console.