How to use this robots.txt generator

Pick the preset closest to your site, adjust the rules, and copy the file. The whole job takes a minute. What this robots.txt generator adds is the part that usually goes wrong afterwards: it writes groups the way crawlers actually read them, it warns you before you block something you need, and it shows which crawlers can still reach your pages before you upload anything.

What this robots.txt file generator does differently

Most robots.txt makers and builders work the same way: a default setting, a list of search engines with Allowed or Refused, and a box for restricted directories. That layout has a flaw that is easy to miss, and it is the main reason this tool was built the way it is.

It repeats your shared rules inside every named group

A crawler follows only one group in robots.txt: the one that names it. As soon as a file contains User-agent: Googlebot, Googlebot stops reading the User-agent: * group completely. So if a generator puts your restricted directories in the * group and then adds a separate "allowed" group for Google, Google is now free to crawl those directories. Nothing warns you.

This generator copies your disallowed paths into every crawler group it creates, and adds a comment saying why. Allow Googlebot and Bingbot on a site that is otherwise blocked, and both still stay out of /wp-admin/.

It shows who can crawl before you publish

Step 4 runs the file you just built through the same matching logic as our tester, based on Google's open-source robots.txt parser, and shows every crawler as allowed or blocked for any URL you type. That catches side effects a plain generator never mentions, such as Googlebot-Image and Applebot following your Googlebot rules when they have no group of their own.

Only crawlers that still exist

Many robots file generators still list DMOZ, which closed in 2017, Alexa, which was retired in 2022, and old Yahoo and MSN crawlers. Yahoo search now runs on Bing's results, so blocking Bingbot is what actually matters. This list covers the crawlers you are likely to see in your logs today, including the AI crawlers most sites now have to make a decision about.

Robots.txt generator for WordPress

WordPress creates a virtual robots.txt file on its own, and it is a sensible starting point. The WordPress preset builds on it: it keeps the admin area out, leaves admin-ajax.php open because themes and plugins use it on the front end, and blocks internal search results, which are thin pages that waste crawling. This is the WordPress robots.txt it produces:

User-agent: *
Disallow: /wp-admin/
Disallow: /*?s=
Disallow: /search/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap_index.xml

The sitemap address depends on how your sitemap is made. WordPress on its own uses /wp-sitemap.xml. Rank Math and Yoast use /sitemap_index.xml. Open the address in a browser before you use it.

WooCommerce robots.txt

The WooCommerce preset adds the cart, checkout and account pages, plus the add-to-cart, sorting and filter parameters. Those parameters are where shops lose most of their crawl budget, because every combination of filters creates a new URL with the same products on it.

Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?add-to-cart=
Disallow: /*?orderby=
Disallow: /*?filter_

How to edit robots.txt in WordPress

• Rank Math: General Settings, then Edit robots.txt. Paste the generated file and save.
• Yoast SEO: Tools, then File editor. It creates robots.txt if the site does not have one yet.
• No plugin: upload a file named robots.txt to the folder that contains wp-config.php. A real file always replaces the virtual one WordPress generates.

Custom robots.txt for Blogger

Blogger lets you replace its default file under Settings, Crawlers and indexing, Custom robots.txt. The Blogger preset builds the usual custom robots txt: it blocks /search, which covers search results and label pages that repeat your posts, allows everything else, and adds your blog's sitemap.

User-agent: *
Disallow: /search
Allow: /

Sitemap: https://yourblog.blogspot.com/sitemap.xml

Blogger's own default file also contains a Mediapartners-Google group for AdSense. You do not need to copy it: that crawler ignores the * group anyway, so leaving it on Default already gives it full access. Only turn the custom robots.txt switch on if you need something different from the default, because a mistake here can hide the whole blog.

Robots.txt syntax: disallow, allow, wildcards and sitemap

You do not need to write any of this by hand, but knowing what each line means makes the generated file easy to check.

User-agent

Starts a group and names the crawler it applies to. * means every crawler that has no group of its own. Several User-agent lines in a row share the rules below them, which is how the generator keeps the file short.

Robots.txt disallow: blocking a directory

Disallow: /private/ blocks the folder and everything inside it. Without the trailing slash, Disallow: /private also blocks /private-offers/ and /private.html, because rules match from the start of the path. To disallow a directory in robots.txt safely, end it with a slash. An empty Disallow: means nothing is blocked.

Allow

Opens a door inside a blocked area. Allow: /wp-admin/admin-ajax.php works because it is longer than Disallow: /wp-admin/, and the longer rule wins.

Robots.txt wildcards: * and $

* matches any run of characters, so Disallow: /*?sort= blocks the sort parameter on every page. $ anchors the end of the URL, so Disallow: /*.pdf$ blocks PDF files but not a page whose address merely contains ".pdf". Both are supported by Google and Bing.

Robots.txt and sitemap.xml

A Sitemap: line tells every crawler where your XML sitemap is, including those you have never set up in a webmaster tool. It must be a full URL, it can point to another host, and you can list several. It sits outside the groups, so its position in the file does not matter. The generator turns /sitemap.xml into a full address using the website you entered.

Crawl-delay

Asks a crawler to wait between requests. Google ignores it and sets its own pace based on how quickly your server responds. Bing follows it. Leave it empty unless a specific crawler is putting real load on your server.

Generate robots.txt rules for AI crawlers

AI companies run more than one crawler, and they do different things. A training crawler collects pages for future models. A search crawler builds the index an assistant uses to answer questions and cite sources. A user fetcher opens a page because someone asked about it. The generator labels each one, because blocking the wrong type has a real cost.

• Training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider.
• Search: OAI-SearchBot, Claude-SearchBot, PerplexityBot. Blocking these removes you from those answers and citations.
• User fetches: ChatGPT-User, Claude-User, Perplexity-User. OpenAI and Perplexity say robots.txt may not apply to these.

The Block AI training preset blocks the training group and leaves search crawlers open, which is the choice most publishers make: keep content out of model training, stay visible and citable in AI search. Google-Extended and Applebot-Extended are control tokens rather than crawlers, and blocking them does not affect Google Search or Apple search.

Robots.txt disallow subdomain: one file per host

A robots.txt file only covers the exact host it sits on. https://example.com/robots.txt has no effect on https://blog.example.com, https://shop.example.com or even http://example.com. So you cannot disallow a subdomain from the main domain's file. To block a subdomain, give it its own robots.txt with Disallow: /. The Block everything preset builds exactly that. For staging subdomains, a password is safer still, because blocked URLs can be indexed from links even though they are never crawled.

Where to put your robots.txt file

Whether you use a robots.txt generator, a robot text generator or write it by hand, the file must be called robots.txt, in lowercase, saved as plain UTF-8 text, and placed at the root of the host, so it opens at https://yoursite.com/robots.txt. A file in a subfolder is ignored. You do not need to submit it anywhere: crawlers request it before crawling. Google usually picks up changes within a day, and the robots.txt report in Search Console lets you request a recrawl sooner.

Shopify robots.txt

Shopify generates robots.txt for every store and does not let you upload a file. To change it, add a robots.txt.liquid template under Online Store, Edit code. Shopify calls this an unsupported customization and warns that mistakes can cost all of your traffic, so add only the lines you need to the existing template rather than replacing it with a generated file.

After you upload it

Open yoursite.com/robots.txt in a private window to confirm the new version is live, then test your most important URLs in the robots.txt tester. It fetches the live file, reports the HTTP status Google sees and shows the exact line that allows or blocks each URL.

Mistakes the generator warns you about

• Blocking Googlebot on a live site, which removes it from Google Search.
• Blocking Bingbot, which also removes the site from Yahoo.
• Blocking AI search crawlers when you only meant to stop training.
• Blocking CSS or JavaScript, for example /wp-content/, which stops Google rendering your pages.
• AdSense and Google Ads crawlers ignoring "Block everything", because they skip the * group.
• A relative sitemap address, which crawlers cannot use.
• Expecting robots.txt to remove pages from Google. It only stops crawling. Use a noindex tag for that, and leave the page crawlable so Google can see it.

Other free SEO tools on Calcxi

These sit in the same SEO category as this robots.txt generator.

Robots.txt generator FAQ

Before you upload

The generator writes valid syntax and checks it against Google's matching rules, but it cannot know which pages on your site matter most. Read the file once, test your key URLs, and keep a copy of your old robots.txt so you can put it back if something looks wrong in Search Console. Everything happens in your browser, and nothing you enter is sent to a server or stored.