Summary
A robots.txt file is a plain-text file at the root of a website that tells crawlers which URLs they may fetch. To make one free and privately here, click a template, choose a User-Agent for each block, write Allow and Disallow rules one per line, put your own address in Sitemap URL(s), then click Copy or Download. The file is built in your browser and nothing is uploaded, as the Network tab in DevTools will show. Upload it as robots.txt to the top level of your site; each subdomain needs its own.
Robots.txt manages crawling, not indexing: a blocked URL can still appear in search results, so use a noindex tag from the meta tag generator for pages that must stay out. The built-in tester follows the matching rules Google documents, so you can see whether a path is allowed before you publish. Pair the file with an XML list of pages from the sitemap generator.
Table of Use: Settings and the File You Get
| Goal | Setting | Result |
|---|---|---|
| Let every crawler in | Template: Allow All | User-agent: * and Allow: /, plus a Sitemap line |
| Keep a staging site out | Template: Block All | User-agent: * and Disallow: /, with no Sitemap line |
| Hide admin, private and API paths | Template: Standard Website; test /admin/settings | Disallow lines for /admin/, /private/ and /api/; the tester reports Blocked, matched Disallow: /admin/ |
| Skip sort and filter URLs on a shop | Template: E-commerce Site; test /shoes?color=red&sort=price | Disallow: /*?*sort= is matched and the URL is Blocked |
| Opt out of AI training | Template: Block AI Training; test GPTBot and Googlebot on / | Six User-agent lines share Disallow: /; GPTBot is Blocked, Googlebot is Allowed |
| Block PDF files only | Rule: Disallow: /*.pdf$ | /docs/a.pdf is Blocked; /docs/a.pdf?x=1 is Allowed because $ ends the match |
Features
- Five templates: Allow All, Block All, Standard Website, E-commerce Site and Block AI Training. A template replaces every block and the Sitemap field.
- Per-bot blocks: choose All Bots or one of 19 named crawlers, including Googlebot and its image, news and video variants, Bingbot, Applebot, link-preview bots for Facebook and X, and AI bots such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, or type any name. Several names separated by commas share one set of rules.
- Rules and delay: rules go one per line, and an optional Crawl-Delay is written only when it is above zero.
- Sitemap lines: one URL per line, each written as its own Sitemap line and checked for an absolute address.
- Validation warnings: missing colons, paths that do not start with a slash, an unsupported Noindex line, unknown directives, empty or browser-style user-agent names, and blocks with no rules.
- URL tester: enter a path or full URL and a crawler name to get Allowed or Blocked and the rule that matched. A full user-agent string copied from a server log is reduced to the crawler's token, such as GPTBot. It applies the longest-match rule, lets Allow win a tie, treats * as a wildcard and $ as an end anchor, and always allows /robots.txt itself. Google's special-case crawler list (updated 17 September 2026) says AdsBot-Google, AdsBot-Google-Mobile, Mediapartners-Google and APIs-Google ignore the * group, so the tester applies only a group that names them.
- Output: Copy or Download a file named robots.txt, which begins with two comment lines crediting ZipConvert.
It does not fetch or edit your live file, and it does not remember your entries after a reload. Wildcards in robots.txt are not regular expressions; to experiment with real ones, use the regex tester, and to compare a new file with the old one, paste both into the text diff tool.
How to Use the Robots.txt Generator
- Under Common Templates, click the closest match. To add another crawler, click Add User-Agent Block.
- In each block, choose a User-Agent or type one, write the Rules (one per line), and set Crawl-Delay only if you need it.
- Replace the sample address in Sitemap URL(s) with your own.
- Read the Validation Warnings, or the line "No problems found in the rules".
- Under Test a URL against these rules, type a path and a Crawler and check the verdict.
- Click Copy or Download and upload the file to your site's root.
Link-preview crawlers read your pages too, so do not block them if you want rich shares; set the tags with the Open Graph generator. On Apache, redirects and password protection belong in the .htaccess generator, not in robots.txt.
Who Can Use It
Site owners launching a new site or a staging copy, shop managers trimming filter and cart URLs from crawls, publishers choosing whether AI training bots may read their pages, and developers who just need a correct file without remembering the syntax all get the same result: a tested robots.txt. Sort a long list of rules first with the sort lines tool, or encode a non-Latin path with the URL encoder. Structured data for the pages you do allow comes from the schema markup generator.
FAQs
How do I create a robots.txt file?
Click a template such as Allow All or Standard Website, pick a User-Agent for each block, write Allow and Disallow rules one per line, and replace the sample Sitemap URL with yours. The file updates as you type and warnings flag mistakes. Click Download, then upload the file as robots.txt to the root of your site, for example https://yourdomain.com/robots.txt.
Does robots.txt keep a page out of Google search results?
No. Google says robots.txt is not a mechanism for keeping a page out of Google: a disallowed URL can still be indexed, without a snippet, if other sites link to it. To keep a page out of results, let Google crawl it and add a noindex robots meta tag or an X-Robots-Tag header. A Noindex line inside robots.txt is ignored.
How do I block AI crawlers in robots.txt?
Click the Block AI Training template. It disallows GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent with one shared Disallow: / rule, and leaves every other crawler allowed. Blocking Google-Extended does not affect Google Search. Robots.txt is a request that reputable crawlers honour; it cannot stop a bot that ignores it.
What is the difference between Allow and Disallow, and which one wins?
Disallow asks a crawler not to fetch paths that start with the given text; Allow explicitly permits them. When both match a URL, the longer, more specific path wins, and if they are equally long the less restrictive Allow wins. So Disallow: /downloads/ together with Allow: /downloads/public/ blocks everything in downloads except the public folder.
Where do I put the robots.txt file, and how soon do changes apply?
Place it at the top level of the host, such as example.com/robots.txt. Each subdomain, protocol and port needs its own file, and a copy in a subfolder is ignored. As of October 2026, Google generally caches the file for up to 24 hours, and other crawlers follow their own schedules.
Does Google support Crawl-delay in robots.txt?
No. Google lists crawl-delay among the fields it does not support and manages its crawl rate itself, and YandexBot has ignored it since 2018. Bing and some other crawlers do honour it, so the generator lets you set a Crawl-Delay in seconds per block, written only when the value is above zero.
Why does my generated robots.txt mention example.com?
The templates and the Sitemap URL(s) box start with the placeholder https://example.com/sitemap.xml so the layout is clear. Replace it with the full address of your own sitemap, or clear the box to leave out the Sitemap line; the Block All template leaves it empty. A relative path such as /sitemap.xml triggers a validation warning.
Is it safe to enter my site's paths in an online robots.txt generator?
Yes. The file is built and the URL tester runs entirely in your browser, and nothing you type is uploaded, as the Network tab in DevTools will show. Your entries are not saved either, so reloading resets the form to Allow All; copy or download the file first. Remember that a robots.txt file is public, so never list secret URLs in it.