Robots.txt Generator — build a valid robots.txt in seconds

A robots.txt generator takes the guesswork out of one of the oldest and most misunderstood files on the web. This free tool builds a syntactically correct robots.txt in seconds — pick a preset for WordPress, ecommerce, or a fully custom rule set, decide whether to block the new wave of AI crawlers, and copy the result straight into your site's root directory.

At Arb Digital, we review dozens of robots.txt files a year during technical SEO audits, and the same handful of mistakes show up again and again: blocking CSS and JS files that Google needs to render the page, using robots.txt to try to "hide" a page that's already indexed, or leaving crawl-delay directives that quietly throttle how fast Google can discover new content. This robots.txt generator is built to avoid those traps by default.

What This Robots.txt Generator Does

The tool builds a plain-text robots.txt file made up of one or more User-agent blocks, each followed by Allow and Disallow directives, an optional Crawl-delay, and a Sitemap line pointing search engines to your XML sitemap. Choose a preset to auto-fill sensible defaults, toggle checkboxes to add blocking rules for specific AI crawlers, and the generator assembles everything into a clean, standards-compliant file you can copy with one click.

Every rule you add is visible before you copy anything, so nothing is hidden or auto-injected. What you see in the output box is exactly what gets uploaded to your server.

How to Use It

1. Pick a preset. "Allow all" is the safest default for most small sites. "WordPress standard" blocks admin and login paths while keeping content open. "Ecommerce standard" adds cart, checkout, and account paths.
2. Adjust the User-agent and paths. Leave User-agent: * to target every crawler, or specify a single bot if you need a special rule.
3. Add your sitemap URL. This single line helps every search engine find your sitemap without you submitting it manually everywhere.
4. Decide on AI crawlers. Check the boxes for any AI bots you want to keep out of your training data pipeline.
5. Generate and copy. Click Generate, review the output, then copy it into a file named exactly robots.txt at your domain's root.

How Robots.txt Actually Works

Robots.txt follows the Robots Exclusion Protocol, a voluntary standard that well-behaved crawlers check before requesting any page on your domain. The file lives at https://yourdomain.com/robots.txt — nowhere else — and each search engine bot reads it first, matches its own User-agent name against your rules, then decides which paths it's allowed to request. Google, Bing, and other major engines respect this protocol closely, as documented at robotstxt.org, the original specification site.

Crawling vs. Indexing — The Mistake That Breaks Sites

This is the single most important thing to understand before you touch robots.txt: robots.txt controls crawling, not indexing. Those are two different processes, and confusing them causes real damage. Crawling is Google's bot requesting a page and reading its content. Indexing is Google deciding to store that page and show it in search results. Disallowing a URL in robots.txt only stops the crawl — it does not guarantee the page stays out of search results.

Here's the trap: if other sites link to a page you've disallowed, Google can still index the URL based on that external signal, showing it in search results with no title or description, just the bare link — often worse for your brand than if you'd left it fully crawlable. Worse still, once a page is disallowed, Google's crawler can never fetch it again, which means it can never read a noindex meta tag on that page either. A page blocked in robots.txt is invisible to the very mechanism that would properly remove it from the index.

The correct fix for "I don't want this page in search results" is almost always a <meta name="robots" content="noindex"> tag on the page itself, left fully crawlable so Google can see and obey that instruction. Reserve robots.txt Disallow rules for pages you don't want crawled at all — internal search results, filtered/faceted URLs that create infinite duplicate combinations, staging environments, or admin areas — not for pages you're trying to remove from search.

The AI Crawler Question Every Publisher Now Faces

Since 2023, a new category of crawler has shown up in server logs: bots that don't index your page for search, but scrape your content to train or ground large language models. GPTBot (OpenAI), CCBot (the crawler behind the Common Crawl dataset used by many AI labs), Google-Extended (a separate signal from Googlebot that controls whether your content trains Gemini or feeds Google's AI Overviews), ClaudeBot (Anthropic), and PerplexityBot (Perplexity AI) are the ones site owners ask about most.

There's a real trade-off here, not a clean right answer. Blocking these crawlers keeps your original content out of AI training sets and, in the case of Google-Extended, out of AI Overviews and Gemini responses — which some publishers want, especially if AI-generated answers are satisfying searches without sending a click to the original page. But blocking them also means your brand, your data, and your expertise won't surface when someone asks an AI assistant a question your content could have answered. For a business built on being found — which is most businesses — that trade-off deserves a deliberate decision, not a default. This generator makes each AI crawler an individual checkbox precisely so you choose consciously rather than copy-pasting a blanket block list you found online.

Common Robots.txt Directives Explained

• User-agent — identifies which crawler the following rules apply to. * matches all crawlers that don't have a more specific block.
• Disallow — tells the specified crawler not to request URLs matching this path. An empty value means nothing is blocked.
• Allow — carves out an exception inside a disallowed path, most commonly used to let a specific file (like a CSS or JS asset) through even though its parent folder is blocked.
• Crawl-delay — asks a bot to wait a number of seconds between requests. Google ignores this directive entirely (use Search Console's crawl rate settings instead); Bing and some others still honor it. Use it sparingly — an aggressive delay can slow down how quickly new content gets discovered.
• Sitemap — an absolute URL pointing to your XML sitemap. This is the one directive every search engine reads regardless of which User-agent block it sits under.

What NOT to Put in Robots.txt

Robots.txt is a public file. Anyone — a competitor, a security researcher, a curious visitor — can visit yourdomain.com/robots.txt and read every path you've listed. Never use a Disallow rule to try to "hide" an admin panel, a staging subdirectory with sensitive data, or an unlaunched product page. Listing the path in robots.txt often draws more attention to it than leaving it unlisted, since it publishes the exact URL structure to anyone who checks. If something genuinely needs to stay private, protect it with authentication or a firewall rule, not a text file that politely asks well-behaved bots to look away — because not every bot is well-behaved, and robots.txt is purely voluntary.

Common Mistakes to Avoid

• Blocking CSS/JS folders. Google renders pages like a browser; blocking stylesheet or script directories can make Google see a broken, unstyled page and misjudge its quality.
• Using Disallow to remove indexed pages. It doesn't work reliably and can backfire — use noindex tags on a crawlable page instead.
• Forgetting the sitemap line. It costs one line and helps every crawler find your sitemap without manual submission.
• Typos in the file name or location. It must be exactly robots.txt, lowercase, at the domain root — not in a subfolder, not Robots.TXT.
• Blocking everything on a live site by accident. A leftover Disallow: / from a staging environment is one of the most common causes of a site suddenly vanishing from search results.
• Assuming robots.txt keeps a page out of search results. It only blocks crawling — see the crawling vs. indexing section above.

Related Free Tools From Arb Digital

Once your robots.txt is live, pair it with the XML sitemap generator to build the sitemap you referenced above. If you're planning URL changes, the htaccess redirect generator handles 301s the right way. Check whether a page is actually indexed with the Google cache checker, review your on-page tags with the meta tag generator, add structured data with the schema markup generator, or browse our full free online tools hub.

Frequently Asked Questions