robots.txt Generator — Build a Crawler Rules File Online
Build a robots.txt with User-agent, Allow, Disallow, Crawl-delay and Sitemap directives, combining multiple crawler rule blocks in one file.
User-agent: * Disallow: /admin Disallow: /private
Test a URL against these rules
What this robots.txt generator does
A robots.txt file lives at the root of your domain and tells search engine crawlers which parts of your site they may or may not fetch. This generator builds that file for you from plain inputs: set the User-agent the rules apply to (use * for all bots or a name like Googlebot), list the paths to Allow and Disallow one per line, optionally add a Crawl-delay in seconds, and point crawlers to your Sitemap URL. Add a crawler and you get a second block with rules of its own, so a common file — let Googlebot in and shut every other bot out — is written here in one pass instead of generated twice and pasted together. To skip the typing, the Start from… menu fills the crawler blocks with a ready-made file — allow everything, block everything, Googlebot only, block AI crawlers such as GPTBot and ClaudeBot, or the rules WordPress, Drupal and Joomla ship with — and keeps your sitemap URL. Beside it, Add rules… adds instead of replacing: “Let crawlers fetch CSS and JavaScript” and “Let crawlers fetch images” put Allow lines for those files into every crawler block, skipping any rule a block already has — search engines need a page’s CSS and scripts to render it. Under those menus, Block AI crawlers one by one lists each crawler the AI preset blocks, with the company that runs it: tick one to add a group that closes the whole site to it, untick it to take that group away, and nothing else in the file changes; a crawler you have already given other rules of its own is greyed out and left for you to edit in its block. Whenever a block disallows / and allows nothing, the page warns that it closes the whole site to that crawler, or, for *, to every crawler without a group of its own. The output updates live as you type and one click copies the whole file, ready to save as robots.txt in your site root.
How the file is assembled
Each crawler you add becomes its own block: a User-agent: line first, then every Allow: path, then every Disallow: path, keeping a predictable order so your file stays readable and diff-friendly. A blank line separates one block from the next to keep the file readable; a crawler starts a new group at the next User-agent line. Blank and whitespace-only lines are ignored, so a stray empty row never becomes an invalid directive. A Crawl-delay: line is added only when you enter a positive number, and a trailing Sitemap: line is appended after a blank separator when you supply a sitemap URL. The result follows the widely supported robots exclusion format that Google, Bing and other crawlers understand.
Frequently asked questions
Where do I put the generated robots.txt file?
It must sit at the root of your domain, reachable at https://yourdomain.com/robots.txt. Crawlers only look there — a robots.txt inside a subfolder is ignored. Copy the output, save it as a plain text file named robots.txt, and upload it to your site root.
Does Disallow actually hide a page from Google?
No. Disallow asks well-behaved crawlers not to fetch a path, but it does not guarantee secrecy and a blocked URL can still appear in results if other pages link to it. To keep a page out of search results use a noindex meta tag or password protection instead.
Can I check whether a URL would be blocked?
Yes. Under the file, type a crawler name and a URL or path: the page says Allowed or Blocked and names the rule and the user-agent group that decided it, matching the way the robots exclusion standard does.
Is there a ready-made robots.txt for WordPress, Drupal or Joomla?
Yes. Pick one in the Start from… menu and the crawler blocks are replaced with the rules that platform's own file ships — for WordPress, the admin folder closed with admin-ajax.php left open. The same menu starts you from allow everything, block everything, Googlebot only, or AI crawlers blocked with everyone else allowed. Your sitemap URL is kept, and every rule stays editable afterwards.
Is there a quick way to let crawlers fetch CSS, JavaScript and images?
Yes. Add rules…, beside Start from…, adds a block of Allow lines to every crawler group instead of replacing the file: “Let crawlers fetch CSS and JavaScript” writes /*.css$, /*.css?, /*.js$ and /*.js?, and “Let crawlers fetch images” writes a line for each common image type, from /*.avif to /*.webp. A rule a group already has is not written twice, and every line stays editable. Search engines need a page’s CSS and scripts to render it, which is why these are the usual files to open; the URL check under the file shows which rule wins for any address you type.