All tools

robots.txt Generator & Validator

Network

Generate and validate robots.txt files, and build or parse robots meta tags and X-Robots-Tag headers, in one place

Was this tool helpful?
robots.txtrobotsseocrawlergeneratorvalidatornoindexnofollowmeta-robotsx-robots-tagsitemapgooglebot

About robots.txt Generator & Validator

Everything that tells crawlers what to do, on one page with four modes. Generate a robots.txt from User-agent groups, Allow and Disallow rules, an optional Crawl-delay and your Sitemap URLs; validate an existing robots.txt and get flagged on rules placed before any User-agent, unknown directives, relative Sitemap URLs, malformed Crawl-delay values and duplicate rules; and build or parse the page-level controls, the robots meta tag and the X-Robots-Tag response header, from directives like noindex, nofollow, nosnippet, max-snippet and unavailable_after. Keeping them together matters because they do different jobs: robots.txt controls crawling, while noindex controls indexing, and a page blocked in robots.txt can still be indexed because the crawler never sees its noindex. Everything runs in your browser.

How to use robots.txt Generator & Validator

  • 1.Start in Generate robots.txt to build a file from crawler groups and rules, then copy it to the root of your site.
  • 2.Switch to Validate to paste an existing robots.txt and see each User-agent group parsed out with any syntax problems flagged by line.
  • 3.Use the Robots meta tag mode for per-page HTML control, or the X-Robots-Tag mode for files that have no HTML head, such as PDFs and images.
  • 4.Scope any meta tag or header to a single crawler (for example googlebot) when you only want to change how one search engine treats a page.

Frequently asked questions

Should I use robots.txt or noindex to keep a page out of Google?
Use noindex (a robots meta tag or an X-Robots-Tag header) and leave the page crawlable. robots.txt only stops crawling; a blocked URL can still appear in results if other sites link to it, because the crawler never fetches the page and never sees the noindex.
When do I need X-Robots-Tag instead of a meta tag?
For anything that is not an HTML page, such as PDFs, images or API responses, since there is no head element to put a meta tag in. It is also convenient for applying a rule to a whole path from server config.
Does Google respect Crawl-delay?
No. Googlebot ignores Crawl-delay; Bing and some other crawlers honor it. The validator still checks that the value is a valid number.
Does the validator fetch my live robots.txt?
No. Paste the file contents; nothing is fetched or uploaded.
Why did an old Utilix tool page redirect me here?
Utilix merged several single-purpose tools into the robots.txt Generator & Validator. Each old URL now opens this page with its mode preselected (via ?mode=<old-tool-slug>), so bookmarks and links keep working. The old tools' REST endpoints, Node.js and Python SDK functions, and MCP tools are unchanged.

Related tools

Sitemap.xml Validator & SplitterValidate a sitemap.xml or sitemap index against the sitemaps.org protocol, and split a URL list over the 50,000-URL cap into multiple sitemaps plus an indexNetwork.editorconfig ValidatorParse an .editorconfig file into its root flag and glob sections, and flag structural mistakes: unknown properties, invalid values for known properties, a root declaration outside the preamble, and duplicate properties within a sectionCodeARIA Role & Attribute ValidatorCheck every role and aria-* attribute in a pasted HTML snippet against a curated WAI-ARIA 1.2 role table: unknown roles, attributes not permitted on a given role, and widget roles missing a required aria-* stateCode