All tools

HTTP X-Robots-Tag Header Builder

Network

Build an X-Robots-Tag response-header value from directives like noindex, nofollow, nosnippet, max-snippet, and unavailable_after, optionally scoped to a specific crawler user-agent, or parse an existing header

Check the flag directives you want (noindex, nofollow, noarchive, and the rest), optionally add a max-snippet, max-image-preview, max-video-preview, or unavailable_after value directive, and get back an assembled X-Robots-Tag header with a plain-English note explaining what each directive does. It uses the same directive vocabulary as the meta name robots tag, but delivered as a header, which is the only option for a resource that isn't HTML, like a PDF or an image, since you can't put a meta tag inside one. You can also scope the whole header to a single crawler by naming a user-agent like googlebot, and it flags obviously contradictory combinations, like pairing all with a restrictive directive. It also parses an existing header value back into its parts, useful for checking what a live response is actually sending.

httpheadersseorobotscrawlingindexing

How to use HTTP X-Robots-Tag Header Builder

  • 1.Check the flag directives you want, and fill in any value directives like max-snippet or unavailable_after.
  • 2.Optionally scope the header to a single crawler by entering a user-agent name like googlebot.
  • 3.Copy the generated header and add it to your server or CDN's response config, or paste an existing X-Robots-Tag value into the parse box to see it explained.

Frequently asked questions

How is this different from a robots meta tag?
Both use the exact same directive vocabulary, but the meta tag only works on an HTML page's head. The X-Robots-Tag header applies to any HTTP response, which is the only way to apply these directives to a PDF, image, or other non-HTML resource.
What does scoping to a user-agent actually do?
Prefixing the header value with a crawler name, like googlebot: noindex, tells only that specific crawler to honor the directives, while every other crawler ignores them. Omit it and the directives apply to any crawler that reads the header.
Does this tool submit anything or crawl my site?
No, it only assembles or parses a header value as a string. You still need to add the generated header to your actual server or CDN response yourself.
What does max-snippet: -1 mean versus 0?
-1 means no limit on snippet length, which is the default behavior with no directive at all. 0 tells search engines to show no text snippet whatsoever, not even a short one.

Use via API, SDK, or MCP

cURL# Free: 1,000 req/day · Pro: 10,000 req/day
curl -X POST https://api.utilix.tech/v1/tools/x-robots-tag-builder \
  -H "Authorization: Bearer utx_live_..." \
  -H "Content-Type: application/json" \
  -d '{"flags":["noindex","nofollow"],"userAgent":"googlebot","header":"noindex, nofollow"}'

Get an API key from your dashboard · Full API docs →

Related tools

HTTP Link Header BuilderBuild an HTTP Link response header from URL/rel pairs (preload, preconnect, prefetch, canonical, pagination) with as/type/crossorigin attributes and a plain-English explanation of each entryNetworkAccept-Language Header Parser / BuilderBuild an Accept-Language request-header value from locale/quality pairs, sorted by preference, or parse an existing header into its ranked locale/quality entriesNetworkCache-Control Header BuilderBuild and explain an HTTP Cache-Control response header from directives like max-age, s-maxage, no-store, and immutableNetwork