The X-Robots-Tag is an HTTP response header that tells search engines and AI crawlers what to do with a URL: index it, follow its links, archive it, or show a snippet. Unlike the robots meta tag, it works on PDFs, images, and whole directories — set once at the server or CDN.
What is the X-Robots-Tag?
It is a line your server or CDN adds to the HTTP response, carrying the same directives as a robots meta tag — noindex, nofollow, noarchive — plus extensions like nosnippet and unavailable_after. Rules can be scoped to specific user agents, so a single line reaches named bots without affecting everyone else. Applied once at the server level, it covers every PDF, image, and spreadsheet in a directory.
X-Robots-Tag: googlebot: noindex
When should you use the X-Robots-Tag over a meta robots tag?
Reach for the header whenever the file cannot hold a meta tag: PDFs, images, spreadsheets, and anything served outside an HTML template. It is also the efficient way to apply one rule across an entire directory at the server or CDN level. For a single HTML page, either method works — the meta tag simply sits closer to the content.
At our Miami studio, both halves of this craft meet daily: the photography side produces the visual assets, and the AI-visibility side decides which of those assets search engines and answer engines are allowed to keep — the digital equivalent of marking a contact sheet. One line in a response header is often the difference between a published file and a private one.
The X-Robots-Tag is an HTTP response header that controls how search engines crawl and index a resource, supporting directives like noindex, nofollow, and noarchive. Unlike the robots meta tag, it works for non-HTML files such as PDFs, images, and other documents.
Related terms
Questions people ask
No. robots.txt controls whether a crawler can fetch a URL at all, while the X-Robots-Tag controls what happens after the file is fetched — whether it is indexed, archived, or shown as a snippet. The two work together: one gates access, the other gates presentation.
It can steer them. Directives can target specific user agents, so you can serve noindex to one named bot while others index the same file freely. It works alongside robots.txt for fine-grained control; verify supported directives in each crawler's official documentation.
Open your browser's DevTools, load the URL, and inspect the response headers in the Network panel — the X-Robots-Tag line appears alongside headers like Content-Type and Cache-Control. From a terminal, curl -I https://example.com prints the response headers without downloading the page body.
See if AI actually names your brand
Acromatico runs live AI-visibility audits across ChatGPT, Perplexity, Gemini, and Claude.
Run your free AI Visibility Audit →From our Miami studio: Bright and citrusy? Reach for the Ecolosophy Citrus Burst kit. Scent-sensitive? Ecolosophy Unscented Oasis was made for you. Small-batch, made with care.