header set x-robots-tag noindex nofollow htaccess

Share

If your website pages or XML sitemaps show the HTTP header x-robots-tag: noindex, nofollow, it means that search engines like Google are being told not to index those pages — even if they are live and accessible.
This issue can quietly prevent your site from appearing in Google Search or Google News.

In this article, we’ll explain what the X-Robots-Tag header is, why it appears, and how to fix or control it properly using the .htaccess file on your website.

🧠 What Is the X-Robots-Tag Header?

The X-Robots-Tag is an HTTP header sent by your web server.
It gives instructions to search engine crawlers — similar to the <meta name="robots"> tag — but at the server level.

For example:

x-robots-tag: noindex, nofollow

This tells Google and other crawlers:

  • ❌ Don’t index this page (noindex)

  • ❌ Don’t follow any links on this page (nofollow)

So even if your sitemap is accessible or your posts are SEO-optimized, Google will ignore them if this header is present.

⚠️ Why Does “x-robots-tag: noindex” Appear?

There are several common reasons why your website or sitemap may show this header:

1️⃣ Cache or Security Plugin Settings

If you use plugins like WP Fastest Cache, LiteSpeed Cache, Loginizer, or Rank Math, they sometimes block access to certain files (like /sitemap.xml) by automatically adding “noindex” headers for performance or security reasons.

2️⃣ Server Configuration or CDN Rules

Some hosting control panels, CDNs, or firewalls (like Cloudflare or LiteSpeed) automatically inject the header for XML, JSON, or temporary pages.

3️⃣ Manual .htaccess Rules

If someone previously added a line like this in your .htaccess file:

Header set X-Robots-Tag "noindex, nofollow"

…it will affect every file served by your server — including sitemaps, pages, and media files.

🔍 How to Check if Your Site Has the “X-Robots-Tag: Noindex” Header

You can easily check using the cURL command or an online header checker tool.

🧪 Using cURL (Command Line):

Run this in your terminal or command prompt:

curl -I https://example.com/

If you see this line:

x-robots-tag: noindex, nofollow

that means the page is not indexable.

🛠️ How to Fix “x-robots-tag: noindex, nofollow” in .htaccess

Here’s how to remove or control it safely.

✅ Step 1: Open Your .htaccess File

You’ll find this file in your website’s root directory (usually /public_html/).

You can edit it via:

  • cPanel → File Manager

  • FTP/SFTP Client (like FileZilla)

  • Hosting File Editor

✅ Step 2: Search for Any Line Containing Header set X-Robots-Tag

You might find something like:

Header set X-Robots-Tag "noindex, nofollow"

or:

<FilesMatch "\.(xml|txt)$">
Header set X-Robots-Tag "noindex"
</FilesMatch>

👉 Delete these lines if you want Google to index those files.

✅ Step 3: Allow Indexing Only Where Needed

You can keep the header for private or sensitive files and remove it for public ones.

Example configuration:

# ❌ Block indexing for admin and system files
<FilesMatch "(wp-login\.php|wp-admin|readme\.html|license\.txt)$">
Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

# ✅ Allow indexing for sitemaps and public pages
<FilesMatch "\.(xml|html)$">
Header unset X-Robots-Tag
</FilesMatch>

This way, crawlers can access your XML sitemaps and posts normally.

✅ Step 4: Save and Test Again

After saving the .htaccess file, clear your cache (both plugin and browser), then re-check using:

curl -I https://example.com/sitemap.xml

If it no longer shows x-robots-tag: noindex, you’ve fixed the issue successfully. 🎉

⚙️ Optional: Add a Custom Robots Header (For Better SEO Control)

If you want fine-grained control, you can add different headers for specific file types.

Example:

# Index HTML pages, but block PDF files
<FilesMatch "\.html$">
Header set X-Robots-Tag "index, follow"
</FilesMatch>

<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex, noarchive"
</FilesMatch>

🚀 Bonus Tips

  • Always clear your caching plugin after editing .htaccess.

  • If you use LiteSpeed Cache, purge all caches via the admin bar.

  • Re-fetch your sitemap in Google Search Console after fixing.

  • Wait 2–3 days for Googlebot to recrawl your pages.

🧩 Final Thoughts

The “x-robots-tag: noindex, nofollow” header is a powerful SEO tool — but when applied incorrectly, it can completely stop your website from being indexed.

If your site’s XML sitemaps or posts are returning this header, remove or adjust it in .htaccess immediately.
This single fix can help restore your website’s visibility in Google Search and Google News.

Keywords
x-robots-tag noindex htaccess, header set x-robots-tag nofollow, fix sitemap noindex, WordPress sitemap not indexing, remove noindex header, robots header htaccess

© 2024 by Concept360web.com