Is Google Misreading My robots.txt? A Curious Case of Unintended Blocking

I’ve encountered a puzzling issue with a client’s robots.txt file that I could use some insights on. Despite not explicitly blocking the /uk/ folder in the robots.txt file, Googlebot is persistently reporting that it’s being blocked.

This bug is not just a technical inconvenience—it’s directly impacting my client’s sales and conversions.

Here’s the setup:

• The robots.txt does not disallow /uk/

• I checked the robots.txt Tester in Search Console, and it confirms that the /uk/ folder should be accessible.

• I used the URL Inspection tool to live test the robots.txt, and it shows that the folder is not blocked.

• 

• I verified that the non-www version of the domain correctly redirects to the canonical page and doesn’t have a different robots.txt file that could cause confusion.
• I also tested the robots.txt file using different user-agents via WebSniffer to ensure there’s no cloaking or alternate rules for specific bots.

Additionally:

• Every live test of the /uk/ folder URL’s using the URL Inspection tool in Search Console returns an error.

• When I submit an indexation request, I get another error message

Despite all indications showing that the /uk/ folder should be accessible, Google Search Console still flags it as “blocked by robots.txt” under both “Crawl allowed” and “Page fetch.”

Why This Matters

We’re not talking about some unimportant pages with little organic traffic here. This is an eCommerce site that relies heavily on the UK market.

1. Impact on Link Equity
   If Google keeps interpreting the /uk/ folder as blocked, any links pointing to it won’t pass link equity . This could have a broader impact on the site’s overall performance and rankings in search results.
2. Sales and Conversions
   Even if UK users manage to find the store via the main URLs instead of the localized ones, this issue could still negatively affect user experience, resulting in lost sales and conversions in the UK market.
3. Google Merchant Center
   The /uk/ products are also blocked in Google Merchant Center, affecting both paid and organic listings. We already contacted support, but their response didn’t address the issue. They suggested modifying the robots.txt file to allow a full-site crawl as follows:

User-agent: Googlebot
Disallow:

User-agent: Googlebot-image
Disallow:

However, this wouldn’t help since the /uk/ block doesn’t exist in any scenario I’ve checked.

Conclusion

I’ve exhausted all possible troubleshooting steps, and I’m convinced this is a bug on Google’s end. If any Googlers or experienced SEOs have encountered this issue or know of a workaround, your help would be greatly appreciated!

via GIPHY

Update

After receiving feedback from John Mueller , we identified that the issue was due to a country redirect on the site. The /uk/ folder was being redirected to /stores/store/switch/?__from_store=uk(...), which made it challenging for Googlebot (which crawls from the US) to access the proper content. This country-based redirection was causing SEO issues.

To resolve it, we removed the following line from the robots.txt file:

Disallow: /*?

This line was inadvertently blocking the URL pattern /stores/store/switch/?__from_store=uk, which is essential for directing users to the correct store version. Removing it allowed Googlebot to properly crawl the content. While the URL was indexed, it wasn’t with crawled content, and fixing this has ensured that the /uk/ folder is now properly indexed, helping restore visibility in search results.

Thanks to this adjustment, the issue is no longer affecting the site’s sales and conversions.