Google’s Fetch & Render new feature (that you’ve probably missed out)

TL;DR – I just found out that Google Search Console brought back the fetch & render Ajax hash bang URLs! On the following post I’ll explain why this silence update that went under the radar is so important.

Don’t have access to Google Search Console? Here are alternative ways to fetch & Render

Hash Bang URLs? Say what?

JavaScript indexing and crawling had probably been one of the most debated topics among SEOs during 2017. However, I could not find any article specifically related to Fetch & Render of Ajax hash bang’s URLs out there.

Just a quick reminder – hash bang (also named Shebang, which is much cooler if you ask me) is a number character sequence that basically looks like this – #!

So, for example, here’s (a not-so-beautiful) hash bang URL:
http://www.example.com/bin/#!/sh/

Fetch & Render Hash Bangs – a brief history

The last mention I actually found on Google, specifically related to this topic was on SEROUNDTABLE on May 2014, an ancient history in terms of the SEO pace, when Google first launched the new shiny fetch & render tool.

However, there was a “temporary” bug – the new feature did not support rendering for Hash bang URLs. Back then; this bug was probably overlooked, as Angular-coded websites were not very common.

It turned out that this temporary flaw became almost permeant, and the only option to render these URLs was to view the HTML snapshot versions of them, or in other words, escaped fragment.

Therefore, the only option to properly fetch & render until very recently was to change the URLs, in our example:
http://www.example.com/bin/_escaped_fragment_ /sh/

So in the past, whenever I wanted to fetch & render the hash bang URL it would only work with the letters before the #! , in our example:
http://www.example.com/bin/

Which is of course NOT what we would expect to see.

On October 2015, Google had made a very clear declaration that they are deprecating the old Ajax crawling scheme. Two months ago, on December 2017 Google officially announced that our beloved Googlebot will (finally) be able to render the #! URL directly

Earlier this week, I innocently tried to render one of my client’s Ajax pages. It actually worked. I tweeted John Muller from Google, and here’s his answer:

I think this happened in the fall, but my sense of time has been skewed recently :). It's useful to be able to test #! URLs directly.

— John ☆.o(≧▽≦)o.☆ (@JohnMu) February 27, 2018

So why is this change so important?

One of Google’s guidelines on this post was as follows:

Test with Search Console’s Fetch & Render. Compare the results of the #! URL and the escaped URL to see any differences.

Well, if before that we could not fetch & render with these URLs, how would we do it now?
I suspect that this change is in accordance to the new update Google had rolled out last December regarding rendering the #! directly.

In other words, if you are still using the escaped fragment, make sure to also fetch & render both versions in order to make sure there is no cloaking.

Moreover, and most important – if Google is supposed to be rendering the hash bang URL directly, we must be able to check these URLs and not only the outdated HTML snapshot.

For example, here is an example from one of my client’s URLs – using escaped fragment:

And this is my latest fetch and render with the hash bang URL:

Drilling down into it, I saw that the blocked resources are completely different on these URLs, an issue that I was not aware of until I had the ability to do it.

Before:

After (same page, this time the ajax script is blocked with medium severity):

Update

A hint for this could be find back in February 2017 as he answered on Twitter:

We’ll support #! for the foreseeable future (others use them too), but we’ll likely render rather than use escaped-fragment

We’ll support #! for the foreseeable future (others use them too), but we’ll likely render rather than use escaped-fragment.

— John ☆.o(≧▽≦)o.☆ (@JohnMu) 22 de febrero de 2017

So what do you think? Would you find it useful?