7 Alternatives to Google Search Console Fetch & Render

In this post I’m going to show you all possible ways to fetch & render your website externally.

All of these alternatives had been tested and working GREAT right now (in 2020).

Contents

Fetch & Render – Why it is important?
What is the difference between Fetch & Fetch and render?
1. Google mobile Friendly test
2. Screaming Frog SEO Spider
3. Using iframe
4. Technical SEO’s Fetch & Render tool
5. Fetch & Render a page on staging environment
6. Google Chrome
7. Cross-domain redirects

Google Fetch & Render – Why it is important?

Nowadays, one of the most basic 101 SEO checks I perform, is Fetch & Render via the Google Search Console. It’s highly important to check your homepage and other internal pages for the following:

• • • Are any important sources such as CSS, JS files or images blocked and can’t be displayed to Googlebot?
    • In general, are there any significant differences between how Googlebot views the site and between the user?
    • Checking there are no other issues we aren’t aware of, such as malware, cloaking or hidden text.

Unfortunately, we don’t always have direct access to Google Search Console for different reasons (potential clients, delay with the code deployment) and we need to get our way around it.

In case of a dead end, In this post, I will review the list of all the possible ways to fetch & Render a page externally.

What is the difference between fetch & fetch and render?

While fetch the URL will display you with the basic HTML that comes from the server (which means – it doesn’t use any other sources such as CSS & JS) fetch & render will run the HTML with all the other resources on the URL.
So in case we got a very heavy reliant JS website, only the render version is supposed to show us how Google actually sees the page (and if it is similar to what the users are seeing).

For the sake of this post, I wrote only about tools that will enable us to visualize to DOM, therefore I won’t mention some tools such as rich results and the structured data testing tool, as they will render the page, but won’t display how the page will look like.

1. Google mobile Friendly test 

From my experience, this is the most reliable external tool, in order to see how Google fetch & renders the page. The reason is obvious –this is an official testing tool by Google, so it should display how Google actually renders your mobile website.

As it name suggests, the test is only relevant for mobile websites. However, now with the mobile first index is rolling out it’s highly important so make sure you can cover that. Second, if your site is responsive – mobile rendering should be similar to the desktop and if there are issues with the mobile display, it will be applicable to desktop as well.

2. Screaming Frog SEO Spider

Probably the most powerful SEO tool (alongside SEMRUSH) I use for my SEO audits. On one of their latest updates, they added a highly useful feature that enables you to render angular.

Please do as follows:
2.1. Configuration –
Go to configuration – > Spider configuration – > Rendering

Change the default option (Old AJAX crawling scheme, as it was depreciated by Google) to JavaScript.

Make sure that “enable render page screenshots” is ticked and choose between Googlebot desktop or Googlebot mobile. For now, keep the Ajax timeout on the 5 seconds default as this is usually the maximum that Googlebot is willing to wait (more on that in a second).

Our tests indicate Googlebot is willing to wait (approx) 5 secs for their snapshot of rendered content btw. Needs to be in well before then.

— Screaming Frog (@screamingfrog) October 13, 2016

Then, go to configuration -> Basic
And make sure that the CSS, JS, images and external links are ticked:

2.2. Crawling – like every crawl on screaming frog, just paste your root domain above and click start. Bear in mind that this process will be a bit slower than usual, as the spider is waiting for all the resources to be fetched and rendered.

2.3. Check for render page and blocked resources –

For every desired URL, go to lowest tabs and change to “rendered page” to check for the actual rendering. Also, it’s very important to change the filter on the left to “blocked resources” to make sure there are no important internal resources (with an emphasis to CSS & JS files) which are blocked.

2.4. In case the rendered page is blank or missing some crucial elements, adjust the Ajax timeout to 7-10 seconds:

If you see that after changing the Ajax timeout the page is rendered properly, it might indicate that your page is loading too slow, and hence Googlebot won’t be able to render your content. I’ve done this test myself with one website and saw the differences between the default 5 seconds (blank screen) and a few seconds more (full rendering of the page).

All and all, it’s a great way to check Angular sites for rendering as well as blocked resources.

3. Using iframe

Another killer tip from Screaming Frog blog.

Do as follow:
3.1. Find a domain that you already are the owners of its GSC and have access to its FTP.

3.2. Add the following PHP file from github

3.3. Change the URL on the 11th line from example.com to your domain.
Preferably load an insecure domain, because you can’t load iframe of HTTPS on HTTP.

3.4. Open a folder on your FTP (called it for the example sake – /render/) and upload the PHP file to it.

3.5. Now check any external URL, just change the example.com to your preferable URL.

In the screenshot below, my sample site was musichrager.com which I own (and deploys HTTP) , the folder I opened for the sake of it called /render/ (you can name it whatever you would like to) and I rendered a Wikipedia page. It looked like this:

Also, you can see the blocked resources:

Although on the original post they added the parameter “type”, from my checks it’s unnecessary. All you have to do is like a normal site rendering –to choose between desktop and mobile. That will do the job.

4. Technical SEO’s Fetch & Render tool

It simple as can be. Just paste the URL, choose the user-agent and then fetch & render. You can also see the blocked resources and fix them accordingly.

I asked Max Prin, the developer to explain about the methodology of the tool:

The page is rendered with PhantomJS (headless browser) and each asset (img, CSS, JS…) URL is checked against the robots.txt file to see if it’s blocked. I’d say the script is 99% reliable. Some syntaxes in the robots.txt file lead to errors but it’s extremely rare.

One disclaimer – although I use this tool very often for my audits, I would recommend not to rely solely on this, as I found out at least a few times some discrepancies between the rendering in here and rendering in GSC (for websites I had access to).

5. Fetch & Render a page on staging environment (via opensourceseo)

While we should have an access to staging site, usually they are blocked (if not – they should, otherwise expect it to get indexed quite easily, even without any given links). The most common approaches are by robots.txt or by IP.

According to opensourceseo, Google always using IP that start with 66. Therefore, all we need to do is to whitelist it. In case your site is hosted with an Apache server, please add the following line to your HTACESS file:
ErrorDocument 403 /error.php
Order Allow,Deny
Allow from 66.

Now you can fetch & render your staging site. Just make sure that it doesn’t require authentication, otherwise Googlebot won’t be able to access it.

After finish with your testing, is the best to remove the line from your htaccess .

6. Google Chrome

Update (May 2019) – Google announced that from now on Googlebot will regularly update its rendering engine to the latest version of Chromium

According to Google documentation, web rendering is based on Chrome 41:

Rendering on Google Search: https://t.co/bgacWwNyz8 – web rendering is based on Chrome 41; use feature detection, polyfills, and log errors! pic.twitter.com/pNptcvpjFo

— Ilya Grigorik (@igrigorik) 4 de agosto de 2017

So although you can’t see the blocked resources (because the web browser will render it anyway) , it’s a highly powerful tool to experience and investigate rendering issues firsthand, as this is the equivalent of the web rendering service (WRS) Google deploys.

Do as follow:
1. Download and install Chrome 41. While you can’t download it directly from the official site, you can download it from here. It’s recommended to uninstall the newer version before , and make sure to backup all your bookmarks and files.
2. In Google Chrome, enter any URL or website you would like to investigate. Right click and click “investigate” will open the Chrome Developers Tools. Then click on the “console” tab and there you will see all the errors:

3. Send it to your development team to investigate the issues.

7. Cross-domain redirects

This method is a bit “grey-hat”, so please be aware of it and use it with caution. Oliver from OHGM had come up with a brilliant hack:

• Own a search console account, any account (doesn’t have to be related to the website you need to fetch and crawl)
• Add a redirect from any given URL on the domain you own to the property you don’t own.
• Inspect the redirect URL with the URL inspection via the Search Console.
• Time to test the live URL

At the image below, you can see how he managed to fetch and crawl Google’s documentation page:

Final Thoughts

I tried to reviews all the external tools and tricks I’ve worked with. Did I miss anything? Do you have another nifty trick? Hit me in the comment section.