5 Ways to Fetch & Render Sites (without having access to GSC)

Nowadays, one of the most basic 101 SEO checks I preform, is Fetch & Render via the Google Search Console. It’s highly important to check your homepage and other internal pages for the following:

      • Are any important sources such as CSS, JS files or images blocked and can’t be displayed to Googlebot?
      • In general, are there any significant differences between how Googlebot views the site and between the user?
      • Checking there are no other issues we aren’t aware of, such as malware, cloaking or hidden text.

Unfortunately, we don’t always have a direct access to Google Search Console for different reasons (potential clients, delay with the code deployment) and we need to get our way around it.

In case of a dead end, here is the list of all the possible ways to fetch & Render a page externally:

1. Google mobile Friendly test 

From my experience, this is the most reliable external tool, in order to see how Google fetch & renders the page. The reason is obvious –this is an official testing tool by Google, so it should display how Google actually render your mobile website.

As it name suggests, the test is only relevant for mobile websites. However, now with the mobile first index is rolling out it’s highly important so make sure you can cover that. Second, if your site is responsive – mobile rendering should be similar to the desktop and if there are issues with the mobile display, it will be applicable to desktop as well.

Google Mobile Friendly tool

2. Screaming Frog SEO Spider

Probably the most powerful SEO tool (alongside SEMRUSH) I use for my SEO audits. On one of their latest updates (February 2018) they added a highly useful feature that enables you to render angular.

Please do as follows:
2.1. Configuration –
Go to configuration – > Spider configuration – > Rendering

screaming frog 1

Change the default option (Old AJAX crawling scheme, as it was depreciated by Google) to JavaScript.

screaming frog 2

Make sure that “enable render page screen shots” is ticked and choose between Googlebot desktop or Googlebot mobile. For now, keep the Ajax timeout on the 5 seconds default as this is usually the maximum that Googlebot is willing to wait (more on that in a second).

Then, go to configuration -> Basic
And make sure that the CSS, JS, images and external links are ticked:

2.2. Crawling – like every crawl on screaming frog, just paste your root domain above and click start. Bear in mind that this process will be a bit slower than usual, as the spider is waiting for all the resources to be fetched and render.

2.3. Check for render page and blocked resources –

For every desired URL, go to lowest tabs and change to “rendered page” to check for the actual rendering. Also, it’s very important to change the filter on the left to “blocked resources” to make sure there are no important internal resources (with an emphasis to CSS & JS files) which are blocked.

2.4. In case you your rendered page is blank or missing some crucial elements, adjust the Ajax timeout to 7-10 seconds:

If you see that after changing the Ajax timeout the page is rendered properly, it might indicate that your page is loading too slow, and hence Googlebot won’t be able to render your content. I’ve done this test myself with one website and saw the differences between the default 5 seconds (blank screen) and a few seconds more (full rendering of the page).

All and all, it’s a great way to check Angular sites for rendering as well as blocked resources.

3. Using iframe

Another killer tip from Screaming Frog blog.

Do as follow:
3.1. Find a domain that you already are the owners of its GSC and have access to its FTP.

3.2. Add the following PHP file from github

3.3. Change the URL on the 11th line from example.com to your domain.
Preferably load an insecure domain, because you can’t load iframe of HTTPS on HTTP.

3.4. Open a folder on your FTP (called it for the example sake – /render/) and upload the PHP file to it.

3.5. Now check any external URL, just change the example.com to your preferable URL.
http://www.yourdomain.com/render/?url=http://www.example.com

In the screenshot below, my sample site was musichrager.com which I own (and deploys HTTP) , the folder I opened for the sake of it called /render/ (you can name it whatever you would like to) and I rendered a Wikipedia page. It looked like this:

Also, you can see the blocked resources:

Although on the original post they added the parameter “type”, from my checks it’s unnecessary. All you have to do is like a normal site rendering –to choose between desktop and mobile. That will do the job.

4. Technical SEO’s Fetch & Render tool

It simple as can be. Just paste the URL, choose the user-agent and then fetch & render. You can also see the blocked resources and fix them accordingly.

I asked Max Prin, the developer to explain about the methodology of the tool:

The page is rendered with PhantomJS (headless browser) and each asset (img, CSS, JS…) URL is checked against the robots.txt file to see if it’s blocked. I’d say the script is 99% reliable. Some syntaxes in the robots.txt file lead to errors but it’s extremely rare.

One disclaimer – although I use this tool very often for my audits, I would recommend not to rely solely on this, as I found out at least a few times some discrepancies between the rendering in here and rendering in GSC (for websites I had access to).

5. Fetch & Render a page on staging environment (via opensourceseo)

While we should have an access to staging site, usually they are blocked (if not – they should, otherwise expect it to get indexed quiet easily, even without any given links). The most common approaches are by robots.txt or by IP.

According to opensourceseo, Google always using IP that start with 66. So all we need to do is to whitelist it. In case your site is hosted with an Apache server, please add the following line to your HTACESS file:
ErrorDocument 403 /error.php
Order Allow,Deny
Allow from 66.

Now you can fetch & render your staging site. Just make sure that it doesn’t require authentication, otherwise Googlebot won’t be able to access it.

After finish with your testing, is the best to remove the line from your htaccess .

TL;DR

I tried to reviews all the external tools and tricks I’ve worked with. Did I miss anything? Do you have another nifty trick? Hit me in the comment section.

Posted by Roy Skif

Leave a Reply