One Of The Best SEO Tools – Screaming Frog – Part 1
Screaming Frog has been my first choice SEO tool for a while now. It has the power to uncover almost all potential issues with your site for improved analysis. It is important to improve website health by finding problems and fixing them before optimising other areas of your site. Let’s see how Screaming Frog helps you achieve that.
Main Benefits:
Screaming Frog makes it easy to find broken links (404s) and server errors on your site. Fixing these allows search engine spiders to seamlessly crawl your site and navigate every page. It’s also always beneficial to spot (and update on-site) temporary and permanent redirects, which can be done with Screaming Frog without any hassle.
Checking and analysing page title, meta tags and headings to resolve potential authority issues is no longer difficult either, and duplicate content is much easier to find with Screaming Frog’s data.
As an added plus, it also allows for XML Sitemap generation and integration with Google Analytics.
Versions:
A free version is available but it has limitation to crawl just 500 pages – this is sufficient for smaller sites. For larger sites, it is worth buying the full paid version especially when we see the benefits and ease of use when crawling much larger sites. A new version for Screaming Frog (9.0) has recently been released with several useful features that complement the ones we’ve already been utilising. The most useful new features include the following.
The ability to store the HTML source code with the crawl file that can be used to take websites’ HTML backups and for future reference.
In addition to that, further configuration options have been added to the XML sitemap export function. e.g. pages with ‘noindex’ tag or pages with server errors or redirection can easily be excluded from the resulted sitemap.
Moreover, a powerful search function has been introduced to quickly locate pages with the custom field search.
Now, let’s take a look at some useful configurations:
The newly added feature of storing HTML, that we just discussed above, can be found under Configuration > Spider > Advance Tab.
Another useful configuration is the speed, especially when your hosting server is lacking resources or there are heavy traffic or load on the website. It allows you to decrease the maximum number of threads when Screaming Frog crawls the website to adjust the load accordingly so that when it crawls, it doesn’t hit the website hard.
What if the web server is blocking Screaming Frog spider or the Googlebot to crawl the website? Or if there is a customised behaviour depending on the type of browsers, or if you have a separate mobile site? In this scenario, Configuration > User Agent is the section required.
Custom Search under Configuration helps to find up to 10 search phrases in the source code per crawl. It can be useful to find empty category pages from an e-commerce website or to find opportunities to improve internal linking for a specific key phrase.
Finally, there is an option to combine the power of Google Search Console, Google Analytics, Majestic, Ahrefs and Moz with Screaming Frog crawl, which is very useful to figure out the pages with less traffic or clicks, or the ones driving the most traffic to the website to further optimise them.
This doesn’t end here…
So, we’ve gotten this far with Part 1 of the series: “One Of The Best SEO Tools – Screaming Frog” Although we’ve covered the newly added features and useful configuration, there is still a lot more to cover and learn. So, don’t miss the next article if you want to find out how to use this powerful software. Stay tuned for the next post, PART 2.
After completing a Masters of Software Engineering in 2005, Ghulam started his career as a web developer and SEO specialist. A committed and dedicated team player with a keen sense of development and management, he focuses on results, client satisfaction and company growth. His goal is to broaden his experience and utilise strong engineering and web development skills in a challenging and progressive environment while at the same time achieving pecuniary benefit.
Ghulam has been working at StewArt Media as an SEO Web Developer & Analyst since 2016.
