Screaming Frog is a widely loved SEO toolkit by the online marketing community. This tool is extremely powerful – if utilized well, it can improve a website’s SEO strength and rankings. Below, we provide guidance on using Screaming Frog to perform typical tasks when auditing a website.
What is Screaming Frog?
Screaming Frog is a comprehensive SEO toolkit that allows you to perform tasks related to SEO optimization for a website. This software works like an “SEO spider”, also known as the search bots of search engines (like Google, Bing, etc.)
This way, you will know how search engines evaluate your website. All the unoptimized details that negatively impact SEO performance will be listed out and easily searched using filters. You just need to fix those errors until the error count goes back to 0.
Of course, Screaming Frog also has many other in-depth tools and applications to improve SEO – the secret techniques only known by experienced online marketing experts.
User Interface
The UI (User Interface) of Screaming Frog is very intuitive, it only takes a few minutes for you to quickly grasp it. You will immediately notice the address bar to enter the URL of the website you want to audit. Above the address bar is the Menu bar containing the main categories of the software.
On the other hand, right below the address bar is a tab bar with each tab representing an aspect that needs attention in SEO. Specifically, you have:
- Internal and External: internal links and outbound links of the website
- URL: paths to pages within the website
- Meta keywords: view the page’s keywords
- Meta description: view the page’s description
- Content: statistics on the content of each page
…
There are many other tabs that allow you to audit a website comprehensively and in detail.
Note: Below the tab bar is a details window that lists all the crawled URLs along with all their metrics.
In addition, there is another tab bar located at the bottom corner of the Screaming Frog window that allows you to examine different aspects of each page. The content of this tab also changes according to the different URLs you click on in the details window above.
Looking to the right column, you will see the top half is the Filter and the bottom half is the statistics window. Both of these tools change based on the tab you select on the main tab bar. When auditing a website, the filter is a powerful tool to help you filter out issues and handle them quickly.
How to Use Screaming Frog to Audit a Website
1. Customize website data collection (crawl)
You need to select the items you want to crawl. This will help ensure the research results are as accurate as possible. The specific steps are as follows:
Step 1: To customize the types of data to crawl, you can go to the Configuration item on the toolbar.
Step 2: Then select Spider, the search bot customization window will appear.
Step 3: Here you can select/deselect all the desired items belonging to the Resource Links group (Images, CSS, …) and Page Links group (Internal Hyperlinks, External Links, Canonicals, …).
2. Audit links
Also in the Spider Configuration window, you can also perform the task of auditing the website’s links. This task consists of 4 steps:
Step 1: In the Crawl Behavior section, select [Crawl All Subdomains].
Step 2: If you want to also check nofollow links, you can tick the 2 boxes [Follow Internal “nofollow”] and [Follow External “nofollow”].
Step 3: Click [OK] and start crawling a website, after getting the results, press the [Export] button next to the List/Table View button.
Step 4: Choose the type of export file you want, such as .csv or Excel file (.xlsx) … then click OK.
This way you have a list of backlinks pointing to the website being audited.
3. Check response codes
To check the response code, you need to crawl the website first. All the features of Screaming Frog mentioned below also require this. So from now on, assume that you have the results of crawling the website before performing any task.
Switch to the Response Codes tab from the top tab bar. Screaming Frog will list all response codes and status of all links on your website.
Then use the filter right below the tab bar or the column on your right to quickly filter the response codes. Accordingly, links with 2xx codes will work normally and smoothly. Meanwhile, 3xx, 4xx and 5xx codes signal abnormalities and require your intervention. Specifically:
- Code 301: Permanent Redirect
- Code 302: Temporary Redirect
- Code 404: URL not found – URL pointing to a non-existent page
- Code 500: Server Error – server error (usually due to overload, out of bandwidth, CPU overflow, …)
- Code 501: Unavailable – Website server is temporarily down
4. Check URLs
Optimizing URLs is an important part of SEO. So Screaming Frog also supports you in checking URLs on the website. The steps are as follows:
Step 1: To check the URLs of the website, especially the homepage and other important pages, switch to the URL tab.
Step 2: You will see the link details window doesn’t change much. But the filter frame will give you many types of bad URLs that negatively affect SEO.
Among them:
- Uppercase, Multiple Slashes and Containing Space: URLs are recommended not to contain uppercase letters, not to have more than 1 consecutive / and not to contain spaces (spaces are converted by computers to % when displayed in URLs).
- Underscores: underscore (_) is a character not allowed to appear in URLs.
- Non ASCII characters: characters not included in the ASCII encoded character table (such as “đ”, “â”, “ớ”, …), making it difficult for computers to recognize URLs.
- Over 115 characters: URLs longer than 115 characters, according to research, affect SEO efficiency and reduce the user’s ability to remember URLs.
5. Check title tags
Title tags play an extremely important role in SEO. To check the title tag, switch to the Page Titles tab. You will see some common title errors listed in the filter frame, including:
- Missing: The page is missing a title tag
- Multiple: The page has more than one title tag
- Over 65 Characters: The page title is over 65 characters long
- Duplicate: The page title is duplicated with some other page
- Below 30 Characters: The page title is less than 30 characters long
Each type of error will affect the SEO effect differently. You should fix them as soon as possible for the best results.
In summary, Screaming Frog is an extremely powerful SEO tool trusted by many marketers. Hopefully the instructions on how to use Screaming Frog to audit websites above will help you make the most of this tool.
6. Check meta descriptions
The meta description is the snippet of text that appears under the page title in search results. It plays an important role in attracting users to click through to your website. To check the meta descriptions, switch to the Meta Description tab. Similar to title tags, you will see some common meta description errors listed in the filter frame, including:
- Missing: The page is missing a meta description
- Multiple: The page has more than one meta description
- Over 160 Characters: The meta description is over 160 characters long
- Duplicate: The meta description is duplicated with some other page
- Below 70 Characters: The meta description is less than 70 characters long
Try to write compelling, unique meta descriptions of appropriate length for each important page to improve click-through rates from search results.
7. Analyze page content
Content is king in SEO. Screaming Frog provides a quick way to analyze the content of your web pages. Switch to the Content tab and you’ll see metrics like word count, number of headers (H1, H2, etc.), and content that appears in the viewport when the page first loads (1st Content).
Some things to look out for:
- Pages with thin content (low word count)
- Pages missing H1 tags or with multiple H1s
- Key pages where the main content doesn’t appear high up in the 1st Content
Beef up thin pages, ensure proper header tag usage, and optimize content structure to get your key content appearing as early as possible.
8. Generate an XML sitemap
An XML sitemap helps search engines properly crawl and index all the important pages on your site. Screaming Frog can generate one for you.
After crawling your site, go to Sitemaps > Create XML Sitemap in the top menu. Choose which types of pages to include, select a Last Modified date option if desired, and click Next. Choose a save location and your sitemap will be generated. You can then submit it to Google via Search Console.
9. Discover orphan pages
Orphan pages are pages on your site that have no internal links pointing to them. This makes them hard for search engines to find. To locate them:
After crawling your site, go to Bulk Export > Response Codes > Client Error (4xx) Inlinks. This exports a list of pages that returned a 4xx error when Screaming Frog tried to crawl them.
Any pages on your site in this list may be orphan pages (assuming the 4xx error isn’t just from a broken link). Inspect them manually to confirm, then either add internal links to the page, direct it to another relevant page with a 301 redirect, or delete it if it’s unimportant.
10. Audit hreflang tags
If you have a multilingual site, hreflang tags are crucial for telling Google which page to show users of different languages. Screaming Frog can audit these tags.
After crawling, go to Configuration > Spider > Crawl and check “Crawl Hreflang”. Then go to the Hreflang tab after the crawl finishes. You’ll see which pages have hreflang tags and can identify any issues like missing return links, incorrect language codes, or hreflang conflicts in the filters pane on the right.
As you can see, Screaming Frog is a comprehensive website auditing tool that can help identify many common technical SEO issues. While there’s still no replacement for manual analysis, Screaming Frog is an essential part of any technical SEO’s toolkit. Master its many features and it will go a long way in helping you optimize your websites.