What is a server log file?
A server log file is a file that is created and maintained by a web server, and records information about the server’s activity. Server log files are a raw, unfiltered look at traffic, bot and human, to your site. They’re a text document automatically created and stores the record data on your web server for a period of time.
What Information does the log file contain?
The log file contains information about client requests, the date and time of the request, the client’s IP address, the requested URL, server responses (such as 200 OK or 404 Not Found), the amount of data transferred and any errors that occurred during the process of handling the requests.
Types of Log Files:
There are several types of log files that may be generated by a website’s server, including:
- Access log: This log file records information about all the requests made to the server, including the date, time, and IP address of the user making the request, the request method (such as GET or POST), and the status code of the request (such as 200 for a successful request or 404 for a page not found error).
- Error log: This log file records information about any errors that occur on the server, such as broken links or missing pages.
- Referral log: This log file records information about the sources that are sending traffic to a website.
- Search query log: This log file records information about the specific keywords and phrases that users are searching for in relation to a website.
- Server log: This log file records information about the server’s activity, such as CPU usage, memory usage, and disk space usage.
What is SEO Log File Analysis?
SEO log file analysis is the process of analyzing log files generated by a website’s server to gain insights into how search engines and users interact with the site. We can use these log files to provide information on things such as search engine crawler activity, referral traffic, search query data, and server errors.
Importance of log file analysis for SEO
There is only one true way Googlebot and other search engines process your website by looking at your website server log files.
3rd party crawlers, Google Search Console and search operators won’t give us the whole picture of how they interact with websites as only log file access can.
By analyzing this data, website owners and SEO professionals can identify issues that may be affecting the site‘s performance and visibility in search engine results and take steps to address them. It allows you to track website traffic, identify crawl errors, and improve website ranking.
Additionally, log file analysis can help to better understand user behavior and tailor the site’s content and SEO strategy to meet the needs of its audience.
Tools Required for Log File Analysis
To analyse the log files you must first get access to them from the developer or website administrator. Once you have access to the log files, you will need a log file analyzer tool to upload the log file to.
Screaming Frog Log File Analyser is our preferred tool of choice although you can use other ones like Splunk, Logz.io. and GamutLogViewer.
5 Ways Log File Analysis can be used for SEO:
1. Know Where to put your Crawl Budget
Crawl budget refers to the number of pages that a search engine will crawl on a website within a specific period of time. Log file analysis can be used to identify a website’s crawl budget by analyzing the log files to determine how frequently search engines are crawling the website.
Here are a few ways log file analysis can be used to identify crawl budget issues:
- Crawl Frequency: If search engines are crawling your website too frequently, this may indicate that you have a large crawl budget.Log file analysis can help you identify if search engines are crawling your website too frequently, which can cause indexation issues.
- Crawl Rate: Knowing the crawl rate search engines are crawling your website can help you determine if your website’s crawl budget is sufficient to cover all the pages on your site.
- Crawl Time: See how long it takes for search engines to crawl a page on your website, Are there any pages that are taking too long to crawl, which may impact your crawl budget?
Once you have identified the crawl budget using log file analysis, you can take steps to optimize it. This can include optimising the site’s structure and navigation, fixing crawl errors, and improving the performance of your site to make it easier for search engines to crawl it.
2. Are the desired search engine bots accessing your pages?
The log file requests are usually made by bots from major search engines such as Google, Bing, and Yahoo. Additionally, log file analysis can also record requests from other crawlers such as those from social media platforms, analytics tools, and other third-party services. Log file analysis can be used to determine if all of the targeted search engine bots are accessing the pages on a website.
Here are a few ways to use log file analysis to see if all of your targeted search engine bots are accessing your pages:
- Accessed pages: Determine which pages on your website are being accessed by search engines. This information can help you identify which pages are being crawled and which pages are not.
- User agents: Log file analysis can also help you identify the user agents (search engine bots, such as Google user agents and Bing user agents) that are accessing your pages. By comparing the user agents to a list of targeted search engine bots, you can determine if all of your targeted search engine bots are accessing your pages.
Filter the log file data by User Agent to see which search engines are visiting your site the most. Hopefully, you will see Googlebot Smartphone and Googlebot Desktop as the most popular user agents.
3. Check if Your Important Pages Are Crawled properly?
Log file analysis allows you to study the pages on your website that are being accessed by search engines. With this information, you can distinguish which pages are being crawled and which pages are not, helping you to understand the behaviour of the search engine bots on your website..
Here are a few ways to use log file analysis to check if your important pages are being crawled:
- Identify pages being accessed to determine which pages on your website are being accessed by search engines. This information can help you identify which pages are being crawled and which are not.
- Compare to important pages: Compile the list of pages that are being accessed and then compare them to a list of the important pages on your website. If important pages are not present in the list of accessed pages, they may not be being crawled by search engines.
Maybe pages are too deep within the website structure to be crawled or log file crawl time might indicate that the page is taking too long to crawl which will impact their crawlability.
Additionally, you can also check for crawl errors or low crawl rates which can also indicate that your important pages are not being crawled properly. If you find any issues, you can take steps to optimise your website to make it easier for search engines to crawl your important pages.
4. Highlight Any Indexability Issues
Log file analysis can be used to identify indexability issues on a website, which refer to problems that prevent search engines from crawling and indexing a website’s pages.
Here are a few ways log file analysis can be used to identify indexability issues:
- Pages not being crawled: By determining which pages on your website are not being accessed by search engines may mean they may not being crawled or indexed indicating an indexability issue.
- Crawl depth: If search engines are not crawling deep enough into the website, this may indicate an indexability issue.
- Redirects: Log file analysis can help you identify if there are too many redirects or if they are redirecting to the wrong pages.
By studying the log files, you can get an idea of which pages on the website are being accessed by search engines and which pages are not. This information can help you pinpoint any indexability issues that may be preventing the search engine from crawling certain pages on the website. Once you have identified the indexability issues, you can take steps to fix them such as:
- Optimise site structure and navigation to improve the crawlability of the site.
- Improve the XML sitemap. Add pages not being crawled to the XML sitemap, this way search engines can easily find and crawl those pages.
- Use of robots.txt: Log file analysis can help you understand which pages search engines are trying to crawl but are blocked by the robots.txt file, you can then remove those pages from the robots.txt file, this way the search engine can crawl those pages.
5. Discover Orphan Pages
Log file analysis can be used to discover orphaned pages on a website. Orphaned pages are pages on a website that are not linked to by any other pages on the site. They can be caused by broken links, changes to the site’s structure, or errors in the website’s code.
To identify orphan pages using log file analysis, you can follow these steps:
- Identify the orphan pages: Once you have a list of all the URLs that were accessed, you can compare them to the URLs of the pages on your website. Pages that are not present in the list of accessed URLs are considered orphaned pages.
- Analyse the reason: Once you have identified the orphan pages, you can analyse the reason why they are orphaned. This can be done by checking for broken links, or by reviewing changes to the site’s structure or code that may have led to the pages being orphaned.
- Fix the orphan pages: Once you have identified and analysed the orphan pages, you can take steps to fix them. This can include adding internal links to the orphan pages, redirecting to other pages, or removing the pages altogether if they are no longer needed.
Final Thoughts:
Once you have analysed the log file data, gathered the insights and completed the action items, you can now see the SEO Benefits: website keyword rankings improvements, increased organic traffic, increased conversions and revenue.
As you may already know, SEO log file analysis is an essential tool for understanding and optimising your website’s performance. It allows you to track website traffic, identify crawl errors, and improve website ranking. However, many website owners are not aware of the full potential of log file analysis and how it can be used to improve their SEO.
Here at Legacy Communications we have SEO Specialists who can take you through the steps of SEO log file analysis, dig deep into the opportunities that will boost your website performance and help increase your business overall goals.
If you’re interested in including SEO in your 2023 marketing plan, get in touch with our talented Search team who will work with you to find the best strategy for you and your business.