Log file analysis
Log file analysis is the review of a web server's access logs to see exactly which pages search engine bots and AI crawlers request, how often and with what responses.
Every request to a server can be recorded in a log, including the URL, time, status code and user agent. Filtering for bots like Googlebot shows real crawling behaviour rather than estimates.
In more detail
Log analysis answers questions other tools cannot: are important pages crawled regularly, is crawl budget wasted on parameter URLs, do bots hit errors, and are AI crawlers visiting at all? Verifying bots by reverse DNS avoids being fooled by fake user agents. Large sites often stream logs into analysis tools for ongoing monitoring.
Why it matters
For large or complex sites, logs are the ground truth of how search engines and AI systems interact with you. They often reveal problems that crawls and Search Console miss.
Logs show Googlebot visits a retailer's faceted URLs 40,000 times a day but its new product pages only twice a week. Blocking the facets in robots.txt reverses the pattern within a month.