ToolsHub4u

Read a server log for crawl activity, errors and wasted requests

Paste a slice of an Apache or nginx access log. The analyser counts requests, IPs, crawler hits, status codes and top paths, and lists the errors worth fixing.

  • Apache and nginx
  • Bot detection
  • Nothing uploaded
Ready SEO & Web
User agents can be spoofedA request claiming to be Googlebot is only labelled as Googlebot here. Real verification needs a reverse DNS lookup or Google's published IP ranges, which this page does not do.
1Enter
2Adjust
3Copy result

Log File Analyser

1

Log lines and options

Combined log format or common log format, one request per line. Query strings are kept, so parameter variants are visible.

No data yet Live
2

Log summary

10 requests from 5 IPs, 70% bot

Requests parsed

10

  • 2 responses in the 4xx and 5xx range (20%)Crawlers that repeatedly hit errors may slow down. Check the paths below and fix or redirect them.
  • 7 bot requests (70% of the slice)Bot requests are labelled from the user agent string. A spoofed agent is counted as the bot it claims to be.
  • Every line was parsedAll lines matched the combined log pattern.
  • Most requested path: /tools/seo-tools/robots-txt-generator.html2 requests. A single path taking a large share can be a crawler loop or a hot asset.

Requests by crawler

Googlebot5
Bingbot2
Most requested paths
#PathHitsErrorsShare
1/tools/seo-tools/robots-txt-generator.html2020
2/old-page.html1110
3/tools/calculator/percentage-calculator.html1010
4/tools/pdf-tools/merge-pdf.html1010
5/tools/seo-tools/core-web-vitals-calculator.html1110
6/tools/seo-tools/keyword-density-checker.html1010
7/tools/seo-tools/keyword-density-checker.html?utm_source=newsletter1010
8/tools/seo-tools/meta-tag-generator.html1010
9/tools/text-tools/1010
  • /old-page.html404, 1 hit, 1

    A missing page. Restore it, redirect it, or remove internal links to it.

  • /tools/seo-tools/core-web-vitals-calculator.html500, 1 hit, 1

    A server error. Fix it: repeated 5xx responses can slow crawling.

Unique IPs
5
Bot requests
7
Errors (4xx/5xx)
2
Transferred
0.1 MB

This summary covers the lines you pasted, not the whole log. Use a full day or a week for crawl-budget questions, and remember that user agents can be spoofed. Bot and human totals are shown separately for context.

This page reads a slice of an Apache or nginx access log and summarises what is in it: how many requests were parsed, how many distinct IPs made them, how much of the traffic came from crawlers, which paths were requested most and where the 4xx and 5xx responses are. Everything runs in the browser tab.

It is a quick second opinion, not a replacement for a full log analyser. A pasted slice is useful for a specific question, such as whether Googlebot is hitting a dead page or whether a bot is spending its budget on parameter URLs. It is not a crawl-budget report for the whole month.

What it can read

The parser expects one request per line in common or combined log format. A combined line has the client IP, the ident and user fields, a timestamp in brackets, the request in quotes, the status code, the response size, then the referer and user agent in quotes. The common format drops the last two fields, and the parser still accepts it.

Query strings are kept, so /page and /page?utm_source=newsletter are counted as separate paths. That is useful when you want to see parameter variants, and it also splits what a crawler may treat as one page. The path list can show between 3 and 40 rows; the default is 10.

The first 20,000 lines are parsed. Anything beyond that is ignored, so a multi-day log needs to be cut down before you paste it. JSON logs, W3C extended logs and custom formats are not understood, and their lines are counted as unparsed rather than guessed at.

A worked example

Twelve lines were pasted: eleven valid combined-format requests and one line of prose. The summary read 11 requests from 7 IPs, 54.5% bot. Six of the eleven requests carried a recognisable crawler user agent, and four ended in a 4xx or 5xx status.

Top paths in the eleven parsed requests
PathHitsErrorsShare
/missing-page2218.2%
/tools/pdf-tools/merge-pdf.html2018.2%
/blog/119.1%
/tools/calculator/percentage-calculator.html119.1%
/tools/seo-tools/keyword-density-checker.html?utm_source=newsletter109.1%

Two paths tied at two hits. The tool put /missing-page first because it breaks ties alphabetically, and both were 18.2% of the slice. The bot breakdown was AhrefsBot 2, Googlebot 2, Bingbot 1 and GPTBot 1. The four errors sat on three URLs: two 404s on /missing-page, one 404 on /blog/ and one 500 on a calculator page.

That is the useful pattern the page is designed to surface. A dead URL taking the joint-largest share of requests, requested twice by a crawler, is a crawl-budget leak you can fix with a redirect. The 500 on a live page is a server problem, not an SEO one, and it is worth separating from the 404s.

Eleven requests is a sample, not a trendA single 500 is not proof of instability, and 54.5% bot traffic over eleven lines says little about the month. Use the summary to find a path worth investigating, then query the full log for that path over a week.

How bots are counted

The tool matches the user agent against a short list: Googlebot, Bingbot, GPTBot, ClaudeBot, AhrefsBot, SemrushBot and Bytespider. Anything else containing bot, crawler, spider, slurp, curl, wget, python-requests or headless is grouped as Other bot. A request with a normal browser string is counted as human.

This is a label from a string the client chooses. It is not verification. Anyone can send a Googlebot user agent, and a real crawler can occasionally send a different one. The page says so in the note, and the Separate bot and human requests checkbox does less than its label suggests: the two totals are always reported, and ticking the box only adds a sentence to the note.

If the distinction matters for a decision, verify the IP ranges or use a tool that does reverse DNS checks. This one deliberately keeps the request local and does no network lookup.

Reading errors and crawl waste

An error rate above 5% turns the checklist amber and above 15% turns it red. Those bands are editorial, not Google thresholds. What matters is which URLs are failing and who is asking for them. A 404 requested once by a person is a broken link; the same 404 requested repeatedly by a crawler is a page that needs to exist or redirect.

The error list shows the path, the status, the number of hits and, when the request was a bot, a second number. That second number is a count, not a name: the line reads 404, 1 hit, 1 rather than 1 Googlebot hit. It is a display bug, and the crawler table above is the place to see which bot was involved.

One grammar defect appears when a line is skipped: the warning reads 1 line was skipped because they did not match the combined log format. The count and the reason are right; the pronoun is not.

Turning the slice into a fix

Start with the paths that combine errors and bot traffic. Redirect dead URLs that still receive requests, fix or remove internal links to them, and check whether a query-string variant is being crawled separately from the canonical page. If a parameter is only for tracking, the robots.txt generator can block the pattern, though blocking a crawl does not remove a URL from the index.

Then check the sitemap. A URL that appears in the XML sitemap but returns a 404 is a signal you can fix at the source. A URL that is requested but missing from the sitemap may be an orphan worth adding, or a page worth removing.

Finally, keep the slice honest. Paste a full day or a full week rather than a handful of lines, note the date range beside the numbers, and do not compare a mobile-heavy slice with a desktop one. The page reports what you gave it and nothing more.

Questions

Common questions

Short answers about this tool and its limits.

Which log formats are supported?

Apache and nginx common or combined format, one request per line. Combined adds the referer and user agent. JSON, W3C extended and custom formats are not parsed and are counted as unparsed lines.

How many lines can I analyse?

The first 20,000 lines are read; the rest are ignored. That is enough for a day on a small site. For a week or a month, cut the log to the paths or dates you need before pasting.

Are the bot numbers verified?

No. Bots are identified from the user agent string. A spoofed Googlebot request counts as Googlebot, and a genuine crawler using an unusual string may be missed. No IP or reverse DNS check is made.

Why does the error line end with a number instead of a bot name?

That number is the count of bot hits on the URL, printed where the crawler name should be. It is a display bug. The crawler totals at the top of the result show which bots were present.

What does the bot and human checkbox change?

Very little. The bot total, percentage and crawler breakdown are always reported. Ticking the box adds one sentence to the note explaining that the totals are shown separately.

Why is the same page counted twice?

Because query strings are kept. A URL with a tracking parameter is a different string from the clean URL, so it appears as its own row. That is deliberate: it shows parameter variants that crawlers may treat as separate pages.

Does a high error rate mean a penalty?

No. The amber and red bands are the tool's own and describe the slice, not a ranking signal. Repeated 404s and 500s can waste crawl attention and frustrate visitors, which is reason enough to fix them.

Is my log uploaded?

No. The file is read in the browser and the summary is generated locally. There is no upload, account or stored history, which also means the full log never leaves your machine.