Site SEO AI AuditInternet Solutionsilt

SEOAuditBot: How the Crawler Treats Your Website

13. november 20268 min lugemistTehniline SEO
SEOAuditBot: How the Crawler Treats Your Website

Short answer: SEOAuditBot is the crawler behind Site SEO AI Audit. It starts from your home page and your XML sitemap, follows internal links the way a search engine does, respects your robots.txt and makes about three requests per second, so it does not slow a normal website. It reads the pages your server returns, records what it finds on each one and then turns the result into a score and a fix list. Knowing how it behaves helps you read the report and avoid blocking it by accident.

Site owners often ask the same question before they run an audit: what exactly will this tool do to my website? It is a fair question. A crawler visits every page, and a careless one can overload a small server, ignore your rules or leave strange traces in your logs. The product’s FAQ answers the main worry in one line: the crawl does not slow your site, because SEOAuditBot makes about three requests per second and follows your robots.txt. This article explains the rest of the behaviour in plain language.

Where the crawl starts

Every crawl needs a starting point. SEOAuditBot begins with two sources. The first is the home page of the address you entered. The second is your XML sitemap, which it finds through robots.txt or in the usual location. From the home page it follows links to other pages, then the links on those pages, and so on, while recording how many clicks each page is away from the start. From the sitemap it learns which addresses you want search engines to know.

The two sources are compared, and the comparison is useful. A page that is in the sitemap but is not reached by any link is reported as an orphan page. A page that is linked but missing from the sitemap is still crawled, which helps you see whether the sitemap is complete. If you have no sitemap at all, the report says so, because a sitemap helps search engines find new and deep pages.

What it follows and what it skips

SEOAuditBot follows internal links like Googlebot does. Links marked with the nofollow attribute are not followed, so a section that is reachable only through nofollow links may look unreachable in the report. That is a useful signal: if you wanted those pages found, you probably need ordinary links. The report also warns when internal links carry nofollow, because telling search engines not to follow links on your own site is rarely intended.

It reads the HTML that each page returns. This matters for sites that build their content with scripts in the browser. If a page is nearly empty before scripts run, the crawler sees the same near-empty page that many other crawlers see. In the AI visibility area, the issue “Content needs JavaScript to appear” points to such pages, because many AI crawlers do not run scripts. The explanation in the article on JavaScript content and AI crawlers shows why this deserves attention.

Politeness: speed and robots.txt

Two rules make a crawler polite. The first is a modest speed: about three requests per second means a site of a few hundred pages is covered in minutes, and a large site takes longer but never floods the server. The second is obedience to robots.txt. If your robots.txt disallows a path for crawlers in general, SEOAuditBot does not crawl that path. This is the same rule you give to search engines, so it is easy to reason about.

The rule has a consequence that surprises some owners. If robots.txt blocks the whole website, or if no page can be indexed, the crawler has almost nothing to read, and the score is capped at 20. The report does not hide this: the critical issue “robots.txt blocks the whole website” appears at the top of the list. A test site that still carries the blocking rule from development is the usual cause. The robots.txt guide explains what each line does.

What it reads besides pages

A crawl is more than a list of addresses. Along the way, SEOAuditBot also reads robots.txt itself, the XML sitemap, the llms.txt file if you have one and the rules for AI crawlers. It records the status code of every address, the title, description, headings, canonical tag, structured data and language markers of every page, the number of internal links pointing to each page and the size and compression of the response. It checks a sample of external links for errors and measures speed through Google’s PageSpeed tool for mobile, together with the server response time on every page.

What is read Why it matters in the report
robots.txt Blocked paths, a missing file and rules for AI crawlers
XML sitemap Missing sitemap, broken or noindex addresses, orphan pages
Page HTML Titles, descriptions, headings, canonicals, structured data, languages
Lingid Broken links, redirects, click depth, links in
Server responses Status codes, redirect chains, slow responses, compression
llms.txt The AI visibility area

How to recognise it in your logs

The crawler announces itself with its name, SEOAuditBot, in the user agent of each request. If you read your access logs, you can filter for that name to see exactly what it fetched and when. This is useful if you want to confirm that a check really visited the pages you expected, or if you need to explain unusual traffic to a hosting company. If logs interest you, the guide to log file analysis for SEO shows how to read them for crawler behaviour in general.

Remember that a user agent name is a label, not proof. Treat it as a good hint, and combine it with the time of your own check: you started the audit, so you know when to expect the visits.

When a firewall or CDN gets in the way

Some websites sit behind a firewall, a CDN or a security plugin that blocks unfamiliar bots or limits requests. The result in the report is easy to recognise: many pages return errors, or the crawl ends early. If you see a wall of 403 or 429 responses on pages that open normally in your browser, check the protective layer first. Add an exception for the SEOAuditBot user agent, or loosen the rate limit for the time of the check. The same problem hits search engines and AI crawlers, so it is worth fixing even if you were not planning an audit. The article on rate limiting and 429 responses explains how to slow aggressive bots without shutting out useful ones.

A short pre-check checklist

Before you start a check on a site with strict protection, spend two minutes on this list. Open your robots.txt and confirm that it does not block the whole site or the sections you want audited. Make sure the sitemap address works. Look at your security plugin or CDN rules for blocks on unknown bots. Tell your hosting provider or developer that a crawl is coming, so that nobody mistakes it for an attack. After the check, read the status codes in the page list: a healthy site shows mostly 200 responses. If you see many errors, fix the protection first and run the check again, because a report built on blocked pages describes your firewall, not your website.

What it does not do

It does not log in. A page behind a login, a member area or a checkout cannot be read by a crawler that has no account, and the audit will not see it. It also does not change your website: it only requests pages. And it does not run on its own schedule. A check starts when you start it, so the visits in your logs line up with your own actions.

Using the knowledge when you read the report

With the crawler’s behaviour in mind, strange results become easy to explain. A page missing from the page list was probably blocked by robots.txt, never linked or reachable only through nofollow links. A page that is in the list but has few words may need scripts to show its text. A cluster of errors may come from a firewall rather than from your pages. A very deep page was found only after many clicks. Each of these is a lead to follow.

To try it on your own site, create a free account and run your check. The free account runs one check and shows the report locked, and a plan from the pricing page opens the full report with every issue, the affected pages and the fixes.

Related reading

The bottom line

SEOAuditBot starts from your home page and sitemap, follows links like a search engine, obeys robots.txt and keeps to about three requests per second. Know what it reads and what blocks it, and the report becomes easier to trust and to act on.

KKK

Will the crawl slow my website?

Ei. SEOAuditBot teeb umbes kolm päringut sekundis ja järgib teie robots.txt-d.

Does it obey robots.txt?

Yes. Paths that your robots.txt disallows for crawlers are not crawled.

How do I find its visits in my logs?

Filter your access logs for the user agent name SEOAuditBot and compare the times with the moment you started the check.

Why does my report show many errors on normal pages?

A firewall, CDN or security plugin may be blocking or rate-limiting the crawler. Add an exception for the SEOAuditBot user agent and run the check again.

Can it check pages behind a login?

No. It reads public pages that a visitor without an account can open.

#AI crawlers#Crawling#robots.txt#SEO audit
Kontrolli oma veebisaiti.Kas Google näeb teie veebilehte?
Alusta tasuta

Veel blogist

Kõik artiklid →
Internet Solutions

Veel meie meeskonnalt

Loonud Internet Solutions. Proovige ka meie teisi tooteid — iga üks säästab aega omal moel.

internet-solutions.net ↗