#robots.txt
robots.txt is a plain text file at the root of a website that tells crawlers which paths they may request. Articles tagged here explain its syntax, how rules are matched and which directives search engines actually support. They show common mistakes, such as blocking CSS, JavaScript or whole sections by accident. Each guide explains the difference between blocking crawling and preventing indexing. You will also find safe templates for WordPress, shops and staging sites. A careful robots.txt protects crawl time without hiding pages that should rank.
14 lis 2026 · Czas czytania: 8 minAI Assistants Blocked in robots.txt: Check and Fix
What the Site SEO AI Audit warning “AI assistants are blocked in robots.txt” means, how to read your rules, decide what to allow and…
13 lis 2026 · Czas czytania: 8 minSEOAuditBot: How the Crawler Treats Your Website
What SEOAuditBot does when Site SEO AI Audit checks your site: where it starts, what it follows, how fast it goes, what it reads…
10 lis 2026 · Czas czytania: 8 minads.txt, security.txt and humans.txt: Do They Matter?
What ads.txt, security.txt, humans.txt and other well-known files do, whether they affect SEO, which sites need them and how to publish and maintain them.
9 lis 2026 · Czas czytania: 8 minAI Visibility Checks in Site SEO AI Audit Explained
What the AI visibility area of Site SEO AI Audit checks: AI crawlers in robots.txt, llms.txt, JavaScript-only content, structure and dates, and how to…
8 lis 2026 · Czas czytania: 8 minCrawl-Delay in robots.txt: Who Obeys It, Who Does Not
What the Crawl-delay directive does, which crawlers honour it, why Googlebot ignores it, and safer ways to slow aggressive bots with 429 and 503…
3 lis 2026 · Czas czytania: 8 minCrawl Traps: Calendars, Session IDs and Endless URLs
A crawl trap is a set of URLs with no natural end. Learn how calendars, session IDs and loops create them, how to find…
2 lis 2026 · Czas czytania: 8 minrobots.txt for Subdomains, CDNs and Multiple Domains
Each host and protocol needs its own robots.txt. See how subdomains, CDN domains, staging hosts and multi-domain setups are covered and how errors hit…
29 paź 2026 · Czas czytania: 8 minIndexed, Though Blocked by robots.txt: What It Means
Learn why Google lists a page it cannot crawl, how to tell whether you want it indexed, and the correct way to remove it…
20 paź 2026 · Czas czytania: 9 minGooglebot Crawlers Explained: Which User Agent Is Which
Googlebot is not one bot. Learn which Google crawlers visit your site, what each user agent does and how to verify, allow or limit…
1 paź 2026 · Czas czytania: 8 minPay-Per-Crawl and AI Licensing: Options for Site Owners
Beyond allow or block: pay-per-crawl, licensing terms and usage preferences for AI crawlers. What exists, how mature it is and what site owners can…
1 paź 2026 · Czas czytania: 8 minCommon Crawl and CCBot: What It Means for Your Website
What Common Crawl is, what its CCBot crawler collects, how the data is used for AI training and research, and how to decide whether…
27 wrz 2026 · Czas czytania: 8 minShould You Let AI Train on Your Content? A Decision Guide
A practical guide to deciding whether AI companies may train on your website content: benefits, risks, what blocking can and cannot do, and how…