ENEnglish
LTLietuviškai
DEDeutsch
FRFrançais
RUРусский
ESEspañol
PLPolski
ITItaliano
UKУкраїнська
CSČeština
PTPortuguês
JA日本語
LVLatviešu valoda
ETEesti
HUMagyar
IDBahasa Indonesia
TRTürkçe
HIहिन्दी
ELΕλληνικά
ARالعربية
THไทย
VITiếng Việt
RORomână
MSBahasa Melayu
BNবাংলা#robots.txt
robots.txt is a plain text file at the root of a website that tells crawlers which paths they may request. Articles tagged here explain its syntax, how rules are matched and which directives search engines actually support. They show common mistakes, such as blocking CSS, JavaScript or whole sections by accident. Each guide explains the difference between blocking crawling and preventing indexing. You will also find safe templates for WordPress, shops and staging sites. A careful robots.txt protects crawl time without hiding pages that should rank.
1. okt 2026 · 8 min lugemistPay-Per-Crawl and AI Licensing: Options for Site Owners
Beyond allow or block: pay-per-crawl, licensing terms and usage preferences for AI crawlers. What exists, how mature it is and what site owners can…
1. okt 2026 · 8 min lugemistCommon Crawl and CCBot: What It Means for Your Website
What Common Crawl is, what its CCBot crawler collects, how the data is used for AI training and research, and how to decide whether…
27. sept 2026 · 8 min lugemistShould You Let AI Train on Your Content? A Decision Guide
A practical guide to deciding whether AI companies may train on your website content: benefits, risks, what blocking can and cannot do, and how…
24. sept 2026 · 7 min lugemistnoai and noimageai Meta Tags: Do They Actually Work?
What the noai and noimageai meta tags are, which systems respect them, why they are not a standard, and what actually controls AI use…
13. sept 2026 · 8 min lugemisthreflang Conflicts with noindex, robots.txt and Redirects
hreflang only works between pages that can be crawled and indexed. How noindex, robots.txt, redirects and errors break clusters, and how to resolve each.
6. sept 2026 · 7 min lugemistStaging Site Indexed by Google? How to Fix and Prevent It
What to do when a staging or development site shows up in Google, how to remove it safely, and how to stop staging settings…
24. aug 2026 · 7 min lugemistIs Your CDN or Firewall Blocking AI Crawlers? How to Check
Robots.txt may allow AI crawlers while your CDN, firewall or security plugin blocks them. How to find hidden blocks and fix them without inviting…
13. aug 2026 · 7 min lugemistGoogle-Extended Explained: What Blocking It Does and Doesn’t
Google-Extended is a robots.txt token, not a crawler. Learn what it controls in Gemini, why it does not affect Search or AI Overviews, and…
6. aug 2026 · 7 min lugemistNoindex vs Disallow: How to Keep Pages Out of Search
Noindex and robots.txt Disallow solve different problems. Learn which removes pages from search, which saves crawl time, and why combining them backfires.
4. aug 2026 · 8 min lugemistHow to Allow or Block AI Crawlers in robots.txt
A practical guide to AI crawler rules in robots.txt: which bots train models, which power AI search, and copy-ready patterns to allow or block…
1. aug 2026 · 8 min lugemistRobots.txt for SEO: What to Block and What to Leave Open
A practical robots.txt guide: how rules are matched, what to block, what never to block, and how to test the file before one line…