#AI crawlers
AI crawlers are the automated bots that AI companies use to fetch web pages, either to train models or to answer user questions in real time. Each one identifies itself with a user-agent name, such as GPTBot, ClaudeBot or PerplexityBot. Site owners can allow or block them in robots.txt, and sometimes they are blocked by accident by a firewall or CDN setting. Knowing which bot does what helps you decide which ones to let in. Articles under this tag cover crawler names, access rules, logs and common mistakes.
2026/11/21 · 8分で読めますSite SEO AI Audit for SaaS Marketing Websites
How a software company can use Site SEO AI Audit on its marketing site: script-only pages, pricing and docs, canonicals, share images, speed and…
2026/11/20 · 8分で読めますContent Needs JavaScript: The AI Visibility Warning
What the “Content needs JavaScript to appear” warning in Site SEO AI Audit means, which pages trigger it, how to test your own pages…
2026/11/20 · 8分で読めますNo llms.txt Notice: What the Audit Checks and What to Do
What the “No llms.txt” notice in Site SEO AI Audit means, how much it counts in your score, what a useful file contains and…
2026/11/14 · 8分で読めますAI Assistants Blocked in robots.txt: Check and Fix
What the Site SEO AI Audit warning “AI assistants are blocked in robots.txt” means, how to read your rules, decide what to allow and…
2026/11/13 · 8分で読めますSEOAuditBot: How the Crawler Treats Your Website
What SEOAuditBot does when Site SEO AI Audit checks your site: where it starts, what it follows, how fast it goes, what it reads…
2026/11/09 · 8分で読めますAI Visibility Checks in Site SEO AI Audit Explained
What the AI visibility area of Site SEO AI Audit checks: AI crawlers in robots.txt, llms.txt, JavaScript-only content, structure and dates, and how to…
2026/11/08 · 8分で読めますCrawl-Delay in robots.txt: Who Obeys It, Who Does Not
What the Crawl-delay directive does, which crawlers honour it, why Googlebot ignores it, and safer ways to slow aggressive bots with 429 and 503…
2026/11/07 · 8分で読めますAI Bot Traffic: Keep It Out of Your Analytics Data
How to tell AI crawlers and bots from real visitors, why they distort reports, and how to use filters and server logs to decide…
2026/10/29 · 8分で読めますRate Limiting and 429 Errors: Handling Search and AI Bots
How server rate limits and 429 or 503 responses affect Googlebot and AI crawlers, how to protect your site without blocking good bots, and…
2026/10/24 · 8分で読めますAI Search for Media, Entertainment and Gaming Sites
Fans ask AI assistants about releases, reviews, guides and schedules. Learn how media, entertainment and gaming sites can earn citations and still keep readers.
2026/10/17 · 8分で読めますAI Cloaking and Hidden Text: Why Tricking Crawlers Backfires
Serving different content to AI crawlers or hiding instructions in a page may look clever, but it breaks guidelines and trust. Learn what is…
2026/10/10 · 8分で読めますChanging URLs Without Losing AI Search Citations
AI assistants keep citing old URLs long after a redesign or migration. How to change URLs so AI answers follow you, and what to…