#faq
8 posts tagged with faq.
-
Can robots.txt Stop AI Scrapers?
No. robots.txt only asks compliant bots to stay away — non-compliant AI scrapers ignore it. To actually stop them you need a WAF, IP/ASN blocking, and bot verification at the edge.
-
Do AI Crawlers Respect robots.txt?
Some do, many don't. Reputable AI crawlers like GPTBot, ClaudeBot, and PerplexityBot honor robots.txt; non-compliant scrapers ignore it. robots.txt is a request, not enforcement.
-
Does Blocking AI Bots Hurt Your SEO or AI Visibility?
Blocking AI training bots doesn't hurt traditional Google SEO. But blocking AI search/retrieval bots does hurt your AI visibility — you can't be cited in answers from a bot you've blocked.
-
Does Cloudflare Block AI Crawlers by Default?
Yes. Since July 2025 Cloudflare blocks AI crawlers by default for new sites and offers one-click blocking plus pay-per-crawl. If you're on Cloudflare, check this setting — it can override your robots.txt.
-
GPTBot vs OAI-SearchBot: What's the Difference?
GPTBot is OpenAI's training crawler (turns your content into model weights, no traffic back). OAI-SearchBot is its search crawler (cites you in ChatGPT answers with a link). Block one, allow the other.
-
Is llms.txt Worth It?
llms.txt is low-effort and may help AI systems understand your site accurately — but adoption is inconsistent and it's advisory, not enforcement. Worth adding; don't expect it to control access or guarantee citations.
-
Should I Allow AI Crawlers?
Allow AI search and user-fetch crawlers — they cite you in AI answers and bring real visitors. Consider blocking only training crawlers, which take content for model training with nothing back.
-
Should I Block GPTBot?
Block GPTBot if you don't want your content training OpenAI's models for free — it gives no traffic back. But blocking GPTBot doesn't affect ChatGPT search visibility; that's a separate bot you can allow.