aeogeo.site
Rule aeo-013 · crawler_ops · high confidence
Configure robots.txt per AI crawler purpose (e.g. GPTBot vs OAI-SearchBot) — do not block search indexing bots by mistake.
Applies to: all
Why this matters
Official crawler docs separate training vs search use; wrong blocks remove you from retrieval.
Implementation
Type: not_automatable
Anti-pattern
Blanket Disallow on all bots without reviewing OAI-SearchBot / Perplexity-User.
Get the full rule + automated checks
Subscribe to access evidence, remediation steps, and configurable check descriptors for this rule.