← Back to home

crawler_ops

1 rule

• Configure robots.txt per AI crawler purpose (e.g. GPTBot vs OAI-SearchBot) — do not block search indexing bots by mistake.

  aeo-013 · high confidence · not_automatable