Crawler ops covers 1 rule in the aeogeo rulebook.

Open a rule for the full statement, the evidence behind it, and the check that can verify it on a page.

← Back to home

crawler_ops

1 rule

• Configure robots.txt per AI crawler purpose (e.g. GPTBot vs OAI-SearchBot) — do not block search indexing bots by mistake.

  aeo-013 · strong evidence · not_automatable