The scanner runs deterministic AI-search hygiene checks plus an evidence-graded rulebook. A grade or a pass is not a citation. Shipped today: 30 automated checks on L0–L3 and 167 rules in the rulebook.

Scanner API

The scanner evaluates HTML or text against the deterministic AI-search hygiene checks plus an evidence-graded rulebook. Same deterministic engine powers MCP, CLI, GitHub Action, and the web UI. A grade or a pass is not a citation.

Keyed results: judge field

Detailed (keyed) scans attach a judge on each check and on work-list items. The label is assigned by the deterministic judgeForCheck mapping — not an LLM on our side.

judge: engine — trust the pass/fail from the deterministic check (access, rendering, and mechanical structure).
judge: agent — your agent re-reads the rule text and decides; the label comes from the deterministic judgeForCheck mapping, not an LLM on our side.

Rules: causality is the causal signal

Every full rule from fetch_rules, search_rules and keyed GET /api/rules carries causality. It is established only when at least one experimental, causal source backs the rule. Read not_established as an association even when the rule is worded as advice ("add X so engines cite you"): the field, not the wording, says what the evidence shows.

How the grade is computed

Each group and the page get a letter from the share of evaluated checks that failed. A failed warn check counts 50% as much as a failed error. Any failed error rules out an A. Warnings alone keep an A until their weight reaches 5% of the checks run. After that the grade is B under 25%, C under 50%, D under 75%, and F above. Two caps apply to the page and to each group. A failed error at level 0 — an indexing blocker such as a robots.txt deny or a page-level noindex — grades F. Otherwise failed error checks cap the grade: one at best B, two at best C, three or more at best D. Advisory checks (BLUF, stuffing, paragraph length, FAQ headings) are reported but do not count toward a grade or a cap. Skipped checks do not count, but a page the scanner could not read never grades A.

Free tier (grades + fundamental fixes)

MCP: scan_url with no API key — grades plus findings: each failed L0–L3 check with its fix. No rule IDs, evidence or severity.
Web: /scan — paste a public URL
HTTP: POST https://aeogeo.site/api/scan with {"url":"https://example.com"} — CORS open; grades plus L0–L3 findings

Keyed tier (detailed)

MCP: lint_html, lint_text — per-check results with executable fix objects
API: POST /api/scan with Authorization: Bearer <key>

Machine-readable contract

The customer REST surface (POST /api/scan, keyed GET /api/checks, keyed GET /api/rules) is published as a parseable OpenAPI 3 document for agents and tooling:

https://aeogeo.site/openapi.json — static apex (same file is also served by the Worker at /api/openapi.json)

Our crawler and robots.txt

The scanner fetches pages as aeogeo-scanner/1.0 (+https://aeogeo.site/docs/scanner). It reads the site's /robots.txt before it fetches a page and follows RFC 9309. The longest matching rule wins, and * and $ work as wildcards.

One page someone asked for (scan_url, /scan, POST /api/scan, lint_html with a URL). A group naming aeogeo-scanner can refuse it, and then the page is not fetched: the answer is blocked_url with the rule that blocked it. User-agent: * does not refuse a page a person asked for, the same way user-triggered fetchers such as ChatGPT-User treat it. The same rule covers every later request of that scan: a redirect to another URL (read against that host's robots.txt) and the check that the page's canonical URL answers.
A whole-site scan (scan_site) is a crawl, so robots.txt applies in full: the aeogeo-scanner group, or * when the site has none. Disallowed pages are not fetched. They stay in the results with skipped: "robots_txt" and the matching rule, so you can see what the scan could not grade. They are counted as pagesSkipped, not failed, and they do not use your daily page quota. If robots.txt does not answer (HTTP 5xx or a network error), the whole site counts as disallowed. Crawl-delay is honoured up to 5 seconds, one page at a time. A sitemap you pass is fetched only if robots.txt allows it.

To keep the scanner off your site, or off part of it:

User-agent: aeogeo-scanner
Disallow: /

Grading is separate. The crawler_policy checks still report what your robots.txt tells GPTBot, OAI-SearchBot, PerplexityBot and the other AI crawlers.

Limits

Every scan runs inside fixed budgets. Past one, the scan stops with an error and a fix instead of running on.

Page size: 2 MB, counted as the page downloads, whatever Content-Length says: payload_too_large.
Time: 10 seconds for the page, 5 for robots.txt: fetch_failed, "request timed out".
Redirects: at most 3. Only public http(s) addresses are fetched, checked again after every redirect: blocked_url.
Markup: a page nested more than 512 elements deep, with more than 25,000 elements and comments, or more than 100,000 character references cannot be graded safely: page_unusable with the reason. Lighthouse already flags a page above 1,400 elements.
Free scans: 5 per domain and 20 per IP address a day, on the website, the API and MCP alike: rate_limited with the wait time. A key raises this to the plan quota.

Invariants

Deterministic: same input → same output
Transient: raw page HTML and draft text are never persisted. Derived findings are kept only for a shareable report you create (up to 90 days) or a whole-site scan (7 days); see privacy
No prompt-tracking or LLM calls in aeogeo runtime
Visibility checks (BLUF, FAQ, SSR) inspect the initial HTML only: the hidden attribute, aria-hidden="true", inline display:none or visibility:hidden, and <template>. Stylesheet cascades are not evaluated.