# Agent Readiness Scanner — spec v0.1

A visitor enters a domain. The server opens it in a fresh headless browser, runs
fixed, repeatable checks and probes, and returns **"Your site is X% agent-friendly"**
with a score per task, deductions for blocking agents, every check's measured value
and the top three fixes.

No LLM runs during a scan. The scanner is a robot that perceives and acts the way
browsing agents do: it reads `innerText` and the accessibility tree, and it only
clicks or types into elements it finds by **role and accessible name**. It never
uses site-specific selectors. If that robot cannot search a site, an agent will
probably struggle too.

## Why these checks

The sources below are paths in the `common-thread-commerce` repository, where these
experiments were run.

Each check comes from friction we measured in this repo's agent runs:

| Observation | Source | Check it motivates |
|---|---|---|
| Airbnb agents without an adapter still succeeded, but every `innerText` read on a listing waited ~6.2 s and each run made ~4 page loads | `airbnb/reports/LUNA_MEDIUM_5PAIRS_2026-10-14.md`, bare `commands.jsonl` | Settle time, URL state; *not* "JS framework = bad" |
| With page tools (WebMCP), Airbnb took 48% less time and 38% fewer tokens; WooCommerce used 4.3× fewer tokens; store-choice workers needed 5–9 browser actions instead of 16–20 | Airbnb 5-pair, Woo real-catalog and store-choice reports | WebMCP tools are the main built-for-agents path |
| The browser-use agent clicked an ambiguous "Add to cart" and finished 0/5 | `output/browser-use-luna-woocommerce-20261008/transcript.txt` | Unique accessible names |
| Truncated accessibility snapshots cost 53–74 s per Glovo run | `glovo/benchmarks/v0.3/transcript-audit.md` | Snapshot size |
| `querySelectorAll('button')` returned `[]` because controls sat in a shadow root; Airbnb adapter v0.1.0 sat outside `<body>` and no agent noticed it | Glovo audit, Airbnb pair report | Hidden surfaces |
| 17/100 target sites returned 403/429/202 to plain HTTP; 35/100 show a consent manager | `experiments/agentface/data/test-sites.csv` | HTTP access, blocking pop-ups |
| The Woo Store API was the backbone of `shopping.js` | `shopping.js` | Public API |

## Universal by design

Nothing is inferred from the domain. A site type is detected from the page itself,
and modules that don't apply are marked **N/A**, which removes their points from the
total instead of counting them as failures.

- **Core checks** (Access, Read, Act, Built for agents) run on every site.
- **Commerce module** turns on when the homepage shows a shop: a known platform
  fingerprint (Shopify, WooCommerce, Magento, PrestaShop, BigCommerce, Salesforce
  Commerce), **or** at least two of: a cart/basket/bag link, ≥3 price-like strings,
  an add-to-cart button, Product/Offer structured data. Several independent signals
  are used so that a poorly marked-up shop is not excused by failing detection.
- **Search probe** is N/A on a non-shop without a search box, but **fails** on a shop
  without one.
- **A site that is down is not scored.** HTTP 500/502/504, Cloudflare 52x/530 or a
  refused/unresolvable connection ends the scan with "the site looks down" (503 is
  excluded because bot managers use it for challenges).
- **Not every URL is an HTML page.** When the response is Markdown, plain text or JSON,
  HTML-only checks (`structured-data`, `landmarks`) are N/A. A page with no controls
  at all makes `semantic-controls` and `hidden-surfaces` N/A — there
  is nothing to operate, so nothing to fail. An agent endpoint (e.g. a remote MCP
  server that also answers GET with Markdown) is judged by the checks that apply.

## Scan procedure

One scan = one fresh browser context (cookies, storage and cache isolated) in a
shared headless Chromium. 1280×900, `en-US`, UTC, service workers blocked.
Budget: ≤7 page loads (home, search, deep link, up to 3 product pages), 90 s hard timeout.

1. **HTTP fetches** (plain `fetch`, honest UA `AgentReadinessScanner/0.1`):
   start URL, `/robots.txt`, `/llms.txt`, sitemap, public API endpoints, and the
   discovery, sign-in and payment documents listed under optional integrations.
   DNS-AID and ARD records are queried over DNS-over-HTTPS (`cloudflare-dns.com`),
   because Node's resolver cannot read SVCB/HTTPS records.
2. **Page A — homepage.** Load, wait for text to settle and for at least 5 s after
   navigation (most pop-ups appear late), then measure everything static, take a
   screenshot and pick probe inputs (nav link, search term, product links).
3. **Clear pop-ups, then the navigate probe** on page A. Before every probe the
   scanner does what an agent would: click a consent "Accept"/"I agree" button
   (in the page or a consent iframe), then a "Close" button, then press Escape,
   up to 4 rounds. Whatever it had to dismiss is noted in the probe's detail; the
   cost is scored once, by `overlays`, not again by each probe.
4. **Page B — search probe.** Reload home, find the search box by role/name, type
   a term taken from the page, press Enter.
5. **Page C — deep link.** Open the search result URL in a new page.
6. **Pages D — product pages** (commerce only). Product links come from the home
   page and the search results. Up to 3 are tried, because the first is often
   sold out or "coming soon"; the best product-data and add-to-cart results are kept.

The browser uses a standard Chrome UA with ` AgentReadinessScanner/0.1` appended
(real agent browsers look like Chrome; we still identify ourselves). A block is
reported as the finding: "your site blocks automated visitors, which is what agents are."

### Never

Submit forms (contact, newsletter, login), start checkout, call MCP tools or send A2A messages
(the MCP check runs only `initialize` and `tools/list`, and only against an endpoint
the site itself advertises on the same site; A2A is read from the Agent Card by GET), register, sign in or pay (OAuth,
Auth.md, Poppy and payment checks only GET public discovery documents; an HTTP 402 is observed, never answered), or follow links to other sites (a
search form that submits to another host is stopped and scored as a hand-off). The add-to-cart click is the only write, into a guest
cart that is thrown away with the context.

## Score: tasks, best path

The score answers *how directly can an agent do the main tasks here?* Each task is
scored by the best path an agent has. **The ordinary page earns at most 50** — an agent
can get it done, slowly and fallibly, by reading HTML and clicking by name. **Something
built for agents earns up to 100.** A site with zero agent features but a flawless
human UI therefore lands near 50%, not near 100%.

| Task | Weight | Ordinary-page path (≤ 0.5) | Built-for-agents path (≤ 1) |
|---|---:|---|---|
| Read | 20 | 0.5 × weighted mean of the Read checks | `markdown` ratio, or a WebMCP tool for reading/asking |
| Find | 20 | 0.5 × weighted mean of the Act checks | WebMCP tool for search/browse/catalog |
| Product details (shops) | 15 | max(0.5 × Act mean, 0.8 × `product-data`) — Product JSON-LD is machine-readable, so it can reach 0.8 | WebMCP tool for products/items/variants |
| Take action (shown as "Add to cart" on shops) | 20 | Shops: 0.5 × `add-to-cart-probe`. Other sites: none — the scanner never submits forms, so the task is N/A unless tools exist | WebMCP tool for an action (book, reserve, order, request, subscribe, contact…) or, on shops, cart/basket/checkout |

A WebMCP tool covers a task when a word in its name, title or description starts
with one of that task's words (reading and actions: name and title only, since words like "page" or "order" appear in any description). Its path value is the
tool's quality: 0.6, +0.2 if described (≥20 chars), +0.2 with an object input schema.
On the site as it is, tools are never called, so coverage is what the site claims.
Comparison scans (adapter or router) plan up to 12 eligible tools after the UI probes,
then call each once in a fresh browser context, at most four concurrently. Tools must
have `readOnlyHint: true`, explicit `openWorldHint: false`, a callable `execute`, and no
`destructiveHint: true`. Missing open-world hints default to open-world and are skipped.
Calls use the smallest input their schema allows (the probe term for required strings).
During execution the context intercepts write requests, WebSockets and requests to
other origins. Bootstrap sockets are allowed during registration, then closed before
execution. Eligible site-owned tools are included. Hints are untrusted; same-origin
GETs may have side effects. This is not a guarantee of harmless arbitrary page code.

Discovery has a 25 s preparation budget. The subsequent execution batch has 30 s,
including each tool page's setup; each actual invocation gets its own 8 s Node timer.
The timer can close a context even when page JavaScript is in a busy loop. A tool
that throws, reports an error, returns nothing or HTML, loses its execution context,
or exceeds a deadline after its call started covers no task. Completed results are
stored in Node and survive failures in other contexts. A failure before any call
starts (including navigation or discovery errors) stays uncalled. Uncalled tools
keep advertised coverage, with reasons in the comparison table: unsafe or missing
hints, missing execute, unsupported input, the tool limit, or an exhausted batch.

`score = Σ weight × task ÷ Σ weight of applicable tasks × 100 − deductions`, floored at 0.
Tasks with no measurement are N/A (Product details on non-shops, Take action on non-shops without action tools; Find on a page with
nothing to operate and no tool).

**Countermeasures are deductions, not credit.** Not blocking agents earns nothing;
blocking them costs points: `deduction = max × (1 − ratio)` — browser access failure up to −15,
plain HTTP failure up to −7.5 when the browser succeeds, and blocking pop-ups up to −10. Browser and HTTP access penalties do not stack. A browser-level block also caps the
total at 20%. When blocked, the page checks can't be measured; they count as 0 in
their tasks (a blocked agent can't do them either), are marked "Not measured", are
never deducted or offered as fixes, and the only fix offered is browser access.

`robots.txt` directives do not affect the score. The file is fetched only to discover sitemap URLs.

**Optional agent integrations** (`agent-protocols`, `public-api`, `agent-docs`, `agent-discovery`, `agent-auth`, `agent-payments`) are informational only. Their presence or absence never changes task scores, deductions, caps or suggested fixes. Observed services and documentation remain visible in a separate section labeled “not scored”.

**Top fixes** = the three checks whose fix (ratio → 1; for WebMCP, tools covering every
task) would raise the score the most, with the points gained and a one-line remedy.

## Checks

Status: **pass** ≥ 0.9, **warn** ≥ 0.4, **fail** < 0.4, **n/a** when not applicable.
"Weight" is the check's weight inside its task's mean; for Access it is the most it
can deduct.

### Access — deductions

| Check | Weight | Measures | Scoring |
|---|---:|---|---|
| `browser-access` | 15 | Automated browser status, navigation failures and challenge markers | Failure → −15 and total capped at 20%. This is an observed access failure, not proof of deliberate bot blocking |
| `http-access` | 7.5 | Ordinary HTTP GET status, errors and challenge markers | Failure → −7.5 only when browser access succeeds; no additional deduction when browser access fails |
| `user-agent-headers` | 0 | Diagnostic GETs with `Claude-User` and `ChatGPT-User` headers, with the scanner token appended | Informational only, regardless of response. These requests do not originate from the real vendor services and never affect scores, caps or suggested fixes |
| `overlays` | 10 | "No blocking pop-ups": ≥5 s after load, a 9×9 hit-test grid finds fixed/sticky layers (site headers excluded) and classifies them as cookie consent (incl. known consent managers), region picker, newsletter pop-up, age gate or generic modal; scroll lock | None → 1. Small banner → 0.7. Covers ≥30% of the viewport or locks scrolling → 0 |

### Read (ordinary page)

| Check | Weight | Measures | Scoring |
|---|---:|---|---|
| `ssr-text` | 7 | Share of distinct rendered (`innerText`) words that already appear in the served HTML | ≥0.7 → 1, ≤0.15 → 0, linear |
| `settle-time` | 5 | Time from navigation until `innerText` stays within ±2% for 1 s | ≤2.5 s → 1, ≥10 s → 0 |
| `snapshot-size` | 5 | Accessibility snapshot (`ariaSnapshot`) size in tokens (chars ÷ 4) | ≤8k → 1, ≥40k → 0, log scale |
| `structured-data` | 5 | Valid JSON-LD (0.6, or 0.4 if any block is invalid), Open Graph title+description (0.2), meta description (0.2) | Sum |
| `landmarks` | 3 | `<main>` landmark (0.4), an `<h1>` (0.3), `<html lang>` (0.3) | Sum |

### Act (ordinary page) — Find path, and the Product path

Control text labels and browser-generated reference IDs are not scored. Element references are supplied by the browser tooling and do not measure a property of the website or an improvement made by the adapter.

| Check | Weight | Measures | Scoring |
|---|---:|---|---|
| `semantic-controls` | 5 | Clickable-looking elements (`cursor: pointer` roots) that are not links, buttons or ARIA widgets. Empty overlay layers sitting next to a real link (card hover overlays) are ignored | Fake share 0 → 1, ≥20% → 0 |
| `hidden-surfaces` | 3 | Controls reachable by `document.querySelectorAll` vs. inside shadow roots (open and closed, counted via an `attachShadow` hook); large cross-origin iframes; content outside `<body>` | Light-DOM share, −0.25 per large cross-origin iframe |
| `navigate-probe` | 5 | Click the first internal nav link (in `nav`, else `header`; not home/logo/login, not the current page or selected tab, not inside an `article`) by the role and accessible name the accessibility tree reports (`link`, or `tab` etc.) | New URL and new title/heading → 1. URL only → 0.8. Click fails → 0 |
| `search-probe` | 6 | Find search by role/name: a search box, else a same-site "Search" button/link (accepting the input it focuses), else inside the menu ("Menu", "Open menu", "Toggle main menu"…). Type a term from the page, press Enter; if the term isn't shown, click a "Search" button | Navigates and results contain the term → 1. Same page, term shown → 0.7. Page changed without the term → 0.4. Nothing → 0 |
| `url-state` | 3 | Open the post-search URL in a new page | Same results → 1. Results only in page state, not in the URL → 0. N/A if search is N/A or failed |

### Built for agents

| Check | Weight | Measures | Scoring |
|---|---:|---|---|
| `webmcp` | — | Tools registered on `navigator.modelContext` / `document.modelContext` (recorded by an init script that wraps or provides the API before page scripts run), plus declarative `<form toolname>`. The only tools a visiting browser agent can call without per-site setup. Feeds the Read/Find/Product/Cart agent paths by coverage (see Score) | Any tool → 0.6, all described (≥20 chars) → +0.2, all have object input schemas → +0.2 |
| `markdown` | — | A Markdown/plain-text version an agent can read instead of HTML: the URL serves text itself, or answers `Accept: text/markdown`, or advertises `rel="alternate" type="text/markdown"` (Link header or `<link>`), or has one at the `.md` URL | Text served / negotiated / advertised → 1. Only at the `.md` URL → 0.8. None → 0. The Read task's agent path: agents read prose more reliably than HTML or JSON |

### Optional agent integrations (not scored)

| Check | Weight | Observations | Score impact |
|---|---:|---|---|
| `agent-protocols` | 0 | Agent protocols the site advertises. **MCP:** a `Link` header or `<link>` with `type="application/mcp+json"`, or a server card naming an endpoint — SEP-1649 `/.well-known/mcp/server-card.json` (`transport.endpoint`) or `/.well-known/mcp/server-cards.json`, or `/.well-known/mcp.json` / `/.well-known/mcp`; the scanner runs `initialize` + `tools/list`. An endpoint answering 401 is reported as requiring sign-in (see `agent-auth`). **A2A:** an Agent Card at `/.well-known/agent-card.json` (or legacy `/.well-known/agent.json`), read by GET only. Optional integration: both need a connected or protocol-aware client | None; informational only |
| `public-api` | 0 | "Machine-readable API or feed", for any kind of site: an RFC 9727 API catalog (`/.well-known/api-catalog`), or advertised in a `Link` header or `<link>`: `rel="service-desc"` (RFC 8631, e.g. OpenAPI), OpenSearch description, RSS/Atom/JSON Feed, a JSON alternate, `rel="service-doc"`. MCP-typed links (`application/mcp+json`) are left to `agent-protocols`. Platform APIs also count — Shopify `/products.json`, Woo Store API, Magento GraphQL, WordPress REST — on the landing host, falling back to the typed domain | None; informational only |
| `agent-docs` | 0 | Presence of `/llms.txt` that isn't an HTML page and a valid XML sitemap. Landing host first, then the typed domain; a file found on another host is named | None; informational only |
| `agent-discovery` | 0 | Manifests that list what agents can use. **Agent Skills** index at `/.well-known/agent-skills/index.json` (RFC v0.2.0; skills with a name and url count, and those with type, description and `sha256:` digest are reported as complete). **ARD** catalog at `/.well-known/ai-catalog.json`, or wherever a robots.txt `Agentmap:` line, a `rel="ai-catalog"` link or a `_catalog._agents` TXT record points (entries need an identifier, displayName, type and exactly one of url or data). **DNS-AID** SVCB/HTTPS records at `_index`, `_a2a` and `_mcp` `._agents.<domain>` (DNSSEC validation is reported; skipped for IP targets). **Poppy** (`/.well-known/poppy.json`): organization.domain must be this site, interfaces other than `web: {}` need `auth.issuer`, and the issuer's RFC 8414 metadata must name it exactly and list the domain in `poppy_domains` | None; informational only |
| `agent-auth` | 0 | OAuth metadata an agent follows to sign in to an API or MCP server, in that order: an MCP endpoint's 401 and its `WWW-Authenticate` `resource_metadata`; RFC 9728 protected resource metadata (named by the 401, path-specific for the MCP endpoint, or `/.well-known/oauth-protected-resource`); the authorization server's RFC 8414 / OpenID Connect metadata (the first server the resource names, else this site's), reporting dynamic client registration, client ID metadata documents and an Auth.md `agent_auth` block; and `/auth.md` (Markdown with an H1 naming auth.md). Only relevant where there is something to sign in to | None; informational only |
| `agent-payments` | 0 | Ways an agent can check out or pay. **UCP** profile (`/.well-known/ucp`), **ACP** discovery (`/.well-known/acp.json` with `protocol.name: "acp"` and `api_base_url`), **AP2** as an extension in the A2A Agent Card, **x402** (an HTTP 402 with a `PAYMENT-REQUIRED` header or an `x402Version`/`accepts` body) and **MPP** (an HTTP 402 with a `WWW-Authenticate: Payment` challenge, or `x-payment-info` with intent, method and amount in `/openapi.json`). 402s are looked for on the homepage, `/api` and `/api/v1` only, so paid routes elsewhere are missed | None; informational only |

### Commerce (only if a shop is detected)

| Check | Weight | Measures | Scoring |
|---|---:|---|---|
| `product-data` | — | Product page JSON-LD: Product with name (0.25), price (0.35), currency (0.15), availability (0.25) | Sum. N/A if no product page could be located |
| `add-to-cart-probe` | — | Find the add-to-cart button by name. If it is missing or disabled — or the click does nothing — choose the first available option (select, radio group incl. visually hidden swatches, or a size button; never "Size guide") and click once more. Evidence: a non-GET request to a cart/basket/bag URL or a GraphQL cart mutation, a cart-count change, a cart dialog, or landing on the cart page. Analytics beacons don't count | Confirmed → 1. Clicked, nothing seen → 0.3. Not found → 0 |

## Not covered

The MVP check catalogue is frozen at the checks above. These are deliberately not checked, so reports make no claim about them and do not match every isitagentready.com check:

- **Content Signals** (`Content-Signal` lines in robots.txt declaring search / AI-input / AI-train preferences): a usage-policy statement, not something that changes whether an agent can read or operate the page. Consistent with robots.txt directives being unscored.
- **Web Bot Auth** (HTTP Message Signatures identifying a bot, `/.well-known/http-message-signatures-directory`): it authenticates the agent to the site, so it is a property of the agent and the site's bot manager, not of the page. The scanner does not sign its requests and cannot observe whether a site would admit signed agents.

## Known limits

- Live sites vary by A/B test, geography and time. The scanner pins `en-US` and UTC,
  but consent walls and region pickers also follow the server's IP (and some CMPs the
  timezone), so overlay results depend on where it runs. Reports are timestamped;
  "deterministic" means same procedure, not same answer forever.
- When the typed domain redirects (bbc.co.uk → bbc.com, gymshark.com → www), the
  browser checks measure the landing page; HTTP checks follow the rules above.
- Built-for-agents checks look at the homepage; WebMCP tools registered only on deeper
  pages (e.g. checkout) are not seen.
- Heuristics: `cursor: pointer` misses handler-only elements; product page discovery
  uses common URL shapes (`/product/`, `/products/`, `/p/`, `/dp/`…).
- WebMCP coverage is read from tool names and descriptions; a tool named vaguely
  ("shop_agent") covers nothing. Only comparison scans call tools, and only those
  explicitly annotated read-only and closed-world. Baseline and uncalled coverage
  remains advertised; annotations do not prove the absence of side effects.
- Probes measure whether the *affordances* exist, not whether a model reasons well
  over them (choosing the "red-ish tee" out of 30). Weights should be calibrated
  offline against bare-agent token/time costs on a sample of sites (the `agentTest`
  field in `experiments/agentface/registry.json`).
- SSRF: the server refuses private, loopback, link-local and CGNAT/tailnet targets unless
  `ALLOW_PRIVATE=1`. Browser traffic goes through a local egress proxy (`src/egress.mjs`)
  and Node fetches through a pinned lookup, so each connection goes to the exact address
  that was checked — a DNS answer that changes after the check (rebinding) can't reach
  the LAN.
- Public use: one running scan per visitor, `SCANS_PER_HOUR` (30) per visitor, at most
  `MAX_QUEUE` (10) waiting; the visitor is the first `X-Forwarded-For` address.

## Interface

- `GET /` — client page. `?url=` prefills and starts a scan (the scanner's own
  results are linkable too).
- `GET /api/scan?url=…` — Server-Sent Events: `queued`, `start`, `stage`, `check`
  (one per check as it completes, with the running task scores and deductions), `screenshot`, `done` (full report) or `error`.
  Add `&inject=adapter` to scan with the server's configured adapter (an `injection` event
  reports whether it loaded, and `error` when it threw while starting); without one the
  request ends with an `error`. If the page crashes with the adapter injected, the scan
  ends with an `error` instead of scoring the site as blocked.
- `GET /api/checks` — catalog, limits and `adapter` (`{name, version}` or `null`). The page
  runs the without/with comparison only when an adapter is configured.
- `npm run scan -- example.com [--json] [--no-cart] [--inject adapter.js]` — the same
  scan from the terminal. `--inject` adds a script to every page after
  DOMContentLoaded, to measure what an adapter would change before shipping it.
- Server settings: `ADAPTER_JS` (path to one self-contained adapter script; optional; a
  script that does not parse stops the server at start-up), `ADAPTER_NAME` (otherwise the
  name in the script's `/*! Name 1.2.3 */` banner, or its file name), `CART_PROBE=0`
  (disables the cart probe).

## Server result cache

Completed baseline and adapter-injected scans are cached separately on disk for one hour by
default (`SCAN_CACHE_TTL_MS`). Repeating a domain (e.g. `example.com` and
`https://EXAMPLE.com/#fragment`) reuses the full report, check results, page type
and screenshot, without starting a browser or consuming the hourly allowance.
Explicit paths, query strings, protocols, ports and subdomains remain distinct
because they may expose different pages. The scanner version and cart-probe
setting are also part of the cache key. Adapter results additionally include the
SHA-256 hash of the installed bundle, so changed adapter code cannot reuse an old result.

Opening `/?url=example.com` or submitting a new URL reuses both results when available.
After a completed scan the button becomes **Scan again** for that URL; clicking it
invalidates/replaces both its baseline and adapter results (`refresh=1` on each request).
Changing the URL restores **Scan site** and normal cache use. Refresh failure leaves a cache miss;
an older in-flight scan cannot repopulate the invalidated entry.
Failed or cancelled scans are not saved. Cached reports retain their
original measurement timestamp and include `cache.hit`, `cache.storedAt` and
`cache.expiresAt`; the UI labels them, including in comparisons. A cached baseline
and fresh adapter scan may differ because the site changed between measurements.

The cache survives restarts in `.cache/scans.json` (`SCAN_CACHE_FILE` can override
the location), with at most 100 entries / 32 MB of entry data; oldest entries are
evicted first. Missing or damaged cache files are treated as misses. The CLI
continues to run fresh scans.