How this is measured, and where it stops
The parts a vendor usually leaves out are the parts that tell you whether to trust the rest. This page is that.
What happens when you type a domain
Origin resolution
We work out your canonical origin ourselves — https against http, www against apex, following redirects and recording them. Sites are frequently not serving what their owners think they serve.
robots.txt, parsed properly
Not a yes/no lookup. Group structure is preserved so we can quote the exact line that allowed or blocked each crawler, including malformed directives that crawlers silently discard — the rule you think you wrote is not always the rule being applied.
Sitemap and page sampling
Your sitemap is fetched and followed one index level deep, then a few deep content pages are sampled alongside the homepage, about page and contact page. Answer-extraction usually fails on article and service pages, not on the front door.
Live crawler probe
The homepage is requested once as each real AI crawler user-agent, with a browser user-agent control fetch before and after the sweep, and an unknown-bot control user-agent alongside it. What your edge does is a different question from what robots.txt says, and both go in the report.
Scoring
46 findings, each with a weight, rolled into five weighted bands. Partial credit is allowed, and a check that could not be evaluated is excluded rather than scored as a failure.
Three rules written into the engine, not into a policy page
A wrong number is more expensive than no number. These are enforced in code because good intentions do not survive a deadline.
A rate limit is never reported as a block
429, 503 and burst-triggered 403 responses get a back-off and a retry. If it still cannot be settled, the crawler is marked inconclusive and excluded from scoring. Otherwise a shared host's throttle ends up in a client report as "this site blocks GPTBot", and that is the easiest lie in this category to tell by accident.
robots.txt and the live edge are separate findings
A site can allow every AI crawler on paper and refuse them all in production. Both answers are reported side by side for every crawler, and they are never merged into one verdict.
Anti-spoofing is a question, not a finding
If known AI-crawler user-agents are refused but an unknown bot user-agent is served, the edge is matching AI signatures. From outside the network that is indistinguishable from correct anti-spoofing — a real crawler arrives from published IPs. We report it as something to confirm in your CDN logs, not as a block.
A request we chose not to send is "not tested"
There is a hard per-host request budget and a circuit breaker. If either stops the run, the remaining crawlers are marked not tested and left out of the score. We built that guard after our own audit got our office IP banned by a shared host.
What this deliberately does not do
No JavaScript rendering
On purpose. AI crawlers do not execute JavaScript either. A client-rendered site scoring badly here is the correct result, not a gap in the test — it is exactly what the crawler sees.
Not a full-site crawl
Homepage plus a small sample of pages chosen from your sitemap. Enough to characterise the site, not enough to audit every page. The report says which pages were sampled.
No Google AI Overview measurement
That requires SERP scraping, which is a paid data source with its own accuracy problems. Rather than estimate it, we leave it out and say so.
Brand details are derived, not verified
Company name, category and location come from your own markup. If your markup is wrong, our reading of it will be wrong in the same way — which is itself a finding.
No recommendation measurement in the free check
25 prompts across 4 engines, repeated three times with a published noise floor, is a paid tier because it costs API credit on every run. Merging it into the free score would mean quoting you a single unreplicated result.
No traffic estimate
Your traffic exists in your logs. We give you a tracker and a GA4 recipe rather than an estimate, because an estimate of your own data is worth less than your data.
Why one check per domain per day
The five bands are weighted by what a failure in each actually costs. Crawler access is 30 points because a blocked crawler cannot cite you at all, at any score. Trust signals are 10 because they shift the odds rather than the outcome. Inside each band, individual checks are weighted the same way — a blocked OAI-SearchBot outweighs a missing breadcrumb by an order of magnitude, and the report orders your fix list by points recoverable rather than by how alarming the label sounds.
What the checks are based on
Every crawler token, permission rule and index behaviour below comes from the operator's own published documentation rather than from folklore. If one of these pages changes, the corresponding check changes.
| Source | What we take from it |
|---|---|
| OpenAI — bots | The GPTBot, OAI-SearchBot and ChatGPT-User tokens, and which surface each one feeds. |
| Perplexity — crawlers | PerplexityBot and Perplexity-User, and the difference between indexing and live fetch. |
| Google — crawlers and fetchers | Googlebot and the Google-Extended permission token used for Gemini grounding. |
| Anthropic — web crawling | ClaudeBot, Claude-User and Claude-SearchBot. |
| Microsoft Bing — crawlers | bingbot, and the index that ChatGPT Search and Copilot both read from. |
| IndexNow | Key-file placement and submission, checked as part of discoverability. |
| Schema.org | The vocabulary behind every structured-data check, including sameAs and FAQPage. |
| RFC 9309 — Robots Exclusion Protocol | How groups, wildcards and precedence are meant to be parsed, including the malformed cases crawlers discard. |
| llms.txt proposal | The file format checked under machine readability. |
Verify any of it yourself.
Every finding carries the evidence we measured, and all of it is reproducible with curl. If you find one that is wrong, tell us — we would rather fix it than defend it.