Watch AI crawlers work your site.
GPTBot, ClaudeBot, PerplexityBot, Googlebot and the rest — what they read, what they're blocked from, and what that costs you in answers. One generated file on your server, live in minutes.

01
Log-level truth
Real edge requests, not synthetic checks — by agent, path, status and day.
02
Config alerts
robots.txt and llms.txt misconfigurations flagged with the exact rule — and the fix.
03
AI referral attribution
The human traffic AI sends back — by platform and landing page.
Setup
Live in minutes, not sprints
1
Add the generated file
Copy the Cloudflare Worker or Next.js proxy file from AI crawlers › Install — there is no package to install.
2
Crawlers appear as they arrive
Known AI crawlers told apart by purpose — training, search or user-triggered — and unknown bots flagged.
3
Misconfigurations become actions
Every blocked path is listed with the robots.txt rule behind it and the engines it shuts out.
Edge snippet~1 MIN
// cloudflare worker · copied from AI crawlers › Install
// no dependencies — the endpoint and your key are already in it
import { reportRequest } from "./rankyourbrand-sender";
export default { async fetch(request, env, ctx) {
const response = await fetch(request);
ctx.waitUntil(reportRequest({ status: response.status, … }));
return response; } };
// no dependencies — the endpoint and your key are already in it
import { reportRequest } from "./rankyourbrand-sender";
export default { async fetch(request, env, ctx) {
const response = await fetch(request);
ctx.waitUntil(reportRequest({ status: response.status, … }));
return response; } };
No request blockingReports after the responseMost recent events
Coverage
By purposeAI agents
GPTBot, ClaudeBot, PerplexityBot, Googlebot, Bingbot and the rest — training, search and user-triggered told apart
Attribution
Per visitAI referrals
Human traffic AI sends back, tied to landing pages
Found on first scan
Blocked pathswith the rule behind each
Each one traced to the robots.txt line that causes it
Find out what the crawlers can't read.