pay per crawl
Pay per crawl lets a site owner charge AI crawlers for access instead of blocking them. A crawler that has not paid receives HTTP 402 with a price; a crawler willing to pay repeats the request carrying a price of its own, inside the components its request signature covers.
The documentation describes it as a feature of AI Crawl Control that enables site owners to control and monetize AI crawler access to content by setting a price per zone. A crawler without payment receives an HTTP 402 Payment Required response with pricing, carried in a crawler-price response header.
The crawler answers with a price header of its own — crawler-exact-price to match the quoted price, or crawler-max-price to accept anything at or below a ceiling it sets. On success the response carries crawler-charged with the amount billed. Those payment headers must sit inside the components its request signature covers, so what a crawler offers is bound to that signature rather than floating free of it.
That binding is what sets it apart from the other HTTP 402 rails. Payment rides on crawler identity, and Cloudflare acts as the merchant of record for the transaction, so no facilitator and no on-chain settlement are involved. The payer is a crawler fetching content rather than a client buying an API call — the same status code, a different trade.
→ probe402 obeys the crawl-consent rules published by every host it reads, and identifies itself in each request. Both are set out in the method.
Sources
- What is pay per crawl?Cloudflare documentation
- Pay per crawl: crawl pages as an AI ownerCloudflare documentation
Related payment challenge · rate limiting vs payment gating · HTTP 402 · pay-per-call API