Skip to content
Primores AI wiki

Pay-Per-Crawl — Definition

Pay-Per-Crawl

TL;DR: Pay-per-crawl is Cloudflare’s mechanism that lets a site charge AI crawlers for access, using the long-dormant HTTP 402 Payment Required status code and crawler-price headers. Instead of “block or allow,” it adds a third option: “allow, for a fee.”

What it means

Pay-per-crawl, launched by Cloudflare in July 2025, reframes AI bot access as a transaction. Rather than only blocking or allowing a crawler, a site can respond to an AI crawler’s request with HTTP 402 Payment Required and a stated price (via a crawler-price header); crawlers willing to pay signal so with a crawler-max-price header, and the access clears. It revives a status code that was reserved in the original HTTP spec but never widely used — repurposed for the machine-to-machine economy of AI crawling.

Why it matters

Pay-per-crawl signals where AI bot access is heading: away from a per-site config problem and toward an infrastructure-layer marketplace. Cloudflare fronts a large share of the web and began blocking AI crawlers by default in 2025, which puts it in a position to broker access at scale. For publishers, it offers a way to be compensated for training crawls that previously took content for free. For site owners generally, it underscores that the real control point is increasingly the CDN, not a text file in your repo.

How it works / examples

The flow is roughly:

  1. An AI crawler requests a page behind Cloudflare.
  2. The site (via Cloudflare) returns 402 Payment Required with a price.
  3. A crawler configured to pay (within its crawler-max-price) completes the transaction and receives the content; one that isn’t gets blocked.

This sits at the firewall/edge layer (see glossary/waf) — the same layer that does enforcement — which is why it can charge non-compliant requests that robots.txt could never have stopped.

Sources