Cloudflare spent the last thirteen months quietly building a checkout flow for machines. Pay Per Crawl arrived in July 2025 alongside a default that blocks AI crawlers on new Cloudflare sites. The Monetization Gateway followed on July 1, 2026, turning the ancient HTTP 402 status code into a live price tag on any URL. Then on August 4 came Cloudflare Wallets, which hands AI agents actual spending money with guardrails.
Here is my position, and I’ll defend it below with arithmetic: the charge button is the least useful of the three switches for a typical WordPress site. If your site earns under six figures a month from content, charging AI crawlers in 2026 pays you pocket change while quietly cutting you out of AI answers. The real story is the one almost nobody is covering: millions of sites are now blocked by default, their owners never chose it, and the AI engines have simply stopped seeing them.
Get the decision right and the machine web works for you. Get it wrong by accident and you vanish from the fastest-growing referral channel there is.
What is Cloudflare Pay Per Crawl?
Cloudflare Pay Per Crawl is a closed-beta feature that lets a site owner charge AI crawlers a set price for each successful page fetch. When a crawler requests a page, it either presents payment intent in its request headers and gets the content with an HTTP 200, or it receives an HTTP 402 Payment Required response carrying the price. No payment, no page.
The mechanics, from Cloudflare’s own documentation:
- Price is set per zone (your whole site), with $0.01 per crawl as the floor.
- Cloudflare is the merchant of record. The crawler company pays Cloudflare, Cloudflare pays you.
- It is a closed beta. You sign up through the Pay Per Crawl waitlist, or through your account executive on Enterprise.
- Firewall rules win. If your WAF or Bot Management already blocks a crawler, it never reaches the paywall.
The three announcements, in order
The machine checkout was built in three moves. Each one widens who pays and what gets paid for.
| Date | Launch | What it does | Status today |
|---|---|---|---|
| July 2025 | Pay Per Crawl + blocked-by-default | Charge AI crawlers per page fetch. New Cloudflare sites block AI crawlers unless the owner opts in. | Closed beta |
| July 1, 2026 | Monetization Gateway | Price any resource behind Cloudflare: pages, datasets, APIs, MCP tool calls. Rules like “$0.01 per GET on this route”. Settles in stablecoins over x402. | Waitlist |
| August 4, 2026 | Cloudflare Wallets + cloudflare.pay | Spending money for AI agents. Virtual Wallets with allowances, allow lists and per-transaction caps set by a human. | Handles claimable now, functionality rolling out |
Notice the direction. First crawlers pay for pages. Then anyone pays for anything. Then the buyers stop being companies and start being agents with pocket money. The x402 Foundation behind the payment protocol lists more than 25 contributing companies, so this is not a solo Cloudflare bet.
Allow, block, or charge: the decision for WordPress owners
Every AI crawler hitting your site gets one of three treatments. Pick per crawler, not sitewide, because a search crawler, an agent fetching a page for a user, and a training scraper are three different trades.
| Allow | Block | Charge | |
|---|---|---|---|
| What you get | Citations, AI referrals, brand presence in answers | Content stays out of AI systems | Cents per fetch, from crawlers enrolled in the beta |
| What you give up | Content feeds answers without a guaranteed click | Every citation and referral from that engine | Same as blocking, for any crawler that declines to pay |
| Right for | Most WordPress sites competing for AI visibility | Private areas, member content, commodity pages you’d rather not train competitors’ models on | Large archives on Enterprise plans with real bargaining power over AI companies |
Here is the part that gets missed: a price a crawler refuses to pay is a block with extra steps. URL accessibility sits at the top of the 23 AI citation ranking factors at 9.5 out of 10, because an engine that cannot fetch your page cannot quote it. Nothing else on the checklist matters if the crawler bounces off a 402 it won’t settle.
The revenue math nobody shows you
Run the numbers before you get excited. Our own site logged 152 AI crawler visits in the last 7 days, roughly 660 a month. At the $0.01 floor that is $6.60 a month, assuming every crawler enrolls and pays, which none are obliged to do. A mid-size publisher doing 100,000 AI crawls a month grosses $1,000. Meaningful for a newsroom with millions of archive pages. Irrelevant for a 60-page business site.
Now price the other column. Semrush’s 2026 research measured AI search visitors converting at 4.4 times the rate of organic search visitors. One AI referral that becomes a customer is worth more than a year of crawl fees for most small sites. That asymmetry is the whole argument, and it is why the charge switch should be the last one you reach for, not the first.
How to control AI crawlers on WordPress today
You do not need to wait for a beta invite to take control. The order matters:
- See who actually crawls you. Decisions made blind are guesses. The free AI Crawler Log in AEO God Mode records every AI bot visit, which pages they fetch, and how often. Read it for a week before touching anything. Our guide to reading WordPress AI crawler logs covers what normal looks like.
- Sort crawlers into three jobs. Cloudflare’s own taxonomy splits them into Search (indexes your pages for AI search), Agent (fetches a page because a user asked), and Training (feeds model training). You can welcome the first two and refuse the third. That is a policy, not a mood.
- Enforce it per crawler. The AI Crawler Allowlist manages 18 named crawlers (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and friends) through robots.txt and headers, free, on any host.
- On Cloudflare? Check what the default already decided for you. If your zone was created after early July 2025, AI crawlers may be blocked right now without you ever choosing it. Open AI Crawl Control in the Cloudflare dashboard and look. If you run a large archive on Enterprise, this is where you join the Pay Per Crawl beta.
- Verify the result monthly. Blocking a crawler should remove you from that engine’s answers, and allowing one should eventually get you cited. Check it instead of assuming it. That is exactly what a citation tracker is for.
Agents with wallets change what a website is
The Wallets announcement is the tell for where this goes. An agent with an allowance, an allow list and a per-transaction cap is a buyer. It will pay a third of a cent for a data lookup without blinking, because a human set the budget and the job needs the data. Websites stop being pages that hope for visits and start being resources that answer requests, some free, some priced.
You position for that future the same way you win AI citations now: answer-first pages, question-shaped headings, FAQ blocks that match real buyer questions, clean structure a machine can parse without archaeology. Free access is your distribution. The day charging genuinely pays, you can add a price to something valuable. Until then, being the source engines quote beats being the site that charged them a penny to leave.
Sources
- Cloudflare: Announcing the Monetization Gateway (July 1, 2026)
- Cloudflare: Announcing Cloudflare Wallets (August 4, 2026)
- Cloudflare Docs: Pay Per Crawl
- TechCrunch: Cloudflare’s new policy pushes AI companies to pay for publishers’ content
Cloudflare Pay Per Crawl: quick answers
Is Cloudflare Pay Per Crawl available to everyone?
No. Pay Per Crawl is in closed beta. You request access through Cloudflare’s signup form, or through your account executive if you are on an Enterprise plan. The Monetization Gateway, the wider version that prices any resource, has its own waitlist as of August 2026.
How much money can a WordPress site make from Pay Per Crawl?
Do the arithmetic with your own crawler log. At the $0.01 per crawl floor, a site receiving 660 AI crawls a month grosses $6.60, and only from crawlers that agree to pay. A publisher with 100,000 monthly AI crawls grosses about $1,000. For most WordPress sites the visibility you surrender is worth more than the fees you collect.
Do I need Cloudflare to charge AI crawlers?
Yes, for the payment part. The 402 challenge, x402 settlement and merchant-of-record billing all run on Cloudflare’s edge. Without Cloudflare you can still allow or block each crawler through robots.txt and headers, which the free AEO God Mode plugin manages for 18 named AI crawlers on any host.
Will blocking AI crawlers hurt my Google rankings?
Blocking AI crawlers does not touch your Google rankings if you block the right bots. Googlebot handles search rankings. Google-Extended is the separate token for AI training, and Google’s AI Overviews are fed by normal search indexing. Block GPTBot and you leave ChatGPT’s answers, not Google’s results. Block Googlebot and you have a very different problem.
What is x402?
x402 is the payment protocol behind Cloudflare’s machine checkout, built on the HTTP 402 Payment Required status code. A server prices a request, the client pays in stablecoins and retries with proof, and Cloudflare verifies at the edge. An x402 Foundation with more than 25 contributing companies maintains it.
Should news publishers charge AI crawlers?
Publishers are the one group with a real case. Large archives, high crawl volume and licensing negotiations give them bargaining power a small site does not have, and per-crawl fees stack up at newsroom scale. If that is you, join the beta and price deliberately. If your archive is 80 posts, visibility pays better.
What is the difference between AI Crawl Control and Pay Per Crawl?
AI Crawl Control is Cloudflare’s free dashboard for seeing AI crawler traffic and allowing or blocking each bot. Pay Per Crawl is the closed-beta charging layer on top, which answers crawlers with an HTTP 402 price instead of a plain block. Control ships to everyone, charging is invite-only for now.
Does AEO God Mode handle crawler payments?
No, and that is deliberate. Payments only work at the network edge, where Cloudflare sits. AEO God Mode covers the parts every WordPress site needs regardless of host: a free log of every AI crawler visit, per-crawler allow and block controls, and citation tracking so you can see what your access decisions actually do to your AI visibility.