How do I stop AI bots from scraping my website?
Short answer: You can block them with robots.txt or a firewall, but blocking also keeps you out of AI answers. The alternative is to let AI bots in on your terms: they pay a small fee per page with x402, people keep reading for free, and search engines keep indexing.
Try Penni free in test mode · Check any URL with the free x402 checker
AI crawlers such as GPTBot, ClaudeBot and PerplexityBot read websites to train models and to answer questions in real time. For a site owner that means server load, and your work showing up in answers without a visit or a cent in return. There are three ways to respond.
Option 1: Ask them to stay away (robots.txt)
Add the crawlers to robots.txt:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /The large AI companies say their crawlers respect this. It costs nothing, but it is a request, not a lock: crawlers that ignore it, or that you didn't list, keep coming. And it is all or nothing.
Option 2: Block them at the door
A firewall or CDN rule (Cloudflare, your host, a security plugin) can refuse known AI user agents and suspicious traffic. That works better than robots.txt, but it has the same downside: when AI assistants can't read you, they don't cite you, link to you or recommend you. As more people ask ChatGPT, Claude or Perplexity instead of searching, being invisible there has a price too.
Option 3: Let them in, for a fee
The web has had a status code for this since 1997: 402 Payment Required. With x402, the open standard built on it, your site answers an AI bot with a price instead of the page. A bot that wants the content pays, a few cents or less, in a stablecoin, and gets the page in the same request. A bot that doesn't pay gets nothing.
- People keep reading for free. Browsers get the normal page; only machine traffic is asked to pay.
- Search engines keep indexing. Googlebot and Bingbot are not AI crawlers in this sense, so your SEO is untouched.
- You set the price, per page or per section: free news, €0.01 per article, €0.25 for the archive.
- The money goes straight to your own wallet, not to a platform that pays out later.
Which one should you pick?
If you never want your content in AI systems, block (option 1 and 2 together). If you want to stay visible in AI answers but stop giving your work away, charging is the middle ground. You can also combine them: block the crawlers you don't trust, and let the rest pay.
How to do it with Penni
Penni is a European x402 facilitator (why Penni): it checks each payment and settles it on the blockchain, straight to your wallet. On WordPress you install the Penni plugin, choose "AI agents and apps only", and set your prices in the dashboard. For any other site there is a Cloudflare Worker that sits in front of it, or middleware for your own server. Penni charges €0.002 per payment; test mode is free. Afterwards, paste one of your URLs into the free x402 checker to see exactly what an AI bot now gets.
Do AI crawlers actually pay?
Crawlers that only scrape will usually leave when they get a 402, which is the same result as blocking. AI agents and apps built with x402 support, a growing group, pay automatically. Either way, nothing is served for free to machines unless you allow it.
Will charging AI bots hurt my Google ranking?
No. Search engine crawlers and link previews are not charged by default, and people in a browser see your normal pages.
How does the site know a visitor is an AI bot?
By the request: known AI user agents, requests that don't ask for an HTML page (scripts and API clients), and requests that already carry an x402 payment. The list of AI user agents can be changed.
What does it cost me?
Penni charges €0.002 per successful payment from prepaid credits, with no subscription. The visitor's payment goes to your wallet in full.