Web Bot Auth: Cryptographic Bot Verification Explained

Web Bot Auth is an emerging standard that lets a bot cryptographically sign its HTTP requests, so the server can verify the bot’s identity from the signature itself instead of trusting a user-agent string or an IP address. It applies RFC 9421 HTTP Message Signatures to crawler traffic: the bot signs each request with an Ed25519 private key and publishes its public keys at a well-known URL, and the server verifies the signature against those keys. A valid signature is proof; a user-agent header never was.

Here is why the old way of identifying bots is fragile, how signature verification fixes it, where the standard actually stands in 2026, and how Kitbase verifies signatures automatically when a crawler sends them.

Why user-agent + IP verification is fragile

For years, the way to confirm a bot was who it claimed to be involved two steps: read the user-agent string, then check the source IP against the vendor’s published ranges. Both steps have problems.

• The user-agent is just text. Any client can send Mozilla/5.0 … GPTBot/1.4. The string is a claim, never proof — which is why spoofed “AI crawler” traffic is a real and growing category. A scraper wearing a trusted bot’s identity slips past user-agent-based rules trivially.
• IP ranges are a moving target. Vendors publish their egress ranges as JSON, but those lists change, some vendors publish none at all, and cloud IPs get recycled between tenants. Matching an IP to a CIDR block is stronger than trusting a header, but it is bookkeeping that breaks quietly when a range updates.
• Neither survives the agent era. As AI assistants start dispatching autonomous agents that fetch pages on a user’s behalf — often from shared cloud infrastructure — “which IP did this come from” stops being a reliable identity signal at all. The request might originate from the same data center as a thousand unrelated services.

The fundamental issue is that both signals are circumstantial. Web Bot Auth replaces circumstantial evidence with a cryptographic proof that travels inside the request.

How signature verification works

The mechanism is standard public-key cryptography applied to HTTP:

1. The bot operator generates an Ed25519 key pair and keeps the private key secret.
2. It publishes the public keys in a directory at /.well-known/http-message-signatures-directory on a domain it controls.
3. On every request, the bot signs selected components of the message (per RFC 9421) with its private key, attaching signature, signature-input, and signature-agent headers. The signature-agent header points to the directory where its public keys live.
4. The server fetches the signer’s public keys (once, then caches them) and verifies the signature. If the math checks out, the request provably came from the holder of that private key — no IP lookup, no user-agent guessing.

Because the private key never leaves the operator and the signature covers the request itself, there is nothing to spoof. An attacker cannot forge a valid signature without the key, and cannot replay a captured one against a different request.

flowchart TD
A["Incoming request<br/>claims to be a known bot"] --> B{"Signature<br/>present?"}
B -->|"Yes"| C["Fetch signer public keys<br/>from /.well-known directory"]
C --> D{"Signature<br/>valid?"}
D -->|"Yes"| V["Verified — cryptographic proof,<br/>unspoofable"]
D -->|"No"| S["Rejected — forged signature"]
B -->|"No signature"| E{"Source IP inside<br/>vendor CIDR ranges?"}
E -->|"Yes"| W["Probably genuine<br/>circumstantial, ranges drift"]
E -->|"No"| S2["Likely spoofed"]

The top path is deterministic and durable; the bottom path is the fragile, best-effort fallback everyone relied on before signatures existed.

Where the standard stands in 2026

It is important to be precise here, because the space moves fast and marketing copy tends to overstate adoption.

• The base is real and ratified. RFC 9421, HTTP Message Signatures, is a published IETF standard. It defines the general signing and verification mechanism that Web Bot Auth builds on.
• Web Bot Auth itself is still an Internet-Draft, not an RFC. It is a set of individual IETF drafts led by Cloudflare (authored with a Google contributor) — an architecture draft (draft-meunier-web-bot-auth-architecture, since evolved into a protocol draft titled HTTP Message Signatures for automated traffic), a directory draft defining the /.well-known/http-message-signatures-directory format, and a registry draft. Internet-Drafts are proposals under active discussion; they carry no formal standards status and can change.
• Cloudflare is driving deployment. In mid-2025 Cloudflare folded HTTP Message Signatures into its Verified Bots program, shipped open-source signing/verifying libraries, and began encouraging bot operators to adopt them. Cloudflare signs some of its own crawlers as reference implementations.
• Broad AI-crawler adoption is still early. As of 2026, signing is more common among agentic clients (AI assistants fetching a page on a user’s behalf) than among the big training crawlers. Some vendors have documented signing for their agent products, and an industry key-registry effort has emerged to publish operator keys at scale — but many major crawlers still identify themselves the old way, with a user-agent and an IP range. Treat Web Bot Auth as a rising signal to verify when present, not as a universal one you can require.

The trajectory is clear even if the timeline isn’t: cryptographic verification is where bot identity is heading, because it is the only approach that survives spoofing and the shift to agent traffic. It complements, rather than replaces, the IP-range verification you still need for the bots that haven’t adopted it yet.

How Kitbase uses it

Kitbase’s bot & crawler attribution pipeline treats Web Bot Auth as the strongest signal in a layered verification hierarchy, short-circuiting on the first decisive result:

1. Web Bot Auth — RFC 9421 Ed25519 signatures, verified against the signer’s /.well-known/http-message-signatures-directory. Unspoofable, so it wins outright when present.
2. Exact CIDR membership — the client IP must fall inside the vendor’s published ranges (not merely the same ASN, which spoofers share).
3. Forward-confirmed reverse DNS — for vendors that publish no IP list.

The practical payoff: you don’t configure any of this. When a crawler signs its request, Kitbase verifies the signature automatically and labels the visit verified with method web_bot_auth; when it doesn’t, Kitbase falls back to CIDR and rDNS. Either way, every stored crawler visit carries a verified-or-spoofed verdict and the method used to reach it — so a scraper impersonating GPTBot gets flagged as spoofed_bot rather than trusted, and a properly-signed request gets the cryptographic gold star it earned. Setup is a copy-paste forwarder for your platform; the verification runs server-side on every request.

To see how the older verification methods work for bots that don’t yet sign, see GPTBot explained and the broader list of AI crawlers.

FAQ

Is Web Bot Auth an official standard? Its foundation is: RFC 9421 (HTTP Message Signatures) is a ratified IETF standard. Web Bot Auth itself — the profile applying signatures to bots — is still a set of IETF Internet-Drafts, not yet an RFC, though it is being actively developed and deployed.

How is Web Bot Auth different from checking IP ranges? IP verification is circumstantial: it asks “did this come from an address the vendor uses?” and breaks when ranges change or infrastructure is shared. Web Bot Auth is a cryptographic proof carried inside the request — it can’t be spoofed and doesn’t depend on where the request originated.

Do AI crawlers sign their requests today? Some do, especially agentic clients that fetch pages on a user’s behalf, and Cloudflare signs some of its own crawlers. But as of 2026 many major training crawlers still rely on user-agent and IP. Adoption is growing but not universal, so verification systems need a fallback.

What does a Web Bot Auth request look like? It carries signature, signature-input, and signature-agent HTTP headers. The signature-agent points to the operator’s public-key directory at /.well-known/http-message-signatures-directory, which the server uses to verify the Ed25519 signature.

Does Kitbase require crawlers to use Web Bot Auth? No. Kitbase verifies signatures when they’re present (the strongest possible verdict) and falls back to IP-range and reverse-DNS checks when they’re not — so you get the best available verification for every bot, signed or not.

Want every AI crawler visit labeled verified or spoofed — by signature when available, by IP range when not? Start your free trial — 7 days, no credit card required — and let Kitbase handle the cryptography for you.