Web Bot Auth

Web Bot Auth is a proposal for letting an automated HTTP client sign its requests so a server can tell who is asking. It defines a header field that points at the client's own key directory, so the identity is checkable without the server having been told about that client in advance. It answers who, and deliberately nothing else.

The draft describes a protocol for identifying automated traffic using HTTP message signatures, so that automated clients can cryptographically sign outbound requests and servers can verify their identity with confidence. It defines a Signature-Agent header field for in-band key discovery, a key directory format, and a well-known address at which that directory is served.

Its own scope section is unusually explicit about what it does not do, and the exclusions are the useful part. It does not authenticate human users, does not provide anonymous authentication, and defines neither authorization nor delegation. It does not define how trust is accrued, held or exchanged, and it gives no mechanism for one origin to pass an opinion about an agent to another. A client chooses whether to sign, and an origin chooses how it treats signed and unsigned requests.

Its standing is worth stating plainly, because the address does not carry it. This is an individual Internet-Draft on the standards track, not an IETF standard and not endorsed by the IETF; an earlier architecture draft under a different name is marked replaced and expired. A draft has a stated expiry date, and citing one is citing a document that is meant to move.

It is the identity half of two things this glossary describes on the payment side. Cloudflare aligns its pay-per-crawl signing with the same mechanism. The Visa Trusted Agent Protocol states that its own agent-recognition signature is aligned with this work and built on the same RFC 9421 fields. Knowing who is asking and knowing what it costs are separate exchanges that happen to share a request.

→ probe402 identifies itself in every request it makes rather than signing one, and the user agent it sends is published in the method.

Sources

Related RFC 9421 · pay per crawl · robots.txt · agent

Glossary