RSL

It extends what a site can say from permission to terms. Where the Robots Exclusion Protocol lets an operator say which paths a crawler is asked to leave alone, this says on what conditions content may be used — naming attribution, pay per crawl and pay per inference compensation among them. The terms are expressible in robots.txt and in web pages, HTTP headers, media files and feeds.

Pay per inference is the part with no equivalent on the HTTP payment rails. A per-fetch price is a charge for retrieving something, which is an event at an address that anybody can observe; a per-inference term is a condition on what happens to the content afterwards, inside somebody else's model. The second is a licensing claim rather than an exchange over HTTP, and it is not visible in any response.

So the crawl-consent vocabulary has three layers and they are worth keeping apart: robots.txt states access, this states terms, and pay per crawl states a price in a live response.