robots.txt
robots.txt is the file at the root of a site that tells automated clients which parts of it they are asked not to fetch. It is the Robots Exclusion Protocol, standardised as RFC 9309, and it is a request rather than an enforcement. It is also the layer the paid-crawl work is built above.
The standard describes itself as specifying and extending a method for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Its own framing of the problem is modest: it may be inconvenient for service owners if crawlers visit the entirety of their address space, so the document sets out the rules crawlers are requested to honor. A crawler that downloads the file successfully must follow the rules it can parse.
The word requested is doing real work. Nothing in the mechanism stops a client that ignores the file; what it establishes is that the operator stated a preference in a machine-readable place, which is what makes ignoring it a decision rather than an accident.
That is why it is the floor the paid-crawl work builds on. Cloudflare's pay per crawl turns the binary into a price, answering an unpaid crawler with a charge instead of a refusal, and RSL expresses licensing terms in the same file. Consent is the older layer, and the price is built above it.
→ probe402 reads each host's robots.txt before it probes and honours what it finds there; how that is done, and how often, is set out in the method.
Sources
- RFC 9309: Robots Exclusion ProtocolIETF
- What is pay per crawl?Cloudflare documentation
- Really Simple LicensingRSL Internet Collective
Related pay per crawl · RSL · Web Bot Auth · agent