Identifying an automated client: User-Agent, contact address, robots rules and rate-limit etiquette

Goal

Be an automated client that a site operator can identify in a log line, contact when something goes wrong, and control through the mechanisms that already exist, instead of one that is indistinguishable from abuse and gets blocked wholesale.

Prerequisites

Control over the HTTP headers your client sends and over its request rate. A page that describes what the client does and how to reach its operator.

Steps

1. Send a User-Agent on every request with a product token, a version and a link to the description page, for example Mozilla/5.0 (compatible; ExampleBot/0.1; https://www.example.com/bot.html). RFC 9309 uses exactly this shape and says the identification string should describe the purpose of the crawler; RFC 9110 defines the header as a sequence of product identifiers with the most significant first and discourages needlessly fine-grained detail.
2. Choose the product token so that it can appear in a robots.txt user-agent line: RFC 9309 restricts it to letters, underscores and hyphens and says it should be a substring of the User-Agent header, so ExampleBot in the header lets an operator write user-agent: ExampleBot and address your client alone.
3. Before fetching pages of a site, read /robots.txt and apply the group whose token matches yours case-insensitively; if none matches, apply the * group; if there is none, access is unrestricted. Match paths from the first octet and use the most specific rule, as the RFC requires of crawlers.
4. Do not disguise the client as a browser to get around a rule; if a site blocks your token, the answer is to contact the operator through the address you published, not a different header.
5. Treat 429 Too Many Requests (RFC 6585) as an instruction: stop, wait at least the Retry-After value if one is given (RFC 9110 allows seconds or an HTTP date), then resume at a lower rate. Apply the same to 503 with Retry-After.
6. Keep a per-host budget: a small number of concurrent connections and a pause between requests to the same host, lower for small sites. Cache what you fetched and use conditional requests so repeated reads cost the operator nothing.
7. Send an Accept header that matches what you can process, and prefer machine formats the site offers (feeds, sitemaps, an API, llms.txt) over crawling HTML.

Expected result

Your requests appear in access logs under one recognisable name with a working contact address; an operator who wants to slow you down or shut you out can do so with one line in robots.txt or with 429 responses, and never has to block an address range that also serves other people.

Limits and test basis

This is etiquette and standards synthesis, not a legal analysis; site terms and local law are separate questions. RFC 9309 governs crawlers reading public pages; an API you call under a key has its own documented limits that take precedence.

Crawl or user-triggered fetch

RFC 9309 is scoped to crawlers. Distinguish two cases in the client: a crawl (traversing links, building an index, collecting training material) obeys robots.txt without exception and identifies itself with a crawler token; a user-triggered fetch (one page, because a person asked for it now) identifies itself with a token that says so, fetches only that page, does not follow links, does not retain the content beyond the session and does not use it for training. Several large operators publish separate tokens for the two cases; an operator who has disallowed crawlers has usually decided about crawling, not about a single reader acting for one person. Where a site blocks the user-fetch token too, that decision is respected as well.