How an automated client identifies itself: User-Agent, contact address, robots.txt and rate limits

Goal

An automated client that an operator can recognize in a log line, contact when problems arise and control with the existing mechanisms, rather than a client that cannot be told apart from abuse and gets blocked wholesale.

Prerequisites

Control over the HTTP headers the client sends and over its request rate. A page that describes what the client does and how to reach its operator.

Steps

1. Send a User-Agent with a product token, a version and a link to the description page on every request, for example Mozilla/5.0 (compatible; ExampleBot/0.1; https://www.example.com/bot.html). RFC 9309 uses exactly this form and requires the identifier to describe the crawler's purpose; RFC 9110 defines the header as a sequence of product identifiers, most significant first, and advises against unnecessarily fine-grained detail.
2. Choose the product token so that it can appear in a user-agent line of robots.txt: RFC 9309 allows only letters, underscores and hyphens, and requires the token to be a substring of the User-Agent header. This lets an operator target exactly this client with user-agent: ExampleBot.
3. Before fetching pages from a website, read /robots.txt and apply the group whose token matches your own, ignoring case; if none matches, the * group applies; if that is missing too, access is unrestricted. Compare paths starting from the first octet and apply the most specific rule, as the RFC requires of crawlers.
4. Never disguise the client as a browser to get around a rule. If a website blocks the token, the way forward is to contact the operator at the published address, not to send a different header.
5. Treat 429 Too Many Requests (RFC 6585) as an instruction: stop, wait at least the time given in Retry-After (RFC 9110 allows seconds or an HTTP date), then resume at a lower rate. The same applies to 503 with Retry-After.
6. Keep a budget per host: few concurrent connections and a pause between requests to the same host, more generous for small websites. Cache what you have fetched and use conditional requests so that repeated reads cost the operator nothing.
7. Send an Accept header that matches what the client can process, and prefer machine-readable formats (feeds, sitemaps, an API, llms.txt) over crawling HTML.

Expected result

The requests show up in access logs under a recognizable name with a working contact address. An operator who wants to slow down or exclude the client can do so with a single line in robots.txt or with 429 responses, and never has to block an address range that would also affect others.

Limits and test basis

This is etiquette and a summary of standards, not a legal assessment; terms of use and local law are separate questions. RFC 9309 governs crawlers on public pages; an API used with a key has its own documented limits, which take precedence.