What share of a small site's requests come from crawlers and automated agents, and which classification method held up over a year?

Open question

RFC 9309 has each crawler set its own name, which the specification calls a product token, and use it to find the group in robots.txt that applies to it. That only describes crawlers that choose to identify themselves. A site's access log also contains uptime probes, security scanners, link checkers, feed readers, headless browsers run by data collectors, and, increasingly, agents driven by language models that fetch pages on behalf of a user and may or may not announce what they are. It is commonly reported that a small site's traffic is "mostly bots"; the share is rarely measured, and the measurement is harder than it looks: user-agent strings are self-declared, address ranges change, and a request from a real browser can still be scripted.

What the wiki lacks is a set of measured shares with the method stated. For a small site (a documentation site, a blog, a small web application), what share of requests over a month came from clients declaring a known crawler token, from clients declaring a language-model agent, from clients with a generic library user agent, and from clients that looked like browsers but never fetched a stylesheet or executed a script? What share of the site's bandwidth and server time went to each class? Which classification signals held up: declared tokens, published address ranges, reverse DNS, behavioural patterns (no assets, no cookies, uniform timing), robots.txt fetches preceding the visit? And one year later, did the same method still classify the traffic, or had the mix shifted enough (new agents, new tokens, browsers embedded in agents) that the numbers were no longer comparable?

The answer matters for capacity, for the decision whether to serve agents deliberately, and for reading analytics that assume a human behind each request.

What a useful answer contains

The site type, monthly request volume, and whether a CDN or a reverse proxy sat in front and filtered anything before the log. The classification rules in full, including the list of tokens treated as crawlers and agents, and the order in which rules were applied. The resulting shares by requests, bytes and server time, with the unclassified remainder stated rather than folded into "human". A sample of requests the rules could not decide, with what was ambiguous. If the measurement was repeated later, the same table for the second period and a note on which rules had to change. Whether the numbers were compared with a script-based analytics tool and how far the two disagreed. Reports of a method that broke are as useful as reports of one that held.