Collapsing redirect chains to single hops raises the share of crawler requests that end in a 200 on a large site

Hypothesis

Google's crawl-budget documentation advises site owners to avoid long redirect chains, stating that they have a negative effect on crawling, but gives no measure of the effect. This hypothesis states one: on a large site whose crawler traffic is bounded per day, each hop in a chain is a request that returns a 3xx instead of a page, so the fraction of crawler requests that end in a 200 response is depressed in proportion to the average number of hops per redirected URL. Collapsing chains to single hops, with no other change to the site, raises that fraction, and the additional 200 responses go to pages that were previously reached less often or not at all. The mechanism is request accounting, not ranking, and the hypothesis says nothing about search visibility.

Prediction

After the chains are collapsed, in server logs filtered to verified crawler user agents, the share of requests with status 200 rises by roughly the share previously spent on intermediate hops, the count of distinct URLs crawled per day rises, and the median time between successive crawls of a given page falls. No such change is predicted for a control site of similar size where chains are left in place, nor for the human-traffic share of the same logs.

Proposed test

1. From access logs, identify crawler requests by user agent and reverse DNS, and compute per day: total requests, share ending in 200, share ending in 3xx, and hops per redirected URL by following the chains offline.
2. Collapse every chain in the redirect map so each old URL answers with one redirect to the final target; change nothing else for the test period.
3. Compare four weeks before and after, excluding the first week after the change; use a second site, or a section of the same site left unchanged, as control.
4. Report the shares with their day-to-day variation, and check whether the extra 200 responses fell on previously rarely crawled URLs.

Status

No result is claimed. Crawl volume is set by the crawler and varies with site health and content changes, which could mask or mimic the effect; the hypothesis would be weakened if the 200 share rose equally on the control, or if total crawler requests simply fell by the number of removed hops.