Paging the GitHub REST API and reading its primary and secondary rate-limit headers

What it is

GitHub's pagination guide shows a link response header such as <...?page=2>; rel="prev", <...?page=4>; rel="next", <...?page=515>; rel="last" and instructs clients to use those URLs directly rather than build page URLs themselves: "The URL for the next page is followed by rel="next"." Depending on the endpoint, those URLs use page, before/after or since. GitHub's docs note that "for most endpoints, the maximum value of per_page is 100," and a higher value is silently reduced rather than rejected. Primary rate limits are reported on every response via x-ratelimit-limit, x-ratelimit-remaining, x-ratelimit-used, x-ratelimit-reset (UTC epoch seconds) and x-ratelimit-resource (which bucket the request counted against). Separately, GitHub enforces secondary rate limits — documented ceilings include no more than 100 concurrent requests, 900 points per minute for REST API endpoints, 90 seconds of CPU time per 60 seconds of real time, and no more than 80 content-generating requests per minute (500 per hour). Exceeding either kind returns 403 or 429; a secondary-limit response may carry retry-after, and when it does not, GitHub says to wait until x-ratelimit-reset if x-ratelimit-remaining is 0, "otherwise, wait for at least one minute before retrying", backing off exponentially on repeats. GitHub's best-practices page adds that a conditional request "does not count against your primary rate limit if a 304 response is returned and the request was made while correctly authorized with an Authorization header."

Why it matters

Constructing page URLs by hand breaks whenever GitHub changes query parameters (it already differs between endpoints using page numbers and endpoints using cursors); ignoring the secondary-limit retry-after header on top of a healthy primary-limit balance is a common way to get temporarily blocked even while primary quota remains.

How to apply

• Follow the link header's rel="next" URL verbatim; stop when no next relation is present.
• Request per_page=100 for read-heavy jobs to minimise round trips, and never assume a larger requested value was honoured.
• Track x-ratelimit-remaining and x-ratelimit-reset; slow down well before remaining reaches zero rather than reacting only to a 403/429.
• On any response carrying retry-after, wait at least that many seconds regardless of primary-limit headroom; on a secondary-limit error without it, wait at least a minute.
• Send write requests serially and at least one second apart, as GitHub recommends against secondary limits.
• For polling workloads, send If-None-Match with a stored ETag; an authorized 304 is free against the primary limit.

import requests

def get_all_pages(url, headers):
    items = []
    while url:
        r = requests.get(url, headers=headers)
        r.raise_for_status()
        items += r.json()
        url = r.links.get("next", {}).get("url")
    return items

Pitfalls

Paginating with page= numbers hand-built from per_page, which breaks on endpoints that only expose cursor-style pagination. Treating a retry-after header as optional because the primary x-ratelimit-remaining still shows headroom. Assuming every rate-limit error is a 429: GitHub also uses 403, so check the headers and error message before treating a 403 as a permissions problem.