Google API pagination and quota errors: pageToken, 403 rateLimitExceeded and 429

What it is

Google's Calendar API pagination guide shows the common Discovery-API pattern: control page size with maxResults, and detect more pages by checking the response for nextPageToken — "incomplete results can be detected by a non-empty nextPageToken field... maxResults does not guarantee the number of results on one page." The next call repeats the same request and adds pageToken set to the previous nextPageToken; a response with no nextPageToken is the last page. Errors that mean "slow down" arrive in two documented shapes. The Drive API's error guide describes rateLimitExceeded ("this error occurs when the project's rate limit has been reached") and userRateLimitExceeded ("this error occurs when the per-user limit has been reached"), both returned as HTTP 403 with the reason field in error.errors[] distinguishing them; the same guide also lists rateLimitExceeded under 429 errors. A third 403, sharingRateLimitExceeded, is a sharing/email limit that quick retries do not fix. Drive's usage-limits page summarises: "if you exceed a quota, you'll receive a 403: User rate limit exceeded... Additional rate limit checks on the Drive backend might also generate a 429: Rate limit exceeded response. If this happens, you should use an exponential backoff algorithm and try again later."

Why it matters

A client that only special-cases 429 will miss Google's 403-coded quota errors entirely, since the HTTP status alone does not distinguish "forbidden" from "rate limited" — the JSON body's reason field does. Because both shapes call for the same remedy, a robust client needs to inspect the body, not just the status code, to decide whether to back off versus fail immediately.

How to apply

• Page with pageToken/nextPageToken (or the API-specific maxResults/pageSize name) and stop only when nextPageToken is absent.
• On any 403, parse the JSON body's error.errors[].reason; treat exactly rateLimitExceeded and userRateLimitExceeded as retryable, and other 403 reasons (a permissions problem, sharingRateLimitExceeded) as not. Match reasons exactly: a case-sensitive substring test such as "RateLimitExceeded" in text misses rateLimitExceeded yet matches sharingRateLimitExceeded.
• On 429, retry with exponential backoff, per Google's own guidance for Drive.
• Consider batching related calls and requesting a quota increase through the Cloud console before scaling request volume, both explicitly listed as fixes in Google's error docs.

import random, time

RETRYABLE_403 = {"rateLimitExceeded", "userRateLimitExceeded"}

def reasons(resp):
    try:
        return {e.get("reason") for e in resp.json()["error"].get("errors", [])}
    except (ValueError, KeyError, TypeError, AttributeError):
        return set()

def call_with_backoff(fn, max_tries=5):
    for attempt in range(max_tries):
        resp = fn()
        retry = resp.status_code == 429 or (
            resp.status_code == 403 and reasons(resp) & RETRYABLE_403)
        if not retry or attempt == max_tries - 1:
            return resp  # caller decides whether to raise
        time.sleep(min(2 ** attempt + random.random(), 64))

Pitfalls

Retrying every 403 indiscriminately, which will silently loop on a genuine permission error instead of surfacing it. Assuming maxResults was honoured exactly and using the returned item count, instead of nextPageToken, to decide whether more pages remain.