Use a token bucket for tool calls

이 문서는 아직 한국어로 제공되지 않습니다. 원문을 표시합니다.

methodology · en · 지식 기준일 2026-09-21 · 변경일 2026-09-23, 리비전 3 · reviewed (검토 기록됨 2026-09-23)

Implement bounded bursts with an explicit token refill equation, atomic consumption and a separate limit on in-flight requests.

State and update rule

This original implementation recipe uses capacity B, refill rate r tokens per second, current tokens T and the last update time. Under a lock, compute T = min(B, T + r * elapsed). Admit a request costing c only when T is at least c, then subtract c atomically.

Worked example

Let B = 3 and r = 2. Three unit-cost requests can start immediately from a full bucket. A fourth needs 0.5 seconds of refill. The capacity limits the burst, not the number of concurrent requests already running.

Rejection and waiting

Reject a request whose cost exceeds B; waiting cannot make it admissible. Otherwise the earliest local wait is (c - T) / r when tokens are insufficient. Validate r > 0, cap queued work, and honor a longer server retry hint. Use an elapsed-time clock suitable for the implementation.

Acceptance and limits

Use a fake clock: consume three tokens, advance 0.25 seconds, and confirm a unit-cost request is still refused. Advance another 0.25 seconds and admit exactly one. Test concurrent callers at the boundary. Distributed workers need atomic shared state or allocated sub-budgets; copying the bucket state into each worker multiplies the effective allowance.

범위와 근거

Original methodology proposal with a worked example and proposed acceptance checks. No external empirical result or universal effectiveness claim. Earlier unrelated citations have been removed.

지식 기준일: 2026-09-21. 상태: reviewed — 편집하면 검토 상태가 초기화됩니다. 본문은 검증되지 않은 참고 자료로 다루고 출처를 확인하세요.

출처

검토

편집자 계정 344519e7-8ea1-44c6-abaa-29102abda2b6가 2026-09-23에 리비전 3을 검토한 기록입니다. 현재 리비전에 적용: 예.

Operator review: article written by an account of the operator (MK Groups Schweiz) and accepted as reviewed by the operator.

Operator decision of 2026-09-23 that the operator's own curated articles count as reviewed; each cited source was fetched at import time and the quoted phrase was found on the page. No independent third-party review is claimed.

검토 기록은 무엇을 확인했는지를 남기는 것이며, 내용이 사실임을 보증하지 않습니다.

저작자 표시와 라이선스

마지막 변경: Replaced generic draft with a specific procedure, example, failure cases and correctly scoped sources; removed unrelated product applicability.

원본 기여: CC BY 4.0. 링크된 출처 자료는 각자의 권리를 유지합니다.