Budgeting cost and latency for model calls in an agent

Goal

Know before deployment what a run of the agent costs and how long a user waits, keep both inside limits that are enforced in code, and see when a change moves either.

Prerequisites

The per-call usage fields the provider returns (input tokens, output tokens, cache reads and writes); the OpenTelemetry GenAI semantic conventions (in development, maintained in a separate repository; the registry page on opentelemetry.io lists them as moved) name them gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.usage.cache_read.input_tokens. A price list for the models in use, and replayable run logs to attribute cost to steps.

Steps

1. Measure a baseline on the evaluation set: per run, the number of model calls, total input and output tokens, wall-clock time and the longest single call.
2. Set budgets per run: a token ceiling, a step ceiling and a deadline. Enforce them in the loop; when a budget is reached, the agent gets one final turn to report what is done and what is not.
3. Cut input tokens first. Put the system prompt, tool definitions and reference documents at the front of the request and mark them for caching. The vendor documentation describes cache_control breakpoints with a 5-minute or 1-hour lifetime, reports hits in cache_read_input_tokens, and states that changing any block at or before the breakpoint produces a different hash; keep timestamps and per-request values after the cached prefix.
4. Cut output tokens: ask for the shortest output the next step can consume (a tool call, an identifier, structured data), not a narrative.
5. Trim the context: drop or summarise old tool results instead of resending them on every step.
6. Route offline work (evaluation runs, nightly extraction) to a batch endpoint. The cited batch documentation states that batch usage is charged at 50% of the standard API prices and that batches expire if processing does not complete within 24 hours.
7. Use a smaller model for steps the evaluation harness shows it passes; keep the larger model where the harness shows it is needed.
8. Alert on cost per completed task and on p95 latency, not on total spend alone; a cheaper model that needs more retries is not cheaper.

Expected result

A documented cost and latency per task type, budgets enforced in code, and a dashboard in which a prompt change that doubles token use is visible the same day.

Limits and test basis

Prices, cache lifetimes and batch terms are the provider's and change; the figures above are quoted from the cited documentation at the time of writing. Batch processing is unsuitable for interactive use. Budgets stop runaway runs but do not make a run cheaper; only steps 3 to 7 do.