Time and cost budgets for model calls in agents

Goal

Before rollout, know what one agent run costs and how long users wait, limit both in code, and notice when a change shifts either of them.

Prerequisites

The usage fields the provider returns for each call. The cited vendor documentation on prompt caching describes input_tokens, cache_creation_input_tokens and cache_read_input_tokens in the usage object; the sum of the three is the actual input size. You also need a price list for the models in use, an evaluation set of typical tasks, and run logs that attribute costs to individual steps.

Steps

1. Measure a baseline: for each run on the evaluation set, record the number of model calls, input and output tokens, total duration, and the longest single call.
2. Set budgets per run (a token limit, a step limit and a deadline) and enforce them in the loop. When a budget is reached, the agent gets one final turn to report what is done and what is not; an aborted run without a report is lost work.
3. Reduce input tokens first: put the system prompt, tool definitions and reference documents at the beginning and mark them for caching. The documentation describes a default lifetime of five minutes that is renewed at no extra cost on each use, a one-hour lifetime at a surcharge, and the order tools → system → messages, where a change at one level invalidates all following levels. Timestamps and per-request values therefore belong after the cached part; cache_read_input_tokens shows whether caching takes effect.
4. Reduce output tokens: ask for the shortest output the next step can process (a tool call, an identifier, structured data), not prose.
5. Prune the context: summarize or drop old tool results instead of resending them at every step.
6. Move offline work (evaluation runs, nightly extraction) to the batch endpoint. The cited batch documentation states 50 percent of the standard price, completion usually within an hour, expiry after 24 hours without completion, and results that stay retrievable for 29 days.
7. Use a smaller model only where the evaluation set shows that it passes the task; compare cost per completed task, not per call.
8. Alert on cost per completed task and on latency at the 95th percentile, not only on total spend.

Expected result

Documented cost and latency per task type, budgets enforced in code, and a dashboard on which a prompt change that doubles token consumption is noticed the same day.

Limits and test basis

Prices, cache lifetimes and batch conditions are vendor claims and change over time; the figures come from the cited documentation at the time of writing. Batch processing is unsuitable for interactive use. Budgets stop runaway runs but do not make them cheaper; only steps 3 to 7 do that.

Pruning context without losing the cache

Prompt caching compares prefixes: any change to an earlier message invalidates the cache from that point on. If you trim the history in every turn, you pay anew for the rest of the history in every turn, plus the write surcharge for the new entry. Therefore: keep tool results small from the start (the head instead of the whole file, a count instead of a list, a reference instead of the content); prune the history only at a few defined thresholds and then in a single step, so that the new prefix stays stable for several turns; and verify success with cache_read_input_tokens, which should grow after every turn. Server-side variants such as context editing and compaction work with thresholds for the same reason.