Replayable run logs for agents: recording every model and tool call

What it is

A replayable log is a record of an agent run from which each step can be reconstructed: the exact request sent to the model (system prompt, messages, tool definitions, sampling parameters), the response (content, tool calls, stop reason, usage), and for each tool call its arguments, result and error status, in order and with timestamps. OpenTelemetry's GenAI semantic conventions define attribute names for this; they are still marked as in development and now live in a separate semantic-conventions-genai repository, while the attribute registry on opentelemetry.io lists them with a note that they have moved. Among them are gen_ai.request.model, gen_ai.request.temperature, gen_ai.request.seed, gen_ai.response.id, gen_ai.response.finish_reasons, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.tool.call.id, gen_ai.tool.call.arguments, gen_ai.tool.call.result and gen_ai.conversation.id for grouping. Evaluation frameworks keep the same information per sample; Inspect, for example, writes a log file for each task evaluated and exposes it as an EvalLog object.

Why it matters

Model outputs are not deterministic, tool results change with the world, and a run may take hundreds of steps. Without the log, "the agent deleted the wrong file" cannot be traced to the step where the wrong path entered the context. With it, the request that produced a bad decision can be re-sent verbatim to test a prompt fix, and tool results can be stubbed from the log so that the rest of the run replays without side effects.

How to apply

• Log at the boundary where the request leaves the process, not from the prompt template: the bytes actually sent are what matter.
• Give every run an ID and every step a sequence number; link tool results to the tool call ID rather than relying on order.
• Store full content, but redact secrets and personal data before writing (the same rules as for application logs) and set a retention period.
• Record the response ID and, where the provider offers one, a version or fingerprint of the served model.
• Build a replay mode: given a log, the harness serves recorded tool results for identical calls and flags calls that diverge from the recording.
• Use the same IDs in traces and metrics so that cost and latency dashboards link back to a transcript.

Pitfalls

Logging only the final answer. Truncating tool results in the log, which hides the injected instruction or malformed payload that caused the failure. Treating replay against a live model as deterministic: replay reproduces the inputs, not the decision. Logs that repeat the whole document set on every step grow fast; deduplicate by content hash.