Run usage diagnostics

Holaryn records a small, provider-neutral summary for every core agent-loop invocation. The summary is designed for troubleshooting, capacity planning, and cost estimation without turning diagnostics into a second conversation archive.

The summary is exposed through the canonical event stream and durable run journal. Models with
prompt caching enabled also show bounded, metadata-only cache diagnostics in Settings →
Providers & Models.

What is recorded

Every invocation gets a stable run_id and trace_id. Each call into a model adapter gets a stable attempt_id. Core loop events carry the run and trace IDs; model streaming and retry events also carry the active attempt ID.

The terminal run_usage_summary event contains:

schema version (1 in this release), run/trace/session IDs, start/finish times, and monotonic duration;
terminal reason: completed, failed, cancelled, or max_turns;
one record per model call: provider, actual model identifier, optional connection ID, duration, time to first output, retry count, terminal reason, and exception type on failure;
provider-reported input, output, reasoning, cache-read, and cache-write token counts when available; and
an optional estimated USD cost calculated from an operator-supplied price snapshot.

Cache-enabled attempts additionally record a content-free plan/result and the terminal run summary
aggregates hits, misses, bypasses, unknowns, read/write tokens, savings, and invalidation reasons.
See Prompt caching.

Unknown data is null; Holaryn does not manufacture token counts or prices. Time to first output is also null when the adapter produces no observable text or tool-call output before completion.

Token categories are non-overlapping. For providers whose totals include cached input or reasoning output, Holaryn subtracts those specialized counts from ordinary input/output before storing the normalized result. This keeps aggregates from double-counting tokens.

Response metrics in conversations

Every model-backed response in the shared Chat and Code conversation surface shows a
compact footer after the run finishes:

Run time is the full wall-clock duration, including model calls, tools, approvals, and other
work performed by that run. It is formatted as milliseconds, seconds, minutes, or hours as
appropriate.
Provider-reported tokens is the sum of the non-overlapping token categories the provider
actually returned. Expand Run details for the model-call count and the available input,
output, reasoning, cache-read, and cache-write categories.

The footer is restored with saved chat history because it is correlated to the response by
run_id. If the provider does not report usage, the footer says Unavailable; Holaryn never
estimates tokens from the prompt or response length. The terminal interface presents the same
duration and token total after its category breakdown.

Privacy boundary

Usage summaries do not contain prompts, responses, reasoning text, tool names or arguments, file paths, URLs, exception messages, provider wire payloads, or generated content. Failures retain only the exception class, such as ProviderTransportError.

The ordinary transcript and canonical tool events still contain the conversation and tool data required to operate the agent. The usage summary is an additional content-free diagnostic record, not a privacy filter over those existing records.

Retry and cancellation semantics

A transient provider retry emits model_retry with the failed/next ordinal, delay, reason, and exception type. OAuth refresh retries use reason auth_refresh. Error messages and endpoint details are excluded.

Cancellation always produces a terminal usage summary for that invocation. A run paused on a durable approval remains resumable in the run journal even if the current in-memory invocation is cancelled; the telemetry record truthfully says that invocation ended while the durable workflow stays available for resume.

Cost estimates and historical prices

Holaryn does not ship or silently fetch a vendor price table. Prices change, providers have tiers, and locally hosted models may not have a per-token price. A cost is calculated only when the selected model has an explicit versioned pricing snapshot in providers.json, for example:

{
  "id": "primary",
  "connection_id": "openai",
  "model": "example-model",
  "capabilities": {
    "context_window": 128000,
    "native_tool_calling": true,
    "parallel_tool_calls": true,
    "vision": true,
    "prompt_cache": true,
    "system_prompt": "message"
  },
  "pricing": {
    "source": "operator-config",
    "version": "rate-card-2026-07-01",
    "captured_at": 1782864000.0,
    "input_usd_per_million": 2.0,
    "output_usd_per_million": 8.0,
    "cache_read_usd_per_million": 0.2
  }
}

The snapshot provenance is copied into the estimate, so a historical run remains explainable after rates change. The result is always marked estimated. If a nonzero token category has no configured rate, the estimate is marked incomplete and names the missing category. With no snapshot, cost is null.

Persistence and access

The host stores summaries in its existing run-journal SQLite database, keyed by run_id and indexed by session and finish time. The store exposes usage_summary(run_id) and usage_summaries_for(session_id) for operator integrations. Re-recording the same run ID is idempotent. Deleting a saved run/chat deletes its usage summaries with the rest of that run's journal data.

Canonical event consumers should treat run_usage_summary as a diagnostic trailer. task_complete or error remains the user-facing outcome; the Chat and Code interfaces attach the trailer's content-free measurements to that outcome instead of creating another message or activity item. Subagent status likewise keeps its meaningful outcome while retaining the summary in event history.

The payload is versioned so exporters can reject or migrate an unfamiliar schema instead of guessing. Generated TypeScript and JSON Schema definitions include the correlation fields and both new event kinds.

Provider coverage

The normalizers cover Anthropic Messages,
Google Gemini, and OpenAI-compatible Chat Completions
and OpenAI Responses. Providers may
omit categories, especially cache or reasoning counts; missing fields remain explicit rather than
being inferred from text length or latency.

← Previous Troubleshooting Next → Voice

Run usage diagnostics