Prompt caching

Prompt caching

Prompt caching can reduce latency and input cost when several model calls share a large, stable
prefix such as tool definitions, repository instructions, or a long conversation. It does not
cache the model's final answer in Holaryn.

Enable it

Open Settings → Providers & Models, edit a model, and confirm that Prompt cache is listed as
a capability. Under Prompt cache policy choose:

• Off — the compatible default. Holaryn uses the ordinary provider adapter.
• Automatic — Holaryn chooses provider-safe stable-prefix boundaries.
• Expert breakpoints — you provide comma-separated diagnostic segment ids; Holaryn still rejects
  unsafe, non-prefix, oversized, or unsupported boundaries.

For Automatic or Expert, check the provider-retention consent. Optional fields select a supported
TTL, expected number of reuses, and a stricter minimum prefix. If you leave TTL/minimum blank, the
selected provider/model capability supplies them. Unsupported values produce an explicit
diagnostic instead of silently changing the request.

Provider retention is not local response storage. The provider may retain an ephemeral internal
representation of the marked prefix under your account's data policy. Review the current
OpenAI,
Anthropic, or
Gemini documentation before consenting.

What is safe to reuse

Holaryn canonicalizes visible tools, system instructions, and prior conversation turns. The current
user turn is dynamic. Tainted/restricted data stops the cacheable prefix. Reuse keys include the
principal, tenant, workspace, model/provider, policy, profile, privacy revision, and session
partition, so two users or tenants cannot share a Holaryn-directed provider cache entry.

Changing instructions, tool schemas, profile/persona, policy, model/provider, workspace,
authorization/privacy revision, cache schema, or TTL invalidates reuse. Content hashes never grant
access.

Read diagnostics

The model card shows mode and the most recent result. Expand Cache diagnostics (metadata only)
to see the last invalidation reason and stable-prefix token estimate. Common results:

• hit — the provider reported cached input tokens;
• miss — the provider reported a new write or a zero-token read;
• unknown — caching was requested but the provider omitted decisive usage categories;
• bypassed — retention, size, reuse, TTL, sensitivity, or capability rules blocked it; and
• failed / cancelled — the model request did not complete, so Holaryn claims no savings.

Run summaries include cache hits/misses/bypasses, read/write token counts, invalidation reasons, and
estimated savings. Savings appear only when the model has a versioned price snapshot; they are
estimates, not billing records.

Diagnostics and prompt-cache.sqlite3 contain hashes, segment ids, sizes, token aggregates,
latency, and estimates—not prompts, responses, tool arguments, paths, URLs, or credentials.
Deleting a model deletes its cache diagnostics.

Troubleshooting

• Always bypassed / minimum size: the stable prefix is below the provider or configured
  minimum. Do not pad a prompt merely to create a cache.
• Retention forbidden: re-edit the model and consent only after reviewing the provider policy,
  or leave caching off.
• Unsupported TTL: use a TTL listed by the provider capability (Anthropic commonly supports 300
  or 3600 seconds; GPT-5.6 explicit mode uses 1800 seconds).
• Repeated instruction/tool invalidation: keep stable instructions and schemas byte-stable;
  avoid nondeterministic tool discovery order (Holaryn sorts cache-enabled tools).
• Unknown metrics: the provider did not report enough cache usage data. Holaryn does not infer a
  hit from latency.
• Corrupt local ledger: Holaryn preserves it as prompt-cache.sqlite3.corrupt-<timestamp>,
  recreates a clean ledger, and runs cold. If the state directory is read-only, caching still runs
  with in-memory metadata and the model call is not blocked.

The full security, lifecycle, provider mapping, and recovery contract is in the
prompt-caching ADR.