Provider recovery, failover, and circuits

Provider recovery, failover, and circuits

Provider recovery can retry a temporary failure, move a turn to another eligible model, and stop
calling a model that is repeatedly unhealthy. It is opt-in. With recovery disabled, provider
failures preserve the existing fail-fast behavior.

Recovery is intentionally narrower than “try another model whenever anything goes wrong.” Holaryn
can recover automatically only before any response is visible and before a consequential tool
result. It never repeats a publish, payment, message, deletion, or unknown external action.

Prepare fallback models

Under Settings → Providers & Models:

1. Configure and verify at least two enabled models.
2. Edit their capabilities, context/output limits, local/cloud locality, availability, expected
   cost, and latency conservatively.
3. If you use adaptive routing, verify its privacy, allow/deny, provider pin, cost, and latency
   limits. Fallback candidates must pass those same hard gates.
4. Open Provider recovery and enter either:
   - a global ordered list such as backup-a, backup-b; or
   - per-model chains such as primary=backup-a|backup-b.
5. Leave Allow fallback away from an explicitly pinned model off unless you want a provider
   outage to override an exact model choice.
6. Enable recovery and save.

Explicit per-model chains win over the global list. If both are blank and Use eligible routing
candidates is enabled, Holaryn orders eligible registry models using the routing evaluator.

Understand what will happen

Default actions are:

• authentication, invalid request, safety refusal, cancellation, and unknown failure: stop;
• exhausted quota or context overflow: try an eligible fallback;
• rate limit, timeout, or temporary provider outage: retry within the configured bound, then try
  a fallback.

Retries use exponential delay and honor a numeric provider Retry-After when it is longer.
Adapter-local retries are disabled while this outer policy is coordinating the request, so two
retry layers cannot multiply the limit.

After visible output, Holaryn keeps the partial text and labels it incomplete. It does not retry or
switch models automatically. Send a new instruction such as:

The previous answer stopped mid-stream. Continue from the incomplete response without repeating
completed content.

After a consequential tool result, inspect what actually happened before continuing. Do not ask
Holaryn to “retry everything” until you know whether the external action committed.

Configure bounds

The Settings panel exposes:

• maximum pre-response retries: 0 to 10;
• base delay: 0 to 300000 milliseconds;
• failures before a connection/model circuit opens: 1 to 100;
• open-circuit cooldown: 1 to 86400 seconds;
• half-open probe lease: 1 to 3600 seconds.

A circuit is specific to a connection and model. After cooldown, only one session/process using
the same state directory can send a half-open probe. Success closes the circuit; failure opens it
again.

Expert action rules use these forms:

timeout=retry-then-fallback, auth=fail-fast
cloud/timeout=fallback
expert/rate-limit=fail-fast

The first line is category policy, the second is a provider connection override, and the third is
a model override. Model overrides win over connection overrides, which win over category rules.
No override can authorize replay after partial output or a consequential result.

Inspect health and reset state

The Recovery diagnostics tables show content-free circuits and recent attempts. They contain
model/connection ids, outcome, retry number, failure category/stage, elapsed time, and fallback
lineage—not prompts, answers, tools, credentials, or provider error bodies.

CLI equivalents:

holaryn routing health
holaryn routing health --limit 100
holaryn routing reset-circuits --yes

The reset deletes circuit history and recovery attempt traces. It does not change policy,
connections, models, routing telemetry, transcripts, or tool journals. Preserve the state before
reset when diagnosing an incident.

Test the recovery contract offline

Run the committed deterministic fault matrix:

holaryn bench resilience --output-dir resilience-results \
  --baseline benchmarks/baselines/provider-resilience-v1.json --fail-on-regression

It should report 100% recoverable-case recovery, 100% unsafe-boundary stops, zero duplicate side
effects, and a passing baseline. This does not contact a provider and does not predict live quality
or billing.

Environment configuration

Every field has an environment equivalent:

HOLARYN_FAILOVER_ENABLED=true
HOLARYN_FAILOVER_AUTOMATIC_CANDIDATES=false
HOLARYN_FAILOVER_ALLOW_PINNED=false
HOLARYN_FALLBACK_MODELS=backup-a,backup-b
HOLARYN_FALLBACK_CHAINS=primary=backup-a|backup-b
HOLARYN_FAILOVER_MAX_RETRIES=2
HOLARYN_FAILOVER_BASE_DELAY_MS=500
HOLARYN_CIRCUIT_FAILURE_THRESHOLD=3
HOLARYN_CIRCUIT_COOLDOWN_SECONDS=60
HOLARYN_CIRCUIT_PROBE_SECONDS=30
HOLARYN_FAILOVER_CATEGORY_RULES=timeout=retry-then-fallback,auth=fail-fast
HOLARYN_FAILOVER_PROVIDER_RULES=cloud/timeout=fallback
HOLARYN_FAILOVER_MODEL_RULES=expert/rate-limit=fail-fast

Configuration-file values can be managed in Settings; do not put credentials in these fields.

Recovery and rollback

To stop automatic recovery immediately for later messages, turn off Enable policy-controlled
provider recovery or set HOLARYN_FAILOVER_ENABLED=false. Open sessions read changes at the next
message.

If a circuit is open:

1. inspect its category and connection/model;
2. repair credentials or provider configuration for auth/invalid failures;
3. wait for cooldown when the provider is temporarily unhealthy;
4. verify fallbacks still meet privacy, capability, cost, and latency constraints;
5. reset only after the cause is understood.

If the resilience database has an unsupported or malformed schema, preserve
provider-resilience.sqlite3 for diagnosis and restore a compatible state backup. When recovery is
disabled, the database is not opened. See the
architecture and threat model for the complete contract.