You are reading Nightly documentation for 0.12.6.dev0+g563af09.

This documentation may describe behavior that differs from Stable.

Open Stable documentation

Documentation version

0.12.6.dev0+g563af09 · Nightly
Stable Beta Nightly

Evidence-gated self-improvement

Evidence-gated self-improvement

The Improve page is a review desk for candidate changes. It does not patch the
running installation or deploy releases. It keeps the evidence needed to decide
whether a candidate is safe enough for a separate release process.

For reusable skill procedures, use the separate Skill Workshop.
holaryn skill teach creates an explicitly requested scoped proposal from selected text files;
holaryn skill workshop approve requires private attended review and creates only an inactive
library draft. Repeated activity produces suggestions only. The code/configuration requirements
below do not treat Workshop draft approval as independent evaluation or rollout authority.
Legacy unscoped skill proposals require explicit Teach; they are not auto-migrated.

What a proposal must prove

A proposal starts with privacy-safe recurring failure or outcome references and a
complete hypothesis. Before review, it must attach an isolated Git worktree patch,
pass unit/static/build/security checks, compare baseline and candidate HolarynBench
reports with confidence intervals, pass a fingerprinted held-out set, and define a
bounded rollout with automatic rollback thresholds.

The dashboard presents those six evidence stages in order:

  1. hypothesis and rollback;
  2. isolated patch and protected files;
  3. gates and multidimensional benchmark metrics;
  4. independent revision-bound reviews;
  5. deterministic shadow/dogfood/canary evidence;
  6. the verified immutable provenance chain.

The proposal author cannot approve their own work. High-risk, protected, and
evaluation-control changes require two distinct elevated reviewers.

Review protected skill drift

SkillBench can create an inert proposal when two reviewed retained evaluations show aggregate
selection, outcome, cost, latency, dependency or provider/model drift. The Improve page labels
this as protected drift evidence and shows the exact skill-subject, baseline-report,
observed-report, drift-report and policy hashes. It also states that the association is not causal
and that private cases, prompts, trial rows, profile identities and model names are omitted.

These proposals start quarantined. An independent attended reviewer can Defer evidence
review while investigating confounders or Reject drift proposal. A deferred review needs a
different reviewer for rejection. There is no approve, cancel, canary or promotion action, and the
proposal cannot change, publish, trust or activate a skill. Start any justified change separately
through Skill Workshop and its governed lifecycle. Do not copy private evaluation material into the
review rationale.

Typical CLI workflow

Create a strict proposal definition in proposal.json, then:

holaryn self-improve create proposal.json
holaryn self-improve patch impr_123 patch.json --expected-revision 1
holaryn self-improve evaluate impr_123 evaluation.json --expected-revision 2
holaryn self-improve plan impr_123 canary.json --expected-revision 3
holaryn self-improve review impr_123 --expected-revision 4 --reviewer reviewer-a `
  --verdict approved --rationale "All gates and held-out metrics pass."
holaryn self-improve start-canary impr_123 --expected-revision 5
holaryn self-improve observe impr_123 observation.json --expected-revision 6
holaryn self-improve promote impr_123 --expected-revision 7
holaryn self-improve export impr_123

promote should report a promoted proposal and an authorization record. It will not
change source files or deploy anything.

If an observation falls below the plan threshold, the output instead includes
"state": "rolled_back" and a blocker such as "safety threshold breached".

Disable and delete

Use Disable collection on the Improve page or:

holaryn self-improve disable

Disabling prevents new proposal/repetition signals and clears process-local
repetition observations. Existing data remains available for review. Terminal
proposals require their exact id to delete. Delete retained data requires
collection to be off and the exact phrase:

DELETE ALL SELF-IMPROVEMENT DATA

That removes the new improvement ledger and legacy quarantined skill proposals.
Backups and previously exported files remain outside the live store and must be
handled separately.

For the security boundaries and recovery procedure, see
Safe self-improvement architecture.

← Previous Self-host with Docker Compose Next → Settings