Automatic code validation

Automatic code validation

Holaryn can discover and run the checks that fit a code change, then attach proof
to the exact revision and working diff it checked. Open a Code session and
select Validation in the right panel.

Validate from the Code panel

1. Choose Fast, Standard, or Release.
2. Select Review validation plan.
3. Read every exact command, working directory, source, trust label, and
   required/advisory label.
4. If the plan contains package scripts, CI commands, repository instructions,
   or .holaryn overrides, enable Trust repository-provided commands only
   after reviewing them. The plan is regenerated so your approval is bound to
   the exact plan ID.
5. Select Run reviewed plan.

The completion matrix distinguishes passed, failed, skipped, stale, timed out,
not configured, and inconclusive checks. Expand a row to inspect retained
attempts, redacted output, and diagnostics. Keyboard focus follows normal form
order; status changes use an aria-live status region.

verified means all required gates passed for that recorded workspace state.
It does not mean the program is completely correct. An edit, branch change, or
checkpoint restore makes old evidence stale; run the plan again.

Command line

Preview without running:

holaryn validate --workspace /path/to/repo --profile standard plan

Run fixed trusted checks while leaving repository commands skipped:

holaryn validate --workspace /path/to/repo --file src/widget.py run

Bound the plan when needed:

holaryn validate --workspace /path/to/repo --max-duration-seconds 300 --max-estimated-cost-usd 1.00 --max-checks 8 run

If the check-count cap excludes a required gate, Holaryn adds a required skipped
row naming the omitted gates. Increase --max-checks; the capped run cannot be
reported as verified.

Supply an explicit trusted argv without invoking a shell:

holaryn validate --workspace /path/to/repo --command-json '["uv","run","pytest","tests/test_api.py"]' plan

Do not place passwords, tokens, API keys, or other credentials in command
arguments. Holaryn rejects secret-like argv so it cannot appear in plans, events,
or retained evidence. Validation also withholds secret-like environment
variables; use an offline check or a separately governed CI job when a gate
requires credentials.

Generate the exact trust-enabled plan first:

holaryn validate --workspace /path/to/repo --profile release --trust-project-commands plan

After reviewing every command, run that exact plan ID:

holaryn validate --workspace /path/to/repo --profile release --trust-project-commands --approved-plan-id vplan_0123456789abcdef --execution-backend docker --docker-image your-validation-image run

Repository-provided commands—and fixed commands that execute repository tests,
plugins, build scripts, or compilation hooks—require both exact approval and
Docker isolation. Parsing-only Ruff and Cargo formatting checks may run without
that approval. Docker receives only the selected workspace, a minimal
environment, and no network for this validation route. With the local backend,
executable repository checks stay visibly skipped even when trust was
requested.

Show current evidence:

holaryn validate --workspace /path/to/repo latest
holaryn validate --workspace /path/to/repo show vev_example

The optional --state-dir must be outside the repository so the evidence
database cannot alter the working-diff fingerprint it is recording.

Run the offline correctness comparison:

holaryn bench validation --output-dir validation-results
holaryn bench validation --baseline benchmarks/baselines/validation-evidence-v1.json --fail-on-regression

On Windows Command Prompt, use a Windows path:

holaryn validate --workspace D:\work\my-repo --profile standard plan

Optional project checks

Create .holaryn/validation.yaml:

version: 1
checks:
  - id: focused-api-tests
    name: Focused API tests
    command: [uv, run, pytest, tests/test_api.py]
    tier: relevant
    required: true
    timeout_seconds: 180
    estimated_cost_usd: 0
    artifacts: [reports/api-test.xml]

The file is treated as untrusted executable configuration. It is never run
merely because it exists.

If validation cannot verify the change

• Skipped: approval or a time budget prevented the check.
• Stale: the workspace changed after the recorded run.
• Timed out: the command exceeded its timeout.
• Not configured: no supported checks were discovered.
• Inconclusive: the executable or working directory was unavailable.

Holaryn reports these states honestly and does not convert them to “passed.”