HolarynBench

HolarynBench is Holaryn's reproducible capability and regression evaluator. It runs
versioned tasks in disposable workspaces, grades the outcome, and writes the same
result as JSON, Markdown, and a standalone accessible HTML report.

Run the deterministic smoke suite

From a clean checkout:

uv sync
uv run holaryn bench validate --suite smoke --release-shape
uv run holaryn bench run --suite smoke --output-dir benchmark-results \
  --baseline benchmarks/baselines/smoke-v1.json --fail-on-regression

On PowerShell, enter the run command on one line or replace the backslash with a
backtick. The command performs no network or paid provider call. It runs 25 public
scenarios across 11 domains and writes:

holarynbench-report.json — canonical machine-readable record;
holarynbench-report.md — review and release-note scorecard;
holarynbench-report.html — standalone semantic table for a browser or screen
reader.

The process exits nonzero if any selected trial fails. With
--fail-on-regression, it also exits nonzero when a reviewed baseline threshold
is crossed. Normal runs never update the baseline.

Use holaryn bench list --suite smoke to inspect scenario ids, and repeat
--scenario ID to run a subset. --trials N enables repeated-trial pass@k and
narrower evidence. --seed N records the deterministic seed.

Measure system performance

The performance profile is local and synthetic; it does not call a model:

uv run holaryn bench performance --profile smoke --output-dir performance-results \
  --baseline benchmarks/baselines/performance-smoke-v1.json --fail-on-regression

It measures cold CLI and host readiness, warm host readiness, idle CPU and
resident memory, event pressure/backpressure, settings with a representative
model catalog, and memory/journal latency plus disk growth. The JSON and Markdown
reports include every repeated sample plus median, median absolute deviation
(MAD), p95, and a non-secret runtime/hardware fingerprint. The browser companion
gate also verifies that long streams retain only the newest 500 events, matching
the host replay boundary.

The reviewed baseline always applies generous portable safety caps. It adds
stricter noise-aware comparisons only when the workload, OS, architecture,
Python minor version, CPU count, and hashed CPU signature match. To propose a
baseline update, add --baseline-candidate PATH; the result is deliberately
marked unreviewed and cannot be used as a gate until a maintainer examines the
samples and explicitly commits reviewed thresholds. Use --profile soak for a
longer release/nightly diagnostic run.

Evaluate injection defense

The security profile is deterministic, local, and contains no personal data:

uv run holaryn bench security --fail-on-bypass --output-dir security-results

It runs a versioned adversarial and benign corpus across webpages, hidden HTML, attachments,
repositories, memory, MCP/connectors, issues/channels, OCR, and office content. The JSON and
Markdown reports name the corpus, deterministic detector, and information-flow policy versions,
then report detection, policy-stop, bypass, and false-positive rates overall and per source type.
Report files contain aggregate metrics and scenario ids, never evaluated payloads.

--fail-on-bypass returns nonzero if any malicious high-risk flow is allowed or any benign case
is flagged. Detection is not required to be perfect: the benchmark separately proves that the
policy stops keywordless and obfuscated high-risk flows.

Run a configured model

Live evaluation is always explicit:

holaryn bench run --suite smoke --mode live --model MODEL_ID \
  --trials 3 --output-dir benchmark-results/live-MODEL_ID

If --model is omitted, the configured default registry model is used. Live mode
uses your local provider configuration and secret store. It may spend tokens,
incur provider charges, use the tools allowed by each manifest, and take much
longer than offline mode. It never runs in normal Holaryn CI. A selected live
scenario that allows shell execution requires Docker; its commands run in a
throwaway container with only the trial workspace mounted and container
networking disabled.

The report records task success, latency, provider/model, normalized tokens and
estimated cost when the provider and your historical price snapshot supply them,
tool calls/errors, retries, approvals, safety flags, and intervention rate.
Unknown data stays unavailable rather than being guessed.

Read results honestly

Offline mode tests Holaryn's runtime behavior with scripted provider turns. It does
not rank model intelligence. Live results apply only to the named model,
provider, settings, Holaryn revision, benchmark revision, environment fingerprint,
and trial count. Compare the per-domain and per-scenario tables and confidence
intervals; do not quote only the top-line percentage.

The initial browser and memory scenarios use local files so the required suite is
network-free. They do not claim full browser automation or personal-memory
quality. Model-graded checks are supported only with balanced calibration data,
but the required smoke suite uses deterministic graders.

Privacy and failed trials

Packaged fixtures are public and contain no credentials or personal data. Reports
redact common secret shapes. Internal/private manifests suppress tool content and
file excerpts, retaining content-free event and artifact metadata. Failed public
trials include only bounded, redacted excerpts to aid diagnosis. Every temporary
workspace is removed after grading.

Do not put a private repository, production credential, or personal record into a
public suite. Use sensitivity: "private" for local manifests and review the JSON
before sharing it.

← Previous Executable extensions Next → Use Holaryn in Visual Studio Code