Local models

Local models

Holaryn can manage the path from “will this model fit?” to a routed local provider
without turning Ollama or llama.cpp into special cases in the agent loop. Use
Settings → Providers & Models → Local inference desk or the
holaryn local-model command.

Start with an existing runtime

The shortest path on a fresh machine is an existing Ollama model:

holaryn local-model runtimes
holaryn local-model hardware
holaryn local-model add-ollama qwen2.5:7b --id local-qwen
holaryn local-model adopt ollama http://127.0.0.1:11434/v1 --id ollama-local
holaryn local-model register ollama-local local-qwen
holaryn local-model role general local-qwen
holaryn local-model benchmark ollama-local

runtimes only checks this machine. If a runtime is missing, Holaryn reports an
official installation page instead of executing an installer. adopt performs an
explicit health check, but never takes ownership of the process. An adopted
server's Stop control is disabled because its owning runtime must stop it.

The same flow works in Settings: record an Ollama model, adopt the endpoint, scan
it, register the selected model, and assign a role. Once registered, it appears in
the same model pickers and routing paths as every other provider model.

Import a llama.cpp model

For a GGUF file already on disk, supply the license and any provenance you know:

holaryn local-model import D:\models\qwen.gguf --id local-qwen --license Apache-2.0 --quantization Q4_K_M
holaryn local-model start local-qwen --port 8081 --autostart
holaryn local-model health llamacpp-local-qwen
holaryn local-model register llamacpp-local-qwen local-qwen

The import is hashed and copied into content-addressed Holaryn storage. Re-importing
the same bytes reuses the blob. --sha256 can enforce a digest supplied by the
publisher. Holaryn-owned servers bind 127.0.0.1; choose a different port when the
requested one is occupied.

For a large existing file that you do not want Holaryn to copy, run llama.cpp
yourself and use adopt. Adopted endpoints and external model stores are
read-only.

Verified downloads and resume

A download is available only when you explicitly provide immutable artifact
metadata:

holaryn local-model download https://models.example/qwen.gguf --sha256 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef --id local-qwen --license Apache-2.0

Holaryn rejects non-HTTPS URLs, reserves disk space, stores a partial file, resumes
with an HTTP range request, and publishes the model only after the SHA-256 matches.
The model manifest records URL, checksum, publisher, license, and trust source.
The built-in catalog works offline and deliberately does not invent artifact URLs
or checksums for mutable upstream releases.

Hardware fit and roles

holaryn local-model catalog --fit
holaryn local-model role coding local-qwen
holaryn local-model status --json

Fit labels are advisory. Each result shows the parameter and quantization estimate,
requested context, concurrency, memory headroom, disk reserve, probe limitations,
and a confidence label. Apple unified memory is treated as shared memory. Missing
or ambiguous GPU telemetry produces an unknown/low-confidence result rather
than a guess. Expert override remains available because runtime offload behavior
can differ from the estimate.

Roles are general, coding, vision, embedding, and background. They become
ordinary registry tags; general sets the default chat model and embedding
selects embedding modality.

Supervision and diagnostics

Managed processes write logs beneath
<state-dir>/local-models/logs. Health checks record actionable errors. Crashes
can restart up to the recorded limit; port conflicts are reported and Holaryn never
kills the process occupying the port. Autostart restores Holaryn-owned servers with
the host and host shutdown stops only live child handles it owns.

holaryn local-model diagnostics
holaryn local-model health llamacpp-local-qwen
holaryn local-model restart llamacpp-local-qwen
holaryn local-model stop llamacpp-local-qwen

After a Holaryn process restart, an old PID is marked orphaned instead of being
terminated—the operating system may already have reused that PID. Relaunch it or
adopt the endpoint deliberately.

Safe removal

holaryn local-model remove local-qwen --yes

Removal refuses active models and external storage. For an owned/imported model it
deletes only unshared bytes under Holaryn's blob root; another manifest referencing
the same SHA-256 keeps the shared blob. Settings presents the same scope in a
confirmation dialog.

Offline and privacy behavior

Status, diagnostics, hardware probing, catalog search, fit estimation, roles, and
stored benchmark results are offline. Endpoint scans, health checks, benchmarks,
and downloads happen only when you invoke that action. Holaryn never uploads a
hardware inventory, model file, prompt, or response from this subsystem.

The fixed benchmark sends Reply with exactly: local smoke ok to the selected
local endpoint. Durable results contain only time-to-response, approximate output
throughput, context and declared capability flags, stability, and an error if
present. Prompt and response text are not persisted.

Maintainers can opt into the real-machine acceptance smoke when a loopback Ollama
with at least one model is already running:

$env:HOLARYN_REAL_LOCAL_MODEL_SMOKE = "1"
uv run pytest tests/test_local_models.py -k real_ollama

The test is skipped by default and never installs, downloads, or selects a remote
runtime. It adopts the local endpoint read-only, registers the first reported
model, and runs the same fixed chat benchmark.

For LAN-hosted Ollama guidance and exposure warnings, see
Configuring Ollama. For provider routing details, see
Providers & models.