You are reading Nightly documentation for 0.12.4.dev0+g50bde75.

This documentation may describe behavior that differs from Stable.

Open Stable documentation

Documentation version

0.12.4.dev0+g50bde75 · Nightly
Stable Beta Nightly

Voice

Voice

Local, private voice I/O: hold a captioned conversation from the chat page and, optionally,
have replies spoken. The supported speech-recognition path runs on your machine.

Enabling voice

Go to Settings → Speech → Install voice (one click). The same install works from the terminal or pip:

holaryn memory install-extra voice
# or
pip install "holaryn-agent[voice]"

The extra brings two engines:

  • faster-whisper — local speech-to-text on the CPU. The Whisper model (HOLARYN_STT_MODEL, default base; tiny, small, and medium also work) downloads once on first use, then dictation is fully offline.
  • pyttsx3 — the operating system's speech voice for spoken output: SAPI on Windows, NSSpeech on macOS, eSpeak on Linux.

Start a real-time voice session

The Voice and dictation disclosure sits above the chat message box and starts closed. Its
summary states whether the microphone is off, listening, speaking, waiting for approval,
disconnected, or in error. Expanding or collapsing it never starts a session or requests
microphone permission.

  1. Expand Voice and dictation.
  2. Pick a mode. Push to talk is recommended and is the default.
  3. Choose whether to keep captions, use text-only output, or temporarily retain raw audio.
  4. For automatic, wake phrase, or continuous mode, acknowledge that the microphone may remain
    active.
  5. Choose Arm voice. This does not start the microphone.
  6. Choose Start turn or Start listening when ready.

The Listen → Think → Speak rail always states the active phase. Live captions appear below it,
and End voice remains available as a one-action stop. While the disclosure is collapsed,
Finish turn or Stop decision remains visible during capture, Interrupt remains visible
during playback, and End voice remains visible for every active session. Approval, device-loss,
and error notices also remain outside the hidden detail.

Modes

  • Push to talk: you start and finish each turn.
  • Automatic turn: the turn finishes after speech followed by the configured silence; capture
    then stops.
  • Wake phrase: capture can continue, but only a final transcript containing Hey Holaryn
    (configurable through the API) is sent to the agent.
  • Continuous conversation: capture restarts after each detected turn until you pause or end.

Wake phrase is convenience, not speaker authentication. Do not use it as proof of identity.

Captions, devices, and interruptions

The microphone picker refreshes when devices connect or disconnect. If the selected device is
lost, the card says Reconnecting and the typed fallback remains available. Active sessions
recover Paused after a host restart; Holaryn never silently reopens the microphone.

While Holaryn is speaking, choose Interrupt or begin speaking in an active automatic mode to
cancel browser speech. Echo cancellation, noise suppression, and automatic gain control are
requested from the browser, but results vary by device.

Privacy and data controls

  • Raw audio is discarded by default.
  • If you opt in to Keep raw audio, the web UI expires it after five minutes. It is never put
    in a session export.
  • Caption retention can be disabled independently.
  • Text-only output stops browser speech while keeping captions.
  • Export session saves the versioned state/event evidence and audio metadata.
  • Delete voice data deletes the session, captions/events, and any retained audio.

Safe narration does not read tool arguments, raw tool output, code blocks, links, or
credential-shaped text aloud.

Voice decisions

An approval pauses the voice loop and the card names only the tool and consequence category.
Review exact arguments in the normal approval dialog.

By default, a voice response can deny only the exact pending request. You can opt into
two-step approval for reversible, low-impact requests: say approve, then confirm approve.
Irreversible or high-impact actions always require visual approval. If the request changes or is
already settled, a stale/replayed phrase does nothing.

Typed fallback

Use Typed fallback inside the voice card whenever audio is unavailable. It uses the same
chat/model/tool/approval loop as a final spoken transcript.

One-shot dictation

Surfaces that enable one-shot dictation keep the original microphone button. Press to record,
press again to stop, review the recognized text, and send it manually.

Screen readers always win

Spoken output flows through a screen-reader-aware wrapper: when JAWS, NVDA, or Narrator is running (detected via the Windows SPI_GETSCREENREADER flag), the agent never speaks over it — replies stay text for your assistive tech to voice. This is engine wiring, not a setting; installing the voice extra cannot regress it.

For per-event audio cues (approvals, completions, errors) that work independently of speech, see Sounds.

Limits

Latency, available voices, echo control, and device behavior depend on the browser, OS, optional
speech engine, and hardware. This release does not add phone calls, cloud STT, biometric speaker
identity, native speech-to-speech provider transport, or an operating-system background wake
service.

← Previous Run usage diagnostics Next → Web Interface