Native computer use

Native computer use

Holaryn can operate selected Windows applications through Microsoft UI Automation (UIA), the same
semantic accessibility layer used by screen readers. It reads roles, names, values, states,
relationships, focus, and supported actions instead of guessing screen coordinates. Native
computer use is a supervised beta feature and is separate from the
first-party browser agent.

Support in this release

Canvas-only and other custom-drawn controls may not expose useful semantics. A window-relative
visual click exists as a last-resort policy option, but it is disabled by default and always needs
explicit approval. It never becomes an unattended coordinate-click mode.

Verify the adapter

Run the local status check:

holaryn computer --state-dir .holaryn-computer-test status --json

On a supported desktop, "available": true and "platform": "windows-uia" indicate that the
adapter can start. Then run the deterministic demo:

holaryn computer --state-dir .holaryn-computer-test demo

The demo opens Notepad, enters a short value through its UIA value pattern, opens Calculator,
invokes the semantic One, Plus, Two, and Equals buttons, verifies the accessible
Display is 3 result, captures one selected-window screenshot, and closes both applications. Its
output names the screenshot and two redacted trace files and explicitly confirms that no coordinate
click was used.

Export a known session trace with:

holaryn computer --state-dir .holaryn-computer-test trace SESSION_ID --output computer-trace.json

Start and supervise a session

Open Computer in the web or desktop navigation.

1. Enter an exact executable or command, such as notepad.exe.
2. Optionally enter the window title you expect.
3. Read the launch warning, check I approve launching this exact application, and select
   Start application.
4. Use Refresh observation to inspect the selected window. The application name, executable,
   process ID, window title, and window handle identify the trust boundary.
5. Browse the semantic target list by keyboard. Each row exposes its role, name, state, supported
   actions, and a Highlight button. The screenshot is supplementary and has meaningful alt
   text; it is not required to supervise the session.
6. Use Pause to stop agent actions, or Take over to interrupt automation immediately and
   operate the application yourself. Resume reconnects only to the same process and window.
7. Use Stop when finished. Stopping supervision does not silently close the application.

The live step history states the application, semantic target, selected action and transport,
reason, result, and approval decision. Download the redacted trace when you need an audit artifact.

The agent receives five compact tools: start a confirmed application, observe its selected window,
perform one action, inspect status/trace, and pause, take over, resume, or stop. A useful prompt is:

Open Notepad and write “Release notes verified.” Use accessibility controls only.
Pause before closing the application and tell me which role and name you used.

A typical response reports that it launched notepad.exe, selected a document or edit control,
used set_value after approval, and paused with a trace entry. Exact role names vary with the
application and Windows version.

Safety and privacy boundaries

• Each session is bound to the exact process ID and top-level window handle that Holaryn launched.
  Element references are opaque, expire quickly, and cannot be reused across a process, window,
  session, helper restart, or changed UIA identity.
• Application text is untrusted input. It cannot grant itself broader tools, approvals, secrets,
  filesystem access, or another application.
• Password values are never returned. Sensitive control rectangles are blacked out before a
  selected-window screenshot is written. Sensitive window titles are redacted.
• Screenshots are explicit snapshots of the selected application window only. Holaryn does not
  continuously record the desktop and does not take whole-desktop screenshots.
• Typing, setting values, sending keys, closing a window, sensitive targets, and any visual
  fallback use the normal approval policy. The trace redacts submitted text.
• The default policy denies credential brokers, password managers, and secure-desktop targets.
  Holaryn operates only on the ordinary interactive desktop.
• Prefer a real API or MCP integration when an application offers one. Native UI control is the
  fallback for applications without a reliable structured interface.

Emergency stop and recovery

Pause, Take over, and Stop update the host state before contacting the adapter, then
terminate its private helper process. They remain responsive even when UIA is hung or an
application has opened a blocking modal dialog. Taking over leaves the application open.

If the application changes beneath the agent:

• Element reference expired/stale: refresh the observation and select the new semantic target.
• Trust context changed: stop the session and launch a new one; Holaryn will not attach to a
  replacement process or window under the old identity.
• Focus was stolen or a modal appeared: take over, resolve the visible application state, then
  resume and refresh.
• Helper or application crashed: the session reports a failure or interruption. Stop it and
  start a fresh session; the durable redacted trace remains available.
• Secure desktop detected: return to the ordinary desktop yourself. Holaryn cannot and should not
  approve UAC, unlock Windows, or enter system credentials for you.

Platform adapter developers

The platform-neutral contract is ComputerUsePlatform. A future AX or AT-SPI adapter must pass
the shared conformance scenario for process/window identity, untrusted observations, stable bounded
semantic targets, expiring trust-bound references, and semantic highlight actions. Read
docs/computer-use.md before implementing an adapter.