Desktop use

Some work has no API and no web page. It lives in a native application, a simulator, or a tool that only ships a GUI. A workflow can open the run's whole desktop and hand it to an agent that sees the screen and works it the way a person would: look, click, type, look again. For driving a web app, reach for browser useinstead, which grounds on the page's structure and needs no vision.

Open the run's desktop

computer.openDesktop() returns a handle to the machine the run is on. The handle is deliberately thin: you launch applications with shell()and inspect files in plain code, so the session exists to bind the agent's tools and take screenshots.

import { agent, computer, shell } from "@boardwalk-labs/workflow";

export default async function run() {
  await shell("xdg-open /workspace/quarterly.ods");   // whatever you want on screen
  const desktop = await computer.openDesktop();

  const shot = await desktop.screenshot();            // stored as a run artifact
  await agent("Fix the broken formula in column D, then save the file.", {
    session: desktop,
    model: "anthropic/claude-sonnet-5",               // must be able to ground on pixels
  });

  await desktop.close(); // or let the run reap it
}

What the agent gets

Passing the session to an agent() leaf gives it six tools. They work in screen pixels, taken from the screenshot the model is looking at.

The loop keeps the last few screenshots in the model's context and drops the older ones, so a long session stays affordable without losing the recent view it is acting on.

It needs a model that can see

A desktop agent works from raw pixels, so it only makes sense on a model trained to ground on a screen. Name one explicitly on the call. If you name a model that cannot, the call is refused with a message saying so, rather than clicking at coordinates it guessed:

await agent("Open the settings panel and turn on weekly digests.", {
  session: desktop,
  model: "anthropic/claude-sonnet-5",
});

Automatic model routing is refused here for the same reason. A desktop session is the one place a wrong model choice does silent damage instead of returning a bad answer, so the choice stays yours.

One screen, one run

A run has a single desktop, so openDesktop() hands out one session at a time and a second call errors. Close it and you can open a fresh one. A desktop session and a browser sessioncan coexist, and the desktop simply sees the browser's window like any other. As with the browser, an open session is reaped when the run finishes, and it survives a wait for a person with the screen intact.

Where it runs

Every hosted run already boots a graphical desktop, so nothing needs turning on. On your own machines it depends on the platform:

Where a tier is unavailable it declines cleanly: openDesktop() fails at the call with a message naming what is missing, instead of a run that clicks into nothing. See self-hosted runners for the environment each tier expects.

On your own Mac

A self-hosted Mac in --host mode drives the real machine: real Chrome, native applications, an iOS Simulator you launched with shell("open -a Simulator"). The simulator needs no special support, because the agent is working the screen rather than the app.

macOS gates both halves of this behind permissions, and both fail silently at the OS level, so a session checks them up front and refuses with the fix rather than letting a run click into the void. Grant them to the application that launches the runner, then restart it:

The agent sees and clicks the main display only. A second monitor is not captured and cannot be reached, so put the work on the primary screen; the runner says so in its log when it notices another display attached.

What the agent can reach

On a hosted run the desktop is the run's own machine and nothing else: a fresh VM, discarded when the run ends. On a self-hosted runner in --host mode it is your actual screen, which is the reason to self-host and worth deciding deliberately. Container mode keeps a self-hosted run on its own in-container display instead.

A run that opens a computer-use session also gets a tighter default egress posture than a plain run, since an agent acting on what a screen shows it is exactly where a hostile page would try its luck. Declare egress in your descriptor when the work genuinely needs the open web.

Screenshots are part of the run's record and inherit its retention, so anything on screen, including a credential a page is displaying, is stored with the run. Secrets your program holds never reach the model: they are redacted from everything the agent sees.