Cowork in MST

Cowork is the cowork client. mst run --config <eval.json> runs an eval config on it, as for any client, and mst batch runs several configs. A platform-specific desktop driver submits each case, and MST builds each transcript from Claude's own session records. macOS uses managed application setup and Anthropic Computer Use. Linux attaches to an externally prepared desktop and uses bounded AT-SPI actions; it does not provision or authenticate that runtime.

First-time macOS setup

This section is for the default third-party mode. In first-party mode, do not run mst cowork setup: it uses the signed-in installed app and no MST profile (see Cloud sessions).

Run this once on a new Mac before the first default-mode Cowork evaluation:

node --import tsx src/cli/index.ts cowork setup

For an installed package, use the package binary instead:

npx mst cowork setup

The command creates the minimal empty Claude 3P profile only when the profile library is missing. It never overwrites an existing profile. After it completes, launch Claude Desktop, sign in through the normal user-controlled flow, open one normal conversation, and quit Claude Desktop. The next batch run manages a temporary Cowork profile and restores the empty base profile afterward.

If the profile already exists but is not empty, setup stops without changing it. Use a dedicated Claude Desktop user/profile or contact the MST owner; do not manually edit configLibrary files.

The batch preflight reports this setup command immediately when the profile is missing. It does not wait for a case timeout.

Which Claude Desktop runs

By default, MST runs the installed /Applications/Claude.app, whatever its version, and records that version with every case result as clientTelemetry.clientApp:

{ "name": "Claude Desktop", "version": "2.19675.1", "source": "installed" }

MST has no built-in version. To run an exact version instead, for example to reproduce an earlier result or to keep a series of runs on one version, pin it in the eval config:

{
  "client": "cowork",
  "clientOptions": {
    "appVersion": "1.52386.6"
  }
}

With appVersion, MST downloads that release for the session. No app bundle or administrator password is required. It resolves the exact version through Anthropic's release service, downloads into a private temporary directory, and checks the advertised size and SHA-256, Anthropic code signature, bundle ID, and embedded version before launch. An unavailable version or failed check stops the run; MST never falls back to the installed app or the latest release. The recorded source is pinned. The earlier MST_COWORK_APP_VERSION environment variable is rejected with a message pointing to appVersion.

Pinned or not, the version must not change during a run. MST requests disabled app updates in its evaluation profile and rechecks the version before accepting run completion; a changed bundle fails the run. Compare runs on the same clientApp.version: a newer Claude Desktop can change results.

MST does not replace /Applications/Claude.app or write managed preferences. It uses the existing signed-in Claude-3p profile. A managed configuration can override profile settings. MST stops the app it ran, restores the original profile and running app, and removes any download directory. Cleanup failures retain the session receipt for cowork recover; recovery also removes a downloaded app. A hard process crash during acquisition can leave the journaled download until recovery. Do not run two Claude bundles simultaneously.

Claude's workspace disk-space check is separate from MCP authentication. Leave headroom for Claude's VM/workspace setup, and for the download when pinning. A server that passes MST's readiness check does not bypass a "Not enough disk space" error. MST removes its own temporary bundle, bridge credentials, and settings on normal teardown; it does not delete Claude's VM bundles, user workspaces, or caches to free space. After an interrupted process, use explicit recovery before retrying.

Root-owned, non-writable managed preferences are accepted only when every key is on the inference-only allowlist: inference routing, credential-helper and model settings, and the deployment's display name. Managed MCP, plugin, updater, unknown settings, and unsafe files still block setup. The plist is read, never modified.

Inference

Two separate model clients run during a macOS case:

• Cowork's inference (the model under test). By default, MST stages ANTHROPIC_API_KEY for the temporary profile through a private credential helper. When managed preferences set inferenceProvider (for example, an LLM gateway with its own credential helper), they take precedence over the profile: MST then requires no key, stages no inference credential, and sets no provider in the profile. If those managed preferences change during setup, the run fails closed.
• The Computer Use planner (clientOptions.computerUseModel). It uses MST's LLM gateway settings: ANTHROPIC_BASE_URL with MST_LLM_AUTH_COMMAND or ANTHROPIC_AUTH_TOKEN, else ANTHROPIC_API_KEY against the public API. The gateway must accept the Computer Use beta tool.

So on a Mac whose managed preferences route Claude Desktop through a gateway, setting the gateway variables for the planner is enough; no API key is needed.

The temporary profile turns on tool search (toolSearchEnabled), as on the Linux desktop: Cowork loads an MCP tool's definition when the model searches for it. Set coworkSetup.toolSearch: false to send every tool in every request. See Cowork in Docker.

MST_COWORK_APP_PATH=/absolute/path/to/Claude.app runs an app installed somewhere other than /Applications. MST neither downloads nor deletes caller-owned bundles, and records their version too. Don't combine it with appVersion. Linux provisioning is unchanged.

Running an app doesn't prove MCP availability. Managed inference configuration can take precedence over a profile's MCP list. On macOS, MST now exposes declared servers through Claude's supported local Developer MCP surface instead:

• HTTP declarations use a local stdio-to-HTTP bridge with the declared headers.
• Plain and private-file-backed stdio declarations run the caller's command, including vendor dry-run proxies. Plugin-root placeholders remain Linux-only.
• The temporary claude_desktop_config.json contains launcher paths, not tokens. Runtime credentials stay in private per-session files. Existing developer servers are replaced for isolation, then restored with the original file bytes.
• Cleanup tolerates Claude's JSON whitespace rewrite, but rejects semantic edits made concurrently. cowork recover also restores the developer configuration.

MST checks the servers once the app is prepared, before the first case (readiness). A server that isn't ready is checked again after 10 seconds and then after 30 more, as for every client; when one is still not ready, no case is sent and each trial of the batch ends as a retryable setup failure.

Use actual tool-call assertions: a server that passes the readiness check is not proof that the Cowork session received the server. A live pinned run has passed with an eval-endpoint search tool and a login-backed GitHub get_me call.

Cloud sessions

Claude Desktop signed in to claude.ai runs Cowork tasks as remote cse_… sessions. Set clientOptions.mode to "first-party" to run on the account's existing connectors instead of MST-managed servers. The default, third-party (also the result of omitting mode), is unchanged: it supports stdio servers and proxies through MST's gateway setup. First-party mode only selects existing HTTPS account connectors through MST's normal server labels and records those sessions. It supports the local macOS runner only, with concurrency: 1 and no sharding.

First-party mode uses the Claude account that is already signed in to Claude Desktop. MST still submits prompts through visible Desktop Computer Use, using the Computer Use planner credentials you already configure. It reads session events through a separate Chrome recorder profile signed in to the same account and mirrors them into MST's native transcript layout. It does not copy cookies, provision VMs, or submit tasks through an undocumented API.

Start a dedicated Chrome recorder profile with a loopback DevTools port. Use a --user-data-dir that no other Chrome uses, and sign in to the same claude.ai account as Claude Desktop in that window. Do not copy cookies or reuse another profile's session.

"/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" \
  --user-data-dir="$HOME/.mcp-server-tester/claude-ai-recorder" \
  --remote-debugging-port=9222 \
  --remote-debugging-address=127.0.0.1 \
  https://claude.ai

Set clientOptions.env.MST_CLAUDE_AI_DEBUG_PORT to the port (default 9222). Keep the port private to the local machine. First-party mode does not install developer MCP servers, marketplace plugins, or an inference gateway profile. It does not register new connectors or sign in to them. Authenticate each connector in the Claude account first.

Configure remote connectors

These settings are account-wide, not per-task. Run variants sequentially. Set concurrency: 1, and do not edit the same account's permissions during a run. The local desktop lock does not coordinate other machines. Authentication provisioning and parallel cloud workers are not implemented.

{
  "name": "account-connectors",
  "datasets": ["cases.json"],
  "client": "cowork",
  "clientOptions": { "mode": "first-party" },
  "concurrency": 1,
  "servers": {
    "search": {
      "transport": "http",
      "serverUrl": "https://mcp.example.com/mcp"
    }
  },
  "variants": [
    { "name": "no-connectors", "servers": [] },
    { "name": "read-only", "servers": ["search"] },
    {
      "name": "filtered",
      "servers": ["search"],
      "clientOptions": {
        "connectors": {
          "servers": {
            "search": { "policy": "read-only", "tools": { "search": "block" } }
          }
        }
      }
    }
  ]
}

Replace the example URL with the exact HTTPS endpoint of an existing connected account connector. Each selected URL must match exactly one connected connector. Missing, ambiguous, or disconnected matches fail before permission changes. MST does not infer a match from a display name or silently create one.

Selection follows the normal contract: an omitted variant servers list selects all declarations; servers: [] selects none. Unselected account connectors are blocked. Selected connectors default to read-only. Policies in clientOptions.connectors.servers are keyed by the selected MST server label, not an account name or UUID. Put variant-specific policies on that variant; a policy for an unselected label is an error. Reports and tool assertions use the same labels; rawName preserves Claude's original name.

This mode accepts only transport, serverUrl, and label (from the map key). It rejects stdio, auth/headers, TLS/proxy overrides, protocol options, and transport timeouts because those settings cannot configure an account connector. It also rejects the options appVersion, dataDir, plugins, pluginRoots and mcpDataRoot, simulated writes, tool-metadata proxies, and recordServedTools (the run records the account's tool inventory in connector-policy.json instead). Readiness and read-only checks use the account's actual tools during preparation, not a separate unauthenticated HTTP probe.

Server policies are read-only, allow-all, ask, and block; tool overrides are allow, ask, and block. First-party mode requires default: "block" and rejects inherit, so omitted connectors cannot leak into a variant. Configuration cannot override organization permission limits. Claude's built-in tools are outside this option's scope.

read-only explicitly enables and always-approves only tools whose MCP readOnlyHint is true. It blocks write tools and tools with unknown annotations. The hint is supplied by the server; it is not a sandbox or proof that a tool cannot write, so MST does not sandbox on annotations. allow-all explicitly enables and always-approves all tools of the selected server, including writes. An explicit per-tool override takes precedence over its server's policy. Neither policy authorizes an eval to perform writes that the caller has not approved. If a policy permits any write or unknown tool, the server also needs the normal writes declaration: "writes": { "search": "unguarded" }, or live with --allow-live-writes. These are real account tools, not a dry-run proxy; planned/simulated-write assertions are unsupported.

failOnApproval defaults to true, which disables automatic human-in-the-loop handling: MST performs no automatic approval clicks and rejects recorded approval requests. Set it to false for an ask policy; MST then delegates to the existing one-time approval handling. It is not a manual-only ask mode. coworkSetup.approveWriteTools enables one-time approval of writable tools only with an explicit writes declaration and failOnApproval: false; with failOnApproval: true nothing is handled automatically. MST verifies saved permission keys and checks the session's initial catalog, discovered tool references, and actual calls against the policy. Cowork can load connectors after reporting them as pending in its initial catalog. MST records that catalog as incomplete rather than claiming it proves every tool's availability. It saves the check as connector-policy.json beside the audit log, including the prepared tool list when validation fails. If the server changes its catalog during a task, MST fails the check instead of silently accepting new tools. Start a new run to prepare against the changed catalog.

The Chrome recorder profile and Claude Desktop must be on the same account and the same active organization. You must make sure they are; MST does not verify Desktop's identity automatically. Before changing permissions it checks the Chrome session's account and organization, and it checks them again against the recovery journal before restoring. This is not a sandbox guarantee.

One macOS Cowork lease is shared by both modes and every dataDir, so only one Cowork run drives the app at a time. MST reads at most 10,000 events from a session. A longer session fails with an error that names the cap; MST does not truncate it.

The permission API is undocumented and may change. Use the account exclusively during a run or recovery: do not run tasks or edit connector settings in another Claude session. MST's checks detect observed conflicts; they are not an atomic lock against other applications. MST records the original values before applying settings, verifies each write, and restores changed keys on close. Keys absent originally are removed, not replaced with explicit allows. A normal completed run restores automatically.

A private recovery journal at ~/.mcp-server-tester/cowork-claude-ai-connectors.json survives a crash. MST writes it before the first permission write and before it submits the first task. A journal from an earlier development format (version: 1, the journal's own schema version, not an MST package or mst.run/v1 version) is unpublished. It has no write-outcome or task record, so restore and configure reject it. MST does not migrate it; it needs manual resolution (see below).

MST keeps the journal and the desktop lock, and blocks both automatic recovery and a new configured run, in any of these cases:

• A permission write has an unknown or in-flight outcome. This holds even if a later read matches the original value. There is no force option.
• MST persisted the intent to submit a task, but no session is bound to it, or the bound session cannot be read.
• The bound session has no authoritative terminal state. Quitting Claude Desktop does not prove that a cloud task stopped.
• The account or organization differs, or a concurrent edit changed a value.

If an excluded connector cannot list its tools, preparation aborts. Checking a task afterward is not a security boundary against a connector that changes its tool inventory while the task runs.

Recover connector settings

If a run was killed, restore from the journal without starting an eval. The command reads the port only from its own shell, not from the eval config. Export the same port the run used:

export MST_CLAUDE_AI_DEBUG_PORT=9222   # omit if the recorder uses the default
npx mst cowork restore-connectors [--recover-stale-lock]

This macOS command uses the existing Chrome recorder profile sign-in. It runs no eval, changes no policy plan, and does not drive the app. It restores only after all of these checks pass: account and organization identity, no concurrent permission edits, every write outcome known, and the bound task finished with authoritative terminal evidence.

Recovery takes two files under ~/.mcp-server-tester: the shared desktop lease and a separate recovery guard (cowork-desktop-macos.lock.recovery). The guard stops two recoveries from racing. An interrupted recovery can leave it behind. If restoration fails, MST releases the guard but keeps the desktop lease and the journal. A kept lease blocks both new runs and later recovery.

Without the flag, any existing lease fails the command. --recover-stale-lock removes a desktop lease only when its local owner process is provably dead. It does not prove that a cloud task finished or that a permission write settled, and it never reclaims the recovery guard. A live or unknown owner is always rejected.

Stale recovery guard

An interrupted recovery can leave ~/.mcp-server-tester/cowork-desktop-macos.lock.recovery. --recover-stale-lock cannot reclaim it. Before you remove this guard alone, inspect its owner and confirm that no recovery is running. A live, unknown, or unverified owner is not a reason to remove it. Removing the guard only lets another attempt acquire the desktop lease. It does not restore permissions or show that cloud work has stopped. Leave the journal and the main desktop lease in place until you have established that doing otherwise is safe.

Resolve a blocked journal manually

Repeating restore-connectors or using --recover-stale-lock does not attest that a write or task has ended, and neither clears these markers. Do not delete or edit the journal or the desktop lease to retry: the journal is the only record of the original permissions, and no current command performs or verifies manual retirement. There is no force option and no supported delete or marker-edit recipe.

1. Keep the journal and the lease unchanged. Keep a protected copy of their account, organization, and original-permission facts.
2. Ask qualified Claude account or provider support to identify the exact task, to establish that every outstanding write has ended and cannot commit later and every task has finished, and to restore and verify the recorded original permissions, including removing keys that were originally absent.
3. Once those facts are established and no local run or recovery owns the files, ask an MST maintainer for help. Maintainer assistance retires or securely archives the resolved local state and retains the evidence. A retry is not promised to succeed.

The same steps apply to an earlier-format journal that restore and configure reject.

Run on macOS from a source checkout

COWORK_ENV_FILE=/absolute/path/to/existing.env ./scripts/run-cowork.sh --configs /absolute/path/to/eval.json

This builds MST and runs the supplied eval configs through batch. It submits real Claude tasks. The invoking terminal must have Accessibility and Screen Recording permission. Credentials remain in the existing dotenv file or exported environment; the runner never rewrites or shell-sources that file. Node loads dotenv before plugins/custom judges start.

An explicit --configs or --config-dir is required; no default evaluation is bundled. Use the normal file/GCS dataset sources and plugins. Configure custom judges according to their plugin's requirements. --dry-run checks configuration, not GUI execution or model behavior.

Use client cowork. The earlier names cowork_cu and anthropic.claude.cowork.desktop-app.macos fail validation, naming cowork. model selects the Cowork inference model; clientOptions.computerUseModel independently selects the planner. The inference provider is anthropic, and no option changes it: clientOptions.provider fails validation. How Cowork runs follows from the OS of the machine that runs the trial, and no option selects it: on macOS, Computer Use, with computerUseModel as its planner; on Linux, the accessibility automation, with no planner. A config that set clientOptions.computerUseProvider fails validation: remove it.

clientOptions.maxActions is the most actions the automation may take to send a case's prompt, on either OS: 1 to 64, default 24. It was computerUseMaxActions, which now fails validation, naming maxActions. hitlMaxActions is a separate budget, for all of a case's human-in-the-loop inspections after its prompt is sent (approving its tool calls, checking that its task finished): 1 to 24, default 12. timeout is the trial's time limit in ms, default 900000.

Cowork runs a variant's cases in one app, with one set of settings. A case that sets its own model or a clientOptions value that differs from its variant's fails when its dataset loads, before anything runs, and so does a case that names another client. Use a variant for a different setting.

MST applies a fixed inference model list with discovery disabled. On macOS, managed settings can override that profile. The submission driver therefore selects and verifies the requested model in Cowork's visible model picker before entering the prompt. If the exact model is unavailable or cannot be confirmed, the case stops without submission; it never substitutes another model. Native telemetry must still report the requested exact model ID; a mismatch or missing model evidence fails the case. Omitting model preserves the existing application default. Configure servers with HTTP endpoints and literal or environment-backed bearer credentials, or stdio commands with declared env values. Every macOS batch owns one setup transaction, including an empty server set, which exposes no user-added MCP servers. Stdio receives only its declared env; omitted inheritEnv behaves like false, and true is rejected. Credentials are not inferred from arbitrary environment variable names.

coworkSetup.approveWriteTools defaults to false. Mac Computer Use approves only clearly read-only tools by default. Explicit opt-in stages local connector defaults for the declared tool inventory before launch, using the same enabled and content-fingerprint keys as the connector picker (live-verified on Claude Desktop 1.52386.6 and 2.19675.1). If a later version stores them differently, its approval prompts remain and cases fail or time out rather than pass. Both bare server:tool and local:server:tool names are staged because the renderer switches naming paths with cowork_snapshot_sync. Mac write opt-in disables the approval-click fallback: staging must work without clicking a pending approval. Managed wildcard policies do not cover these local Developer MCP connections. The existing third-party account-settings file must be unambiguous. The private session journal restores the original permission entries during cleanup and recovery; unrelated settings edits are preserved, and conflicting edits fail closed. MCP config cleanup also preserves unrelated preference changes, such as Claude's epitaxyPrefs, while refusing conflicting MCP edits. This does not grant permanent permissions or change tool annotations. Organization restrictions can still require approval. Keep vendor servers behind the caller's dry-run proxy. Delayed native tool requests trigger at most three further approval inspections within the original per-case action budget; prompts are never resubmitted. Follow-up usage is recorded under clientTelemetry.computerUse.hitlFollowups.

In third-party mode, MST does not configure or observe connectors added from Cowork's own connector directory (the client's own connectors), so disconnect any that could write to real accounts before a run, or use test accounts. first-party mode does configure and observe the account connectors it selects (see Cloud sessions).

Run an installed release

node --env-file=/absolute/path/to/existing.env node_modules/@gleanwork/mcp-server-tester/dist/cli/index.js run --config /absolute/path/to/eval.json

For several eval configs, use batch --configs <files...> --workers 1.

The Computer Use driver runs in the MST process. For mouse and keyboard input it compiles a small Swift helper, once per process, which needs the Xcode Command Line Tools (xcode-select --install). If Xcode is installed but its license hasn't been accepted, xcrun refuses to build, and MST says so. Run sudo xcodebuild -license accept, or build with the Command Line Tools by setting DEVELOPER_DIR=/Library/Developer/CommandLineTools. The build sees no other variable from your environment. Screenshots use macOS's screencapture and sips. The terminal that runs MST needs Accessibility and Screen Recording permission. Nothing else is installed. The source-checkout wrapper is a convenience, not required by run or batch.

Install an unreleased source pin

A Git dependency must run the package's prepare lifecycle to generate dist. Use an immutable Git commit and explicitly allow this package's build in your package manager. --ignore-scripts is appropriate for prebuilt registry packages, but leaves a Git source dependency without its CLI. No build tooling is required when installing the normal published package.

Attach to a prepared Linux desktop

Use client: "cowork" with mst run (or batch) on the Linux machine. Cowork runs there with the accessibility automation, because the machine runs Linux. computerUseModel and appVersion are macOS options and fail: there is no LLM desktop planner, and the prepared desktop is the app that runs. MST checks them when the config loads, against this machine's OS for --runner local (the default) and against Linux for --runner docker. A plugin's runner checks them on its worker, before any app is prepared.

The caller supplies a working Claude Desktop session with the desired model/MCP settings. MST accesses that session through X11, D-Bus, and AT-SPI (Python gi), and xclip to paste a prompt longer than a deep link carries. Run MST in the prepared desktop's filesystem and session context. DISPLAY and DBUS_SESSION_BUS_ADDRESS are required. MST_COWORK_PYTHON can select a prepared system interpreter; MST does not install Linux desktop dependencies. The packaged cowork-linux-runtime export resolves the Python driver independently of the working directory.

The driver reads /etc/claude-desktop/managed-settings.json (or the absolute MST_COWORK_SETTINGS_FILE) and fails if its model, MCP servers, or wildcard approval policy disagree with the eval config. Other policy-only entries must block all tools ({"*": "blocked"}). With client plugins, allowedPluginMarketplaces must contain exactly one entry per plugin that matches coworkPluginMarketplace(plugin); without plugins, it must be absent or empty. Stdio eval servers and blocked plugin servers follow the contract in Client plugins. Native sessions default to $XDG_CONFIG_HOME/Claude-3p/local-agent-mode-sessions, or $HOME/.config/Claude-3p/local-agent-mode-sessions; clientOptions.dataDir overrides it. Preparation is read-only. MST does not provision or authenticate the desktop, change its launch policy, or manage its lifecycle.

Owned-desktop mode. A worker image that runs several variants on one desktop (ADR 0004) sets MST_DESKTOP_OWNED=1, makes the settings file writable by the desktop user, and sets MST_DESKTOP_RESTART to the absolute path of a command that restarts Claude Desktop and returns once it's back, within 5 minutes (MST_DESKTOP_RESTART_TIMEOUT_S sets another limit, up to 3600 seconds). If the command fails, MST runs it once more before the variant fails. Claude Desktop reads /etc/claude-desktop/managed-settings.json only when root owns it and no one else can write it, so an image points MST_COWORK_SETTINGS_FILE at a staging file the desktop user owns, and its restart command installs a root-owned copy first. For each variant, MST then writes the variant's servers, plugins and model over the image's own settings, keeping the image's other keys. An HTTP server with headers or an access token (such as a connector server's) gets a headersHelper that prints its headers from a private file beside the staging file (mcp-headers/), so the settings hold no secrets. The helper holds none either; it's written to MST_COWORK_HELPER_DIR when the staging file's directory doesn't allow executing files (Docker mounts a tmpfs noexec), else beside the headers. Both are removed when the variant ends. Another HTTP server keeps the image's headersHelper for the same name and URL. It removes AskUserQuestion from the image's disabledBuiltinTools, so HITL can answer clarifying questions, or, with coworkSetup.clarifyingQuestions: "disabled", adds it. It runs the restart command, checks the file as above, and probes. When the variant ends, or if the desktop isn't ready, the file goes back to what it was. Without MST_DESKTOP_OWNED=1, preparation stays read-only.

In a runner. A plugin runner (mst run --runner <namespace>/runner/<name>) can run Cowork shards on Linux workers in owned-desktop mode. Connector servers and variants with tool metadata work there as they do locally: the worker launches each connector with its own paths and gets its access token from the coordinator, and runs the tool-variant proxy itself. The worker image must install the plugin packages the eval loads, at the versions the coordinator runs, where the worker resolves them (MST_PLUGIN_DIR); a worker with another version refuses to run. The built-in docker runner runs shards in containers from MST's own image, and installs the eval's plugins in it: see Cowork in Docker.

Submission opens the native deep link through the prepared environment's xdg-open handler, then attempts one semantic Start task action. Alternatively, set MST_COWORK_URL_OPENER to an absolute executable path that accepts the URL as its sole argument. The caller owns protocol registration and application launch flags; MST does not select an application binary or sandbox policy. The prompt is preserved unchanged, and the opener runs without a shell and with a bounded deadline. Claude keeps only the first 14,336 characters of a deep link's prompt, so a longer one (1–2.5% of the Cortex eval sets' cases, up to 135,000 characters) goes another way: an empty new-task link, then the prompt pasted into the composer (xclip holds it on the clipboard; AT-SPI presses Ctrl+V), checked against the composer's text before Start task. A bound session whose stored prompt is only the prompt's beginning fails as an infrastructure failure (failureKind: "input"): its answer says nothing about the case. There is no keyboard fallback or retry after an uncertain action. The shared native collector still requires a new exact-prompt session. HITL only operates on approval controls while that bound session is pending; it never types, creates tasks, or continues onboarding. Unclassified approval prompts fail closed unless coworkSetup.approveWriteTools explicitly permits approval. Built-in Cowork tools ignore managed MCP toolPolicy and confirm with an inline card whose primary action names the write (Create/Update for artifacts; Schedule/Update/Run/Delete for scheduled tasks) beside Cancel. The 3p desktop offers no Skip all approvals mode, so under approveWriteTools HITL approves that card; read-only runs refuse it. More than one candidate action fails closed.

A task can ask for a folder of the user's computer (request_cowork_directory, often for code it can't find): a card with Choose folder beside Deny. Nobody can choose one in a headless run, and managed settings can't remove the tool (disabledBuiltinTools doesn't apply to Cowork's own MCP tools), so HITL clicks Deny, under either policy, since denying grants nothing. The model is told the user rejected it and answers with the tools it has; the case is graded on that answer. computerUse.hitl.telemetry.folder_requests_denied counts them.

A task can ask a clarifying question (AskUserQuestion). By default (coworkSetup.clarifyingQuestions: "first-option"), HITL answers it the way ACE does: it reads the question's options from the bound task's transcript and clicks the first option of the question on screen (Cowork shows one per page, marked 2 of 3). A control matches by its exact label, or by the label followed by its description, unless a longer option label claims it. Once a multi-select option is checked, HITL presses the page's one Submit/Next control. HITL acts only on one unambiguous control and never types an answer. If the form doesn't take the first option within 60 seconds, or the question has no usable options, the case fails as unanswered (clientTelemetry.awaitingUser). computerUse.hitl.telemetry.questions_answered counts the answered pages. Set clarifyingQuestions: "disabled" to turn the tool off instead, so the model proceeds on its best assumption. The macOS eval profile always turns it off.

clientTelemetry.computerUse records observed semantic actions and elapsed time. New receipts omit the driver identity; readers still accept it in stored results. Planner tokens and planner cost are not applicable, not synthetic zero-usage Anthropic calls. Native usage and cost retain their own scope.

Installation and configuration checks do not establish end-to-end correctness. Validate the prepared desktop with representative cases and audit their captured native evidence.

Audit a saved Linux native run

Use auditCoworkNativeRun from the experimental clients subpath, which may change between minor versions. It is offline: it does not invoke the desktop, MCP servers, or judges. No CLI or private parser import is required.

import { auditCoworkNativeRun } from '@gleanwork/mcp-server-tester/experimental/clients';

const report = await auditCoworkNativeRun({
  rawResultsPath: '/archive/runs/20261007T182504Z-a3f9c1/results.json',
  nativeRoot: '/archive/native',
  expectedCases: 2,
  expectedModel: 'claude-opus-4-6', // optional exact native model assertion
});

if (!report.evidencePassed) {
  console.error(
    report.issues,
    report.cases.map(({ id, issues }) => ({ id, issues }))
  );
}

The input is a run's results.json in the mst.run/v1 format, with the run's summary.json beside it. Each case must have a unique ID and clientTelemetry.nativeSessionId. Retain this archive layout, including the actual saved tool-output bytes:

native/
  local_<UUID>/
    local_<UUID>.json
    audit.jsonl
    .claude/projects/<project>/<CLI UUID>.jsonl
    .claude/projects/<project>/<CLI UUID>/tool-results/<name>.txt

The audit uses MST's existing native parser and both client normalizers. It checks exact case count, unique case/session identities, exact initial prompt, final response, ordered normalized events and tool calls (arguments, output, IDs and provenance), usage/cache/cost, native/API durations, completion/nonerror flags, parsed audit and transcript, no parser warnings, and native-derived telemetry in both saved envelopes. Only live computerUse and hitlWarning fields are excluded from telemetry equality. An expected model must match the single observed native model. Models must use the recognized Claude opus/sonnet/haiku version-ID format.

Output has been saved to ... notices require nonempty real .txt files. Recorded absolute paths are never read directly: only the same local_<UUID> session's .claude/projects/<project>/<CLI UUID>/tool-results/*.txt suffix can resolve below nativeRoot. Traversal, encoded paths, foreign sessions and symlinks fail closed. Attachment records contain tool-call indexes, byte sizes and SHA-256 hashes, never paths or contents. A saved notice alone is not complete evidence.

CoworkNativeAuditReport is exported from the package root:

• schemaVersion: 1, expectedCases, observedCases, issues (fixed codes).
• qualityPassed: boolean | null reflects saved case pass values. A failed judge does not fail evidence. null means missing/unknown quality, not success.
• evidencePassed: boolean; status: 'verified' | 'failed' describes evidence only.
• cases[]: id, pass, sessionId, model, usage, timing, toolCounts, validity, attachments, evidencePassed, issues. Validity includes auditParsed, transcriptParsed, complete, nonError, hasUsage, hasCost, and noWarnings. Unknown fields stay null, not zero.
• totals: null unless the entire evidence audit passes. Otherwise native-only token/cache/cost/duration sums plus costScope: 'native-inference-only'. Unknown cache fields stay null. This excludes controller and judge cost/time.

Reports never include prompts, answers, tool arguments/results, native paths, parser error messages, server URLs, or credentials. Numeric e2e-<digits> and case-<digits> IDs are retained; other case IDs use SHA-256 pseudonyms. Invalid session/model identifiers are not echoed. Errors use fixed issue codes.

Bounds: 1–1,000 expected cases, 32 MiB raw results, 16 MiB per native file, 64 MiB per session, 4,096 directory entries per session, depth 12, and 1,024 output notices per case. The parser reads a private temporary snapshot of the two bounded transcript files, which is removed after parsing. User-controlled symlinks are rejected (including roots and transcript paths); fixed macOS /tmp and /var system aliases are allowed. Audit a quiescent archive under trusted filesystem ownership: portable Node path checks cannot eliminate hostile concurrent directory-replacement races. A verified report establishes internal consistency, not cryptographic authenticity or independent verification of the saved judge result. Attachment hashes describe the bytes present at audit time; notices contain no original digest to authenticate.

Structure and behavior

• desktopBatch.ts: the batch lifecycle shared with the ChatGPT client: the desktop lease, per-case reset policy, native-session de-duplication, redaction, and cleanup.
• coworkClient.ts: selects the concrete OS entry point on the trial's machine; owns setup, one case (submit, bind, HITL, native collection), reset, and transcript conversion.
• cowork/desktop.ts: the native operations shared by the two OS entry points, with no selector.
• cowork/macos.ts: the macOS entry point, using the existing setup, recovery, and Computer Use functions.
• coworkSetup/: profile/MCP settings, private header helpers, and guarded restore. Its macController.ts handles only application start/stop, not UI automation.
• nativeHelper.ts: builds and runs the macOS Swift controllers for Cowork and ChatGPT with one environment policy. The Cowork controller runs with only the system PATH and a private TMPDIR. The ChatGPT controller launches the app with the test environment; see ChatGPT desktop.
• cowork/interaction.ts: shared receipts, error classifications, and scoped telemetry. New Linux receipts omit driver; old receipts remain readable.
• cowork/linux.ts: the Linux entry point, with readiness checks and execution against a caller-owned desktop.
• scripts/cowork_linux.py: bounded semantic UI actions; no lifecycle or answer parsing.
• cowork/anthropicComputerUse.ts: the macOS CU driver entry point, its deadline and its error reporting.
• cowork/computerUseLoop.ts: the planner loop: bounded screenshot navigation, executor-owned query insertion, one submit boundary, model confirmation, focus checks, and bounded HITL and reset handling.
• cowork/computerUsePlanner.ts: the planner's Messages API calls, through resolveLLMEndpoint(), with the Anthropic SDK's retry rules.
• cowork/macDesktop.ts: screenshots, and input through a Swift helper (macDesktopSource.ts) built with nativeHelper.ts.

Cases run sequentially: snapshot, submit, bind, HITL, then native collection. Already-complete native tasks skip unnecessary HITL navigation. An exhausted inspection budget is recorded as a warning: passing still requires native completion and the normal assertions/judges. So is a Linux HITL error that provably left the app untouched (no action, and every driver call returned a receipt; a failed completion check, for one), when the task goes on to finish: it can't have changed the answer. Other HITL errors, those that acted or can't tell, remain failures. A HITL error keeps its cause (Linux HITL failed for the bound native session: …). On Linux, HITL's scan for approval prompts skips tables (an answer's, or a presented spreadsheet's preview, whose cells can number in the thousands) and gives up before its call's deadline; a pass whose scan still can't finish saw no prompt, and HITL scans again (counted in computerUse.hitl.telemetry.incomplete_scans). Native paths support both legacy session folders and the current short-ID/cwd-root layout. Native tool server names are retained; built-in tools are not attributed to the tested server.

The executor inserts the dataset query unchanged. No marker, prefix, or suffix is added. MST binds the case to the session its submission created: a metadata path absent from the per-case snapshot, with a recent creation timestamp (five seconds of clock tolerance). Existing sessions remain excluded even if updated. A new session whose native initialMessage equals the prompt binds at once. Claude can store a prompt changed from what the deep link carried (its surrounding quotes dropped, say), so when no new session has it exactly, the only new session binds, once it has stayed the only one for two polls. Two new sessions without exactly one exact match are ambiguous, and the case fails. The transcript's correlation says which binding held (exact-initial-prompt or only-new-session); for only-new-session, promptDifference gives the stored and submitted lengths and the first differing index, never the text. Answers and tool output are never prompt matches. MST binds within 30 seconds after submission and pins collection to the bound session. A failed case is never retried or resent. Each case is self-contained: after a submit, binding, HITL, or transcript failure, MST records that case's failure and resets the app before the next case. On Linux the reset first stops the failed case's task if it still runs (a task past its deadline, or one waiting on a prompt): it clicks the task's Stop response and waits until it's gone, about 2 seconds. A task that won't stop fails the reset (task_not_stopped), and the old task's view is left as it is. The reset then opens one empty new-task deep link and waits read-only (about 10 seconds) for the Cowork start surface. What the screen showed is captured before the reset (When something fails, in cowork-docker.md). On macOS, the Computer Use planner stops a task that is still running and declines an open permission prompt, within eight actions. The driver refuses typing, drags and key presses other than Escape (a refused proposal does nothing, and the planner may try again within the budget), and the planner is instructed never to click Send, approve, or answer a question. If the planner can't conclude the app is idle, MST quits and relaunches Claude Desktop instead, which stops whatever was running; the session's settings and MCP servers stay installed. Only if that fails too is the reset a failure. The next case takes its own session snapshot, so a late session from the failed case cannot be attributed to it. If the reset fails, the remaining cases are not submitted, and the same happens after three failed cases in a row: a reset can succeed and still leave a desktop that every case fails on. Per-case snapshots distinguish repeated identical prompts. Use a dedicated desktop: do not manually create or switch tasks during evaluation.

The case's answer is the last turn's assistant text from its longest text block on. Cowork often writes its answer, then saves memory or files and ends with a short message ("Saved the context…"); the native result holds only that last message, so grading it would grade the wrap-up. Shorter text before the longest block, the narration between tool calls, is left out. The transcript's answerSource says which held: final-message when the last text block is the longest, else longest-text-onward, with finalMessageChars, the last block's length.

A case has timeout milliseconds from submission (default 900000, 15 minutes) to complete. When the session hasn't completed by then, the case fails with Native Claude transcript deadline exceeded but keeps what the session did: its tool calls, usage, its answer so far, and the session folder, with deadlineExceeded: true in its telemetry. deadlineWait says what the task was waiting on and since when: { "on": "tool", "tool": "mcp__slack__…" } for a tool call with no result yet, else { "on": "model" }, with idleMs since its last native event; the error says the same (It had been waiting on the model's response for 10m 54s.). A long wait on the model is a stalled response or one very long one: Claude logs a message only when it ends, so the transcript can't tell which. Set a longer timeout in the eval config for long tasks.

On macOS the driver acts only while Claude is the frontmost app. A click that misses Claude's window (on the desktop, say) brings another app forward; before every click, key or scroll, the driver checks, and if another app is in front it brings Claude back, skips the action and has the planner take a new screenshot. If Claude can't be brought back, the case stops with navigation_blocked and nothing more is sent. A planner request that can't reach the API (a short network drop) is sent again after 2, 4, 8 and 16 seconds, on top of two quick retries for each request, as the Anthropic SDK does: it has no effect on the desktop. If the API stays unreachable, the case fails with provider_unavailable; the driver log names the error's classes and system code, such as TypeError <- Error(ECONNREFUSED). A request that times out gets the two quick retries, but not the longer ones: it may still be running. MST doesn't print the driver's step-by-step log; set MST_COWORK_CUA_LOG_FILE to a path to keep it. The log holds action and app names, never the query.

Claude Code replaces a large tool result in its transcript with a <persisted-output> placeholder and saves the result to a file, which the model reads. MST reads that file back while it collects the trial, so the transcript (and any judge) sees the full result, up to 1 MB. A server writes the placeholder's text and can know the call's tool_use_id, so MST reads only a regular file whose real path is named for that id, in a tool-results directory, inside Claude Desktop's private temporary directory (on macOS, the MST helper's mst-cowork-native-*) or the session's directory; never another path a placeholder names. The evidence audit replays the transcript without these files, and accepts a stored result that starts with the placeholder's preview. A second form Claude Code uses for oversized results (Output has been saved to …/tool-results/mcp-<server>-<tool>-<time>.txt) isn't read back yet.

Results include clientUsage, clientTelemetry, and telemetry.totalClientUsage. These describe native Claude execution, not the separate Computer Use planner's API cost. A test assertion or judge failure is distinct from a driver failure. clientTelemetry also has, from Cowork's results and audit log:

• timeToFirstTokenMs and timeToFirstTokenStreamMs: the trial's first turn, to its first token and to the first token of the response stream, with firstRequestAt. Each model call's own TTFT is in the client's own telemetry (telemetry.host).
• permissionRequestCount, permissionDenialCount and approvalWaitMs: the tool approvals Cowork asked for, the ones refused, and the time from each request to its answer. The wait is part of durationMs.
• finalContextTokens: the context the last model call read (its input, cache-read and cache-write tokens).
• webSearchRequests: the server-side web searches the results count.

A model call's reasoningText is its thinking summaries, when the transcript has them. Cowork on MST's Linux image has none: its thinking blocks carry only a signature, and Claude Code's showThinkingSummaries in the desktop user's ~/.claude/settings.json doesn't reach the agent in Cowork's VM. Each result's thinking-token count (reasoningOutputTokens) is still recorded.

The shared orchestration is OS-neutral and tested with injected platform doubles. macOS is live-qualified. Linux has separate qualification requirements described above. Windows is unsupported; there is no fallback to a different product.

Do not rerun an ambiguous submission blindly. Inspect native completion and retained setup state. After confirming work has stopped, explicit guarded recovery is:

node --import tsx scripts/recover-cowork.ts --confirm

Never delete managed locks to force a retry. This is the existing managed desktop workflow, not a sandbox or a transactional guarantee over arbitrary UI actions.

One run drives a desktop at a time. MST claims it with a lease file, ~/.mcp-server-tester/cowork-desktop-<id>.lock, where <id> identifies the desktop by its native data directory. A second run on the same desktop, even in another process, refuses to share it; runs on separate desktops (for example, several Linux displays) don't block each other. If cleanup or restoration fails, MST prints why, every case result in that batch says so (an infrastructure failure, since the desktop's state is unknown), and the lease is kept, so later batches stop until you have inspected the desktop and removed the file. In a run with several variants, those batches' trials are recorded as infrastructure failures that name the lease, and the run's results are still written. An interrupted run leaves the lease behind in the same way. If a case fails unexpectedly mid-batch, the cases before it keep their results and the rest are recorded as not submitted.

Client plugins

The config has two independent parts:

• clientOptions.plugins[] installs plugins (their skills). On Cowork, a plugin can also block its own MCP servers with blockMcpServers.
• servers is the MCP servers under test, keyed by label. Both plain stdio and client-resolved stdio entries are supported on macOS and Linux, alongside HTTP. A client-resolved entry can launch a native proxy with private credential files or, on Linux, resolve a file from a declared plugin.

{
  "client": "cowork",
  "clientOptions": {
    "pluginRoots": {
      "acme": "/opt/example-app/plugins/acme"
    },
    "mcpDataRoot": "/config/mcp-data",
    "plugins": [
      {
        "name": "acme",
        "marketplace": {
          "source": "acme/claude-plugins",
          "ref": "<40-character commit SHA>"
        },
        "blockMcpServers": ["acme_plugin"]
      }
    ]
  },
  "servers": {
    "acme-eval": {
      "transport": "stdio",
      "command": "/usr/bin/node",
      "args": ["${pluginRoot:acme}/mcp/start.mjs"],
      "url": "https://mcp.example.com/eval",
      "auth": {
        "accessTokenEnv": "ACME_API_TOKEN"
      },
      "minTools": 4,
      "env": {
        "ACME_MCP_SERVER_URL": "${url}",
        "CLAUDE_PLUGIN_DATA": "${dataDir}",
        "ENABLE_HITL": "false"
      },
      "files": {
        "mcp-credentials.json": {
          "tokens": {
            "access_token": "${bearerToken}",
            "token_type": "Bearer"
          }
        }
      }
    }
  }
}

Stdio eval servers

Cowork accepts plain stdio servers entries without a URL, plugin, or proxy:

{
  "servers": {
    "local-tools": {
      "transport": "stdio",
      "command": "/usr/local/bin/node",
      "args": ["server.mjs"],
      "cwd": "/opt/mcp/local-tools",
      "env": { "NODE_ENV": "test" },
      "inheritEnv": false
    }
  }
}

Use paths that exist on the desktop client. command is required; args, env, and cwd are optional, and the key (local-tools) is the server's label. cwd must resolve to an absolute path. Desktop has no cwd field, so MST converts it to a fixed /bin/sh wrapper with cwd, command, and arguments passed as separate argv values, never interpolated into shell source. inheritEnv: true is unsupported: declare all required server environment variables in env. Omitting inheritEnv has the same Cowork behavior as false.

Client-resolved entries can also set url, auth.accessTokenEnv, files, and minTools (default 1). The readiness client accepts connectTimeoutMs, requestTimeoutMs, callTimeoutMs, and quiet; these are not Desktop settings. Unknown keys fail; Cowork does not currently accept the general MCP client's protocol or probe options. Supported placeholders are:

• ${url}: the declared url; allowed in command, args, env, cwd, and files.
• ${dataDir}: the private per-server directory; allowed in command, args, env, cwd, and files. On Linux it is <clientOptions.mcpDataRoot>/<label>; on macOS it is <transaction staging directory>/stdio/<label>.
• ${pluginRoot:<plugin>}: clientOptions.pluginRoots[<plugin>] on Linux; allowed in command, args, env, cwd, and files. <plugin> must be a declared plugin.
• ${bearerToken}: the value of auth.accessTokenEnv; allowed only in files.

Any other ${...} fails. If url is declared, include ${url} in the launch command, args, or env, not only in files or cwd. requireEvalEndpoint checks that declared endpoint; MST does not invent an endpoint for plain stdio. Resolved bearer tokens stay in private files, not settings, env, args, logs, or receipts. Tool calls appear as mcp__<label>__<tool> and are attributed to label (for example, mcp__acme-eval__search).

Both macOS (Computer Use) and Linux (the accessibility automation) support these stdio forms. The package-root export COWORK_STDIO_PLATFORMS is ['darwin', 'linux']. On macOS, ${pluginRoot:...} is rejected because marketplace installation paths are unknown before Desktop starts. This is not a native-proxy requirement: use a known local command or adapter path instead. For the Linux example above, a Mac config must omit pluginRoots and mcpDataRoot, which are Linux options, and replace the plugin-root argument with a known local adapter path. The private files and URL/env substitutions remain supported, including in mixed HTTP/stdio server sets.

ChatGPT (chatgpt and codex-app) and direct MCP clients support plain stdio, but not Cowork's client-resolved fields. For ChatGPT plugin overrides, use plugins[].mcp as described in chatgpt-desktop.md.

MST's transport does not make a server read-only. Native-proxy write interception is a client/catalog policy responsibility, not a stdio guarantee.

Dry-run proxy

To let a client use a real HTTP MCP server without writing to it, put MST's dry-run proxy in front of it. The proxy is a stdio MCP server. The client sees the upstream server's real tools. Read-only tool calls go upstream. Every other tool call returns a planned-write result and never reaches the server:

{
  "_mst_planned_write": {
    "server": "slack",
    "tool_name": "send_message",
    "arguments": {}
  }
}

It fails closed. A tool is read-only only when it is annotated readOnlyHint: true and not destructiveHint: true. A tool without annotations, or one the server does not list, is a write. readOnlyTools allows reads on a server that annotates nothing; alwaysWriteTools intercepts tools annotated as reads.

A server that handles writes itself, such as an eval endpoint that never changes real data, can still use the proxy for its token: with forwardWrites: true, every call goes upstream. MST then sees the server's writes only when the connector sets plannedWriteKey, the key the server puts in the result of each write it stopped: the proxy records each marked result (the server-planned write mode). Without it, the run doesn't start unless the eval config accepts the server's writes as unguarded (see Writes). A connector whose server runs other tools through one tool names it in dispatchers, with the argument that holds the target ({ run_tool: 'tool_name' }), so the record shows the target. Use the proxy for any connector whose session can outlast its access token. A token in an environment variable is read once, when the client starts, and is never renewed. requestTimeoutMs (default 120 seconds) bounds each request; raise it for tools slower than that, so the client's own tool limit applies instead.

The proxy connects to the upstream server before it serves its client. If that fails, or its token file isn't there yet, it tries again after 2 and 5 seconds, then exits; each failure goes to stderr with its kind (an error class, a network code or an HTTP status), never a token or response body.

dryRunProxyServer() (from @gleanwork/mcp-server-tester/evals) returns the stdio server entry. It holds the path of a token file, never a token. Pass it the launch context's writes unchanged: it says where the proxy records the server's writes and whether it simulates them. A launch that drops it, or changes whether to simulate, fails the run, because MST would see none of the server's writes:

import { dryRunProxyServer } from '@gleanwork/mcp-server-tester/evals';

// In a connector's launch(context):
const slack = dryRunProxyServer({
  label: 'slack',
  upstreamUrl: 'https://mcp.slack.com/mcp',
  tokenFile: context.tokenFile, // { "version": 1, "accessToken": "..." }
  writes: context.writes,
  minTools: 10,
});

The proxy re-reads the token file when it is replaced, so a renewed token takes effect on the next request without restarting the MCP session. After a 401 it waits up to 90 seconds for a new token, then retries once. If none arrives, it logs CONNECTOR_AUTH_EXPIRED to stderr. It never logs a token.

Simulated writes

A planned-write result tells the client its write didn't happen, and a model that knows that may retry, apologize or answer differently. To grade it as it behaves after a real write, set "simulateWrites": true in the eval config. The proxy then answers a write with a success reply. It still never forwards the write, and it records it. The reply, in order of preference:

1. The connector's template for the tool (writeReplies).
2. A minimal object that satisfies the tool's outputSchema, as structured content, when it declares one.
3. { "ok": true, "id": "<id>", "result": <the call's arguments> }.

In a template, a string that is exactly {{arguments.<path>}} becomes that argument's value; inside a longer string, its text. {{id}} is a fresh ID for the call, {{now}} the time (ISO 8601) and {{unixTime}} seconds with microseconds, as in Slack's ts:

const slack = dryRunProxyServer({
  label: 'slack',
  upstreamUrl: 'https://mcp.slack.com/mcp',
  tokenFile: context.tokenFile,
  writes: context.writes,
  // Used only when the eval config sets simulateWrites.
  writeReplies: {
    slack_send_message: {
      ok: true,
      channel: '{{arguments.channel_id}}',
      ts: '{{unixTime}}',
      message: { text: '{{arguments.message}}' },
    },
  },
});

After each trial, MST marks the trial's tool calls the proxy answered (same server, tool and arguments) simulatedWrite: true, in its transcript events and in the client response's events and toolCalls, so judges and reports know the write never happened; output is the reply. A write the proxy can't record gets the planned-write result instead: MST never tells a client a write succeeded without a record of it. The run's run.json has "simulateWrites": true. A regrade keeps it, and a resume collects the missing trials the same way, whatever the eval config now says. Turning the flag on or off starts a new era in the eval's history, since the client sees something else after a write.

The model can't read back what it "wrote": a search for the message it sent won't find it.

The write record

The proxy records every write it stops, planned or simulated, and every marked write a forwardWrites server stopped, in a private JSONL file MST gives each connector's launch (writes.file, mst.write-record/v1). It also records the tools it treats as writes each time it lists the server's tools, and each forwarded call it can't classify: a dispatcher's call, or a call to a tool that isn't read-only whose result has no planned-write mark.

After each trial, MST takes the trial's part of each file and matches it with the trial's transcript, then stores the result with the trial as its writeRecord. The writes grader and judges read it; see Writes. The files are deleted when the run ends.

macOS setup contract

MST validates the full server set before changing the app or profile. HTTP, plain stdio, and private-file-backed proxies all use the local Developer MCP surface, not the managed profile's MCP list. Even an empty eval config temporarily replaces the user's local MCP list. The profile transaction owns inference settings and private stdio files: directories use mode 0700 and JSON files use mode 0600. Credentials are resolved from the supplied runtime environment, not embedded in the eval config or public app configuration.

Setup returns transaction-owned stdioPaths, which MST's readiness check uses after installation. Do not supply Linux-only pluginRoots or mcpDataRoot options on macOS. Setup status remains applied-not-verified: installing settings or starting the app does not prove Desktop adopted the MCP inventory or tool policy.

Cleanup restores the original local MCP configuration before removing private stdio files. The private-file transaction validates inventory, ownership, permissions, and content hashes during cleanup and explicit recovery; it removes only recorded files and empty directories. Unexpected or changed state fails closed and retains recovery state. Do not delete locks or modify staged credential files to force cleanup.

Linux managed-settings contract

The caller stages each plugin from its pinned ref, writes the private files, and writes /etc/claude-desktop/managed-settings.json before it starts Claude Desktop. MST only reads and checks them. For the example above:

{
  "managedMcpServers": [
    {
      "name": "acme-eval",
      "transport": "stdio",
      "command": "/usr/bin/node",
      "args": ["/opt/example-app/plugins/acme/mcp/start.mjs"],
      "env": {
        "ACME_MCP_SERVER_URL": "https://mcp.example.com/eval",
        "CLAUDE_PLUGIN_DATA": "/config/mcp-data/acme-eval",
        "ENABLE_HITL": "false"
      }
    },
    {
      "name": "acme_plugin",
      "transport": "policy-only",
      "toolPolicy": { "*": "blocked" }
    }
  ],
  "isLocalDevMcpEnabled": false,
  "isDesktopExtensionEnabled": false,
  "disabledBuiltinTools": ["AskUserQuestion"],
  "allowedPluginMarketplaces": [
    {
      "source": "github",
      "repo": "acme/claude-plugins",
      "ref": "<40-character commit SHA>",
      "installationPreference": "required"
    }
  ]
}

And one private file, owned by the desktop-session user that runs MST and Claude Desktop:

/config/mcp-data/                                   0700
/config/mcp-data/acme-eval/                        0700
/config/mcp-data/acme-eval/mcp-credentials.json    0600
  {"tokens":{"access_token":"<ACME_API_TOKEN>","token_type":"Bearer"}}

prepare fails closed, before any UI action, unless all of these hold:

• Each stdio eval server is one managed entry named label with exactly transport: "stdio", the resolved command, args, and env. It has no other keys, except toolPolicy: {"*": "allow"} with coworkSetup.approveWriteTools. Its env has exactly the declared keys, so the substituted URL equals url, and plugin env such as ENABLE_HITL: "true" is replaced.
• Each blockMcpServers name is a policy-only entry with exactly toolPolicy: {"*": "blocked"}. Claude Desktop names plugin tools mcp__plugin_<plugin>_<server>__<tool>; the managed entry uses the plugin's own server name from its .mcp.json (for example, acme_plugin).
• HTTP servers match as before, and the pinned marketplace entries match.
• isLocalDevMcpEnabled and isDesktopExtensionEnabled are false (COWORK_MANAGED_ONLY), so Claude Desktop runs only the managed servers: no servers from the user's claude_desktop_config.json, and no desktop extensions.
• With coworkSetup.clarifyingQuestions: "disabled", disabledBuiltinTools contains AskUserQuestion (coworkHeadlessSettings() builds it), so Claude proceeds on its best assumption instead of asking. Other disabled tools are allowed. If a desktop still asks, the task stops as a verified failed case (clientTelemetry.awaitingUser) instead of waiting. By default (first-option) the tool may be on or off: HITL answers a question with its first option. The macOS eval profile MST writes always turns it off.
• Each plugin root is an absolute, real (no symlink), non-world-writable directory. mcpDataRoot and each <mcpDataRoot>/<label> are real 0700 directories owned by MST's user. Each files entry is a regular 0600 file owned by MST's user, with exactly the substituted JSON (the token from auth.accessTokenEnv in MST's environment).

Build the expected entries with coworkManagedPluginSettings({servers, plugins, paths: {pluginRoots, dataRoot}}) and check a file with coworkMcpSettingsMatch. Both are exported from the package root and never include a token. Append HTTP entries as before. materializeClientStdioFiles writes the private files for TS callers.

Readiness

MST runs the same readiness check for Cowork as for every client, with one difference: Cowork's stdio servers resolve only once the app is prepared, so MST checks them after preparation and before the first prompt, not before the batch starts. It launches each stdio eval server itself, with the same resolved command, args, env, and data dir (when used), but without the parent environment. On macOS it uses the paths returned by the setup transaction; on Linux it uses the caller-owned paths checked during prepare. HTTP servers get the same headers Cowork uses. Every server, stdio or HTTP, must connect and list at least one tool, and at least its minTools. When one doesn't, no prompt is sent: each trial of the batch ends as a retryable setup failure whose reason names the server, for example fake-eval listed 1 tool, fewer than its minTools (4). When a variant gives Cowork tool metadata, its servers go through MST's tool-variant proxy, which MST launches itself: MST checks them as declared before the proxy starts, as for any client, and Cowork doesn't check them again after preparation. An adapter with a bad token, for example, may list only its static tools. Desktop-side readiness (for example, a caller's own log check) should also compare each server's toolCount with minTools.

Plugin marketplaces

Cowork installs each plugin from a managed allowedPluginMarketplaces entry, pinned and required. An owner/repo source becomes source: "github"; an HTTPS Git URL becomes source: "git" with url. Local paths are rejected, because Cowork cannot read them. On macOS, MST writes these entries and the blocked policy-only entries into the MST-owned profile. On Linux, the caller writes them, and MST checks them before any UI action. Extra keys such as expectedName are allowed.

Why not plugins[].mcp on Cowork

Cowork rejects plugins[].mcp overrides with plugin_unsupported before any UI action. Claude Desktop 2.7032.0 has no managed way to give a plugin's own MCP server a custom endpoint, credential, or data directory:

• Client-bridged plugin stdio servers get only the plugin's declared env and CLAUDE_PLUGIN_ROOT. Placeholders expand only from HOME, LOGNAME, PATH, SHELL, TERM, and USER. ${CLAUDE_PLUGIN_DATA} is left unexpanded, and a server that uses ${user_config.*} is dropped.
• orgPluginSettings sets only per-tool permissions. A managedMcpServers entry with the same name decides only the tool policy.
• The desktop starts local plugin servers only when allowedPluginMcpServers is unset and local MCP is enabled. Local MCP is enabled when isLocalDevMcpEnabled is not false and the organization feature flag allows it. allowManagedMcpServersOnly is not an input to that check.

So declare the eval server as a stdio servers entry that runs the plugin's adapter, and block the plugin's own server with blockMcpServers.