Upgrading from a 2.0 prerelease

This guide covers what changed after 2.0.0-beta.7. If you're upgrading from 1.x, read Migrating from v1.x to v2.0 instead.

From an earlier prerelease, read the beta.7 migration guide first; it covers the changes between prereleases up to beta.7.

Names in older sections. Each section uses the names of the change it describes. Sections from Result fields use the eval vocabulary on renamed many of them: manifests are eval configs, suites are evals, hosts are clients (claude-cli is claude-code), arms are variants, runEvalSuite is runEval, runVariantExperiment is runToolOptimization, and hostTelemetry.hostApp is clientTelemetry.clientApp. The main guide has the full mapping. Sections before The mst client is now api call the api client mst, and sections before The Codex surface is the codex-app client call it chatgpt with surface: "codex". Sections before The Codex CLI client is codex-cli call the codex-cli client codex. Sections before One name for each client option call maxActions computerUseMaxActions or nativeMaxActions. Use the current names in new code.

These sections of the main guide are also new since beta.7:

• Custom judges are plugins
• Every judge runs the same way
• LLM calls: bearer tokens and streaming
• runToolOptimization needs clear evidence to recommend a variant
• Client cases name a client and model, not mcpHostConfig
• Direct cases are Playwright tests
• The simulator and the external-host runtime are internal
• Eval configs and variants

Runs record which client app and judge version produced them

Affects: code that reads clientTelemetry, run.json's judges or a result store's manifest.json; editors or validators that fetch a schema by its $id.

• Every built-in client records its app. Each trial's clientTelemetry.clientApp is { name, version, source } for the app that ran it: Claude Code's version from its startup event, the Codex CLI's from its session rollout, Cowork's and the ChatGPT and Codex apps' from the installed app (the macOS bundle, or the Linux package). Only Cowork recorded it before. The api client has no app: run.json's mst.version covers it. A client that can't tell its version records no clientApp.
• run.json's judges have a version. Beside each judge's identity, the version a person can read, as mst judges lists it. Runs from before have none.
• manifest.json names its kind. A stored run's manifest keeps "format": 1 and adds "kind": "manifest", with a schema at schema/run/v1/manifest.schema.json. MST still fetches runs whose manifest has no kind. A manifest with a format this MST doesn't know fails the fetch and says to upgrade MST.
• Schema IDs resolve. Every schema's $id is now https://unpkg.com/@gleanwork/mcp-server-tester@2/schema/<path>. The eval config's was a GitHub URL, the dataset's a GitHub Pages URL, and the run and history schemas' an unversioned unpkg URL; none of those returned the schema. @2 resolves once 2.0.0 is published. Until then, point $schema at @rc or at the file in node_modules.

Runs keep what a newer MST wrote; clients' events are checked

Affects: a plugin client whose transcript events don't match the TranscriptEvent type; code that reads mst.run/v1 files with the published JSON Schemas.

MST reads a run written by a newer MST and keeps the fields it doesn't know, at every depth, including through mst grade and --resume. A value it doesn't know in a field whose values the format names, such as a write's mode, a trial's ending or a transcript event's kind, fails the read and says to upgrade MST. schema/run/v1/ now describes each trial's transcript and each case result as well.

MST checks each client's transcript events against the stored shape when the client returns them. An event without a name, with arguments that aren't an object, with an output that isn't a string, or with a kind or source the type doesn't list fails that trial and names the field. Before, MST wrote such a trial into a run it couldn't read back. A durationMs that isn't a finite number is dropped. To migrate, return events that match TranscriptEvent: put a tool's output in output as text.

Score keys are assertion keys

Affects: code that reads a trial's scores or a run's scores/ directory, or uses the GraderType type; OpenTelemetry queries on a grader's name; a stored run or saved baseline from an earlier prerelease.

A code grader now records its score under its assertion's name, as CONTEXT.md and ADR 0013 name it. The report and the scores/<grader>/ directories use the same names. So does OpenTelemetry: the span name evaluate <grader>, gen_ai.evaluation.name and mst.grader.

Before	Now
textContains	containsText
regex	matchesPattern

GraderType drops exact, schema, snapshot and size. Nothing has written them since direct cases became Playwright tests. error stays: it is a trial's score when its grading didn't come back.

MST refuses to read a stored run or a baseline with the old keys, and the error names the new ones. Run the eval again. A baseline from a result store gives a warning, as any baseline it can't read does, and the run has no comparison.

The CLI runs without a terminal

Affects: scripts that run mst init, mst generate, mst login or mst token.

The CLI no longer uses Ink, so these commands no longer fail with Raw mode is not supported outside a terminal. In a terminal, init and generate still ask for what the flags leave out; the prompts are now numbered lists and (Y/n) questions. Without one, they take every answer from flags and fail, naming the flag, when one is missing:

• mst init --command "node server.js" or --url <url>, with --name, --install/--no-install and --force to write into an existing directory.
• mst generate --tool <name> --args '<json>', with --config, --url or --command, records one call. --name, --description, --suite and --exact set what the prompts did.
• --command keeps a quoted argument together, so --command 'node "/Users/a/My Server/index.js"' works. Before, every space split an argument.

mst token changes for CI scripts:

• It writes each value on one line. It wrapped long tokens at 80 columns when its output was piped, so >> $GITHUB_ENV stored a broken token.
• It exits 1, and writes to stderr, when no tokens are stored for the server. It exited 0.
• An unknown --format fails with the allowed values. It printed nothing and exited 0.

Every write is accounted for

Affects: eval configs whose servers list tools that aren't read-only without a dry-run proxy in front; plugin connectors whose launch returns dryRunProxyServer(), or that forward writes; code that reads mst.simulated-write/v1 files.

ADR 0014 gives every server a write mode, records every write the dry-run proxy stops, and grades write cases from that record. See Writes.

• A server MST can't see the writes of stops the run. A server with no dry-run proxy, or behind a proxy that forwards writes without the connector's plannedWriteKey, must list only read-only tools (readOnlyHint: true, not destructiveHint: true). Otherwise the run doesn't start: MST can't see these servers' writes: "docs" lists tools that aren't read-only (save, delete). Annotate the server's read tools, put it behind a connector's dry-run proxy, or accept its writes in the eval config: "writes": { "docs": "unguarded" }. A server whose writes should reach the service is "live", and needs mst run --allow-live-writes.

• Connectors pass context.writes to the proxy. dryRunProxyServer({ simulateWrites }) fails with what replaces it: pass the launch context's writes unchanged, and the reply templates as writeReplies. A launch that drops writes, or changes whether it simulates, fails the run instead of printing a warning.

  Before	Now
  context.simulateWrites in a connector's launch	context.writes, in every run
  dryRunProxyServer({ simulateWrites: { ...simulateWrites, replies } })	dryRunProxyServer({ writes: context.writes, writeReplies })
  mst.simulated-write/v1 file, simulated writes only	mst.write-record/v1 file, every stopped write

• A connector that forwards writes names its mark. Set plannedWriteKey on the connector to the key the server puts in each write it stopped, so MST records those writes (server-planned). Name tools that run other tools in dispatchers.

• Compared variants get the same kind of reply to a write. Variants whose servers stop writes in different ways (planned, simulated, server-planned) no longer run in one eval config.

• Write cases are graded from the record. List a case's writes in expected.writes (server, tool, partial arguments, required, order, exclusive) and the new writes grader checks the trial's write record. Every score of a write case has writeEvidence. A trial whose record and transcript disagree (unreported or unrecorded writes) fails its writes grader. A trial whose client failed keeps its record, and each write in it is unreported. A record with an unclassified call gives called evidence, not requested.

• A read-only server is held to the tools it listed. A trial that calls a tool its read-only server didn't list at the readiness check ends as a setup failure that --resume doesn't retry. Put a server whose tools depend on who signs in behind a dry-run proxy.

• Stored runs gain write data. Each stored trial has its writeRecord, and each variant in run.json its writeModes. A stored run without writeModes regrades as before; it has no write records to grade.

• Judge prompts changed. The built-in judges render the write record and say a planned or simulated write was requested, not made, so their versions changed. Agentic judges' workspaces have writes.json.

A transcript event names a tool as its server lists it

Affects: client plugins whose events set server and put the server label in name ({ server: 'docs', name: 'docs.search' }).

An MCP event that has a server names the tool exactly as that server lists it. MST no longer reads a <server>. prefix off that name, so a server's tool named docs.search stays docs.search rather than counting as search. A plugin that wrote { server: 'docs', name: 'docs.search' } for the tool search now reports a different tool: its write record shows an unreported write, and on a read-only server the trial fails as a call to an unlisted tool. Report { server: 'docs', name: 'search' }. MST gives an event with no server the server of a one-server run, as before; with several servers it can't tell which one the call went to.

What run.json records so runs can be compared

Affects: code that reads run.json; plugins with settings whose judges have calibrations; evals whose runs are collected on several machines (CI, --detach); evals with connector servers.

mst history is new since rc.0. It compares an eval's runs, and starts a new era wherever their setup differs: a server's endpoint or command under the same label, a server's write mode, a plugin's version or settings, a client option, or a judge's identity in a case. See The eval's history. So it can tell, run.json records more than in rc.0:

• Hashed client options under one key per eval. A client option that looks like a credential is still in run.json only as hmac:…. It is now hashed under the eval's key, option-key in its results directory, not a new key for each run, so the eval's runs record an unchanged value alike. run.json's optionKey is a fingerprint of the key. rc.0 runs hashed each under its own key and record no optionKey, so history compares their options by name only. The first run after upgrading starts a new era for each variant with such an option. Where runs are collected on several machines, such as CI, give each the same MST_OPTION_KEY (64 hex digits) from a secret, or every run starts an era. --detach sends the coordinator the key of the machine that started the run. Don't commit or upload option-key.
• A judge's identity covers its plugin's settings. A plugin judge whose plugin's settings are non-empty once parsed (defaults in its schema count) gets a new identity, so its calibrations go stale: run mst judges check again. mst grade --resume of a run graded before upgrading grades such a case judge's trials again. For such an eval config judge it refuses, as for any judge that changed: grade the run in full. Judges whose plugins take no settings, and the built-in judges, keep their identities.
• A regrade records its source run's setup. Its run.json has the servers, client, model, client options (with their hashes), tool metadata, input template, plugins and optionKey of the run that collected its transcripts, not the eval config's or this machine's now.
• A connector server is recorded without credentials. run.json records a connector server as its connector, label and url (the declared URL, or else the connector's default), without the URL's credentials, query or fragment. An rc.0 local run recorded a connector with a launch as the launched proxy's command instead. An rc.0 run on a runner recorded the URL as written, so if a declared connector URL held a credential, it is in that run's run.json, including in a result store: check those runs, and rotate the credential. A regrade of such a run doesn't copy it.
• New fields. run.json has plugins (namespace, name, version, and settings, a hash of the plugin's parsed settings; empty when the run loaded none) and optionKey.

Traces are transcripts

Affects: code that reads a result's trace, toolCallTrace or traceEvidence, or a stored run's traces/ directory; code that imports the trace types; question judges that set needsTrace; agentic judges that read trace/ in their workspace; calibrations of the built-in judges.

ADR 0013 renamed the record of what a client did in a trial from trace to transcript, Anthropic's word for it. "Trace" now means only an OpenTelemetry trace, as in storedRunTraces. None of the old names still works: stored runs, baselines and needsTrace fail and name their replacement, and TypeScript flags the renamed types and fields.

Before	Now
trace, toolCallTrace, traceEvidence on a result or trial	transcript, toolCallTranscript, transcriptEvidence
traces/<variant>/<case-id>/<trial>.json in a stored run	transcripts/<variant>/<case-id>/<trial>.json
Trace, TraceEvent, TraceEvidence, TraceSource	Transcript, TranscriptEvent, TranscriptEvidence, TranscriptSource
traceSource, traceConfidence, traceLimitations	transcriptSource, transcriptConfidence, transcriptLimitations
traced on a report trial	hasTranscript
needsTrace in defineQuestionJudge	needsTranscript
trace/events.json, trace/messages.json in a judge workspace	transcript/events.json, transcript/messages.json

A run stored by an earlier prerelease has a traces/ directory, and a saved baseline has trace fields. This MST can't read either, and says so: run the eval again.

The built-in task-completion, grounded-in-tools, tool-use, criteria and preference judges now render <transcript> in their prompts, so their versions changed. Labels you checked against the old versions are stale: run mst judges check again.

Environments are runners

Affects: commands and scripts that pass --env, --env-option or --grade-env, or run mst env; plugins with environments; code that imports the environment types or reads environment in run.json.

ADR 0013 renamed where trials are collected from environment to runner, because in Anthropic's vocabulary the environment is what the client acts on. The old names fail and name their replacement; none of them still works.

Before	Now
--env, --env-option, --grade-env	--runner, --runner-option, --grade-runner
mst env setup	mst runner setup
A plugin's environments, names such as acme/env/cloud-vm	runners, acme/runner/cloud-vm
EnvironmentDefinition, Environment, EnvironmentContext, EnvironmentKeep	RunnerDefinition, Runner, RunnerContext, RunnerKeep
machineEnvironment, dockerEnvironment	machineRunner, dockerRunner
env, envOptions in runEval, gradeRun and evalConfig options	runner, runnerOptions
environment and grading.environment in run.json	runner and grading.runner

A run stored by an earlier prerelease has environment and no runner. This MST can't read it, and says so: run the eval again.

A variant's coworkSetup applies in Docker and cloud-VM runs

Affects: Cowork evals with coworkSetup in a variant that run with --env docker, on cloud VMs, or in any other environment that collects shards with mst collect.

A shard gave its client the eval's coworkSetup, not the variant's merged over it, so a variant's toolSearch, clarifyingQuestions, approveWriteTools, disabledBuiltinTools and other coworkSetup fields were ignored there. Local runs were right. Now a shard merges them as a local run does. Variants that compared Cowork settings in such a run measured one setting twice; run them again.

On Linux, Cowork also no longer rejects an HTTP server with MST's own fields, such as requestTimeoutMs; it leaves them out of Claude Desktop's settings, as macOS does.

Case inputs must contain non-whitespace text

Affects: datasets for every client, including run-only plugins, whether loaded from files, returned by a plugin, or supplied in memory.

Empty inputs already failed dataset validation. Whitespace-only inputs now fail too, before any client opens or any trial is submitted, even when the blank case follows valid cases. Replace blank inputs with the intended prompt or remove those cases. Valid prompt text is preserved exactly; MST does not trim whitespace, normalize line endings or change Unicode. Direct native-session calls still reject blank prompts before submission.

Client collection uses sessions; runBatch is removed

Affects: plugin clients that implement ClientDefinition.runBatch and callers of the desktop built-ins' batch entry points.

Replace the client extension's runBatch(requests, context) with open(options, context). A definition may have run, open, or both. Run-only plugin definitions need no change; their datasets follow the input rule above. MST prefers open during local and worker collection.

options.config holds the fixed client options and model. options.input holds the servers and environment, without a prompt. Return a session with run(trial), reset(), optional diagnose(), and close(). Each trial has caseId, zero-based trial, and input: { prompt }. Return { result: ClientRunResult, continuation: 'continue' | 'reset' | 'stop' }. MST opens a separate session for different fixed settings.

Move setup into open and cleanup into close. If setup throws, clean up its partial state before rethrowing. Do not loop over cases or call context.reportResult or context.acquire from session operations. MST now owns those calls, readiness, reset and diagnosis, partial-result streaming, and final cleanup. Declared servers are checked before opening; Cowork's preparation-dependent servers are checked after opening. Retry delays and diagnosis deadlines do not change. A client definition that still has runBatch, even beside open, fails validation with this migration instruction.

The session types remain in @gleanwork/mcp-server-tester/evals. ClientRunResult, grader results and stored schemas do not change. runEvalBatch and collection batches are not removed. See Client sessions.

Cowork answers clarifying questions with their first option

Affects: Cowork evals on MST's Linux desktop (the Docker image and cloud VMs), prepared Linux desktops, and cases that failed as awaitingUser.

MST used to turn off Claude's AskUserQuestion tool in headless runs, so the model went on with its best assumption. ACE keeps the tool and answers with the first option, so the two measured different behavior. Now MST does the same as ACE on Linux. It leaves the tool on, and HITL reads the question's options from the task's transcript and clicks each question's first option. For a checked multi-select option, it presses the page's Submit/Next. A question whose form doesn't take its first option within 60 seconds, or that has no usable options, still fails the case as awaitingUser. computerUse.hitl.telemetry.questions_answered counts the answered pages.

• To keep the old behavior, set "coworkSetup": { "clarifyingQuestions": "disabled" } in the eval or a variant.
• Owned desktops (MST_DESKTOP_OWNED=1): MST removes AskUserQuestion from the image's disabledBuiltinTools, unless the eval's disabledBuiltinTools lists it.
• Prepared desktops: the managed settings no longer need "disabledBuiltinTools": ["AskUserQuestion"], except with clarifyingQuestions: "disabled". Settings that still turn it off keep working; Cowork just never asks.
• macOS: the eval profile still turns the tool off.
• Answers, tokens and time change for cases where the model asks.

Cowork has tool search on

Affects: Cowork evals, on MST's Linux desktop (the Docker image and cloud VMs) and the macOS eval profile, and comparisons of their tokens or time with earlier runs.

MST now writes Claude Desktop's toolSearchEnabled as true. Cowork lists each MCP server's tools by name and loads a tool's definition only when the model searches for it, as Cowork signed in to claude.ai does. Before, Desktop left tool search off in gateway mode, so every request carried every connector's tool definitions. Requests are smaller with many connectors, and a trace can now show ToolSearch calls. Token and time results change; answers can too.

• To keep the old behavior, set "coworkSetup": { "toolSearch": false } in the eval or a variant.
• Claude Desktop 1.21459.0 or later knows the setting; an older one ignores it.
• Through an LLM gateway, check that tool search works: the gateway must accept the tool-search-tool-2025-10-19 beta, or requests fail with HTTP 400, and a case's trace should show ToolSearch calls.

See docs/cowork-docker.md.

Linux desktop receipts omit the driver identity

Affects: code reading driver from SemanticDesktopTelemetry, including Cowork's submission and HITL receipts and ChatGPT or Codex app's native setup and submission receipts.

New receipts no longer write driver: 'linux-desktop'. Read the client from the trial's variant or case, and the OS from the run's environment metadata. Accounting, action counts, duration, planner and cost fields are unchanged. The public type keeps driver as an optional legacy field, and readers accept old receipts without rewriting stored files.

Cowork, ChatGPT and Codex app now use concrete macos.ts and linux.ts entry points internally. Product and worker OS select execution; this adds no user option or public runtime export.

Client listings include defaults; known environment OS is checked

Affects: consumers of mst clients text or JSON output, and evals using clients with requirements.os.

mst clients now prints the same requirements and option/default table as mst clients show, for every client. Its JSON entries now include options, the JSON Schema previously returned only by show --json. Literal runtime defaults use default. Dynamic defaults and omission semantics use x-defaultDescription; required and not declared distinguish required options from optional plugin options with no declared default. Text output omits removed options; their rejecting schemas remain in JSON. Update text parsers to use --json.

Timeout behavior is unchanged: omit timeout for no API timeout, or the existing default on other clients. 0 and null are not substitutes for omission. Desktop defaults still resolve on the trial machine.

MST now checks a client's requirements.os when loading an eval, including variants and case overrides. local means the executing machine; built-in docker means Linux. Choose a compatible environment if the new error names an unsupported OS.

Plugin environments may add os: 'macos' | 'linux' to EnvironmentDefinition when every trial machine uses that OS. MST validates the field when loading the plugin. Omit it for unknown or mixed machines; existing undeclared environments remain compatible. MST opens no machines to discover their OS. Run-only client plugins need no change.

Desktop clients expose sessions; batch entry points remain

Affects: code implementing or calling ClientDefinition.

Cowork, chatgpt and codex-app now implement open(options, context). It fixes the client options, model, servers and environment for one app session. session.run({ caseId, trial, input: { prompt } }) returns the unchanged ClientRunResult plus a typed continuation: continue, reset, or stop. The session also has reset(), optional diagnose(), and close(). See Client sessions for the types exported from ./evals and the readiness and lifecycle metadata.

This earlier step added a transitional contract, before dispatch migrated. The desktop clients retained thin runBatch wrappers, and plugin validation still required run or runBatch. Those entry points are now removed; use the session migration above. Run-only plugins need no changes. Readiness order, reset and diagnosis policy, leases, cleanup, reporting, concurrency and retry timings are unchanged.

Built-in clients return the common result

Affects: code importing MstClientSimulationResult, custom judges reading a Playwright eval's response, and code reading the API client's conversation or timing fields.

MstClientSimulationResult is removed from the root and ./types exports. Built-in clients now return ClientRunResult from @gleanwork/mcp-server-tester/evals, with finalText and ordered events instead of a simulation envelope. LLMToolCall remains exported for native trace readers. The internal simulator interface, simulateMstClient entry point and result-conversion functions are deleted.

MST derives the assertion/report response from the common result. Playwright API evals now expose the same events, evidence, and tool-call kind/source fields as mst run. Custom judges on those Playwright evals see this common view. The existing success, response and toolCalls fields remain; grader rules, error classifications and trial endings do not change. Existing custom executeCase implementations can still return their assertion view.

The API client's input and conversation history move to response.telemetry.content; MCP timing moves to response.telemetry.mcpDurationMs. llmDurationMs stays on the response. Stored results drop telemetry.content unless redactStoredResponses: false. Existing skill-load measurements retain their meaning and per-trial fields; both execution paths now carry them through the common result. Old stored runs remain readable without rewriting them.

ChatGPT results don't record a driver

Affects: code that reads driver, driverSlug, capabilitiesUsed, displayName, clientName or clientType from the clientMetadata of a chatgpt or codex-app result (in clientTelemetry.clientMetadata, or a stored trial's clientMetadata), and code that uses the ClientMetadata type from @gleanwork/mcp-server-tester/experimental/clients.

The ChatGPT app's clients run as ordinary client code (ADR 0011). The driver identities (openai.chatgpt.agent.desktop-app.macos and .linux) and the capability bindings they named are gone, and new results stop writing the fields that only named them. The client, the environment and the OS of the machine that ran the trial already say how it ran. What the client recorded about the trial itself is unchanged: correlation, session and turn, evidence, failure kind, and the computerUse and nativeController receipts.

• New results. clientMetadata has no driver, driverSlug, capabilitiesUsed, displayName, clientName or clientType. Read the client from the trial's variant or case, and the OS from environment in run.json: its platform, or each worker's in machines.
• The type. Those fields of ClientMetadata are optional, documented as recorded by earlier versions only. ClientDriverId, ClientCapability and ExternalClientType stay exported, for the old values.
• Stored runs. A run stored before this change keeps these fields, and MST doesn't rewrite it. mst grade, mst run --resume and everything else that reads a stored run accept them, and a failed trial still ends as its failure kind says.
• Messages. A ChatGPT step that throws instead of returning its failure now fails the trial with ChatGPT trial failed: <reason> (failure kind client_run_failed, as before), not External client capability failed: <reason>.

MST checks every client's servers before the first trial

Affects: evals on api, claude-code, codex-cli and plugin clients whose servers don't start, can't be reached or serve no tools; anything that sets MST_COWORK_MCP_PREFLIGHT_RETRY_S; code that reads ChatGPT's batch telemetry mcpPreflight or nativeReadiness.mcpPreflight, or the Linux setup code mcp_preflight_failed.

One readiness check runs before the first trial of each batch, for every client (ADR 0011): each server connects and lists at least one tool, and at least its minTools for a stdio server. Before, only Cowork and ChatGPT checked, each in its own way, and only Cowork retried. See Server readiness.

• Trials against an unready server don't run. On api, claude-code, codex-cli and plugin clients, a trial against a server that doesn't connect, serves no tools or serves fewer than its minTools used to run, and usually failed as a trial. Now no trial of its batch is sent, and each ends as an infrastructure failure with broke: 'setup' and retryable: true, and a reason that names the server, for example MCP server not ready, so no trial was sent: search listed no tools. Checked 3 times. Its case is incomplete instead of failed, so pass rates on such runs change, and mst run --resume collects those trials again.
• Desktop failures are results. A Cowork or ChatGPT batch whose server wasn't ready used to stop a local run, and on a worker its trials came back missing. Now each trial is the same setup failure, locally and on a worker, and the run goes on with its other batches.
• Retries for every client. A server that isn't ready is checked again after 10 seconds and then 30 more; only the servers that failed are checked again. Rename MST_COWORK_MCP_PREFLIGHT_RETRY_S to MST_MCP_READINESS_RETRY_S (comma-separated seconds; empty turns retries off). A run that still sets the old name fails when it starts, and the message names the new one.
• One check per batch. MST checks on the machine that runs the trials, before the batch starts and before anything touches a desktop app. Cowork's stdio servers resolve only once the app is prepared, so its servers are checked after preparation instead, and not twice. A batch whose servers go through the tool-variant proxy (a variant with tool metadata) is checked at its declared servers before the proxy starts, Cowork's included, and its minTools apply; a proxy that still can't reach a server ends the batch the same way instead of stopping the run. MST's check no longer uses input.checkServers. ChatGPT's own preflight is gone: MST checks before it stops your app on macOS, and the Linux setup no longer connects to the servers itself, so mcpPreflight leaves its telemetry and mcp_preflight_failed its setup codes. The Linux app-server status probe, Claude Code's MCP startup check and Codex CLI's required servers are unchanged.
• A time limit for the whole probe. Each server gets 30 seconds to connect, list its tools and close; before, only listing tools was timed. A server that sets a longer connectTimeoutMs gets that to connect, plus 30 seconds. A server that takes longer fails the check.
• Not checked. A stdio server with client-resolved fields (url, auth, files, ${...} placeholders) is checked only by Cowork, after preparation. runEvalDataset and runEvalCase in a Playwright test aren't checked: the test's fixture already connected.

The Codex CLI client is codex-cli

Affects: eval configs, variants, datasets and cases that name the codex client, and code that passes client: 'codex' to runEvalDataset or runEvalCase.

The built-in client that runs the Codex CLI headless is codex-cli (ADR 0011). There are two Codex products: the CLI, and the Codex surface of the ChatGPT desktop app, which is codex-app. codex could mean either. The client's options, its requirements, @openai/codex-sdk and the Docker image don't change, and the judges' runtime: "codex" keeps its name: it is a judge runtime, not a client.

• Configs and cases. Change "client": "codex" to "client": "codex-cli", and mst/client/codex to mst/client/codex-cli. An eval config, a variant or a case that names codex fails with Client "codex" is now "codex-cli". For the Codex surface of the ChatGPT desktop app, use "codex-app"., and so does runEvalDataset or runEvalCase given client: 'codex'.
• Results. mst clients lists codex-cli, and messages from the client name it, such as codex-cli client timed out after 300000 ms.
• Docker. codex-cli is headless, as codex was: its shards run in MST's Linux image without a desktop.
• Stored runs. A run stored before this change keeps client: "codex" in its files, and MST doesn't rewrite them. Everything that reads a stored run reads it as codex-cli, with its options: mst grade, mst run --resume, mst label, mst judges check, the report mst open builds, the previous-run comparison, storedRunTraces, and a tool optimization started before the change. A report built before the change still shows codex until it is built again.
• Telemetry. OTLP spans from the client say codex-cli: the span is invoke_agent codex-cli, and mst.client and gen_ai.agent.name are codex-cli. Codex's own exporters (codex.* records) are unchanged. A collector query that filters on codex matches only spans exported before the change, so match both names across older data.
• Batches. Changing a config from codex to codex-cli changes its contentHash, so mst batch runs that eval again, as it does after any config change.

Client options that had no effect fail

Affects: eval configs, variants, datasets and cases that set clientOptions.server or clientOptions.servers on api or claude-code, clientOptions.apiToken, pluginDir or pluginMcpUrl on claude-code, or clientOptions.provider on cowork, chatgpt or codex-app; code that passes these options to runEvalDataset or runEvalCase.

An option that a client can't use fails validation (ADR 0011). These options were accepted and did nothing, so removing them doesn't change what a run does.

• Servers come from the eval. api and claude-code run with the eval config's servers (or a variant's), and in a Playwright test with the test's MCP connection. Their server and servers options never changed that: every run replaced them with the eval's servers. Remove them, and put the servers in servers.
• Claude Code's legacy plugin settings are gone. apiToken, pluginDir and pluginMcpUrl built a Claude Code MCP config that every run replaced with the eval's servers; pluginDir could still write mcp-server-url.json under ~/.mcp-server-tester/plugins, and it no longer does. The MCP_SERVER_URL, MCP_ACCESS_TOKEN and MCP_PLUGIN_* variables they fell back to no longer reach Claude Code's config either. Put a server's token in its own auth (accessToken or accessTokenEnv), and add a plugin's MCP server to servers.
• A desktop client fixes its provider. cowork took only provider: "anthropic", and chatgpt and codex-app only provider: "openai"; nothing read it. Remove it. Their credentials and LLM gateways come from the environment, as mst clients show <client> lists. provider on api and claude-code is unchanged: it picks the API.
• Messages. Each option fails on an eval config, a variant or a case, and the message names the option and says what to do instead: the servers message names the eval config's servers, and the provider message names the provider the client fixes.
• Shared defaults. An eval config's shared defaults (provider, timeout and the rest) no longer reach a client whose schema rejects that option, a plugin client's included. An eval config that sets provider and whose only clients are desktop clients now fails, saying none of its clients takes it.
• Stored runs. A run stored before this change may keep these options in its variants' clientOptions, and MST doesn't rewrite it. Everything that reads a stored run leaves them out, so the run compares equal to a config written today: mst grade, mst run --resume, the previous-run comparison, and a tool optimization started before the change, whose rounds still match its baseline.

One name for each client option

Affects: eval configs, variants, datasets and cases that set clientOptions.computerUseMaxActions or clientOptions.nativeMaxActions on cowork, chatgpt or codex-app, or a reasoningEffort the client doesn't support; datasets whose cases set their own model or clientOptions on those three clients.

One idea has one option name, and a config that can't run fails when it loads (ADR 0011).

• The action budget is maxActions. On cowork, rename computerUseMaxActions to maxActions: 1 to 64, default 24, on macOS and Linux. On chatgpt and codex-app, rename computerUseMaxActions (macOS) and nativeMaxActions (Linux) to maxActions. Its range and default follow the trial's OS: 1 to 64, default 32, on macOS; 1 to 64, default 24, on Linux. A value outside that range fails when the eval config loads, if MST knows the OS (as for the other options one OS uses), and otherwise on the trial's machine, before the app is prepared. The old names fail on an eval config, a variant or a case, and the message names maxActions. Cowork's hitlMaxActions keeps its name: it is a separate budget, for a case's human-in-the-loop inspections after its prompt is sent.
• reasoningEffort has one list of values: minimal, low, medium, high, xhigh, max, ultra. Each client takes the ones it supports: codex takes minimal to xhigh, and chatgpt and codex-app take low to ultra. A value of the list a client doesn't support fails, and the message names the ones it accepts; any other value fails as unknown. The values each client accepted haven't changed. mst clients show <client> lists them. api and claude-code have no reasoning effort option.
• A case can't change a desktop client's settings. cowork, chatgpt and codex-app run a variant's cases in one app, with the variant's settings. A case that sets its own model or a clientOptions value that differs from its variant's now fails when its dataset loads, before anything runs, naming the case and the field, and so does a case that names another client. Before, these failed only when the batch started, except that in an environment (--env) a case naming another client ran in that client's own batch; it now fails there too, as it did locally. In a variant on another client, cases that name one of these clients with different settings now run in separate batches in an environment too, as they did locally; before, their shared batch failed when it started. Use a variant for a different setting. A case that repeats the variant's settings still runs.
• timeout and appVersion are unchanged. Every client's timeout is the trial's time limit in ms: none by default on api, 180000 on claude-code, 300000 on codex, chatgpt and codex-app, and 900000 on cowork. appVersion is Cowork's; ChatGPT's version is the Docker environment's chatgptVersion.
• Stored runs. A run stored before this change keeps computerUseMaxActions or nativeMaxActions in its variants' clientOptions, and MST doesn't rewrite it. Everything that reads a stored run reads either as maxActions, so the run compares equal to a config written today: mst grade, mst run --resume, the previous-run comparison, and a tool optimization started before the change, whose rounds still match its baseline. A stored ChatGPT variant with both names reads as computerUseMaxActions, the budget it ran with on macOS.

The trial's OS decides how a desktop client runs

Affects: eval configs, variants, datasets and cases that set clientOptions.computerUseProvider on cowork, chatgpt or codex-app; Cowork and ChatGPT runs with --env docker; code that reads run.json's environment.machines.

How a desktop client runs follows from the OS of the machine that runs the trial, not from configuration (ADR 0011). On macOS, cowork, chatgpt and codex-app use Computer Use, with computerUseModel as its planner. On Linux, they use the accessibility (AT-SPI) automation, which has no planner. Before, Cowork's computerUseProvider defaulted to the OS of the machine that read the config, so a run from macOS with --env docker sent the macOS value to its Linux workers, which rejected it.

• Configs and cases. Remove computerUseProvider. Set on an eval config, a variant or a case, on any of the three clients, it fails, and the message says the OS of the trial's machine decides.
• Options one OS uses. On Cowork, pluginRoots and mcpDataRoot are Linux options, and computerUseModel and appVersion are macOS options. On chatgpt and codex-app, computerUseModel and computerUseMaxActions are macOS options, and nativeMaxActions and plugins are Linux options. One the trial's OS can't use fails when the eval config loads, if MST knows that OS: this machine's for --env local (the default), and Linux for --env docker. A plugin's environment checks on its worker, before any app is prepared. Before, Cowork checked these against computerUseProvider, and ChatGPT rejected some of them only when its app was being prepared; nativeMaxActions on macOS was ignored. Move such an option to a variant that runs on the OS that uses it.
• Results. Each shard's machine in run.json (environment.machines) records platform, the OS its worker ran on, and the report shows it under Run details. A local run's OS is environment.platform, as before. At this earlier step, telemetry also recorded driver: "linux-desktop" on Linux and provider: "anthropic-computer-use" in ChatGPT's macOS Computer Use telemetry. New Linux receipts omit the driver identity; use the recorded environment and OS instead.
• Stored runs. A run stored before this change keeps computerUseProvider in its variants' clientOptions, and MST doesn't rewrite it. Everything that reads a stored run leaves it out, so the run compares equal to a config written today: mst grade, mst run --resume, the previous-run comparison, and a tool optimization started before the change, whose rounds still match its baseline. A stored machine has no platform.

The Codex surface is the codex-app client

Affects: eval configs, variants, datasets and cases that set clientOptions.surface on the chatgpt client.

The ChatGPT desktop app's Codex surface is its own client, codex-app (ADR 0011). An option no longer changes which product a client tests: chatgpt always drives ChatGPT Work. The two clients drive the same app, with the same options, setup and Docker image, and they share the app's desktop lease, so a chatgpt variant and a codex-app variant never run at once. codex-app is not the codex client, which runs the Codex CLI.

• Configs and cases. Change "client": "chatgpt" with "clientOptions": { "surface": "codex" } to "client": "codex-app", and mst/client/chatgpt to mst/client/codex-app, and remove surface. Remove "surface": "chatgpt-work" too: it is what chatgpt does. surface on either client, on an eval config, a variant or a case, fails before any app starts, and the message names the client to use. The eval config and its variants fail when the config loads; a case fails when its dataset loads, before the run collects a trial.
• Docker. mst env setup docker --config <eval> builds the ChatGPT image for a codex-app eval, and --env docker runs its shards in that image. chatgptVersion applies to it.
• Results. A codex-app case records request.client as codex-app. Its errors name the Codex app, for example Codex app desktop is locked by another run. mst clients lists codex-app.
• Stored runs. A run stored before this change keeps client: "chatgpt" and clientOptions: { "surface": "codex" } in its files, and MST doesn't rewrite them. Everything that reads a stored run reads such a variant as codex-app, without surface in its options, and that variant's case results as codex-app: mst grade, mst run --resume, mst label, mst judges check, the report mst open builds, the previous-run comparison, storedRunTraces, and a tool optimization started before the change, whose rounds on codex-app still match its baseline. A stored chatgpt without surface: "codex" stays chatgpt, and a stored surface: "chatgpt-work" is dropped the same way. A shard's machine records only the client's name: it reads as codex-app when every chatgpt variant of its run was the Codex surface. A report built before the change still shows chatgpt until it is built again.
• Telemetry. OTLP spans from a codex-app run say codex-app: the span is invoke_agent codex-app, and mst.client and gen_ai.agent.name are codex-app. A collector query that filters on chatgpt for Codex runs matches only spans exported before the change.
• Batches. Changing a config from chatgpt with surface: "codex" to codex-app changes its contentHash, so mst batch runs that eval again, as it does after any config change.

The mst client is now api

Affects: eval configs, variants, datasets and cases that name the mst client, and code that passes client: 'mst' to runEvalDataset or runEvalCase.

The built-in client that runs the model alone, through its provider's API, with the servers' tools, is api (ADR 0011). mst named the tool, not what the client is. The mst CLI, the mst/ plugin namespace and the client's options don't change.

• Configs and cases. Change "client": "mst" to "client": "api", and mst/client/mst to mst/client/api. An eval config, a variant or a case that names mst fails with Client "mst" is now "api"., and so does runEvalDataset or runEvalCase given client: 'mst'. The older names vercel-sdk and anthropic-api now name api.
• Results. A case that names no client in a Playwright test records request.client as api, not mst. mst clients lists api.
• Stored runs. A run stored before this change keeps client: "mst" in its files, and MST doesn't rewrite them. Everything that reads a stored run reads it as api: mst grade, mst run --resume, mst label, mst judges check, the report mst open builds, the previous-run comparison, storedRunTraces, and a tool optimization started before the change, whose rounds on api still match its baseline. A report built before the change still shows mst until it is built again.
• Telemetry. OTLP spans from the client say api: the span is invoke_agent api, and mst.client and gen_ai.agent.name are api. A collector query that filters on mst matches only spans exported before the change, so match both names across older data.
• Batches. Changing a config from mst to api changes its contentHash, so mst batch runs that eval again, as it does after any config change.

Plugin clients' description and requirements are checked

Affects: plugin clients that already set a description or requirements field of their own.

A client definition can now say what it is (description) and what a machine needs to run it (requirements), and mst clients prints both. MST checks their shape when it loads a plugin, and a plugin whose client gets one wrong fails to load, with the reason:

• description must be a string.
• requirements must be an object with only os, install, credentials and setup.
• os must be a non-empty list of macos and linux. Leave it out for a client that runs on any OS.
• install, credentials and setup must be lists of strings.

Before, MST ignored both fields. A client that used either name for something else renames it, or changes it to this shape.

MST redacts every client's result

Affects: plugin clients that return telemetry, code that reads clientTelemetry, and ChatGPT runs.

• Secrets. MST replaces each secret it knows in everything a client returns, and in the error its run throws, before it grades or stores them. Before, it redacted only error and an infrastructure reason. Since the answer and tool calls it grades are redacted too, MST now replaces only values it can tell are secrets: the credentials it uses, and values whose names say they are credentials, from 8 characters. A name says so when one of its words, split at _, -, ., camelCase (an acronym too) and the digits that end it, is token, key, apikey, secret, password, passwd, auth, authorization, cookie, credential, credentials or pat (ACME_API_KEY, GITHUB_PAT, Authorization, APIToken, GITHUB_TOKEN2; not GIT_AUTHOR_NAME or AUTH0_DOMAIN). That covers environment variables, secrets file entries, server headers and stdio servers' env entries alike. Before, every secrets file value and every header value counted, and named environment values counted from 4 characters. Rename a credential stored under another name, such as ACME_PIN or GITHUBTOKEN, so MST replaces it.
• Telemetry content. Stored results drop clientTelemetry.content unless they keep full traces (redactStoredResponses: false). Put text that may quote the prompt, the model or the server there.
• ChatGPT. A trial's conversationHistory moves from clientTelemetry.conversationHistory to clientTelemetry.content.conversationHistory, so a redacted run no longer stores the prompt and the answer in it.

Clients say how an attempt ended

Affects: plugin clients that return an error, runs where a client errored, and code that reads ending or infrastructureError.

MST no longer decides from an error's words whether an attempt was an infrastructure failure. The client that ended the attempt says so (ADR 0010).

• Plugin clients. An error that says 429, 503, ECONNRESET or "prompt is too long" used to make an infrastructure failure. Now an error alone is a trial that failed. To mark one, return ending: { kind: 'infrastructure', broke, retryable, reason }, or set diagnostics.failureKind. A thrown error is a network failure only when it, or its cause, has a Node network code. See Clients.
• The mst client. A provider's rate limit, 5xx or overload is a retryable provider failure, and a failed connection a network one. Before, only a 429 or 503 was. A key the provider rejects, a model it doesn't serve, or a missing provider package is a setup failure a retry can't help: the case is incomplete, and mst run --resume says it needs a fix. Before, these were failed trials.
• Context overflow. A prompt that doesn't fit before the model acts is still setup, not retryable. One that fills the context after the model acted is now a trial that failed; before, it was setup too. Claude Code, Codex and the mst client each read their own provider's errors to tell them apart.
• Claude Code and Codex. A rate limit or other API error they report is provider. Claude Code's turn or budget limit is a trial. So is either client running out of the case's timeout after it started (Claude Code's init, Codex's first turn), its own time; before, that was a retryable deadline failure that mst run --resume collected again. A timeout before it started is still a deadline. Its failureKind is still timeout.
• Cowork. A Computer Use submission that stopped for a permission, a sign-in or a model setting the desktop needs is setup, not retryable. One the planner's provider limited is provider, and one MST's deadline stopped is deadline. Before, every failed submission was a retryable client-app failure.
• Old runs. A stored trial stays an infrastructure failure or a trial, as it was recorded, and its failure kind says what broke. One with no failure kind is a retryable client-app failure, whatever its error says, and so is a Cowork workspace failure stored before ending.

A case short a trial is incomplete

Affects: code that reads a case's pass, a run's passed, failed, total or pass rates, isInfrastructureError or infrastructureError, and runs resumed with mst run --resume.

Each attempt at a case now ends as a trial or an infrastructure failure, and the trial records it as ending (ADR 0010).

• Incomplete cases. A case with a trial that ended in an infrastructure failure, or with one its graders couldn't score, is incomplete. It neither passes nor fails: pass is false, it's counted in incomplete, and it's left out of passed, failed and pass rates, which counted it before. A case with every trial lost to infrastructure counted as failed; it's incomplete too.
• total. runEvalDataset's result and each variant's result still count every case in total, so expect(result.passed).toBe(result.total) fails while a case is incomplete. A run summary's metrics.total counts complete cases only, as it did for cases with missing trials. mst run and mst batch exit 1 while a case is incomplete.
• Ungraded trials aren't infrastructure failures. A trial a judge couldn't score has infrastructureError: false and ending: { "kind": "trial" }. It's still left out of pass rates, and mst grade --resume grades it.
• --resume collects more. It collects every infrastructure failure a retry can help, not just missing and unsubmitted trials. One a retry can't help is kept, and the run says it needs a fix.
• Old runs. A run stored without ending gets one from its old fields when it's read. Its trials keep their classification, except ungraded ones.

Tool optimizations on an eval are stored

Affects: code that calls runToolOptimization({ evalConfig }), or reads the eval's latest.json after it.

An optimization on an eval config is now stored in the eval's optimizations/<id>/, as mst optimize stores it (ADR 0008), so these things change:

• The baseline runs on its own. Static variants used to run in one run with the baseline. Now the baseline is one run and each round another, so the baseline's metric is checked before any candidate runs.
• Its runs don't become the latest. Each run's run.json has optimization (its id and role), and latest.json keeps pointing at your last ordinary mst run. Open an optimization's runs by id, with mst open <run-id>.
• Candidates are checked first. A tool key that names no served tool, a change to an input schema beyond its description and title text, or a repeated id fails before the round runs. So does a server whose tools changed since the baseline, or a change to the datasets, the eval config's judges or the base variant's client, model or options.
• proposeVariants sees the stored evidence. The callback's history is read from the runs on disk, as it is when the optimization is opened again. When the eval redacts stored responses (the default), its trials have no responses, final answers, tool call outputs or judge metadata; their tool calls and arguments, and each judge's score, pass and reasoning, stay. Set "redactStoredResponses": false in the eval config for a callback that reads them.
• Its runs aren't resumed. mst run --resume refuses an optimization's run, because resumed trials would run against tools nobody checked. A round whose run is missing trials is run again (mst optimize try --retry).

maxVariants caps the candidates across rounds on both forms, and evalConfig takes env and envOptions. See optimize.

The Docker environment picks its image and container by client

Affects: runs with --env docker, or a plugin environment built on it (such as cloud VMs), whose client isn't Cowork; and plugins whose environment builds MST's image.

• ChatGPT evals get their own image. mst env setup docker with a ChatGPT eval ("client": "chatgpt", directly or through extends) now builds mst-chatgpt-linux:<MST version> from docker/chatgpt-linux, and its shards and grade shards run in it. Before, setup built the Cowork image even then. See docs/chatgpt-docker.md.
• Headless clients start without the desktop. A collect shard whose clients are all headless (mst, claude-code, the new codex client, or a plugin client with headless: true on its ClientDefinition) runs in a container of the Cowork image without Claude Desktop, VM devices or Cowork inference, as grade shards do. ANTHROPIC_* settings are no longer needed for those runs, and several such shards share one Docker context. Before, every shard booted Claude Desktop. Put the client's own credentials in the run's --secrets-file.
• Environment hooks get the client. An environment's open, setup and coordinator context has client, the eval's client as its extends resolve it. A plugin environment that builds or picks an image passes it on to the docker environment.
• A detached run's file datasets. The coordinator no longer looks for a file dataset's path on its own machine; it uses the datasets shipped with the run. Before, a relative path failed there and the run stayed starting until it was lost.

See docs/cowork-docker.md and docs/codex.md.

The dry-run proxy tries its upstream again at startup

Affects: connectors whose launch returns dryRunProxyServer().

A dry-run proxy that can't reach its upstream server when it starts, or can't read its token file yet, now tries again after 2 and 5 seconds before it exits, and logs each failure's kind to stderr. Before, it exited at once and its client saw only "connection closed". A client that gives up on a server sooner than about 10 seconds still sees the failure. See Dry-run proxy.

Cowork on the Linux desktop reaches public hosts

Affects: Cowork evals on MST's Linux desktop (the Docker image and cloud VMs).

Cowork's web fetch, and the commands it runs in its VM, can now reach any public host. MST writes Claude Desktop's coworkEgressAllowedHosts as ["*"] unless the image or the eval names hosts. Before, the image set no list, so every fetch failed ("not on the network allowlist"), even of pages a prompt linked. Results on cases that fetch web pages can change.

It's one list, by host, for both web fetch and the VM's commands, so it can't allow reads only. Private and local addresses stay blocked. MCP servers aren't affected: their writes still go through the dry-run proxy.

• To keep the old behavior, set "coworkSetup": { "egressAllowedHosts": [] } in the eval or a variant.
• To allow only some hosts, list them: ["docs.example.com"].
• To turn off a Cowork built-in tool, set coworkSetup.disabledBuiltinTools. For example, ["WebSearch"] through an LLM gateway whose providers can't run Anthropic's server-side web search, where every search returns a 400.

See docs/cowork-docker.md.

macOS Computer Use needs no Python

Affects: Cowork and ChatGPT runs on macOS, and code or scripts that used the ./cowork-runtime or ./cowork-requirements exports, MST_COWORK_PYTHON on macOS, or MST_COWORK_DRIVER_ROOT.

The macOS Computer Use driver is now TypeScript and runs in the MST process. MST no longer creates a Python environment or installs anthropic, pyautogui, mss, Pillow or pyobjc. For mouse and keyboard input it compiles a small Swift helper, which needs the Xcode Command Line Tools (xcode-select --install), as MST's other macOS helpers already do. The terminal that runs MST still needs Accessibility and Screen Recording permission.

• The package exports ./cowork-runtime (the Python driver) and ./cowork-requirements are gone. Nothing replaces them: the driver is part of MST's code.
• MST_COWORK_PYTHON no longer applies on macOS. It still selects the interpreter for the Linux driver.
• MST_COWORK_DRIVER_ROOT is gone. It pointed MST at a checkout's Python driver.
• A failed run's error now says stopped before completion; not retrying or deadline reached before completion; not retrying, in place of driver exited unsuccessfully and driver terminated before completion. A blocker or provider code, such as (model_unavailable), still follows.
• A scroll to the left or right now scrolls sideways. The Python driver scrolled up.
• A key name the driver doesn't know is reported to the planner and not sent. The Python driver ignored it without saying so.

Result stores hold whole runs

Affects: eval configs with results.store, plugins with resultStores, and code that read an eval's store with loadStoredEvalRunnerResult or listArtifacts.

A result store set in an eval config, or chosen with the new mst run --results <store>, now gets a copy of each run directory: <eval name>/runs/<run-id>/ and <eval name>/latest.json, the same layout as .mcp-test-results/. It no longer gets eval-runs/ and eval-summaries/ artifacts, and the run summary no longer has caseArtifactPointers. The previous run and mst batch --skip-existing read the store's runs. Runs stored by earlier prereleases aren't read; the first run after the upgrade has no previous run to compare with.

A file store now holds every file of each run, results.json and summary.json included. If its dir is inside the directory MST writes runs to (such as mst batch --output-root), each run is there twice: once where MST ran it and once under <eval name>/runs/. A script that collects runs by searching that directory for results.json finds every run twice. Point the store at a directory of its own, or drop it when the local run directory is all you need.

A plugin's result store returns a RunStore (location, put, get, list) from create, not an artifact store:

// Before
resultStores: { bucket: { schema, create: (options) => new MyArtifactStore(options) } }
// After: MST owns the layout; the store moves bytes
resultStores: {
  bucket: {
    schema,
    create: (options) => ({
      location: `s3://${options.bucket}`,
      put: (key, body) => putObject(options.bucket, key, body),
      get: (key) => getObjectOrNull(options.bucket, key),
      list: (prefix, { shallow } = {}) => listKeys(options.bucket, prefix, shallow),
    }),
  },
}

A store that still returns saveArtifact fails with what to change. The artifact stores runEvalDataset and the Playwright reporter take (resultStore, FileEvalResultStore, GCSEvalResultStore) are unchanged.

Linux Cowork settings use Claude Desktop's own keys

Affects: Linux Cowork desktops prepared by the caller (/etc/claude-desktop/managed-settings.json), and callers of coworkManagedPluginSettings.

Claude Desktop ignores its whole managed settings file when the file has a key it doesn't know, and allowedMcpServers and allowManagedMcpServersOnly are Claude Code settings, not Claude Desktop's. A file with them lost its inference and MCP settings too. MST no longer writes or expects them.

A prepared desktop's settings must now set "isLocalDevMcpEnabled": false and "isDesktopExtensionEnabled": false instead (spread COWORK_MANAGED_ONLY from ./experimental/clients), or preparation fails with "Prepared Linux desktop settings do not match". coworkManagedPluginSettings no longer returns allowedMcpServers. Owned-desktop mode (MST_DESKTOP_OWNED=1) writes the new keys, and drops the old ones from an image's own settings.

Claude Desktop also reads the file only when root owns it and no one else can write it. An owned-desktop image now stages the file for MST (MST_COWORK_SETTINGS_FILE) and installs a root-owned copy in its restart command, which MST now gives 5 minutes instead of 2.

Headless Cowork disables AskUserQuestion

Affects: Linux Cowork desktops prepared by the caller (/etc/claude-desktop/managed-settings.json).

A headless run has nobody to answer a clarifying question, so Cowork evals now disable Claude's AskUserQuestion tool and the model proceeds on its best assumption. A case that asks anyway still stops as a failed case (awaitingUser).

A prepared Linux desktop's managed settings must now include "disabledBuiltinTools": ["AskUserQuestion"] (spread coworkHeadlessSettings() from ./experimental/clients), or preparation fails with "Prepared Linux desktop settings do not match". Disabling more tools is fine. Owned-desktop mode (MST_DESKTOP_OWNED=1) and the macOS eval profile MST writes set it for you.

Cowork takes tool variants

Affects: manifests that give cowork toolOverrides, and plugin clients with runBatch.

Cowork now gets an arm's toolOverrides through MST's tool-variant proxy, so a tool optimization (runVariantExperiment on a suite) can run on Cowork. Before, validation rejected it. ChatGPT still opts out.

For plugin clients:

• serversPerBatch: true says the client connects to one server set for the whole batch, the first request's input.servers. The batch then shares one proxy endpoint, and the check that the client listed the variant's tools is for the batch.
• A proxied request carries input.checkServers, the same servers on an endpoint for the client's own checks (a readiness probe, say), so that traffic isn't taken for the model seeing the variant.

Cowork and ChatGPT options aren't nested

Affects: manifests that use cowork or chatgpt with clientOptions.options.

Their options go directly in clientOptions, like every other client's. A nested options object fails with a message saying so.

// Before
{ "client": "cowork", "clientOptions": { "options": { "appVersion": "1.52386.6" } } }

// Now
{ "client": "cowork", "clientOptions": { "appVersion": "1.52386.6" } }

The client, its model and its options are separate keys

Affects: every manifest, arm and case that sets host, and shared configs (configs in a plugin).

host is now three keys: client names the client under test, model the model it uses, and clientOptions the client's other options. A config that still sets host fails with a message naming them.

// Before
{ "host": { "type": "cowork", "model": "claude-sonnet-4-6", "timeout": 900000 } }

// Now
{ "client": "cowork", "model": "claude-sonnet-4-6", "clientOptions": { "timeout": 900000 } }

• Arms and cases take the same three keys. An arm that sets only model keeps the manifest's client and options; one that sets a different client doesn't inherit the manifest's clientOptions.
• The manifest's model is a default for every arm and case, as before.
• clientOptions can't set type, client or model; each has its own key.
• Shared configs. A manifest that sets client or clientOptions replaces a shared config's client, so one client's options never apply to another.
• Results. A validated manifest (validateManifest) holds each level's resolved client in the same three keys. Result fields are unchanged in this release.

Tool optimization reports a comparison

Affects: code that reads runVariantExperiment results or the reporter's mcp-tool-optimization attachment.

With testInfo, runVariantExperiment also attaches a case-by-case comparison, and the reporter opens on a Comparison tab. Each candidate gains measurement and improvement, whose paired changes carry an assessment of 'better', 'worse' or 'unclear'. Case groups are capability and regression cases (measurement.capability, measurement.regression). The result says where the groups came from in grouping ('declared' from the regression tag, or 'grouping-run', with the extra run in groupingBaseline). On a suite, an arm metric such as input_tokens_mean has no per-case scores, so its improvement isn't tested; a variant that improves it is recommended unless it breaks regression cases.

Clients have canonical names

Affects: manifests and datasets that name a built-in host, plugins that contribute hosts, HostDefinition and the other Host* types, and imports from ./experimental/hosts.

What an eval tests is an MCP client with a model (ADR 0002). Built-in clients have canonical names, and how MST drives one isn't part of its name. An old name fails with a message naming its replacement, for example Client "claude-cli" is now "claude-code".; the 2.0 deprecated aliases (cowork_cu and the long driver names) fail the same way.

Before	Now
vercel-sdk, anthropic-api	mst
claude-cli	claude-code
chatgpt-mac, chatgpt-linux	chatgpt (the platform picks the driver)
cowork	cowork

anthropic-api was a second implementation of what mst does, so it's removed. mst infers the API from the model id (claude-*, gpt-*, gemini-*, claude-*@date for Vertex); set provider to override it.

For plugins and TypeScript:

• A plugin contributes clients under clients (was hosts), and a plugin with hosts fails validation naming clients.
• Types: HostDefinition → ClientDefinition, HostRunInput / HostRunContext / HostRunResult / HostRunOptions / HostBatchRequest → ClientRun* / ClientBatchRequest, HostConfig → ClientConfig, HostEvent → TraceEvent, HostEvidence → TraceEvidence, HostTrace → Trace, HostPlugin → MarketplacePlugin.
• @gleanwork/mcp-server-tester/experimental/hosts is now /experimental/clients.

The config key is client; see The client, its model and its options are separate keys.

Cases and run controls use common eval terms

Affects: every dataset and manifest, runEvalDataset and runVariantExperiment options, CLI host args, and plugin hosts.

MST now uses the vocabulary other eval frameworks share (ADR 0002). Old keys fail validation with a message naming the replacement, for example case "a" iterations: `iterations` is now `trials` ; none of them is an alias.

Before	Now	Where
scenario	input	case
iterations	trials	case, manifest, run
accuracyThreshold	passThreshold	case, manifest, run
expect	assertions	case
canonicalAnswer	expected.answer	case
scenarioTemplate with {{scenario}}	inputTemplate with {{input}}	manifest, arm
{{scenario}}	{{prompt}}	CLI host args (mcpHostConfig.cli.args)
defaultLlmIterations, defaultAccuracyThreshold	defaultTrials, defaultPassThreshold	runEvalDataset and runVariantExperiment options
EvalExpectBlock, EvalExpectBlockSchema	EvalAssertions, EvalAssertionsSchema	types

Plugin hosts receive the case's input as input.prompt (was input.scenario). A batch request numbers its trial as request.trial (was request.iteration). Judges are unchanged: they already read case.input.prompt and case.expected.answer.

A manifest that used a renamed key has a new contentHash once it's updated, so runEvalBatch doesn't resume a run saved before the change. Result fields (iterations, assertionPassRate and the request's scenario and expect) are unchanged in this release.

Cowork runs the installed Claude Desktop unless the manifest pins one

Affects: macOS Cowork runs, and anyone who set MST_COWORK_APP_VERSION.

MST no longer downloads a built-in Claude Desktop version (1.52386.6) for each run. It runs the installed /Applications/Claude.app and records its version with every case result as hostTelemetry.hostApp (now clientTelemetry.clientApp; { name, version, source }). To run an exact version, set clientOptions.appVersion in the manifest (now the eval config); MST downloads and verifies it as before, and records source: 'pinned'. MST_COWORK_APP_VERSION now fails with a message pointing to appVersion. coworkSetup.approveWriteTools no longer requires 1.52386.6. Suite results also record MST's own packageVersion. See Which Claude Desktop runs.

The Claude CLI host runs isolated from your Claude Code setup

Affects: claude-cli hosts.

The host used your own Claude Code configuration, so your skills, plugins and settings shaped what the model did, and added to its input tokens. It now gives each run an empty CLAUDE_CONFIG_DIR, so your settings.json (its env block, apiKeyHelper) doesn't apply either. Authenticate through the environment (provider: 'vertex', or an Anthropic API key), or set isolate: false on the host (now clientOptions.isolate on the claude-code client) to use your own configuration, for example to sign in with a claude.ai account. See Claude Code isolation.

Variant experiments run on suites

Affects: runVariantExperiment result types (see below); otherwise new.

runVariantExperiment({ suite: { manifestPath }, variants }) runs tool variants as arms of a manifest, so experiments reach any host that takes tool variants, not only the SDK host, and can optimize any numeric arm metric (better: 'lower' for tokens or cost). The dataset form, runVariantExperiment(options, context), runs as before, except that metric on its results is typed as a string (any arm metric) rather than ExperimentMetric, and a candidate with no value for the metric is disqualified (metricUnavailable: true) instead of scoring as the baseline. runEvalSuite gains arms, which replaces the manifest's arms with a list or with a function of them.

Internals leave the root

Affects: code that imports BUILTIN_RESULT_SCHEMAS, getResponseSizeBytes, SKILL_LIMITS or skillDigest from @gleanwork/mcp-server-tester.

These were internals:

• BUILTIN_RESULT_SCHEMAS is the expectation engine's table for named schema expectations.
• getResponseSizeBytes is how validateSize measures a response; check a size with validateSize or toHaveToolResponseSize.
• SKILL_LIMITS and skillDigest are the skills client's size limits and digest; mcp.skills and the skills conformance checks apply them.

Desktop hosts have one naming style

Affects: manifests whose host is chatgpt, cowork_cu, openai.chatgpt.agent.desktop-app.macos, openai.chatgpt.agent.desktop-app.linux or anthropic.claude.cowork.desktop-app.macos.

Built-in host names are kebab-case: vercel-sdk, anthropic-api, claude-cli, cowork, chatgpt-mac and chatgpt-linux. The earlier names still work and print a deprecation warning (MST_DEPRECATED_HOST) naming the replacement:

Earlier name	Use
cowork_cu	cowork
anthropic.claude.cowork.desktop-app.macos	cowork
openai.chatgpt.agent.desktop-app.macos	chatgpt-mac
openai.chatgpt.agent.desktop-app.linux	chatgpt-linux
chatgpt (the host for the local platform)	chatgpt-mac or chatgpt-linux

The long names remain the external-host driver identities recorded in results; only the host type changes.

The Claude Agent SDK is an optional peer dependency

Affects: the anthropic-agent-sdk judge provider.

@anthropic-ai/claude-agent-sdk (about 46 MB) was a dependency, installed with MST whether or not a judge used it. It is now an optional peer dependency, loaded only when an anthropic-agent-sdk judge runs. If you use that provider, install it:

npm install --save-dev @anthropic-ai/claude-agent-sdk

Without it, the judge fails with an error that names the package. The other judge providers already worked this way.

Plugins are objects

Affects: every plugin; code that calls registerJudge or the register*, get*, list* and clear* registry functions; METRIC_REGISTRY; loadPlugins; and manifests and datasets that reference plugin extensions.

Plugins now use ESLint's model (Plugins, ADR-0001). A plugin is a plain object that MST reads, instead of a module whose hook calls MST's register functions.

// Before (2.0.0-beta.7)
import { registerJudge } from '@gleanwork/mcp-server-tester';
import { registerDatasetSource } from '@gleanwork/mcp-server-tester/evals';

export function register() {
  registerJudge({ name: 'completeness', schema, evaluate });
  registerDatasetSource({ name: 'legacy', schema, load });
}

// After
import type { Plugin } from '@gleanwork/mcp-server-tester';

export default {
  meta: { name: '@acme/mst-plugin', version: '1.0.0', namespace: 'acme' },
  judges: { completeness: { schema, evaluate } },
  datasetSources: { legacy: { schema, load } },
} satisfies Plugin;

• Default-export a plugin object. The loader no longer calls register, registerPlugins, registerGleanJudges or a default-exported function. A module that still exports one fails with a message pointing here.
• Extension definitions lose name. The map key is the name.
• Reference plugin extensions as namespace/name: { "type": "acme/legacy" } in manifests, and { "judge": "acme/completeness" } in datasets and toPassToolJudge. Bare names belong to built-ins, so a plugin can no longer register a bare correctness or replace file.
• A manifest must load the plugins it references. Referencing acme/... without acme in that manifest's plugins (or --plugins, or the plugins option) is an error, even if another manifest in the same batch loaded it.
• --plugins and the pluginPaths option add to the manifest's plugins instead of replacing them.
• Manifest plugin paths resolve relative to the manifest's directory, then rootDir (--root-dir, the working directory by default), then as package names, resolved as import resolves them (exports conditions, including module-sync, and subpath patterns), so a package subpath must be a file or an exports entry. Before, they resolved against rootDir only.
• Plugins are validated when they load. Unknown top-level keys are rejected, every extension needs a Zod schema and its kind's function (load, run/runBatch/createConfig, evaluate, compute, or create), and a different plugin claiming a loaded namespace throws. An old registerJudge(name, executor) judge becomes { schema: z.object({}).passthrough(), evaluate: executor }.
• Datasets are checked too. A case's client and its judges must name a built-in or a namespace the suite loads.
• Plugins can share manifest settings. configs holds named manifest fragments (PluginConfig) that a manifest applies with "extends": ["namespace/name"]. See Plugins.
• getBuiltinHostConfig resolves built-in hosts only.
• validateManifestRegistrations is now validateManifest (from ./evals).
• Code passes plugin objects where it used to call register functions: runEvalSuite({ ..., plugins }), runEvalBatch({ ..., plugins }), runEvalDataset({ dataset, plugins }, ctx), runEvalCase(evalCase, ctx, { plugins }), or test.use({ mcpPlugins: [plugin] }) in Playwright. Code that calls validateJudge or the matchers outside those uses installPlugins([plugin]), from the package root. For a one-off judge, wrap it in a small local plugin, such as { meta: { name: 'local', namespace: 'local' }, judges: { x } }.
• Removed:
  • From the root: registerJudge (both forms) and the CustomJudgeExecutor and CustomJudgeResult types. A plugin judge's evaluate takes the same arguments as the old executor; it also needs a schema.
  • From ./evals: registerDatasetSource, registerHost, registerMetric, registerResultStore, and their get*, list* and clear* counterparts; METRIC_REGISTRY; and loadPlugins, loadPluginModule, EvalPluginModule and LoadPluginsOptions.
  • The metric-record argument of resolveMetric and computeMetrics. Metrics resolve from built-ins and loaded plugins only, and BUILT_IN_METRICS is read-only.
• The glean-legacy dataset source example was removed. It encoded one organization's legacy dataset schema and judge policy, which belongs in that organization's plugin, not in MST. Copy it from the v2.0.0-beta.7 tag (examples/plugins/legacy-glean-datasets.ts) if you need it, and convert it to a plugin object as above. Dataset sources shows a dataset-source plugin.

A judge's schema sees only its own options

Affects: plugin judges whose schema reads type, name, threshold, reference or reps, mainly judges listed in a manifest's judges.

A manifest judge's schema used to receive the whole manifest entry, so its options included type, reference and the case's settings. It now gets only its own options on every path. Read the reference from case.expected.answer in evaluate's input. A case judge now also passes provider, model and the other LLM fields to a named judge's schema; beta.7 dropped them. A custom-text rubric's metrics key is now rubric, not judge. Two manifest judges of the same type (two rubric judges, say) stay separate, and a case entry overrides the one whose results carry the same name. The built-in rubric judge can also be listed: judges: [{ "type": "rubric", "rubric": "correctness" }].

A judge's evaluate takes ({ case, trial }, options)

Affects: every plugin judge.

// Before
evaluate: async (candidate, reference, options) => grade(candidate, reference);
// After
evaluate: async ({ case: c, trial }, options) =>
  grade(trial.response, c.expected.answer);

candidate is trial.response (its text is trial.text) and reference is case.expected.answer. The input also has the case's prompt or tool call, expected.criteria, tags and metadata, and the run's tool events, messages and evidence. A judge may declare requires paths, and may return pass, skipped, subScores, usage and metadata. A dataset case may set expected (answer, criteria, other keys); expected.answer overrides canonicalAnswer. A score outside 0 to 1 is now an error. See Judge contract.

Hosts take a system prompt

Affects: manifests that compare instructions, such as an organisation's, across arms.

vercel-sdk, anthropic-api and claude-cli accept systemPrompt, as does mcpHostConfig.systemPrompt (a CLI host through a {{systemPrompt}} placeholder in cli.args); see System prompts. anthropic-api used to reject it as an unknown option. Hosts that can't apply one still reject it, and a suite rejects a case prompt that would replace the arm's.

Claude Code host tools are typed events

Affects: expectations and metrics on Claude CLI or Cowork runs that name Skill, SlashCommand, Task, Agent or ToolSearch as tool calls; plugin hosts that report tool searches.

The Claude CLI and Cowork parsers used to record these as host tool calls. They are now typed events: Skill is a skill event named for the skill, SlashCommand a command, Task and Agent a subagent, and ToolSearch a new tool_search event with the tools it returned in results. See Claude Code host-native events.

• Expectations: match a skill with { "name": "triage", "kind": "skill" }, not { "name": "Skill" }. toolsTriggered calls accept kind: "tool_search".
• Counts: these events are no longer tool calls, so they leave tool_count, first_tool, is_no_action (a trial that only searched or loaded a skill is now no action), the toolCallCount expectation and run telemetry's toolCalls. They still count in host_event_count. Every arm reports the new tool_search_hit_rate when its host searched.
• Plugin hosts that search a tool catalog should report { kind: 'tool_search', source: 'host', name, results }.

Tool variants reach plugin hosts and the Claude CLI

Affects: plugin hosts with run or runBatch, and manifests that give claude-cli or such a host toolOverrides.

These hosts used to fail validation with toolOverrides. Now the suite serves them the arm's servers through a local MCP proxy that applies the variant: each host request gets http configs for its own loopback Streamable HTTP endpoints, one per server, with the same labels, so the host must speak Streamable HTTP. See Tool metadata on every client.

• A host request that never lists the proxied tools fails, with an error naming the variant.
• Proxied hosts don't see the variant in context: the proxy applies it. A host that applies variants itself must declare toolOverrides: true, as before.
• HostDefinition.toolSurfaceProxy: false opts a host out, for one that connects to servers other than input.servers; giving it toolOverrides stays a validation error. The Cowork and ChatGPT desktop hosts opt out until they are verified with the proxy.

Tool variants: one rule, and renames

Affects: toolOverrides passed to runEvalDataset(), and error messages matched in tests.

Every host applies a tool variant through one tool surface, so a variant means the same thing on every host:

• Keys: a tool's name on its server, or server.tool to pick one of several servers in a suite; a qualified key wins over a bare one. runEvalDataset() serves one unlabelled server, so it takes bare names, as before.
• Renames: an override can set name. Calls to the new name reach the original tool, and are recorded under the original name, so a dataset's expectations need no change between arms. The trace's rawName keeps the name the model used.
• Errors: an unknown key is toolOverrides variant "v" overrides unknown tool "x"., an ambiguous one ... matches a tool on several servers; use "server.tool". (was references unknown tool(s) / Unknown tool override / Ambiguous tool override). A rename that isn't a valid tool name, or takes a name another tool on the same server has (originally or by rename), is an error.

Plugin hosts that declare toolOverrides: true can apply variants with buildToolSurface() from ./evals, and should record a renamed tool's calls under its original name (resolve() gives it) with the model's name in rawName.

Comparison orchestrators are arms

Affects: code that calls runServerComparison(), saveServerComparison() or runSkillsComparison(), or stores or reads server-comparison artifacts.

All three are removed, along with their types and the server-comparison result-store kind:

• Server comparisons: run two suite arms with different servers. Arms run one after another, and per-case wins come from compareEvalRuns() on the two arms' results. See Server comparisons are suite arms.
• Skills comparisons: run arms that differ in the vercel-sdk host's new skills option (off, catalog, preload), with metrics: ["skill_loaded", "skill_before_tool", "skill_verification_failed"]. A case can't also set mcpHostConfig.skills: the suite rejects it, since the case setting would override every arm's. See Measuring whether skills help.

runVariantExperiment(), which proposes and scores tool-description variants over rounds, stays.

Arms report every metric

Affects: code that reads arm metrics, armDeltas or unavailableMetrics.

• More metrics per arm. Every arm now reports tool counts, MCP calls and host events, tokens, cost, duration and judge results when the host or cases report them, whatever the manifest lists. mcp_call_count and host_event_count are new metrics.
• Unavailable metrics. unavailableMetrics lists only metrics the manifest lists.
• Deltas. armDeltas gain metricDeltas: the change in every numeric metric, keyed like metrics.
• Pricing. A manifest's pricing (USD per million tokens, by model) estimates cost for hosts that report tokens but not cost. Estimates are estimatedCostUsd in usage, kept apart from totalCostUsd; arms report their costSource, the prices they used, and unpricedModels. See Pricing.
• Shared configs can also set pricing, accuracyThreshold, temperature and maxTokens.

Manifest paths are relative to the manifest

Affects: suites run from another directory than their manifest's, and file result stores with a relative dir.

• Datasets. Dataset paths resolve against the manifest's directory first, then rootDir (--root-dir, the working directory by default), as plugin paths already did; they used to resolve against rootDir only. If both directories have the file, the one next to the manifest now wins.
• Stores. A file store's relative dir is relative to the manifest. It used to be relative to the process's working directory.
• Dataset sources get manifestDir in their context.
• Comparison with the previous run. Each run summary has a runId and, when the manifest ran the same arms before, a previousRun comparing the two, arm by arm. With a result store, the previous run is the newest summary of the same manifest, not the store's latest (which any manifest may have written). See Compared with the previous run.

Settings a run would ignore are errors

Affects:

• manifests and datasets with keys MST doesn't define
• manifests with toolOverrides or a concurrency the host can't honour
• anthropic-api hosts with options it doesn't use
• claude-cli arms whose servers set connection policy
• arms or cases that switch host type
• manifest-level temperature, maxTokens or provider

The changes:

• Manifests are strict. An unknown top-level key is an error; it used to be ignored.
  • Newly declared: accuracyThreshold, filterTags, run, temperature, maxTokens, redactStoredResponses and $schema were read or allowed without being declared, and are now part of the schema.
  • accuracyThreshold: a default for host cases that don't set their own, at the top level or as run.accuracyThreshold.
  • Editor schema: schema/eval-manifest.schema.json now declares every key, including extends and coworkSetup, and rejects the rest.
• Datasets are strict. An unknown key in a dataset, a case, or an expect block (including inside responseSize, toolsTriggered calls and toolCallCount) is an error, and so is an unknown option on a rubric judge.
  • Examples: regex (the key is matchesPattern), judge (it's the case's judges), or a misspelt accuracyThreshold. These used to be silently dropped.
  • Named judges are the exception: a plugin judge named by type still gets its own options.
  • Errors name the dataset and case, and suggest the right key for common mistakes.
• Hosts declare what they honour. HostDefinition gains toolOverrides (it shows the model an arm's tool variants) and maxConcurrency.
  • Tool variants: toolOverrides for a host without the flag is a validation error. MST's vercel-sdk and anthropic-api set it, as does any host with only createConfig, because MST's SDK host runs it. Cowork and plugin hosts used to run with the original tools while results recorded the variant; claude-cli and ChatGPT used to fail only once a case ran. A plugin host that applies variants itself, from context.arm, sets toolOverrides: true.
  • Concurrency: ChatGPT declares maxConcurrency: 1.
• Defaults reach only the hosts that take them. A manifest's model, provider, maxToolCalls, timeout, temperature or maxTokens fills in each host option its schema declares. A shared provider no longer reaches anthropic-api, which has none. A default that none of the run's hosts takes is an error.
• A host inherits options only from the same host. An arm, or a case, that switches host type no longer gets the other host's options.
• anthropic-api options are strict. It used to accept and ignore options such as systemPrompt or provider.
• Checks run before anything does, including in --dry-run.
  • Before execution: claude-cli connection settings it can't forward (OAuth, client credentials, TLS, proxies, retries, timeouts), case-level hosts, and host concurrency are all checked first.
  • Dry runs: a dry run now loads datasets, without resolving hosts or secrets.

Suite metrics are per trial

Affects: suites with iterations above 1, code that reads cost_usd, first_tool or per-case metric values, code that reads arm summaries, and code that builds a MetricResult.

• Per-trial means. Arm metrics used to read the last iteration's trace and every iteration's summed usage, so with 10 iterations tool_count_mean counted one trial and input_tokens_mean was ten times a trial's. Usage, timing, tool and answer metrics are now means over each case's trials.
• No infrastructure failures. Runs that failed on infrastructure aren't trials, including a single run.
• Shares. response_success and is_no_action are shares of trials, so their per-case values are numbers, not booleans; their _rate keys are unchanged.
• first_tool names MCP tools server.tool, as multi-server arms already did.
• New per-arm fields. Every arm now reports passed_rate and trial_pass_rate (the share of passing trials), and armDeltas add trialPassRate and trialPassRateDelta. Arms also report evidence (the weakest among their cases) and unavailableMetrics, and MetricResult has unavailable.
• Cost. Hosts that don't know their cost no longer report 0: the Vercel AI SDK and anthropic-api hosts, and CLI or Claude traces without a cost. cost_usd is unavailable for them rather than zero.

See Metrics.

External-host capabilities are built in

Affects: code that calls registerExternalHostCapability, datasets whose externalHost.capabilities bind uses: "module:<specifier>#<export>", and code that imports ExternalHostCapabilityImplementation or ExternalHostCapabilityContext.

The desktop drivers' capabilities are no longer an extension point. registerExternalHostCapability, ExternalHostCapabilityImplementation and ExternalHostCapabilityContext are removed from @gleanwork/mcp-server-tester/experimental/hosts, and uses no longer accepts module: ids. A binding can still choose among the built-in implementations (uses: "builtin:..."). A module: binding fails the case as an unsupported host, with an error that says module: capabilities were removed and points to Hosts.

To run a host MST doesn't ship, provide it from a plugin's hosts and select it in a manifest. See Hosts.

The Computer Use planner uses MST's LLM gateway settings

Affects: Cowork and macOS ChatGPT runs with ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN or MST_LLM_AUTH_COMMAND set, and Cowork on Macs whose managed preferences set the inference provider.

The Computer Use planner resolves its endpoint with the rules in LLM gateways. With ANTHROPIC_BASE_URL set, it sends the MST_LLM_AUTH_COMMAND token or ANTHROPIC_AUTH_TOKEN to that URL as a bearer token, or else ANTHROPIC_API_KEY. Without it, the planner calls the public Anthropic API with ANTHROPIC_API_KEY, and a gateway token is never sent there. In 2.0.0-beta.7 the Python SDK read these variables itself and could mix them; 2.0.0-beta.8 ignored them and always called the public API. If you export ANTHROPIC_BASE_URL for something else and want the planner on the public API, unset it for the run.

Cowork no longer requires ANTHROPIC_API_KEY when managed preferences set inferenceProvider: Claude Desktop takes inference from them, so MST stages no inference credential. Managed preferences may also set deploymentDisplayName. See Cowork inference.

Cowork on macOS continues after a failed case

Affects: Cowork batches on macOS.

After a submit, binding, HITL or trace failure, MST now resets Claude Desktop before the next case, as it already did on Linux: the Computer Use planner stops a task that is still running and declines an open permission prompt. The driver refuses typing, key presses and drags, and the planner is instructed never to click Send, approve or answer a question. Before, a submit or binding failure stopped the batch and the remaining cases were not submitted, while a HITL or trace failure let the next case start with the failed task possibly still running. If the reset fails, the remaining cases are not submitted. See Cowork.

Result fields use the eval vocabulary

Affects: code that reads results, stored artifacts, baselines or pairwise comparisons, and code that imports result, judge or pairwise types.

Results use client names, not host names (hostUsage → clientUsage, mcpHostTrace → toolCallTrace, externalHost → clientMetadata, request.scenario → request.input, and more), and record trials and scores (iterationResults → trialResults, expectations → scores, assertionPassRate → passRate). Judge and pairwise types follow: JudgeVerdict → JudgeScore, PairwiseVerdict → PairwisePreference, PairwiseCaseVerdict → PairwiseCasePreference, and pairwise cases[].verdicts → cases[].preferences. PairwisePreference now names what a pairwise judge returns; the 'baseline' | 'candidate' | 'tie' union it used to name is PreferredSide. Also ToolCallExpectation → ToolCallAssertion and reporter metrics.expectationBreakdown → metrics.graderBreakdown. Eval case results no longer carry toolName: 'mcp_host'. Stored results are schemaVersion: 2; loading an older one explicitly fails with what to do, and history skips it. See Result fields use the eval vocabulary.

Eval APIs and client contracts say eval, client and tool optimization

Affects: code that calls runEvalSuite or runVariantExperiment, client plugins, and code that reads tool-call sources, telemetry, failure kinds or the host_event_count metric.

runEvalSuite → runEval, runVariantExperiment → runToolOptimization (its suite option → evalConfig), and the *Suite*, *Host* and VariantExperiment* types follow. Client plugins get options.client (was host), and a completed executeCase returns kind: 'completed'. Tool calls the client ran itself have source: 'builtin' (was 'host'); host_event_count → builtin_event_count. See Eval APIs and client contracts say eval, client and tool optimization.

Plugin extension names say their kind

Affects: anything that names a plugin's extension.

acme/completeness is acme/judge/completeness: names are <namespace>/<kind>/<name>, and a two-part name fails with the full one. Built-ins may be written mst/<kind>/<name>. See Plugin extension names say their kind.

Servers are a map keyed by label

Affects: eval configs that set servers.

"servers": [{ "label": "acme", ... }] is "servers": { "acme": { ... } }, and a variant lists labels: "servers": ["acme"]. The old shapes fail with the new one. See Servers are a map keyed by label.

Narrowed runs are partial

Affects: code that reads run summaries, and scripts that pass --variant.

mst run takes --filter-tag <tags...> and --max-cases <n> beside --case and --trials, and --variant takes one or more names. A run narrowed by any of them has partial: true, its selection and a selectionHash in the summary. It's compared only with partial runs narrowed the same way, never becomes the result store's latest, and isn't resumed by --skip-existing. RunEvalOptions.variant takes a name or a list, plus filterTags and maxCases.

Case judges sit beside assertions

Affects: datasets whose cases set assertions.passesJudge, and eval configs that want pairwise judges.

A case's judges are a judges list beside assertions, written like an eval config's judges: "acme/judge/completeness" or { "type": "rubric", "rubric": "correctness", "threshold": 0.8 }. assertions.passesJudge fails loading, naming judges. The merge with the eval config's judges is unchanged, and so are judge scores; a result echoes the judges as request.judges. An eval config's top-level pairwiseJudges (also allowed in shared configs) compares each variant with the baseline after the run and records the result as the variant's pairwise; pairwise usage is in telemetry.totalJudgeUsage and, alone, telemetry.pairwiseJudgeUsage. See Case judges sit beside assertions.

Runs are directories in the mst.run/v1 format

Affects: anything that reads mst run output, stored results or baselines.

A run writes runs/<run-id>/ with run.json, traces/, scores/, results.json and summary.json, and the eval's latest.json; schemaVersion: 2 is format: 'mst.run/v1' and no longer loads. See Runs are directories in the mst.run/v1 format.

mst open opens a run's report

Affects: scripts that run mst open.

mst run writes runs/<run-id>/report/index.html with every run, and mst open [path] opens the newest run's report, or the run or eval directory named. See mst open opens a run's report.

The MCP reporter reports evals only

Affects: projects that use the MCP Playwright reporter.

It writes each run's eval results as a mst.run/v1 run with the run report (a variant per project), and leaves tests and conformance checks to Playwright's own report. historyLimit, includeAutoTracking and runId were removed; name names the eval. See The MCP reporter reports evals only.

The last scenario and host names

Affects: code that reads MstClientSimulationResult.scenario.

MstClientSimulationResult.scenario is input, the case's input as given to the model. The desktop drivers' run context says input and submittedInput (was scenario and submittedScenario), and the mst client's errors say "The mst client failed" (was "MCP host simulation failed").