Comparing tool descriptions in Cowork
Design proposal. This is how we intend evals to be run. Some of these commands and config keys don't exist yet; each is marked planned where it appears. The explainer covers the design, the run lifecycle, and what exists today versus what's planned.
This walkthrough runs one eval end to end. It uses a fictional company, Acme, whose private plugin @acme/mst-plugin provides datasets, judges, a VM runner and a result store.
Goal: find out whether rewritten tool descriptions for Acme's own MCP server make Cowork pick the right tools and answer Acme's info-seeking questions better. Every variant uses the same servers (Acme's, Slack's and GitHub's) and the same model; only the descriptions change.
Prerequisites
- Node.js 22+
- Access to your organization's MST plugin package (here
@acme/mst-plugin) - Cloud credentials for the plugin's result store and credential store (only for remote runs, CI, or remote results)
- For local Cowork runs: Claude Desktop on your Mac
- For container runs: Docker (or OrbStack), and Claude Desktop's Linux package or a desktop image your organization publishes
1. Install
Keep eval configs in your own repo, either a personal one or a team repo of use cases.
mkdir cowork-evals && cd cowork-evals
npm init -y
npm i -D @gleanwork/mcp-server-tester@rc @acme/mst-plugin
# Once, on a Mac that will run Cowork locally
npx mst cowork setup
# Once, to run Cowork in containers (section 6)
npx mst runner setup docker
# → built mst-cowork-linux:2.0.0-beta.9
See what the plugin provides (mst plugins is planned):
npx mst plugins --plugins @acme/mst-plugin
@acme/mst-plugin 0.14.0 (namespace: acme, snapshot 2026-10-06)
dataset acme/dataset/info-seeking, acme/dataset/action-taking, acme/dataset/tool-selection
judge acme/judge/correctness, acme/judge/completeness, acme/judge/groundedness
pairwise-judge acme/pairwise-judge/preference
connector acme/connector/acme, acme/connector/slack, acme/connector/github, …
credential-store acme/credential-store/secret-manager (planned kind)
setup acme/setup/reset-test-tenant (planned kind)
runner acme/runner/cloud-vm
result-store acme/result-store/eval-results
config acme/config/cowork
Names are <namespace>/<kind>/<name>, so acme/judge/correctness is a judge.
2. Find datasets and judges
Datasets and judge configs come from the plugin's nightly snapshot by default. You can pull live data or pin an older snapshot instead.
npx mst datasets --plugins @acme/mst-plugin
npx mst judges --plugins @acme/mst-plugin
npx mst judges show acme/judge/correctness
acme/dataset/info-seeking 50 cases snapshot 2026-10-06 tags: source:correctness, tool:*
acme/dataset/action-taking 32 cases snapshot 2026-10-06
acme/dataset/tool-selection 120 cases snapshot 2026-10-06
Extract a dataset to read, diff, or freeze it:
npx mst datasets pull acme/dataset/info-seeking --out datasets/info-seeking.json
npx mst datasets pull acme/dataset/info-seeking --source live
npx mst datasets pull acme/dataset/info-seeking --snapshot 2026-10-01
Each case has an input, an expected answer, tags, and optional default judges:
{
"id": "e2e-0011",
"input": "What is the process for reporting a newly discovered customer-impacting outage?",
"expected": {
"answer": "File a SEV in the incident tool, page the on-call IC, then…"
},
"tags": ["source:correctness"],
"judges": ["acme/judge/completeness"]
}
3. Write the eval config
evals/cowork-tool-descriptions.json:
{
"$schema": "https://unpkg.com/@gleanwork/mcp-server-tester@rc/schema/eval-config.schema.json",
"name": "cowork-tool-descriptions",
"plugins": ["@acme/mst-plugin"],
"extends": ["acme/config/cowork"],
"datasets": ["acme/dataset/info-seeking"],
"client": "cowork",
"model": "claude-opus-4-8",
"trials": 3,
"judges": ["acme/judge/correctness"],
"pairwiseJudges": ["acme/pairwise-judge/preference"],
"metrics": ["passed", "judge_score", "cost_usd", "duration_s", "tool_count"],
"redactStoredResponses": false,
"servers": {
"acme": { "connector": "acme/connector/acme" },
"slack": { "connector": "acme/connector/slack" },
"github": { "connector": "acme/connector/github" }
},
"variants": [
{ "name": "current" },
{
"name": "rewritten",
"tools": {
"acme.search_docs": {
"description": "Search Acme's internal docs: runbooks, policies and how-to guides. Use it first for questions about how Acme does something."
},
"acme.read_doc": {
"description": "Read one doc by the id search_docs returned. Use it when a search result looks relevant but its snippet is too short to answer from."
}
}
},
{
"name": "rewritten-terse",
"tools": {
"acme.search_docs": {
"description": "Search Acme's runbooks and policies."
},
"acme.read_doc": { "description": "Read a doc by id." }
}
}
]
}
serversdefines each server once. The key is its label in transcripts and the report. Each one names a plugin connector, which says where the server is, how to sign in, and how Cowork reaches it (here, through a dry-run proxy). A server can also be a plainhttporstdioentry with a literal or environment-backed token;mst authsigns in only to connector servers.variantsrun the same cases with the same client, model and servers, and only the tool descriptions differ. A variant that lists noserversuses all of them.currentshows the servers' own descriptions.rewrittenandrewritten-terseset tool metadata (tools) for Acme's two tools, keyedserver.tool; calls are still recorded under the tools' original names, so assertions read the same in every variant. The first variant is the baseline; set"baseline"to choose another.datasetsuse the snapshot by default. To pin a snapshot:{ "type": "acme/dataset/info-seeking", "snapshot": "2026-10-01" }. For live data:{ "type": "…", "source": "live" }.run.jsonrecords which copy each dataset came from.redactStoredResponses: falsekeeps full transcripts in the run. Stored transcripts are redacted by default, and a redacted run can't be collected with--no-grade, regraded (section 8) or resumed (section 7). A plugin's shared config can set it for every eval that extends the config.judgesapply to every case, on top of each case's own judges. Pointwise judges score each trial. Pairwise judges compare each variant with the baseline on each case.setup(planned, optional) lists plugin steps that run before collection and tear down afterwards, e.g.acme/setup/reset-test-tenant.
Point write-capable servers at test tenants or test accounts. A connector's dry-run proxy blocks writes, or answers them with a success reply when the eval config sets "simulateWrites": true. Connectors added in the Claude account don't go through the proxy, so disconnect any that could write to real accounts.
Check the plan without running anything. Today --dry-run prints the plan as JSON and doesn't check sign-ins; the text plan below, with its credentials line, is planned:
npx mst run --config evals/cowork-tool-descriptions.json --dry-run
cowork-tool-descriptions
datasets acme/dataset/info-seeking (snapshot 2026-10-06, 50 cases, sha256 3f9a…)
client cowork · model claude-opus-4-8 · 3 trials per case
servers acme, slack, github (every variant)
variants current (baseline) · the servers' own descriptions
rewritten · tool metadata for 2 tools
rewritten-terse · tool metadata for 2 tools
graders acme/judge/correctness, acme/judge/completeness (12 cases)
acme/pairwise-judge/preference (each variant vs current)
runner local (macOS) · 1 shard
results .mcp-test-results/cowork-tool-descriptions/
total 450 trials · 100 pairwise preferences
credentials 1 of 3 servers needs `mst auth` (slack)
The runner line says where the trials would run. With --runner docker --runner-option shards=4, it reads docker (linux/amd64) · 4 shards · image mst-desktop@sha256:9c1e….
4. Authenticate once
npx mst auth --config evals/cowork-tool-descriptions.json --store acme/credential-store/secret-manager
acme oauth ✓ valid
slack oauth → opening browser for consent… ✓ saved
github github-app ✓ client credentials (no consent needed)
Each run refreshes these grants and hands short-lived tokens to the client, then removes them when it finishes. Leave out --store to keep grants on your machine only. Remote runs and CI need a shared credential store. Today --store takes a directory; plugin credential stores such as acme/credential-store/secret-manager are planned.
npx mst auth status --config evals/cowork-tool-descriptions.json
npx mst auth revoke --config evals/cowork-tool-descriptions.json --server slack
5. Run locally
Use small local runs to iterate:
npx mst run --config evals/cowork-tool-descriptions.json --variant rewritten --case e2e-0011 --trials 1
npx mst run --config evals/cowork-tool-descriptions.json --max-cases 5
npx mst run --config evals/cowork-tool-descriptions.json --filter-tag source:correctness
setup auth staged for 3 servers · desktop lease acquired
collect current e2e-0000 ●●● e2e-0001 ●●● e2e-0002 ●●●
rewritten e2e-0000 ●●● e2e-0001 ●●● e2e-0002 ●●!
rewritten-terse e2e-0000 ●●● e2e-0001 ●●● e2e-0002 ●●●
teardown tokens removed · Claude Desktop profile restored
grade current e2e-0000 ✓✗✓ e2e-0001 ✓✓✓ e2e-0002 ✓✓✗
rewritten e2e-0000 ✓✓✓ e2e-0001 ✓✓✓ e2e-0002 ✗✓–
rewritten-terse e2e-0000 ✓✗✓ e2e-0001 ✓✓✗ e2e-0002 ✓✓✓
● collected ! infrastructure error ✓ passed ✗ failed – missing
The client runs the trials first. Judges then grade the collected transcripts.
6. Run in containers
Built, with limits. See Cowork in Docker. Each Docker host runs one Cowork container at a time, so 4 shards need 4 Docker contexts (
--runner-option contexts=a,b,c,d); on one host they run one after another. The progress display below, andmst runs view, are planned.
Run the same config in 4 Linux containers on your machine:
npx mst run --config evals/cowork-tool-descriptions.json --runner docker --runner-option shards=4
setup auth staged for 3 servers · 4 containers from mst-desktop@sha256:9c1e…
collect shard 1 e2e-0003 ●●● ●●● ●●● e2e-0007 ●●● ●●● ●●●
shard 2 e2e-0000 ●●● ●●● ●●● e2e-0011 ●●● ●●! ●●●
shard 3 e2e-0001 ●●● ●●● ●●●
shard 4 e2e-0002 ●●● ●●● ●●● e2e-0005 ●●● ●●● ●●●
teardown tokens removed · 4 containers deleted
grade …
Cowork runs in the containers, so you can keep using your Mac. Each container is a fresh desktop: it has nothing from your own Claude Desktop, and leaves nothing behind. A case's variants and trials all run in one container, so each comparison is made on one desktop, minutes apart.
To use an image your organization publishes instead of building one, add --runner-option image=registry.acme.example/mst-desktop@sha256:….
Watch a container's desktop while it runs (mst runs is planned), or keep a container that failed so you can look at it:
npx mst runs view <run-id> --shard 2
npx mst run --config evals/cowork-tool-descriptions.json --runner docker --runner-option shards=4 --runner-option keep=failed
7. Run remotely
Use the same config, on 5 fresh VMs. They run the same image as the containers in section 6. The limits in section 6 apply here too, and --results keeps the run in your store:
npx mst run --config evals/cowork-tool-descriptions.json \
--runner acme/runner/cloud-vm --runner-option shards=5 \
--results acme/result-store/eval-results
The VMs run the trials. Your machine gathers their transcripts, runs the judges, and writes the results. Ctrl-C deletes the VMs and keeps the transcripts collected so far. Rerun with --resume <run-id> to collect only what's missing. --runner-option keep=failed keeps a VM whose shard failed, as it does for containers.
To walk away from a long run, detach it (ADR 0007). The run's coordinator then runs on a machine the runner creates, with a cloud identity of its own, and your machine can go offline. It needs a result store and, for connector servers, a credential store that machine reaches:
npx mst run --config evals/cowork-tool-descriptions.json \
--runner acme/runner/cloud-vm --runner-option shards=5 --grade-runner \
--results acme/result-store/eval-results \
--store acme/credential-store/secret-manager \
--secrets-file .env --detach
# → Detached run d20261020T181638Z-e33763 is running on …
npx mst runs list -c evals/cowork-tool-descriptions.json --results acme/result-store/eval-results
npx mst runs watch e33763 -c evals/cowork-tool-descriptions.json --results acme/result-store/eval-results
npx mst runs cancel e33763 -c evals/cowork-tool-descriptions.json --results acme/result-store/eval-results
--grade-runner grades the run on the runner's machines once it is collected, so the coordinator runs no judges.
On a schedule
# .github/workflows/cowork-tool-descriptions.yml
on:
schedule: [{ cron: '0 9 * * 3,5' }]
workflow_dispatch: {}
jobs:
eval:
runs-on: ubuntu-latest
permissions: { contents: read, id-token: write, packages: read }
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 22 }
# authenticate to your cloud for the credential store and result store
- run: npm ci
- run: >
npx mst run --config evals/cowork-tool-descriptions.json
--runner acme/runner/cloud-vm --runner-option shards=5
--results acme/result-store/eval-results
8. Regrade without re-running
Judges read stored transcripts, so you can change or add judges and regrade without running Cowork again. This needs the run stored with "redactStoredResponses": false, as the config in section 3 sets:
npx mst run --config evals/cowork-tool-descriptions.json --runner acme/runner/cloud-vm --no-grade
npx mst grade 7f3c2a --config evals/cowork-tool-descriptions.json
# → run 7f3c2a.g2 (transcripts from 7f3c2a)
Grading takes hours when an agentic judge reads every trial. Grade on the runner's machines instead, with the judges' keys in the secrets file (ADR 0005):
npx mst grade 7f3c2a --config evals/cowork-tool-descriptions.json \
--runner acme/runner/cloud-vm --runner-option shards=5 --secrets-file judge-keys.env
A run kept in a result store (section 7) can be graded on another machine, such as your laptop after a scheduled run. mst grade fetches a run that isn't local from the store, checks it against the run's manifest, and keeps the regrade there too:
npx mst grade 7f3c2a --config evals/cowork-tool-descriptions.json \
--results acme/result-store/eval-results
# → [mst] Fetched run 7f3c2a from the result store acme/result-store/eval-results (…)
# → run 7f3c2a.g2 (transcripts from 7f3c2a)
--resume 7f3c2a fetches the same way, then collects what's missing.
Planned: mst grade without --config, and the --plugin-option and --judge flags:
npx mst grade 7f3c2a --plugin-option acme.source=live
npx mst grade 7f3c2a --judge acme/judge/groundedness
9. Find your results
Runs are written locally by default:
.mcp-test-results/cowork-tool-descriptions/
├── latest.json # the newest complete, full run
└── runs/<run-id>/
├── run.json # the config's identity, each variant's setup, resolved datasets and judges,
│ # runner, phase status
├── transcripts/ # one transcript per trial
├── artifacts/ # each trial's client artifacts, for judges
├── scores/ # one score per grader per trial; pairwise preferences per case
├── results.json # transcripts and scores joined
├── summary.json # per-variant metrics and comparisons
└── report/
--results also keeps the run in a result store, in this layout under <eval name>/. Without it, the eval config's results.store applies, if it has one. The run directory stays local either way.
--results acme/result-store/eval-results # a store your plugin provides
--results gs://my-bucket/cowork-evals # any GCS location
--results file:./shared-results # any local directory
npx mst stores --plugins @acme/mst-plugin # planned
run.json records where the run ran: the runner, its options, the number of shards, and each shard's machine (its image digest, worker MST and OS, which decides how a desktop client ran); each trial records its client app. A regrade on a runner's machines (section 8) also records where it graded, in grading.runner. The report shows both under Run details. Recording each transcript's shard is planned for 2.0.
10. Read your results
Every graded run ends with a comparison against the baseline:
cowork-tool-descriptions · run 7f3c2a · 50 cases × 3 trials · baseline: current
variant pass judge vs baseline (pairwise) $/case median tools used
current 0.78 0.80 — 0.71 55 s acme 58% · slack 27% · github 15%
rewritten 0.86 ▲ 0.85 44% win · 30% loss 0.69 51 s acme 74% · slack 17% · …
rewritten-terse 0.75 0.74 ▼ 29% win · 41% loss 0.70 54 s acme 63% · slack 24% · …
▼ ▲ clearly worse / clearly better than baseline
rewritten: 2 cases regressed · 7 improved rewritten-terse: 6 regressed · 3 improved
Open the report:
npx mst open # newest local run
npx mst open .mcp-test-results/cowork-tool-descriptions/runs/<run-id>
# planned
npx mst open --results acme/result-store/eval-results 7f3c2a
npx mst open --results acme/result-store/eval-results --latest cowork-tool-descriptions
The report has the same sections for every eval:
- Result: which variants are better, worse, or unclear compared with the baseline.
- Variants compared: the summary table above. Show statistics adds confidence intervals, p-values and pass^k.
- What differs: each variant's setup next to the baseline's.
- Case by case: every trial, with its transcript, each grader's score, and the pairwise preference against the baseline.
- Why trials failed: failed trials grouped by cause.