What's in 2.0
Playwright tests for tools
The mcp fixture connects over stdio or HTTP. Call a tool, then check the result with matchers for text, schemas, snapshots and errors. Fast, deterministic, free.
Evals on real clients
Run the same cases on the api client, Claude Code, the Codex CLI, Cowork, ChatGPT and the Codex app. Each trial's transcript records what the client did: its tool calls, skill loads and final answer.
Graders you can check
Code graders check tool calls and text. Model graders, MST's judges, read the transcript. Label trials yourself with mst label, and mst judges check tells you whether a judge is usable against your labels.
Every write accounted for
Every server has a write mode before a run starts. A dry-run proxy stops writes and records each one, and the writes grader checks that record, not what the client says it did.
Both protocol eras
Test the initialize handshake and the 2026-07-28 protocol from one config, with conformance checks for the era the server speaks.
Desktop clients in Docker
The docker runner collects Cowork and ChatGPT trials in disposable Linux containers, one per shard, while your machine stays usable.
A report for every run
Every eval run writes a report that says whether each variant beat the baseline, with every case and trial behind the answer.
Plugins for the rest
Datasets, clients, judges, connectors, runners and result stores are all extension kinds. Your organization adds its own in a plugin.
Agent Skills over MCP
Test the skills a server serves through the MCP skills extension, and evaluate whether the client loads them when it should.
Tests and evals
A test calls a tool directly and checks what it returns. An eval has a client act on each case over several trials, and graders score each trial from its transcript. MST is the evaluation harness for both.
A tool test
import { test, expect } from '@gleanwork/mcp-server-tester/fixtures/mcp';
test('read_file returns file contents', async ({ mcp }) => {
const result = await mcp.callTool('read_file', { path: '/tmp/test.txt' });
expect(result).toContainToolText('Hello, world');
expect(result).not.toBeToolError();
});
An eval case
{
"id": "read-config",
"input": "What version is set in the app's config file?",
"trials": 5,
"assertions": {
"toolsTriggered": { "calls": [{ "name": "read_file", "required": true }] },
"containsText": "1.0.0"
}
}
Ask your AI about MST
These docs are also an MCP server. Connect your coding agent and it can search and read every page, and load MST's skills for writing tests and evals.
npx skills add gleanwork/mcp-server-tester