Architecture

This document describes the internal structure of @gleanwork/mcp-server-tester — its module layout, key data flows, and the design decisions that shaped them.

Module Map

This map covers the core modules. AGENTS.md is the full list, including plugins/, proxy/, llm/, skills/, entries/, evals/runners/ and auth/grants/.

src/config/

Defines and validates the MCPConfig discriminated union (stdio | http) using Zod. All transport configuration is typed and validated at the boundary via validateMCPConfig(). Nothing else in the codebase reaches into transport details directly; they go through this module first.

src/mcp/

The MCP client layer. clientFactory.ts owns createMCPClientForConfig(), which translates an MCPConfig into a connected MCP SDK Client instance — choosing between StdioClientTransport, StreamableHTTPClientTransport, or SSEClientTransport based on config. response.ts normalises raw CallToolResult objects from the SDK into the framework's NormalizedToolResponse shape (with .text, .isError, .contentBlocks). The fixtures/ subdirectory contains MCPFixtureApi, the high-level test helper wrapping the client.

src/fixtures/

Playwright test fixtures. mcp.ts exposes { test, expect } — test is the base Playwright test extended with mcpClient (raw SDK Client) and mcp (MCPFixtureApi). assertions is the Playwright assertions extended with all custom matchers. mcpAuth.ts provides an auth-specific fixture variant for OAuth and token auth flows. Configuration is read from project.use.mcpConfig in the consumer's playwright.config.ts.

src/assertions/

The unified assertion layer. Internally split into:

• validators/ — pure functions returning ValidationResult objects. They have no Playwright dependency and can be called from the eval runner, unit tests, or anywhere else.
• matchers/ — thin Playwright adapter functions wrapping each validator. All matchers are registered in a single assertions.extend({}) call in matchers/index.ts, which re-exports the extended assertions.

This separation is intentional: validators can be used programmatically without Playwright (e.g., in the eval runner), while matchers provide the idiomatic expect(result).toContainToolText(...) API in Playwright tests.

src/evals/

The dataset-driven evaluation engine. Key files:

• datasetTypes.ts — Zod schemas for EvalDataset, EvalCase, and all assertion block types.
• datasetLoader.ts — reads and validates JSON datasets from disk.
• evalRunner.ts — runEvalDataset() iterates cases, calls the MCP server (or the LLM client simulator), validates results against each case's assertion block, and returns EvalRunnerResult. Supports multi-trial accuracy tracking and configurable concurrency.
• baseline.ts — saves and loads pass/fail baselines for tracking regressions across runs.
• resultStore.ts — persists eval runs, reporter runs, and comparison artifacts to local files or GCS using a shared JSON envelope.
• mstClient/ — the api client: a model through the Vercel AI SDK, or Claude Code (see data flow below).

src/judge/

LLM-as-a-judge evaluation. judgeTypes.ts defines the Judge interface and JUDGE_PROVIDER_KINDS, from which ProviderKind and the dataset schema derive. llmJudge.ts is the one judge core: the prompts, the verdict parser, the maxToolOutputSize guard and usage defaults. Each provider file (anthropicJudge.ts, vertexAnthropicJudge.ts, claudeAgentJudge.ts, openaiJudge.ts, googleJudge.ts) is only a completion adapter: a prompt goes in, text and tokens come out. judgeClient.ts maps each provider kind to its adapter in createJudge(). rubrics.ts provides built-in rubric definitions and the resolveRubric() helper. questionJudge.ts is the core of the yes-or-no judges (ADR 0009): defineQuestionJudge() asks each question in its own model call, about the trial's transcript as judgeTranscript.ts renders it (the request, each tool call with its arguments and result, and the final answer), and combines the answers: a required no fails, a required unknown makes the verdict unknown. Its prompts are files in prompts/, and a judge's version is a hash of them, the transcript format and its defaults. taskCompletionJudge.ts, groundedInToolsJudge.ts and toolUseJudge.ts are built-in yes-or-no judges, and builtinQuestions.ts gives mst label each built-in yes-or-no judge's question. Custom named judges come from plugins (judges in a plugin object, referenced as <namespace>/judge/<name>); they return a score and don't use the LLM core. agentic/ holds agentic judges, which run an agent (the Claude Agent SDK or Codex) over a workspace of the trial's evidence.

src/auth/

OAuth 2.1 with PKCE and static token utilities.

• oauthFlow.ts — low-level OAuth primitives: PKCE generation, authorization URL building, code exchange, token refresh. Uses oauth4webapi.
• discovery.ts — RFC 9728 protected resource discovery and OAuth authorization server metadata discovery.
• oauthClientProvider.ts — implements the MCP SDK's OAuthClientProvider interface, wiring together discovery, PKCE, and token storage for the SDK's automatic auth handling.
• storage.ts — file-based token persistence (tokens.json, client.json, server.json) with 0700/0600 permissions.
• cli.ts — CLIOAuthClient, which orchestrates the full browser-based OAuth login flow for the CLI and supports CI token injection from environment variables.
• tokenAuth.ts — static access token utilities (header generation, expiry checks).

src/spec/

MCP protocol conformance checks. conformanceChecks.ts runs a set of server-level verifications (server info presence, tool schema validity) and returns a structured ConformanceResult. Used in tests to assert protocol compliance rather than specific tool behaviour.

src/reporters/

Custom Playwright reporter. channel.ts is the contract between the code that records results on a test (the fixture's listTools/callTool, conformance and cross-era checks, runEvalDataset, tool optimizations) and the reporter: attachment names, payload types, one writer (attachReporterData) and a schema-checked reader. mcpReporter.ts is a Playwright Reporter implementation that reads each test's eval results and tool optimizations (and nothing else: tests and conformance checks are in Playwright's own report) and writes them as a mst.run/v1 run with the run report, a variant per project. This repo's own playwright.config.ts runs it once the UI is built. The ui-src/ subdirectory contains the React application that renders the report UI; it is compiled separately and embedded into ui-dist/. build-ui.ts is the build script for the React app.

src/cli/

The CLI, installed as mst and as mcp-server-tester (the same binary). index.ts is the entry point; the commands/ directory contains one subdirectory per command (init, generate, login, token, auth, datasets, judges, run, grade, batch, cowork, open, and the hidden collect a worker runs). utils/prompt.ts holds the line prompts init and generate ask in a terminal; without one, they take every answer from flags. templates/ stores scaffold files emitted by init.

src/types/

Canonical shared type definitions, kept here to prevent drift between modules:

• index.ts — core types: AuthType, ResultSource, GraderType, GraderScore.
• reporter.ts — reporter-specific types: EvalCaseResult, MCPRunReportData, MCPConformanceResultData.

src/reporters/ui-src/types.ts re-exports all types directly from the canonical backend sources (src/types/index.ts and src/types/reporter.ts) — no manual sync is required.

---

Data Flows

Tool Test (a Playwright test with matchers)

test(... async ({ mcp }) => ...)
   ↓  mcp.callTool(name, args)  ← MCPFixtureApi
   ↓  MCP SDK Client.callTool()
   ↓  Transport (stdio or HTTP)
   ↓  MCP Server (under test)
CallToolResult (raw SDK response)
   ↓  matchers/: expect(result).toContainToolText(), toMatchToolSchema(), ...
   ↓  validators/: validateText(), validateSchema(), validatePattern(), etc.
ValidationResult (pass: boolean, message: string)
   ↓  Playwright's own reporter
Playwright report (the MCP reporter reports evals only)

Client Eval Case (the api client in a Playwright test)

EvalCase { input, assertions.toolsTriggered }, run { client: "api", model }
   ↓  caseExecution.ts: a case with input runs on the client
   ↓  mstClient/run.ts: runApiClient()
   ↓  mstClient/adapters/vercel.ts: runVercelClient()
      - Lists MCP tools via mcp.listTools()
      - Sends tools + input to the configured LLM provider (via Vercel AI SDK)
      - LLM generates text and calls tools autonomously (multi-turn, up to maxSteps)
      - Each tool call is forwarded to the MCP server via mcp.callTool()
ClientRunResult (.events[], .finalText, .error)
   ↓  clientTranscript.ts: clientRunToExecution() — assertion view and transcript
   ↓  validators/: validateToolCalls(), validateToolCallCount()
ValidationResult
   ↓  evalRunner.ts: rolled into EvalCaseResult (with a pass rate over N trials)
EvalRunnerResult

External Result Storage

EvalRunnerResult / run summary / comparison result
   ↓  resultStore.ts: createStoredEvalArtifact()
StoredEvalArtifact { format, kind, id, metadata, data }
   ↓
FileEvalResultStore or GCSEvalResultStore
   ↓
eval-runs/, eval-summaries/, or comparisons/
   ↓
latest.json + immutable <id>.json

An eval config's result store is different: mst run copies each mst.run/v1 run directory into a RunStore (src/evals/runStore.ts), which only moves bytes, and reads its earlier runs from there.

The eval runner uses stored eval-runner-result artifacts for baseline regression detection. compareEvalRuns() remains pure; helpers load stored runs and save comparison artifacts around it. The Playwright reporter writes each run's eval results as a run directory with its report, and stores its summary as an eval-run-summary artifact, as mst run does.

Auth Flow

CLI user runs: mst login <server-url>
   ↓
auth/cli.ts: CLIOAuthClient.login()
   ↓
auth/discovery.ts: discoverProtectedResource()   [RFC 9728: GET /.well-known/oauth-protected-resource]
   ↓
auth/discovery.ts: discoverAuthorizationServer()  [RFC 8414: GET /.well-known/oauth-authorization-server]
   ↓
auth/oauthFlow.ts: generatePKCE()                 [code_verifier + code_challenge]
auth/oauthFlow.ts: buildAuthorizationUrl()
   ↓
Browser opens → user authenticates → redirect to localhost callback
   ↓
auth/oauthFlow.ts: exchangeCodeForTokens()        [PKCE code_verifier cleared after exchange]
   ↓
auth/storage.ts: saveTokens()                     [written to ~/.local/state/mcp-tests/<key>/ with 0600 perms]
   ↓
At test time:
auth/credentials.ts: resolveCredentials(config)   [the one place credential precedence is decided]
   ↓  auth.oauth.authStatePath → PlaywrightOAuthClientProvider (Playwright OAuth state file)
   ↓  auth.accessToken → a static header; auth.clientCredentials → ClientCredentialsAuthProvider (re-runs the grant as tokens expire)
   ↓  otherwise (fixtures) a stored login → StoredLoginAuthProvider (refreshes via CLIOAuthClient.tryGetAccessToken() as tokens expire)
mcp/clientFactory.ts: createMCPClientForConfig(config, { authProvider })
   ↓  MCP SDK Client asks the authProvider for a token before each request

---

Key Design Decisions

Why Playwright for e2e / integration tests?

The framework is built to help users write Playwright tests for MCP servers. Using Playwright for its own integration tests (against the mock server) means the framework exercises the exact same fixture, config, and reporter surface that consumers use. It also means the CI pipeline runs the framework tests in the same environment as consumer CI pipelines.

Why Vitest for unit tests?

Playwright's test runner is process-heavy and starts real browser workers. Unit tests for pure logic (validators, config parsing, OAuth utilities) don't need that overhead. Vitest's ESM-native, watch-friendly runner is a better fit, and it supports the same assertions assertions. The two test runners co-exist: npm test runs Vitest, npm run test:playwright runs Playwright.

Why tests and evals (direct vs. client cases)?

Direct tool calls are deterministic: you specify exact inputs and assert exact outputs. They are fast, cheap, and suitable for CI regression gates. Client cases are non-deterministic: a client and its model get a natural-language input and decide which tools to call, given only their descriptions. The two answer different questions (direct calls validate correctness, client cases validate discoverability) and they need to be run and interpreted differently (direct: once per CI run; client cases: N trials, measure the pass rate).

Why a separate assertions/validators layer?

Early versions of the framework coupled assertions directly to Playwright matchers. When the eval runner was added (which needed to validate responses outside of Playwright's expect()), we had to duplicate logic. The split into pure validators (framework-agnostic) and matchers (Playwright adapters) eliminates the duplication: the eval runner calls validators directly, and the matchers are a thin wrapper providing the expect(result).toContainToolText() ergonomics for inline Playwright tests.