Skip to main content

Troubleshooting

Common issues and how to fix them.

"Raw mode is not supported"​

Symptom: Running npx @gleanwork/mcp-server-tester init, npx mst generate or npx mst login fails immediately with an error like Error: Raw mode is not supported on the current process.stdin.

Cause: An older 2.0 beta. Its prompts needed a real terminal, which CI, piped shells and agents' shells don't have.

Fix: Upgrade. The CLI no longer needs a terminal: pass the answers as flags, such as init --command "node server.js" and generate --tool <name> --args '<json>'. See CLI Commands.

OAuth token refresh failing silently​

Symptom: Tests pass the auth setup phase but subsequent MCP tool calls fail with HTTP 401 errors or access-denied responses.

Cause: The cached OAuth access token has expired. hasValidOAuthState returns false, which causes performOAuthSetupIfNeeded to attempt re-authorization. In CI, no browser is available to complete the OAuth flow, so the re-auth silently fails or is skipped, and the expired token is used for calls.

Fix:

  1. Use performOAuthSetup in globalSetup — run the full browser-based OAuth flow once before tests start. The auth state file is written to disk and reused for all tests in the run.

  2. Cache the auth state file between CI runs — persist the file written by performOAuthSetup (typically ~/.mcp-server-tester/auth/<server>.json or the path you configured) in your CI cache. This avoids re-auth on every run.

  3. Switch to a static API token — if your MCP server supports API tokens, use tokenAuth instead of oauth. Static tokens do not expire on a short cycle and do not require browser interaction. See docs/authentication.md for configuration details.

HTTP MCP server not reachable (connection refused)​

Symptom: Tests fail immediately with ECONNREFUSED or a network timeout error pointing at the configured serverUrl.

Cause: One of the following:

  • The MCP server process is not running when the test starts
  • The serverUrl in your MCPConfig has the wrong host or port
  • A proxy or firewall is intercepting the connection

Fix:

  1. Verify the server is running and listening on the expected port before tests execute. Use a globalSetup script to start the server and wait for it to become ready.

  2. Double-check the serverUrl value in your playwright.config.ts. A common mistake is using localhost when the server binds to 127.0.0.1 only, or vice versa.

  3. Add retryAttempts to your MCPConfig to tolerate brief startup delays:

    mcpConfig: {
      transport: 'http',
      serverUrl: 'http://localhost:3000/mcp',
      retryAttempts: 3,
    }

  4. If you are behind a corporate proxy, ensure the proxy is configured to pass through requests to localhost / 127.0.0.1, or set NO_PROXY=localhost,127.0.0.1 in your environment.

Every trial says "MCP server not ready, so no trial was sent"​

Symptom: mst run prints [mst] The <client> batch for variant "<name>" didn't run: MCP server not ready, so no trial was sent: .... Each trial of that batch is an infrastructure failure (broke: 'setup'), and its cases are incomplete.

Cause: MST checks every client's servers before a batch's first trial (Server readiness). The reason names each server and what failed:

  • <server> did not connect (...): the server didn't start, refused the connection, rejected its credential, or took more than 30 seconds to connect, list its tools and close (a server with a longer connectTimeoutMs gets that to connect, plus 30 seconds).
  • <server> did not list its tools (...): the server connected, then answered tools/list with an error.
  • <server> did not list its tools (...) when the tool-variant proxy started: the server passed the check, then connected to MST's tool-variant proxy but failed or timed out listing its tools.
  • <server> listed no tools: it connected but served no tools, which no client can be evaluated against.
  • <server> listed <n> tools, fewer than its minTools (<m>): a stdio server, or a connector's proxy, served fewer tools than it declares. A degraded server, such as one whose upstream credential failed, often serves only its static tools.

Fix:

  1. Start the server, or fix its URL, command or credential, and check that it lists its tools: mcp.listTools() in a Playwright test connects the same way.
  2. For a server that starts slowly, set longer retry delays: MST_MCP_READINESS_RETRY_S=10,30,60 (comma-separated seconds; the default is 10,30).
  3. Complete the run with mst run --resume <run> in its runner: these trials are retryable. A local run's cases run again with --case.

The retry setting was MST_COWORK_MCP_PREFLIGHT_RETRY_S, for Cowork only. A run that still sets it fails when it starts and names the new one.

Eval pass rates much lower in CI than locally​

Symptom: The infrastructureErrorRate field in eval run results is high when running in CI, but the same evals pass consistently on a developer machine. passRate appears low as a result.

Cause: CI environments often have stricter rate limits, slower network paths, and no retry tolerance. MCP tool calls that succeed quickly locally may time out or hit provider rate limits in CI.

Fix:

  1. Reduce concurrency. Lower the concurrency option in runEvalDataset to avoid saturating rate limits:

    await runEvalDataset({ dataset, concurrency: 2 }, { mcp, testInfo });

  2. Add retry attempts. Configure retryAttempts on your MCPConfig so transient network errors are retried automatically:

    mcpConfig: {
      transport: 'http',
      serverUrl: 'https://your-server/mcp',
      retryAttempts: 3,
    }

  3. Increase the call timeout. If your MCP server is slow to respond in CI (cold starts, shared infrastructure), raise callTimeoutMs:

    mcpConfig: {
      transport: 'http',
      serverUrl: 'https://your-server/mcp',
      callTimeoutMs: 30_000,
    }

  4. Inspect infrastructureErrorRate separately from passRate. Infrastructure errors (timeouts, connection resets) are distinct from assertion failures, and a case with one is incomplete, not failed. Each trial's ending.broke says what failed: network, provider and so on. A high infrastructure error rate signals a connectivity or capacity problem, not a quality problem with your MCP server.

Every CI run starts a new era: "client options hashed under another option key"​

Symptom: mst history shows a new era at every run, for each variant with a client option that looks like a credential (a token, a URL, a header).

Cause: run.json records such an option only as a hash under the eval's option key, which MST keeps in .mcp-test-results/<eval>/option-key. A fresh CI checkout has no key, so each run makes its own, and two runs' hashes can't show the value stayed the same.

Fix: Give every machine that collects the eval's runs the same key from a CI secret:

openssl rand -hex 32   # once; store it as a secret
export MST_OPTION_KEY=…  # in the CI job, from the secret

The next run starts one more era (its key is new); later runs compare. Don't upload option-key or the whole results directory as a public artifact. See The eval's history.

Windows: token files not secure​

Symptom: OAuth token files are written successfully, but on Windows the 0o600 POSIX permission mode applied by Node's fs.writeFile is silently ignored. Any local user account can read the file.

Cause: Windows does not enforce POSIX file permission bits. The 0o600 mode has no effect on NTFS.

Fix: Manually verify that the folder containing token files (%LOCALAPPDATA%\mcp-server-tester\ or the path you configured) has Access Control List (ACL) entries that restrict read access to your user account only.

To inspect and tighten ACLs from PowerShell:

$path = "$env:LOCALAPPDATA\mcp-server-tester"
icacls $path
# Remove inherited permissions and restrict to current user:
icacls $path /inheritance:r /grant:r "${env:USERNAME}:(OI)(CI)F"

In shared or multi-user environments, consider storing tokens in a secrets manager rather than on the local filesystem.

"Unsupported Vercel AI SDK provider" error​

Symptom: Running a case on the api client fails with an error such as Unsupported provider: google or Cannot find module '@ai-sdk/google'.

Cause: LLM client providers are optional peer dependencies. The package declares them in optionalDependencies, but npm does not guarantee they are installed in all environments. The specific Vercel AI SDK adapter for the provider you selected is missing.

Fix: Install the ai package and the adapter for your chosen provider:

ProviderInstall command
openainpm install ai @ai-sdk/openai
anthropicnpm install ai @ai-sdk/anthropic
googlenpm install ai @ai-sdk/google
azurenpm install ai @ai-sdk/azure
mistralnpm install ai @ai-sdk/mistral
deepseeknpm install ai @ai-sdk/deepseek
openrouternpm install ai @openrouter/ai-sdk-provider
xainpm install ai @ai-sdk/xai

See docs/mst-client.md for the full provider table and required environment variables.