Traces (OpenTelemetry)

MST can send each trial of a run as an OpenTelemetry trace to a collector, over OTLP. MST builds the traces itself, from the record it keeps of each trial, so every client is traced the same way: the api client, Claude Code, Cowork, ChatGPT and plugin clients, run locally, in Docker or on a cloud VM, attached or detached. No client needs a telemetry setting of its own.

{
  "telemetry": {
    "otlp": {
      "endpoint": "https://collector.example.com",
      "bearerTokenEnv": "MST_OTLP_TOKEN"
    }
  }
}

Field	Meaning
endpoint	The collector's base URL (MST posts OTLP/JSON to <endpoint>/v1/traces) or its traces URL. HTTPS, or HTTP to localhost.
headers	Headers to send, such as a tenant. Not for secrets.
bearerTokenEnv	The environment variable, or --secrets-file key, holding a bearer token for the collector. The token is never written to the run.
resourceAttributes	Added to every trace's resource, for example { "deployment.environment.name": "ci" }.
content	Also send what the trial said: the prompt, the answer, each model request's text, tool arguments and results, judges' reasoning. Off by default. See What a trace holds.
maxAttributeChars	The most characters of one content attribute (default 16384); longer ones are cut and say how much was cut.

A plugin can supply telemetry in a config preset, like any other setting (see extends in evaluation-framework.md).

When traces are sent

The machine that runs mst run (or, for a detached run, its coordinator) sends them, so the collector must be reachable from there, not from the client's machine or VM.

• Each trial, as soon as it's saved: the client's run, its model requests and its tool calls.
• Each verdict, once the run is graded: a span for the trial's pass or fail, one for each judge's score, and one for each pairwise judge's preference (in the candidate's first trial).
• A regrade (mst grade) sends only its verdicts, into the traces of the run it grades: a trial's trace ID comes from the run, variant, case and trial, so it's the same for every grading. Each verdict says which grading made it (mst.grading_run_id).

Sending never fails or slows the run. Requests are queued and sent in the background, retried on a network error, 429 or 5xx, and reported once if they still fail. At the end, the run waits up to 30 seconds for what's left, then says how many requests failed.

A finished run

A run's directory has everything its traces are built from, so they can be sent after the run, to a collector it didn't send to. storedRunTraces(await readRunDirectory(dir), options) (from ./evals) returns the OTLP/JSON requests the run would have sent with telemetry.otlp: each trial's trace, each case's verdicts, and each variant's pairwise verdicts, with the same IDs. options takes content, maxAttributeChars and resourceAttributes, as telemetry.otlp does, and trialsOnly to leave out the verdicts. Times the run didn't record are its finishedAt, so a run always gives the same traces. A regrade's directory gives only its verdicts, in the traces of the run it graded, because that run sent the trials. A run collected with --no-grade gives no verdicts.

MST doesn't send them itself: a plugin's command does, encoded and batched as its collector needs.

What a trace holds

Spans follow OpenTelemetry's generative AI conventions, plus mst.* attributes for the eval.

Span	What it is	Attributes
invoke_agent <client>	The trial: the root. <client> is the variant's client, which a variant can set.	gen_ai.request.model, gen_ai.response.model, gen_ai.usage.*, mst.eval, mst.run_id, mst.variant, mst.case_id, mst.trial, mst.dataset, mst.tags, mst.cost_usd, mst.tool_call_count, mst.model_call_count; an error's message as its status
chat <model>	One model request, when the client reports them.	gen_ai.response.id, gen_ai.response.finish_reasons, gen_ai.usage.*
execute_tool <name>	One tool call: MCP or the client's own.	gen_ai.tool.name, gen_ai.tool.call.id, gen_ai.tool.type, mcp.method.name, mst.tool.server, mst.tool.source, mst.tool.kind; an error status when the call failed
evaluate <judge>	One verdict.	gen_ai.evaluation.name, gen_ai.evaluation.score.value, gen_ai.evaluation.score.label, mst.grading_run_id

A verdict's gen_ai.evaluation.score.label is pass or fail, or unknown when the judge couldn't tell from the transcript. An unknown verdict has no score value and no error status.

gen_ai.usage.input_tokens counts all input, cached or not, as OpenTelemetry does; gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_creation.input_tokens say how much was cached, and mst.usage.uncached_input_tokens is MST's own input count. mst.usage.reasoning_output_tokens is how many of the output tokens were reasoning, when the client reports it: OpenAI's reasoning tokens, or Claude's thinking tokens (Claude Code's and Cowork's result, for the trial as a whole).

With content: true, spans also have gen_ai.input.messages (the prompt), gen_ai.output.messages (the answer, or a model request's text and tool calls), gen_ai.tool.call.arguments, gen_ai.tool.call.result and gen_ai.evaluation.explanation. These can hold anything the servers under test returned: send them only to a collector cleared for that data. redactStoredResponses doesn't apply to traces: a run that redacts what it stores still sends content when content is on.

What each client reports

The trace has what the client recorded. Every client reports its answer and tool calls; model requests and times depend on what the client's own record has:

Client	Model requests	Times
api	Each step's request, with usage and response ID	Each request's; tool calls' from the requests around them
claude-code	Each message, with usage	From when each line of Claude Code's output arrived
codex-cli	Each response in the session's rollout, with usage	From when each of Codex's events arrived; requests' from the rollout
Cowork	Each message in the session's audit log, with usage	From the audit log's timestamps
ChatGPT	Each response's usage record	From the session log's timestamps; tool calls' own
Plugin client	Whatever its ClientRunResult has in modelCalls	Its startedAt and completedAt, and its events' and model calls' times

A span with no times of its own is placed at the start of its trial and marked mst.untimed. A tool call timed from the model requests around it is marked mst.timing: "inferred"; one timed by the client's own telemetry, mst.timing: "host".

The client's own telemetry

A client's record of a trial is its conversation: messages, tool calls and results, usage per message. Some clients also report what they measured while running, over their own OpenTelemetry exporter, and never write it into that record: a request's duration, time to first token and cost, permission decisions, retries and API errors, the hooks and plugins that ran. With telemetry.host, MST records that too, in the same trial:

{
  "telemetry": {
    "otlp": { "endpoint": "https://collector.example.com" },
    "host": {}
  }
}

The telemetry.host setting, the hostTelemetry field and the mst.host.* attributes keep "host" on purpose. The MCP specification calls the client application the host, and this is the application's own telemetry. The numbers MST reads from the client's own records are clientTelemetry.

MST runs an OTLP/HTTP receiver beside the client (one per process, on a random port and secret path) and points the client's exporter at it, asking for OTLP/JSON. Each trial takes the records of its own session: Claude Code runs with a session ID MST chooses, and a Cowork task is matched by its session. The trial's record keeps them in transcript.hostTelemetry: the client's events, spans and metrics (each metric's latest value), with its resource attributes. telemetry.otlp isn't needed: without it, they're only in the run.

Field	Meaning
content	Keep the client's content attributes (prompt, response, tool_input, tool_parameters, gen_ai.*.messages, ...), and ask the client to send them. Off by default: the client is asked for none, and any it sends are dropped. Identity attributes (user.*, organization.id, host.name) are never kept.
traces	Also ask the client for its spans, where it has them (Claude Code's are beta).
forward	Also pass what the client sent, as it sent it, on to a collector: endpoint (MST posts to <endpoint>/v1/<signal>), headers, bearerTokenEnv. Use it to keep a collector that got the client's telemetry before getting it. Sending never fails the run.

In the trial's trace, an event from the client's telemetry joins the span it names: an api_request the model request with its request ID (request_id; the event names the model call by one of its message entries, message_uuid, or by its response ID, response_id), a tool_decision or tool_result the tool call with its ID (tool_use_id). The built-in clients that collect their own telemetry (claude-code, cowork, codex-cli) record a call's request ID by that rule; Codex's own api_request events name neither a request nor a response today, so a Codex trial's model requests carry none. Each is a span event, and the first event of each name is also the span's attributes, as mst.host.<event>.<key> (for example mst.host.api_request.ttft_ms), so a query can filter on them. A request the client's record doesn't have, such as a subagent's, gets a span of its own (mst.model_call.source: "host"). Other events, and metrics, are the root's events; the client's resource is the root's mst.host.*. The client's spans join under the root (mst.span.source: "host", their own trace ID as mst.host.trace_id).

Client	How
claude-code	MST sets CLAUDE_CODE_ENABLE_TELEMETRY and the per-signal OTEL_EXPORTER_OTLP_*_ENDPOINTs, and passes --session-id. A machine's managed settings with an otelHeadersHelper keep Claude Code's telemetry with their own collector by design: Claude Code then ignores MST's endpoints, and MST says so and records none.
Cowork	Linux owned-desktop mode only (MST's image): MST writes Claude Desktop's otlp* settings, in place of the image's, pointing at the receiver. Desktop gives a task in its VM the host's address for localhost, so the receiver listens on every interface of the container, under its secret path. Desktop applies these settings with third-party inference, as MST's image uses.
codex-cli	MST writes Codex's [otel] exporters (logs and metrics, and traces with traces) into the trial's config.toml, pointing at the receiver over OTLP/JSON, with log_user_prompt only with content. A trial takes the records of its Codex thread (conversation.id).
Plugin client	hostTelemetryReceiver(config) (from ./evals) starts or returns the process's receiver: point the client at its endpoint, then collect(sessionIds) after the trial and return it as hostTelemetry.

When the client sends nothing, the first trial waits 15 seconds for it, MST says so once, and later trials wait only briefly.

Try it locally

Run Jaeger and point an eval at it:

docker run -d --name jaeger -p 127.0.0.1:16686:16686 -p 127.0.0.1:4318:4318 jaegertracing/jaeger:2.10.0

{
  "telemetry": {
    "otlp": { "endpoint": "http://localhost:4318", "content": true }
  }
}

Then open http://localhost:16686, pick the service mcp-server-tester, and search by tag, for example mst.run_id=<run id>.