ChatGPT desktop evaluation

MST submits through the authenticated ChatGPT desktop app. It does not use the Codex CLI, Codex APIs, or an inference API to run the evaluated query.

Clients and platform

The desktop app has two products, and each is a client:

• "client": "chatgpt" drives ChatGPT Work.
• "client": "codex-app" drives the app's Codex surface. It is not the codex-cli client, which runs the Codex CLI headless (codex.md).

Both run on macOS and on Linux. How they run follows from the OS of the machine that runs the trial, and no option selects it. clientOptions.computerUseProvider is gone: it fails validation. The earlier names (openai.chatgpt.agent.desktop-app.macos, openai.chatgpt.agent.desktop-app.linux, chatgpt-mac and chatgpt-linux) fail validation, naming chatgpt. clientOptions.surface is gone: it fails validation and names the client to use. clientOptions.provider, which took only openai, is gone too: each client fixes its provider, so it fails validation. The client picks the surface, which controls UI setup and the expected native originator; MST does not add surface instructions to the evaluated prompt. Everything else on this page applies to both clients. Both drive the same app, so they share its desktop lease: a chatgpt variant and a codex-app variant never drive the app at once.

macOS keeps the Anthropic Computer Use submission driver. It receives the selected surface and must verify the current-mode label before filling the composer. It requires an Anthropic credential (ANTHROPIC_API_KEY, or an LLM gateway), the Xcode Command Line Tools for its Swift input helper, and Accessibility and Screen Recording permission. Linux uses native draft deep links and deterministic AT-SPI actions. It requires no Anthropic key and rejects Computer Use planner configuration. It does not fall back to macOS CUA.

Some options apply to one OS only. computerUseModel, the planner, is for Computer Use on macOS. plugins, which install into Linux's fresh profile, are for Linux. maxActions, the most actions the automation may take to send a case's prompt, applies on both, with each OS's range: 1 to 64 (default 32) on macOS, and 1 to 64 (default 24) on Linux. It was computerUseMaxActions on macOS and nativeMaxActions on Linux; both now fail validation, naming maxActions. An option the trial's OS can't use, or a maxActions outside its range, fails when the eval config loads, against this machine's OS for --runner local (the default) and against Linux for --runner docker. A plugin's runner checks it on its worker, before the app is prepared.

timeout is the trial's time limit in ms, default 300000. reasoningEffort is one of low, medium, high, xhigh, max or ultra. minimal, which the codex-cli client takes, fails here, and the message names the values ChatGPT accepts.

Both clients run a variant's cases in one app, with one set of settings. A case that sets its own model or a clientOptions value that differs from its variant's fails when its dataset loads, before anything runs, and so does a case that names another client. Use a variant for a different setting.

On macOS, MST starts and stops ChatGPT through a small Swift helper that it compiles with no caller credentials. The helper launches the app with the test process's environment, so ChatGPT can read every variable the test run has, including API keys. Launch credentials that MST adds travel on stdin, never in process arguments. Run ChatGPT evaluations from an environment that holds only the credentials the run needs. Narrowing the helper to an allowlist is planned, once it is confirmed on a real Mac that ChatGPT still launches and signs in.

Example Linux client settings (in an eval config):

{
  "client": "chatgpt",
  "model": "your-native-model-id",
  "clientOptions": {
    "reasoningEffort": "medium",
    "timeout": 300000,
    "maxActions": 24,
    "requireMcpCalls": true,
    "correlation": "exact_prompt"
  }
}

Model and effort are installed in the native configuration. MST verifies both against the completed native turn. Linux does not guess mappings from model IDs to display labels such as 5.6 Terra Medium. Linux rejects clientOptions.configPath (unless it equals $HOME/.codex/config.toml) and chatgptSessionRoot.

macOS runs in your own profile

On macOS, ChatGPT runs signed in as you, with your configuration. For the length of a run, MST swaps the MCP servers in ~/.codex/config.toml for the eval's servers and turns off everything else that could answer instead of them. It restores the original configuration when the run ends:

• ChatGPT's account connectors (the apps feature and its codex_apps server). Otherwise the model can pick, for example, your account's own connector for a service instead of the eval's server for it.
• Web search and the in-app browser (the in_app_browser feature).
• The bundled computer-use and browser plugins.
• Every plugin you installed from a marketplace other than OpenAI's own (openai-bundled, openai-primary-runtime), with their skills and MCP servers.

Your global ~/.codex/AGENTS.md still reaches the evaluated prompt. Keep personal instructions out of it on a Mac you run evaluations on, or use Linux, whose fresh profile has none.

MST checks the eval's servers before it touches the app (server readiness). When one isn't ready, MST doesn't stop, reconfigure or restart ChatGPT, and each trial of the batch ends as a retryable setup failure.

Before each case, MST chooses File > New Chat itself, since the app stays on the previous case's conversation, and the planner pastes the query rather than typing it. ChatGPT Work makes its own folder for a task. For codex-app, MST opens the draft in an empty, trusted folder it creates for the run and removes afterwards.

When a case stalls, its chat is still mid-turn, and ChatGPT answers MST's quit with "Quit ChatGPT? Active local chats on this machine will be interrupted". During recovery and cleanup, MST confirms that prompt only when every chat with a turn in progress is one it started in this run with one of its queries, read from the session rollouts. Otherwise it leaves the app running and the configuration installed, and reports the failure. MST never confirms the prompt when it first quits the app before a run. Confirming needs the Accessibility permission the driver already uses.

Linux runtime contract

The caller (the automation that starts MST) owns the VM/container, display, D-Bus, keyring, AT-SPI bus, the fresh HOME and the inference credential file (creation and deletion), the package version check, the disposable no-new-privileges container, and uploading MST_CHATGPT_EVIDENCE_DIR. It runs MST as the unprivileged desktop user with this process environment. There is no controller helper, socket, opener, or attestation variable.

Variable	Provided by the caller	MST checks
HOME	Fresh private dir on tmpfs	Absolute, normalized, owned, mode 0700, not a symlink. $HOME/.codex must be absent.
XDG_RUNTIME_DIR, TMPDIR	Private dirs inside the fresh area	Absolute, owned, mode 0700.
XDG_CONFIG_HOME, XDG_DATA_HOME, XDG_CACHE_HOME	Under HOME	Absolute and inside HOME.
DISPLAY, XAUTHORITY (optional)	Xvfb/desktop display	DISPLAY required.
DBUS_SESSION_BUS_ADDRESS	Caller-owned private session bus	unix: address.
AT_SPI_BUS_ADDRESS	From org.a11y.Bus.GetAddress	unix: address.
GNOME_KEYRING_CONTROL	Caller-owned unlocked keyring	Absolute path.
MST_CHATGPT_APP_PATH	/usr/bin/chatgpt	Executable file.
MST_CHATGPT_CODEX_PATH	The package's bundled codex backend	Executable file. Used for login and the app-server status probe.
MST_CHATGPT_API_KEY_FILE	0600 file containing only the API key	Owned regular file, no group/other bits, no symlink, at most 8 KiB, printable ASCII.
MST_CHATGPT_INFERENCE_BASE_URL	An OpenAI-compatible gateway's base	https:, no credentials. Instead of the API key file, with the token file.
MST_CHATGPT_INFERENCE_TOKEN_FILE	0600 file the caller keeps fresh	As the API key file. Read by the app's token command, never written to the config.
MST_CHATGPT_MCP_TOKEN_<n>	Set by MST from mcpServers[].auth	Config uses bearer_token_env_var; never written to disk.
MST_CHATGPT_EVIDENCE_DIR	0700 dir that the caller uploads	Owned, mode 0700. Each session writes to its own session-* subdirectory.
MST_CHATGPT_PYTHON (optional)	/usr/bin/python3	Absolute path. Needs PyGObject with gi.repository.Atspi 2.0.
Process isolation (no variable)	Disposable no-new-privileges container	Not checked. Required: MST turns off the native sandbox (see below).

Any violation fails the case before a prompt with environment_invalid: <NAME> or another fixed code. A set CODEX_HOME must equal $HOME/.codex.

MST then owns, in order:

1. Create $HOME/.codex (exclusive, 0700) and one empty workspace mkdtemp($TMPDIR/mst-chatgpt-workspace-*).
2. Write config.toml with cli_auth_credentials_store = "keyring", model, reasoning effort, MCP entries with bearer_token_env_var, and trust for only that workspace, approval_policy = "never", and sandbox_mode = "danger-full-access".
3. Inference, one of two ways. With MST_CHATGPT_API_KEY_FILE: codex login --with-api-key with the key on stdin only, then codex login status. With MST_CHATGPT_INFERENCE_BASE_URL and MST_CHATGPT_INFERENCE_TOKEN_FILE: no sign-in. config.toml gets a mst_gateway model provider (wire_api = "responses") whose token command reads the token file every minute, and MST checks GET <base>/models with the token before the app starts (inference_unavailable otherwise). nativeReadiness.inference records which (api-key or gateway).
4. A read-only codex app-server probe (initialize, initialized, mcpServerStatus/list only; server requests abort; 30 s, 256 KiB/line, 4 MiB total). Every configured server must be initialized with at least one tool and bearerToken auth (or unsupported for servers without a bearer). MST's own readiness check already connected to each server before the batch started; this probe is a native diagnostic on top of it: it checks what the app itself loaded, which a direct connection can't see.
5. Spawn the app with fixed flags (--no-sandbox --ozone-platform=x11 --force-renderer-accessibility --disable-gpu --disable-dev-shm-usage) and fixed accessibility variables, in its own process group. The app receives only the session variables, CODEX_HOME, and MST_CHATGPT_MCP_TOKEN_<n>.
6. AT-SPI preparation, then per case exactly one draft and one Send.
7. Copy evidence, stop the app group (TERM, then KILL, then verify no member remains), restore config, and remove the workspace. The caller deletes HOME.

Built-in tool policy: code mode hides MCP tools until the model searches for them, so the model uses the first capable tool it can see. On the headless VM, that is the bundled browser (cua_repl) or web search, and the MCP server under test is never tried. On Linux, MST writes [plugins."unified-computer-use@openai-bundled"] enabled = false (the plugin that adds cua_repl) and web_search = "disabled". nativeReadiness.builtinToolPolicy records both values. The rollout records the disabled plugin in turn_context.disabled_plugin_ids. Shell commands stay available. Any built-in tool call is recorded separately from MCP calls. macOS is unchanged.

Client plugins (Linux only): a fresh profile has no plugins, so the model has no skills that point it at the MCP server. The client config has two independent parts: MCP servers (the eval config servers, which may be empty) and plugins. MST has no plugin-specific code. The caller supplies each plugin and, optionally, a declarative override for each of the plugin's own MCP servers:

"plugins": [
  {
    "name": "acme",
    "marketplace": {
      "source": "acme/codex-plugins",
      "ref": "<40-character commit SHA>"
    },
    "mcp": {
      "acme_plugin": {
        "url": "https://mcp.example.com/eval",
        "auth": { "accessTokenEnv": "ACME_EVAL_TOKEN" },
        "minTools": 3,
        "env": {
          "ACME_MCP_SERVER_URL": "${url}",
          "CLAUDE_PLUGIN_DATA": "${dataDir}",
          "ENABLE_HITL": "false"
        },
        "files": {
          "mcp-credentials.json": {
            "tokens": { "access_token": "${bearerToken}", "token_type": "Bearer" }
          }
        }
      }
    }
  }
]

After login, MST runs codex plugin marketplace add and codex plugin add for each entry. Git sources require a full commit SHA; an absolute local path (a pre-staged checkout) is also accepted, and its ref is only recorded. Each mcp key names a stdio server in the plugin's .mcp.json. MST writes a complete [mcp_servers.<key>] table under that same name: command, args, and cwd from the plugin, and the plugin's env merged with the override env. A partial env-only table would make the app reject the transport. files are written as JSON into a private per-server data dir, $CODEX_HOME/mst-plugin-data/<plugin>/<server> (directories 0700, files 0600, exclusive create, no symlinks). Only ${url}, ${dataDir}, and ${bearerToken} are substituted; any other ${...} fails validation. ${bearerToken} is allowed only in files, so it never enters config.toml. The token comes from auth.accessTokenEnv, resolved from the same environment as direct MCP auth.accessTokenEnv; a missing token fails before the app starts. It is never logged, sent to the app environment, or put in a receipt.

Plugin MCP servers are eval MCP servers. The app-server probe checks each one by name: it must be initialized with at least minTools tools (default 1). Native mcp__<server>__<tool> calls are attributed to <server>, and requireMcpCalls accepts them, also when servers is empty. A plugin server must not share a name with a direct server. Without mcp, only the plugin's skills are added. Any install error fails setup with plugin_setup_failed before the app starts. nativeReadiness.plugins records the name, marketplace, version, ref, and overridden server names.

ChatGPT Work and Codex support plain stdio servers entries on macOS and Linux: use command, optional args, cwd, and declared env. No URL or plugin is required. An optional minTools is MST's own readiness rule: the readiness check applies it before the batch starts, and it is never written to config.toml. A dry-run proxy entry is such a server, so connectors behind it work with ChatGPT too.

An HTTP servers entry may carry a bearer token (auth.accessToken or auth.accessTokenEnv), which the app reads from an environment variable (bearer_token_env_var), and nothing else: ChatGPT fails the variant before the app starts for an HTTP server with headers or with OAuth. For a server that needs a header, such as an eval ID, use a connector, whose dry-run proxy adds the connector's headers, or another stdio proxy.

ChatGPT rejects two Cowork-only forms before the app starts: client-resolved stdio entries (url, auth, files, or ${url}/${dataDir}/${pluginRoot:...} placeholders) and plugins[].blockMcpServers. For plugin-specific endpoint and credential overrides, use the Linux plugins[].mcp setup above. See cowork.md for Cowork's macOS/Linux contract.

Execution policy: on Linux, MST writes approval_policy = "never" and sandbox_mode = "danger-full-access". Native command execution needs bubblewrap, which cannot run inside the container, and nobody can answer an approval request in a headless run, so a sandboxed command would hang the turn. This relies on the caller's isolation: run the whole app in a disposable no-new-privileges container with a fresh tmpfs profile. MST does not verify this. nativeReadiness.executionPolicy records the fixed values. macOS is unchanged.

There is no setup-only mode. Setup failures carry fixed codes such as home_not_fresh, login_unverified, mcp_status_unavailable, mcp_server_not_ready, app_exited, and app_stop_failed. Batch telemetry records nativeReadiness (login state and sanitized app-server status) next to nativeSetup (AT-SPI accounting). No native output, URL, token, or key is recorded.

Draft hand-off

MST opens drafts by launching the same app binary briefly with codex://new?path=<workspace>&prompt=<text> (both components encoded like Python quote(value, safe=''); the empty setup draft omits prompt). The hand-off process gets no MCP tokens, runs in its own group with a 20-second limit, and must exit 0. MST then verifies that the original app process is still alive. The URL is one argv element, so prompts are limited to 120 KiB after encoding (prompt_too_large). Exit 0 acknowledges the hand-off only; it is not a submission.

The AT-SPI script does not open drafts itself. When it needs a draft, it writes {"open": "<sha256 of the UTF-8 draft>"} on the socket passed as --open-fd 3. MST accepts one request per invocation, only when the hash matches the draft it expects (empty for setup, the unchanged prompt for submission), performs the hand-off, and replies {"opened": true} or {"opened": false}. The script waits at most 30 seconds (capped by its deadline). After the hand-off, it polls up to 30 seconds for the draft to appear; this is read-only. Nothing is retried.

The app routes each deep link asynchronously and does not order them. A setup route that lands after the first submission's route replaces that draft with an empty new chat, so the composer shows only its placeholder (for example, Work with ChatGPT). Because the UI is already ready before the setup hand-off, readiness cannot show that the setup route landed. Preparation therefore observes read-only for 5 seconds after its hand-off, within its deadline, before it checks the surface. It does not act, read text, or reopen during this time.

If a submission's draft does not appear, the error is state_transition_unobserved. It is draft_prompt_not_applied when, at the end of the wait, the surface is ready and the composer text is exactly the text from before the hand-off. The comparison uses an in-memory hash only. Both codes fail closed with no Send, reopen, or retry.

Before each submission, a visible mode switch set to a different surface fails at once with surface_mismatch, before any action. After a previous turn, the app stays on that conversation, which has no mode switch, composer, or Send. The script observes read-only for up to 2 seconds, then opens the case deep link to a new chat. The draft checks above still require the selected surface and the exact prompt before the one send.

Native UI protocol and limits

The Node adapter uses only the prepare and submit modes of the packaged chatgpt-runtime script. It accepts --timeout-ms and --max-actions, with JSON stdin containing surface and, only for submission, prompt. A successful receipt contains status (ready or submitted), surface, action_count, and duration_ms. The Node adapter caps each invocation at 60 seconds and validates the receipt. It forwards only desktop environment variables, not model or MCP credentials, to Python.

The English selectors come from ChatGPT 26.915.31945 observations. Preparation can select Engineering, advance Continue, skip the specific example-introduction screen, confirm Go to ChatGPT, and select the requested Switch mode menu item. Engineering selection uses the unique observed radio/toggle's public AT-SPI Component WINDOW extents plus its containing frame's SCREEN origin. Integer geometry must fit within the frame and pass the public Component hit test when available. The only mouse command is xdotool mousemove x y click 1, with no --sync, hard-coded coordinates, or fallback checkbox action. MST then polls for enabled Continue on the observed profession page; a stale checked bit does not block it. Unknown onboarding and ambiguous controls fail closed.

Preparation selects the surface and calls open_prompt('') exactly once to open the canonical new chat. After the 5-second read-only settle described in Draft hand-off, it verifies the requested surface, one available editable composer, and one visible Send control (which can be disabled). If the deep link changed mode, one fixed Switch mode correction is allowed. Setup has no user prompt: it does not require or claim an empty Text readback. A visible placeholder can appear as nonempty composer Text. ready means the controls and surface are ready, not that empty text was verified or a model task ran. Preparation never supplies an evaluation query or activates Send.

Submission re-verifies the prepared surface and asks MST to hand off the original prompt unchanged, including all Unicode and whitespace. It opens that draft once and requires matching composer Text before and after any optional mode correction. A match is only text == prompt or text == prompt + '\n': the one additional native terminal LF is a bounded representation, not a query edit. This is the same allowance as native exact_prompt correlation. Two or more additional LFs, changed prefixes or suffixes, and trimmed spaces do not match. If the deep link changed mode, MST permits one fixed Switch mode menu selection back to the requested surface. The draft must still match after that selection. There is no trimming, newline cleanup, other normalization, reopen, rewrite, fresh-chat button, File menu, keyboard shortcut, clipboard input, or fallback if the draft is lost.

The composer must be unique, visible, showing, enabled, sensitive, have EDITABLE state, an allowed non-password role, and a real Text interface. Nested editable paragraphs belong to their containing editor; independent fields stay ambiguous. Readback uses unbound Atspi.Text.get_character_count and Atspi.Text.get_text(node, 0, count) to avoid the PyGObject Accessible/Text binding collision. Bare and org.a11y.atspi.* interface advertisements are supported. For each U+FFFC object character, an advertised Hypertext interface can supply an explicit reference through Atspi.Hypertext.get_link_index(node, offset) and get_link(node, index). Only a link with exactly one anchor is expanded through Atspi.Hyperlink.get_object(link, 0) and recursive unbound Text reads. An empty linked paragraph therefore reads as empty. Invalid or ambiguous links fail closed; they do not trigger a structural fallback.

If Hypertext is not advertised or its link index is -1, a narrow structural fallback applies only when the entire Text value is one U+FFFC. Starting at the owned editable composer, each expanded node must have exactly one direct child. That child must be a paragraph or text with a real Text interface, read with the same unbound methods. A paragraph can instead contain one visible, showing leaf with role static or static text and no children. Only that leaf's accessible name may supply content when it has no Text interface. Text remains preferred when available. This fallback follows direct composer descendants only; a Hypertext target alone does not establish ownership for structural expansion. Arbitrary labels, buttons, images, password fields, and non-Text paragraph/text children are rejected. Static leaf content is input validation only and is never exported as raw diagnostics or used as answer evidence.

Zero or multiple direct children leave the marker literal; there is no child-order or paragraph-joining guess. Cycles, incomplete reads, invalid metadata, and exceeded budgets fail closed. Each read permits at most 256 node visits, 16 levels including the root, and 2 MiB of cumulative source UTF-8 bytes (including object characters and static leaf content) and expanded output, within the driver deadline.

All other literal text is unchanged, including whitespace, line feeds, and unresolved U+FFFC. No paragraph separators are invented: missing reported separators cause the comparison to fail if the prompt contains them. Send stays blocked until the expanded text equals the unchanged prompt or that prompt plus one terminal LF, and exactly one enabled Send control exists. The one exception is inline code: the composer shows `x` as a code span, so its accessible text omits those two backticks. A draft that equals the prompt with every single-backtick span (no backticks or line feeds inside, not part of a longer backtick run) shown without its backticks also matches. The sent message is still matched with the exact prompt in the native transcript. This also applies when the original prompt already ends in LF; only one additional LF is allowed. MST activates Send once. It never presses Enter or retries an uncertain Send.

The action count records each draft hand-off and each native selection/click attempt separately. Typical preparation is one action; typical submission is two (open draft, Send). A mode correction adds two menu actions. These are controller actions, not model requests or MCP calls. A ready/opened receipt is not evidence of a submission. Read-only polls do not increment the action count.

An acknowledged action without an observed state change gets bounded resnapshots, not another click. Read-only snapshot traversal can restart at most twice on transient GLib errors; partial trees never authorize actions. Authentication and permission requests are not handled automatically.

Failed receipts retain status: 'failed' and an allowlisted error code. Unexpected AttributeError, TypeError, and GLib.Error map to desktop_attribute_error, desktop_type_error, and desktop_glib_error; other exceptions map to desktop_driver_failed. Raw exception text is never returned. Hand-off failures are helper_missing (no --open-fd), helper_failed, and helper_timeout; MST adds open=<code> to its error when its own hand-off failed; invalid profession geometry returns profession_geometry_invalid. Fixed composer failure steps are draft-open, draft-surface, draft-readback, and send. Failure receipts carry only error, phase, step, and draftState; never prompt text or accessible names. Failure-only draftState measurements use the same bounded, expanded readback: textLength, embeddedObjectCount, and newlineCount count observed Unicode code points; textSha256 hashes the exact UTF-8 bytes. Resolved object markers are not counted. Unreadable text omits these measurements. Readback failures return composer_text_unavailable, never raw RPC errors. The successful receipt schema is unchanged.

During draft transitions, the controller polls temporary composer/Send overlap and unreadable text within the existing deadline. It acts only after a fresh snapshot confirms the exact prompt, configured surface and unique enabled Send. Persistent ambiguity or unreadable text still fails without sending; the draft is opened once and Send is never retried.

Offline tests cover these controller contracts. Verify the app flags and hand-off against the caller's pinned image. A native fresh-session binding remains mandatory even after a successful UI receipt.

Evidence and continuation

The existing strict native transcript contract is shared across both platforms: unchanged exact_prompt matching (only exact text or one additional native terminal LF, reported as native_terminal_lf; on Linux only, Markdown punctuation backslash-escaped by the app, reported as native_markdown_escaped), a fresh session and turn for each query, and complete native final-answer/model/effort evidence. UI text is never treated as an answer. Explicit prompt_marker correlation remains opt-in.

If a truncated JSONL record is followed by an unfinished native replacement with the same next ordinal, collection waits read-only for the replacement. It never accepts that fragment as evidence. A completed replacement must pass the normal parser; corruption, mismatched ordinals, or a rewrite that never finishes still fails within the binding/run/stall deadlines. A rewrite left unfinished for most of the wait is a parse_failure, not no_matching_session, and keeps the tool calls the bound turn recorded before it; one caught only on the last polls is an ordinary timeout. A Linux binding timeout also saves a diagnostic screenshot; it is not native answer or tool evidence and does not trigger a resubmission.

Native originator acceptance is surface-specific and case-sensitive:

• chatgpt (ChatGPT Work) requires exactly session_meta.payload.originator: 'codex_work_desktop'.
• codex-app (the Codex surface) requires exactly 'Codex Desktop', as observed in preserved Linux run 35881948713 (source: 'vscode').
• Neither surface accepts the other surface's originator, CLI identities such as codex_cli_rs or codex_cli, case changes, whitespace changes, or name variants. Native metadata never selects or changes the configured surface. source alone does not establish the surface.

The internal parser API preserves existing callers: findChatgptTranscript accepts surface in its fifth-argument options; parseChatgptTranscript accepts an optional fifth argument { surface, mcpServers } after observedBeforeMs. Both default to Work and no configured labels. expectedChatgptOriginator(surface) provides the same fixed mapping.

On Linux, MST copies the matched (or bound but failed) transcript into MST_CHATGPT_EVIDENCE_DIR before teardown, as <caseId>-<random>/matched-<name> (32 MiB limit). The artifact references the copy and includes its sha256. Copies read only regular, owned, single-link files inside the session root, without following symlinks, and are verified after writing. A successful case whose matched transcript cannot be preserved fails as client_run_failed. Unmatched sessions are not copied. macOS does not copy evidence; its artifact references the native transcript path.

The preserved Codex transcript above contains gpt-5.6-terra, medium effort, and the native terminal-LF prompt form. It ends in turn_aborted, without task_complete. It is evidence for originator binding, not a completed or passing evaluation. The parser's complete flag means terminal: an abort also sets error, and the adapter returns client_run_failed (with partial telemetry) before accepting any final answer. Offline tests use a minimized, redacted abort fixture; no model calls are needed.

Native telemetry

The parser reads only native structure in the matched turn, never tool-result text:

• response_item function_call / custom_tool_call, paired with their *_output by call_id. Duration is the create_time difference. A call without output is marked pending in toolProvenance.
• A function_call namespace mcp__<label> is an MCP call. Configured labels map back with the app's namespace form (non-alphanumeric → _, so acme-eval is mcp__acme_eval). executed_tool_calls must agree. cua_repl.js, codex.list_mcp_resources and codex.list_mcp_resource_templates stay built-in tools, including when recorded as native McpToolCall items. Resource listing does not satisfy requireMcpCalls. The cua_repl and codex server labels are reserved for the app; rename configured MCP servers using either label. A targeted lookup can instead record the requested server as item.server. MST recognizes this shape only when a completed exec contains a standalone await tools.list_mcp_resources({server: "..."}) (or list_mcp_resource_templates) with a literal server, optionally assigned and printed with text(result) or text(JSON.stringify(result)), allowing whitespace between tokens. Exactly one matching native item must fall inside that call's timing, with no overlapping enclosing call. The lookup's server, arguments, error and duration remain intact. Mixed/dynamic code, unmatched items and real MCP tools with the same name retain their MCP classification; a failed lookup does not become a successful call or satisfy requireMcpCalls.
• Work code mode: a custom_tool_call exec is one built-in call. MST keeps only the nested built-in tool names (for example web__run, exec_command), input length, and sha256, never the code. Each nested tools.mcp__<label>__<tool> reference (or executed_tool_calls entry) is also an MCP call on that label. These nested calls have no separate arguments or latency, and a transcript limitation says so.
• item_completed McpToolCall, CommandExecution, and Extension web.search items (other builds) remain supported; a shared id counts once.
• Usage comes from token_usage_record (turn_token_usage).

A bound turn that times out or aborts still fails (timeout or client_run_failed). The result keeps the recorded tool calls, usage, and conversation, with telemetry.partial: true, transcriptConfidence: 'low', and the limitation "Turn did not complete; tool calls and usage are partial." Partial usage duration is the native elapsed time so far.

Each case is self-contained. After a case fails for a native, controller, or evidence reason, MST records that failure, then stops and restarts the owned app with the same profile and settings and verifies the surface again before the next case (batchLifecycle.recoveryCount). The failed case is never retried or resent. The next case still requires its own exact prompt and fresh native session, so a late answer from the failed case cannot be attributed to it. If the restart fails, the remaining cases are not submitted. A completed, reliably attributed turn can fail the configured-MCP measurement without a restart. Built-in tool calls remain distinct from MCP calls; unexpected or unattributed MCP servers fail measurement, and requireMcpCalls requires a call on the configured server selection.

Linux controller accounting is in clientMetadata.nativeController, with provider: 'linux-atspi', the selected surface, and planner/cost marked not-applicable. It is separate from native model usage and from macOS clientMetadata.computerUse planner accounting. New Linux receipts omit the old driver identity; readers still accept it in stored results.

Implementation

chatgptClient.ts selects chatgpt/macos.ts, chatgpt/linux.ts, codex-app/macos.ts or codex-app/linux.ts from the product and worker OS when its batch opens. Codex app reuses ChatGPT's native operations with its own product identity. All four entry points use the same app transaction (chatgptSetup/session.ts) and turn body (chatgpt/turn.ts). The Linux AT-SPI helper remains in chatgpt/linuxDesktop.ts; macOS uses the existing shared Computer Use loop. There is no separate platform-selector module.