← Building Glasstrace

The Design and Execution of Study E

Building Glasstrace, Part 9 · August 27, 2026

By the middle of August the program had an odd profile. The execution infrastructure was in place (Part 7). The preregistered study had one robust result and one fragile one (Part 5). The screening corpus was full of suggestive but flawed cells. And Study D (the letters index the program’s internal design series), which ran 72 result-bearing cross-runtime trials, had found nothing statistically significant: Claude 4 of 12 versus 5 of 12 (Fisher p = 1.0), Codex at high effort 2 of 12 versus 6 of 12 (p = 0.193), Codex at low effort 3 of 12 versus 3 of 12. The results were directionally encouraging and statistically silent. Study E was designed to end the ambiguity in one properly powered attempt, and to survive the review that would follow if it succeeded.

Statistical power as a design decision

The frozen question: under the current production server and activated account configuration, does the replicated Study D package improve repair outcomes on the flagship bug (MFG-RLY-021, the stale-cache defect of Part 4) versus concurrent control, across the Claude and Codex runtimes?

Four cells (Claude Opus 5 at medium effort and Codex gpt-5.6-sol at high effort, each under control and treatment conditions) at 64 trials per cell: 256 trials in total. The design’s own power table dictated the claim structure. At an assumed effect of 20 percentage points, the power of the stratified Cochran–Mantel–Haenszel test across runtimes was 0.93. The power for both per-runtime tests to reject individually was only 0.37. The primary endpoint was therefore pooled and stratified by design, with per-runtime Fisher tests as Holm-corrected secondary endpoints. The claim the budget could afford was thus committed to in advance rather than discovered afterward from whatever the data happened to support.

The protocol also fixes its own vocabulary. Allocation was deterministic balanced four-cell interleaving, and the protocol prohibits the word “randomized,” because nothing was randomized and a hostile reviewer would notice immediately. The treatment is designated a package — the Model Context Protocol (MCP) tools together with the observability note of Part 8 — because nothing in the design isolates the tools from the note, and the claim language must not imply otherwise.

The envelope: a dedicated account, a hard cap of $5,000, 20 execution lanes, a runner image pinned by SHA-256 digest, and the product deployment pinned to an exact Vercel deployment and commit (glasstrace-product@038cce55…). A boundary was accepted at the outset. The Study D artifacts recorded only the interface alias, multiple deployments occurred during that era, and no prior endpoint exists for a concurrent comparison. Era comparisons are therefore association only, and attributing any improvement to the new server release is out of reach by construction.

The execution record

Execution began on August 23 (the authorization file records the instruction verbatim: “Please execute study E”) and finished on August 25. It did not go smoothly; what follows is the record.

The first launch largely failed. Only shard credentials 1 through 4 were mapped to the activated account; trials on the remaining lanes failed with allowlist_denied on the decisive evidence fields. Eight accepted artifacts survived. Recovery re-launched on the 4 correctly mapped lanes under a preregistered amendment.

A silent-success defect surfaced during recovery preparation. The type-checking child process in the Codex trials exhausted the Node.js default heap of roughly 2 gigabytes and died with exit code 134, while Codex exited 0 and reported success. The correction, a pinned 4-gigabyte heap across the agent process tree, was applied to all subsequent trials. The 8 early rows ran on the default heap and are disclosed as a mixed instrument rather than quietly excluded.

Twenty expanded workflows then produced no data at all, because a stale unit test asserted the absence of the very cutover marker the expansion introduced. The defect was corrected in commit 8bfd45f3. All 20 failures occurred before any provider call, so the cost was time rather than money. A readiness gate also produced one unexplained intermittent failure (shard 9 passed once, then failed with no update trace), consuming a diagnostic amendment; the expansion required a third all-shard readiness attempt, and a fourth was explicitly forbidden.

The final execution mix: 8 initial, 64 recovery, and 184 expanded-lane trials, with the expansion’s interleaving boundary (repetition 18) computed outcome-blind before any efficacy field was read.

The last deviation looks bad in summary and is defensible in full. A prelaunch smoke test exercised the treatment arm end to end, and the Claude smoke failed: its two tool calls used exact locale-prefixed addresses and missed the decisive trace, a query-shaping behavior later shown to match a pre-existing Study D pattern rather than any server regression. I then authorized removing the “successful natural smoke” launch criterion — after the failure had been observed. The recorded rationale is that the criterion measured a study outcome, and conditioning a launch on an outcome selects on the outcome. The statistics are correct and the sequence is uncomfortable. The design requires the sequence to be disclosed wherever the result is used, and creating a record that described the smoke test as passing was explicitly forbidden.

The result

Claude Opus 5: control 3 of 64 (4.7 percent); treatment 25 of 64 (39.1 percent). Risk difference +34.4 percentage points (Newcombe 95 percent interval +20.6 to +47.0), Holm-adjusted p = 5.3 × 10⁻⁶.

Codex gpt-5.6-sol at high effort: control 13 of 64 (20.3 percent); treatment 30 of 64 (46.9 percent). Risk difference +26.6 percentage points (+10.2 to +41.1), Holm-adjusted p = 0.0025.

Stratified Cochran–Mantel–Haenszel: common odds ratio 5.47 (95 percent interval 2.87 to 10.41), p = 4.5 × 10⁻⁸. The registered verdict, verbatim: overall current package benefit — yes; benefit in both runtimes — yes.

This is the first statistically significant repair result in the program’s history, and it held in each runtime independently — the outcome the power table gave only a 37 percent chance if the true effect were 20 points. The true effect was larger.

The preregistered robustness checks: the effect was positive in all 4 repetition quartiles; no leave-one-shard-out or leave-one-repetition-out split reversed its sign; paired McNemar returned p = 3.0 × 10⁻⁶ and 0.0033; and the direction survived adversarial treatment of missing outcomes. Control-arm leakage was 0 of 128 — no control trial issued a single tool call.

There was no duration benefit: paired median differences of +12.6 seconds (Claude) and +10.1 seconds (Codex), both non-significant after Holm correction. That finding retired every speed percentage from the program’s forward-looking claims. The era question — did the new server improve outcomes over Study D? — returned “not established” in both runtimes (Holm p = 0.073 and 0.82). And Codex’s 20.3 percent control baseline, the highest control arm in the corpus, ended an obsolete internal belief that Codex could not repair this bug; Studies D and E together recorded 57 verified Codex repairs. That baseline is also why Codex’s lift is smaller than Claude’s, so the deltas license no claim of cross-runtime superiority.

The scope is narrow: one seeded hard bug, one behavioral oracle, pinned models, a bundled treatment, deterministic allocation. Within it, the answer is affirmative.

What made the affirmative possible was not the effect size. Numbers had appeared before — a 75-point screening cell, the July study’s fragile 6.3 points — and each contained an opening a reviewer could walk through. This design was built by enumerating those openings (power, allocation honesty, bundling honesty, outcome-blind boundaries, disclosed deviations) and closing them before the data arrived. The distinctive property of Study E is that no amendment touches the analysis, every deviation is disclosed, and none is outcome-driven.

Supporting data and commit references

Evidence availability: references to glasstrace-sdk are public — the repository, commits, and files are linked in the series evidence manifest. All other repositories cited above are private; artifacts may be shared as needed, and the manifest publishes each cited artifact’s SHA-256 digest so that any shared copy can be verified against the version these articles describe.