← Building Glasstrace

An Unresolved Instrument Effect: The Observability Note

Building Glasstrace, Part 8 · August 27, 2026

One measurement in the screening corpus says the observability note halves Claude’s engagement with the tools. The largest study in the program, run with the note enabled, measured 100 percent engagement on the same runtime. Both measurements are real. What the note actually does remains an open question, and the program has chosen to leave it open rather than force an answer from inadequate evidence.

The finding

Glasstrace ships with a short “observability note”: a neutral, informative paragraph in the agent’s context explaining that runtime evidence tools are available. Guidance text of this kind is the cheapest product surface there is (no code path, only words), which is why the August screening data about it mattered.

Section 49 of the findings log records the following, drawn from the screening corpus on the flagship bug: Claude Opus 5 under natural prompting, with Model Context Protocol (MCP) tools available. Without the note: 88 of 88 trials engaged the tools (100 percent), and 56 of 88 repaired the bug (64 percent). With the note: 13 of 26 engaged (50 percent), and 8 of 26 repaired (31 percent). Fisher exact tests: p = 2.4 × 10⁻¹⁰ on engagement and p = 0.0037 on repair. The findings log described this as the largest and best-powered effect in the corpus at the time.

The note exists for a concrete reason, which matters to everything that follows. The Codex command-line runtime (version 0.144 and later) hides MCP tools behind a tool_search step, and without the note Codex did not discover the tools at all — 0 of 3 trials engaged, against 2 of 3 with it, per Section 49 of the same log and the resolution record of the July Codex-regression investigation. Claude received the identical paragraph for cross-runtime parity, not because Claude needed it.

From the product’s perspective the reading was painful. The paragraph written to encourage tool use was associated with engagement falling by half. A plausible mechanism exists: a model told that observability tools are present may treat the situation as managed, satisfy the instruction minimally, and return to reading source code, whereas a model that discovers the tools in the middle of difficulty treats them as leverage. Plausible is not the same as tested.

The complication

Study E (Part 9) ran with the note enabled in every treatment-arm trial. This was deliberate. The note was part of the replicated Study D instrument, and the design document states that its measured suppression of Claude was part of the instrument being replicated, not a reason to alter the instrument mid-series. Given the Section 49 figures, I braced for a weak Claude arm.

The observed result was different. Claude’s note-carrying treatment arm engaged 64 of 64 times — 100 percent, Wilson 95 percent interval 94.3 to 100 — completed searches in 64 of 64 trials, and obtained candidates in 64 of 64. The halving did not appear. At 64 trials per arm under the current instrument, engagement was indistinguishable from perfect.

Neither result can currently be declared correct; the question is unresolved. The two observations are separated by nearly everything: a different production server pin, different account activation (full capture fidelity), a post-recovery harness, a pinned 4-gigabyte heap, different shards, a different era. Study E contains no note-disabled arm, so it cannot test the note. It can only fail to reproduce the context of Section 49. And Section 49 itself was weakened on two fronts afterward. The August 25 methodology roadmap ruled that a historical note-disabled cohort compared against historical note-enabled runs cannot estimate a note effect; only a concurrent randomized comparison can. Provenance narrowing then found that only 21 of the 88 note-free trials explicitly record the matched effort tier, so the clean-comparison subset is smaller than the headline implied.

What survives with confidence is directional and modest. Study E’s note-enabled repair rate (39.1 percent) lies between the Section 49 note-enabled figure (31 percent) and the note-free figure (64 percent). That ordering is consistent with a note effect, and it proves nothing.

The decisions inside the ambiguity

Three decisions were made inside this uncertainty.

The instrument was not changed mid-series. When Section 49 appeared, the tempting response was to remove the note immediately, on the theory that it might be suppressing our own treatment effect. But comparability between Study D and Study E depended on a frozen instrument, and exchanging guidance text between studies converts a measured series into two anecdotes. The note stayed enabled and the suppression risk was written into the design. Study E then succeeded with the note enabled, an outcome that would have been uninterpretable had the instrument also changed.

Our own finding was downgraded. The roadmap ruling that removed Section 49’s decision-grade status is the program’s own rule, applied retroactively to its result. The downgrade removed a tidy narrative (a guidance defect found and fixed) and left a methodology with no exemption for findings we are fond of.

The experiment that would settle the question was deferred. The clean test exists and is designed: Study E version 2, a concurrent randomized comparison with the note enabled and disabled. It is postponed, not from lack of curiosity, but because the prelaunch sequence prioritizes capture correctness and guidance-content repairs ahead of another repair study, and because the decision the note question would change is not blocking anything shipping this month. Any such design also has to respect the note’s original purpose: switching it off may simply break Codex’s tool discovery, so a note-off arm is not the free simplification the halving story makes it look like. Experiments compete for the same budget as everything else, and “interesting” is not a priority class.

Why publish an unresolved finding

Because the alternative is worse. Research programs are damaged less by wrong results than by the quiet disposal of results that stopped being convenient. Section 49 stands in the log with its open status, its methodological downgrade, and its failed-reproduction context. The contradiction is retained.

The object-level lesson generalizes to anyone building agent products. Guidance text is a treatment, not documentation. It has effect sizes, and those effects can be large, negative, unstable across instruments, and confounded with everything else changed in the same period. Few engineers would ship a load-bearing code path without a test. Prose in an agent’s context deserves the same suspicion, and so does the first experiment that tells a dramatic story about that prose.

The note, as of this writing, remains enabled and under suspicion, pending the concurrent comparison described above.

Supporting data and commit references

Evidence availability: references to glasstrace-sdk are public — the repository, commits, and files are linked in the series evidence manifest. All other repositories cited above are private; artifacts may be shared as needed, and the manifest publishes each cited artifact’s SHA-256 digest so that any shared copy can be verified against the version these articles describe.