← Building Glasstrace

Diagnosing a Ten-Millisecond Span-Export Race in Next.js

Building Glasstrace, Part 4 · August 27, 2026

In July, a stochastic asymmetry in the pilot studies nearly became a finding about agents: trials using one agent runtime were capturing evidence that trials using another appeared to miss. It was actually a finding about roughly ten milliseconds inside Next.js, and about the narrow business of stabilizing a measurement instrument without concealing the defect the instrument exists to observe.

A symptom that misattributed its own cause

The flagship validation bug, MFG-RLY-021, is a stale-cache defect in Rallly. A settings change should invalidate a registration gate; a cache-invalidation flaw allows a subsequent read to observe the stale state. The validation fixture exercises it end to end: change the setting, then immediately read the registration route.

Intermittently, and with different frequency across pilot cells, the captured trace for that flow was missing its spine. The route-level span for the settings action never appeared, no trace carried the decisive instance_settings.update to registration_gate.evaluate chain, and trials failed with the reason symptom_without_ordered_trace_evidence. Because the miss rate differed between a Claude pilot cell and a Codex pilot cell, the tempting reading was that the agents behaved differently. The correct reading, established by the July 8 diagnostic, was that the capture gate ran before any agent was involved. The asymmetry was a pre-agent instrument fault presenting as an agent difference.

The race condition

The mechanism, once isolated, is a streaming-lifecycle hazard. The fixture’s context close and immediate follow-up read land within roughly 0 to 10 milliseconds of the settings action’s response. When the client abandons the response body that quickly, the Server Action’s route-level span is torn down before export. The span never exports, so the trace that should carry the causal chain does not exist; the child spans survive while the parent is lost.

A delay-series experiment made this quantitative. Inserting a pause between the settings action and the read:

The third line is the constraint that matters. Too little delay leaves the instrument unreliable; too much masks the bug by scheduling around it, leaving the study measuring nothing. Ten milliseconds sits in the narrow band that stabilizes capture without disturbing the defect. The delay was not sufficient on its own: the passing revalidation combined it with a fixture-ordering correction, and on July 8 the flagship bug passed 10 of 10 attempts at the live capture gate whose origin Part 3 described.

Three scope notes apply. The delay is a validation-harness fixture pacing default (DEFAULT_MFG_POST_SETTINGS_DELAY_MS = 10 in glasstrace-validation/scripts/lib/runtime-fixture-trigger.mjs), not product code, and it stabilizes fixture capture only; the reproducer work, described below, later established that the underlying trigger leaves spans unended at every delay tested. It is validated for this bug specifically and is not inherited by other scenarios. And a residual remains even at 10 milliseconds, designated Mode 1b: a later, fast render of the registration route can still lose its render route subtree and its side-effect child. Separately, the ambient capture-gate miss rate — the pooled route-span loss without artificial pacing — is approximately 1.66 percent, and the July 31 findings log records that this figure had not yet been incorporated into the wave plan’s missingness budget. The repair brief discloses one further residual: when no carrier span survives at all, no server-side reconstruction is possible, and only a framework-level Next.js fix would help.

The product-side defect: DISC-2009

The race is a genuine capture-path defect rather than a harness peculiarity, tracked in the product repository as DISC-2009: a never-ended server route span whose child spans survive. The characterization work shows how demanding the word “rare” can be.

Pooled across two instrumented runs, the count stood at 6 failures in 6 qualifying events with zero counterexamples — and the verdict was still, formally, inconclusive, because the preregistered threshold required 5 qualifying events per run and neither run reached it (roughly 2 in 200 attempts, then 4 in 161). Six of six feels conclusive. The denominators nonetheless fall short of the preregistered threshold, and the protocol takes precedence.

The reproducer ceiling is approximately 1 to 2.5 percent event frequency on Next.js 16.2.1, which means the race cannot be brute-forced quickly; it has to be trapped. (The validation fixture concentrates the triggering pattern far above this ambient rate, which is how it could fail 1 attempt in 10.) The trigger was pinned to response-body cancellation and reproduced across Turbopack development mode (16.2.1 and 16.2.10), webpack development mode (16.2.10), and a forced production server. That last reproduction eliminated my comfortable initial theory that the behavior was an artifact of the development server.

The repair program (ING-034, the server-side route-span reconstruction wave) is architectural rather than a point fix. If the runtime can cancel a parent span before export, the ingestion layer must be able to reconstruct route spans from surviving evidence. Most of the wave has shipped: the wave plan (glasstrace-product@3b81b5ef); the SCHEMA-043 partial-evidence surface (8eaa6d93, with the implementation in pull request 1291 adding the orphaned_partial_evidence reason and a “partial” status on get_root_cause — deliberately scoped to that one tool, a scoping tracked as its own open item, DISC-2025); write-side route recovery marked routeSource: "reconstructed" (pull request 1292, Slice 1b; the Slice 1a read-time predicate shipped with the SCHEMA-043 implementation in pull request 1291); the provenance-propagation predicate walkers (pull request 1293, merged July 14); an observe-only span-lifecycle diagnostic shipped in the software development kit (SDK) as SDK-063, version 1.29.0; and the internal Next.js reproducer that pinned the trigger — the wave’s highest-numbered slice, closed by d2cc7e17 on July 15.

What remains open is narrow and specific. Slice 4, the building-account dogfooding exercise — a second vanilla-OpenTelemetry exporter pointed at the production endpoint, with a self-observation filter and a kill switch — is gated on a human-performed account-provisioning step and has seen no commits since July 15. And DISC-2009 itself remains open at priority 1, pending a powered post-fix regression proof at tuned pacing and a standing canary to run on every Next.js version bump; no such run exists after the July 9 sweeps, and the tracking brief states that the delay mitigation stays in place until that proof lands. Current studies stand on the ten-millisecond fixture mitigation and the disclosed Mode 1b residual.

Three decisions

Mitigate in the fixture versus repair in the product. The fixture delay came first because study validity could not wait for an ingestion redesign. The cost is that every subsequent study carries a disclosed instrument choice while the product defect stays open under its own tracking. I consider the ordering right, and only because both halves are documented.

Trap the race versus trust the pool. Six of six with a formal verdict of “inconclusive” is the price of preregistered thresholds. The alternative, declaring conclusions on insufficient denominators, is the exact failure the validation program exists to prevent.

Diagnose before attributing. The program was about one incurious week from publishing an agent-comparison artifact whose headline difference was a span-lifecycle race. Since then, every cross-condition difference gets a pre-agent instrument check before anyone interprets it. The rule cost nothing to adopt and has already prevented one wrong interpretation.

Part 3 concerned evidence that did not exist where we believed it did. This article concerns evidence that still goes missing about 1.66 percent of the time even after mitigation, which is in some ways the more dangerous condition: a failure rate that low is easy to overlook and still large enough to bias a study.

Supporting data and commit references

Evidence availability: references to glasstrace-sdk are public — the repository, commits, and files are linked in the series evidence manifest. All other repositories cited above are private; artifacts may be shared as needed, and the manifest publishes each cited artifact’s SHA-256 digest so that any shared copy can be verified against the version these articles describe.