← Building Glasstrace

Validation Infrastructure Before Features: Building the Study-Execution System

Building Glasstrace, Part 7 · August 27, 2026

On July 9, 2026, I seeded a new repository whose sole purpose is to execute studies of Glasstrace on rented infrastructure. Seven weeks later it holds 1,845 main-branch commits through August 26, 1,719 of them from August, and the two validation repositories together now out-commit the software development kit (SDK) by roughly five to one. For a one-person company, that allocation needs justifying.

Why local validation stopped scaling

The May and June validation work (Parts 2 and 3) ran on machines I administer personally. By July the program had outgrown them. Paired studies require dozens of concurrent, identical, disposable environments — application, seeded database, mail-catching service, agent runtime, capture pipeline — and a personal workstation’s notion of “identical” degrades with every software update. The remedy was to move the entire harness onto GitHub Actions: ephemeral runners, a pinned container image, parallel execution lanes.

The seed commit (glasstrace-remotevalidation@2a8a5e2e) carried the harness scripts, oracles, bug manifests, and the virtual-machine matrix plan. The commit immediately after it (a913163a) was a Phase 0 portability, packaging, and provenance audit together with a runner-simulation container smoke test.

Two constraints shaped the design. My personal GitHub plan provides no larger runner classes, so the system had to scale wide — many standard lanes, eventually 20-lane sharding — rather than tall. And every trial consumes real provider money, so the system needed financial brakes as much as throughput.

The architecture, at the level that matters

Three subsystems carry the weight.

The first is the spend ledger: an append-only ledger in which every paid dispatch requires a committed reservation beforehand. The order of operations is reserve, commit, push, then dispatch, and the in-workflow gate reads the committed ledger, so an unpushed reservation blocks execution. Accounts carry hard stops and authorization caps that the harness enforces rather than merely documents. This can look bureaucratic until it operates under load. During Study E the ledger recorded 302 reservations and 302 settlements, 604 rows with no orphans, and the study consumed a ledger-booked $593.46 against a $5,000 authorization (the Codex share of that figure is flat-rate proxy accounting; Part 10 explains). Worst-case liability was readable from a file at any moment, which is what allowed 20 lanes to spend money overnight without supervision.

The second is sharded execution with exclusion: 20 managed application and database shards, a shard lease providing exclusion across workflows, a study-level lock providing mutual exclusion between studies, and a deployment guard that pins the product revision a study is permitted to address. The runner image is referenced by its SHA-256 digest rather than by tag, because floating tags are how instrument drift gets into a treatment effect.

The third is evidence retention as the default. Every trial ships its artifacts — transcripts, Model Context Protocol (MCP) call ledgers, oracle payloads, span exports — into retained bundles. The post-hoc mining that later became central to the program (Part 10) was possible only because retention was never optional.

The ledger meets reality

The first reconciliation against the actual provider console, on August 2, showed the ledger carrying $57.99 where the console showed $38.06 — an overstatement, which is the error the estimates were designed to make. The entire gap sat in estimated rows: trials that die without artifacts are charged a conservative flat estimate, set high on purpose ($1.00 per trial for the smallest model, against measured costs of $0.30 to $0.48). That over-reservation, compounding across the screening waves’ 21 lanes (Study E later ran 20), is also what forced three cap raises in a single day, from $65 to $150 to $250 to $350. Conservatism in a spend gate is a feature. It also has carrying costs, and both facts are now documented.

A second modeling lesson ran the other way, and its first version did not survive its own evidence. Scaling the Sonnet unit cost up from the Haiku measurement by list-price ratio under-predicted by roughly a factor of two, because token usage rose with tier as well, and the findings log generalized accordingly. The first Opus trial refuted the generalization within the hour: Opus’s usage ratio came in at 0.97, its list-price arithmetic accurate to about 1 percent. Usage is not monotonic in tier. What actually changed were two constants — the Sonnet per-trial estimate, $0.83 to $1.80, and the orphan-settlement estimate, $1.20 to $2.15 — and the log records the retraction beside the withdrawn claim.

What the system made possible

The concrete answer is Study E (Part 9): 256 accepted trials across four cells, the trials themselves running unattended across three phases in under two days (phase transitions and amendments were human-authorized; Part 9 covers them), with complete artifact coverage (2,688 of 2,688 critical files), for a ledger-booked $593.46, or 11.9 percent of its authorization. Its predecessor, Study D, ran 72 trials for $124.71 in provider spend. None of these numbers is large; a study has become an ordinary event, and a single founder can commission a preregistered, multi-arm, cross-runtime one with roughly the overhead of a continuous-integration run.

The less obvious gain is symmetric skepticism. The same machinery that runs studies also audits them: independent ledger regeneration (128 of 128 MCP ledgers replayed, 587 of 587 tool rows matched in the Study E audit), oracle-payload contract checks (256 of 256), and control-arm leakage checks (0 instances across 128 control trials). When the machinery that produces a study also audits it, a statistical claim stops being a number in a presentation. It becomes an object that can be re-derived from the retained artifacts. Those artifacts are private today; the series evidence manifest publishes their SHA-256 digests, so a copy shared later can be verified against what existed when this was written.

A reader of the commit history will notice one aspect of the operating model quickly: execution of this program is heavily delegated to artificial intelligence (AI) coding agents. Agents write most of the harness code, run the studies under the controls described above, and draft the analyses and reviews. The ledger discipline, shard leases, and deployment guards exist precisely so that delegated execution cannot outrun human authorization — every study start, cap raise, and publication is authorized by me, and the instruction that started Study E (“Please execute study E”) is recorded verbatim in its authorization file. Where this series describes a review as independent, the word means an independent implementation against hash-bound inputs on a separate software stack. It does not mean a disinterested third party, and no claim in these articles should be read as if it did.

The trade-off

The five-to-one commit ratio is deferred product work. The browser extension sat pending store review and the billing integration remained unstarted, both of them tracked, while lanes and ledgers were built. The bet underneath is that in this product category, proof is the rate-limiting step: many AI tools claim to help, and few of the claims survive adversarial review. Whether that pays commercially is open. What can be stated now is that every claim Glasstrace makes has a run identifier, a ledger row, and an artifact bundle behind it, and that the marginal cost of the next claim keeps falling.

Supporting data and commit references

Evidence availability: references to glasstrace-sdk are public — the repository, commits, and files are linked in the series evidence manifest. All other repositories cited above are private; artifacts may be shared as needed, and the manifest publishes each cited artifact’s SHA-256 digest so that any shared copy can be verified against the version these articles describe.