Engineering case study · Original production incident and successful successor, September 2026 · Historical evidence
An AI coding run can have a valid branch, an open pull request and passing CI checks, yet still fail the final acceptance gate. That is not necessarily an inconvenience to remove. In one OpsTruth–DoneState production canary, the gate caught a concrete contract defect: GitHub's repository ID was being serialised as a string when the signed verification report required an integer or null.
The correct fix was not to make DoneState accept either type. It was to repair the independent evidence adapter and preserve the exact-subject contract.
Why the verifier must have its own boundary#
OpsTruth is a read-only inspection and evidence-verification tool. Its public surfaces include repository checks, exact-commit CI evidence and signed Evidence Graph capabilities. The relevant hosted verification path has a distinct role: observe an already prepared DoneState handoff, compare the external GitHub evidence with that exact subject and sign a decision under a fingerprint pinned before execution.
DoneState owns execution, publication of an authorised pull request and durable state. It must not certify its own outcome. OpsTruth must not acquire repository write or deployment permission merely because it is verifying another tool's work.
The handoff contract, donestate.verification-contract.v2, carries a complete response rather than a reassuring boolean. It includes the contract version, a structured report and a cryptographic attestation. The report binds repository identity, expected and observed head, check requirements and the sealed run. The attestation binds the report digest and handoff to the verifier's separate key.
A real request was refused#
In the production canary identified by DoneState run c4a07fa6-90b2-4597-a4c6-eae66de5a3e8, the independent OpsTruth response reached the v2 path, but report.subject.providerRepositoryId contained a string.
The schema expected the numeric GitHub repository ID. A value can look visually correct in logs and still be structurally wrong. In this case the distinction mattered: silently converting or ignoring it at the trusting end would have widened what the executor accepted as proof about a repository's identity.
DoneState rejected the malformed response and did not manufacture VERIFIED. The existing implementation, branch and pull request were not launched again. The repair was limited to the evidence adapter that had changed the provider's numeric ID into text.
OpsTruth PR #26 preserved the numeric type, added a regression assertion and deliberately left the sealed-subject validation unchanged. It merged as eef00ca4f242cf99d6b39e8c37ae4b84970a86e4.
The successful successor observation#
After the bounded adapter repair, the same sealed exact-head subject was re-observed. The production verifier returned the required complete v2 response with matching subject/report/attestation fields. DoneState accepted the independently signed decision and reached terminal VERIFIED.
The retained code-review subject is DoneState PR #115, head 41f1ae3b0fed670e64bd99f1bcb1aea9c9e7e869. The exact-head CI run 33806832575 passed core (22), core (24) and hosted-plugin. The pull request deliberately remained unmerged: DoneState's automatic authority was to prepare reviewable work, not approve or merge it.
Two kinds of correctness were tested here. The implementation and GitHub checks had to describe the intended revision; the verifier response had to identify that revision under the accepted contract. Neither could replace the other.
What was preserved as failure evidence#
Earlier DoneState runs had failed or become ambiguous for different reasons, including interrupted process-control operations and unavailable validation capability. Those original histories were not retroactively labelled successful when the later verifier response passed. The malformed repository-ID response similarly remained part of the debugging record.
This matters for research as well as operations: a dashboard showing only the eventual green result would miss the original class of fault and create a false picture of what the system had always been capable of verifying.
Boundary and repeatability#
The narrow result is strong: a real, versioned production contract refused malformed independent evidence and accepted a later corrected response on the same exact subject. It does not prove that every OpsTruth CLI probe executes code, that a green CI check establishes deployment, that an arbitrary third-party repository is supported by the selected verifier App or that a cryptographic signature guarantees truth absent appropriate evidence.
The repository-ID regression can be reproduced with the public contract and fixture. The hosted canary remains a dated observation of selected runtime identities and access; re-running a live workflow solely to recreate an old bad response would not add useful proof.
Original record#
- Numeric repository-ID correction — OpsTruth PR #26.
- DoneState PR #115 — retained exact-head verification subject.
- Exact-head CI run 33806832575.
- DoneState's implementation of the v2 verification boundary.
- OpsTruth hosted source and signed evidence-graph capabilities.
- Existing OpsTruth project overview.
- Why I kept the failed result.
This is a historical, source-and-receipt-linked incident study, not a claim of universal verification, fresh service uptime or independent academic review.
Technical discussion#
I am inviting criticism of this specific evidence boundary: Should independently verifiable execution-result contracts be shared across tools, or are repository-specific checks sufficient? The cross-project conversation is open in Proof & State Discussion #44. No product integration has been agreed; counterexamples, simpler alternatives and reproducible failures are welcome.