Thirteen deferred vaults were not thirteen lock collisions

Engineering case study · Evidence observed 28–29 October 2026 · Historical result, not a claim of current system health

The original symptom looked simple: an automated collector deferred thirteen of its eighteen enabled content vaults. An engineer could reasonably read that headline as thirteen independent lock collisions. The records showed something more specific—and more useful. Two acquisitions actually encountered a busy runner. A rule that stopped after two consecutive refusals then prevented eleven other vaults from even being attempted.

That distinction changed the fix. Replacing a scheduler, buying another worker or increasing timeouts indiscriminately would have treated the aggregate symptom rather than the control-flow defect. The work stayed inside Post-Once's existing durable execution paths.

The operational setting#

Post-Once is a local, agent-operable publishing utility used for portfolio content. It collects material from approved vaults, maintains durable scheduling and publication receipts, and interacts with X, Threads and LinkedIn under owner-specific permissions. It is not OneClickPostFactory, the separate hosted product, and it is not a blanket guarantee that every third-party platform is readable or writable.

For this incident, the relevant boundary was between concurrent vault import, an empty run-due scan and a shared RunnerLock. The collector must make progress while preserving the existing publication safety gates. A missing source item may be retried as a collection operation; an ambiguous external publication may not be resent just to make a dashboard green.

What the incident actually demonstrated#

The original deferral count decomposed into:

The concurrent run-due invocation reported zero due results and zero consumption, but it still occupied the critical runner boundary long enough to interfere with importers. The second problem was continuation: the collector's two-consecutive-busy rule abandoned the remaining work instead of preserving a bounded, durable retry worklist.

Crash tests exposed related failure paths. A terminated process could leave an incomplete lock marker, and interrupted receiver initialisation could leave partially constructed local state. That combination mattered because the system had to distinguish a dead owner from a live owner without clearing the permanent publishing guard or duplicating D1/SQLite work.

Repairing the existing layer#

The reviewed change used several small mechanisms rather than a new architecture.

First, the empty-queue path avoided unnecessarily holding the runner lock, but retained a guarded recheck to protect against work arriving around that decision. Importers received bounded waits rather than uncontrolled contention. Deferred vaults were carried in a finite, persisted retry worklist so the next eligible collection could continue from known state.

The crash-recovery changes made lock markers atomic and bound their ownership to actual process and boot identity. Receiver initialisation assembled complete state before publication with a no-replace filesystem operation. The important invariant was not merely that a restart worked in one happy-path test; it was that a restart could not mistake another owner's work for abandoned state or promote incomplete files into an authoritative receiver.

A separate reviewer then identified an evidence-classification defect. Legitimate verified_discovered provider evidence was not being recognised through the shared strict validator. Another independent review identified a rotating-readiness problem: a clean subset could hide an outstanding error in a different project. The correction retained complete project coverage in each cycle's snapshot. Neither repair gave the publisher new authority or changed any original publication receipt.

What passed in production#

After the compatible-reader bridge and guarded no-stop installation, the exact merged application revision was 7c14b5612b75a4a84811d849382f7640c5479142. The installer switched at 19:28:11 UTC on 29 October 2026, preserving publishing services and timer identities without manually forcing a publication.

Two naturally scheduled collector cycles subsequently produced the acceptance evidence:

The independent observer recorded two_healthy_observed_cycles=true at 19:50:07 UTC. Both snapshots retained all ten required readiness subjects and reported no incomplete collection stages.

The read-only cross-store audit inspected all eighteen selected D1/SQLite receivers, including stock and cursor alignment, inbox/receipt/acknowledgement counts and SQLite quick_check. It found the selected receivers aligned. This was a bounded read-only observation, not an atomic cross-system snapshot or a demonstration of recovery from physical disk loss.

The final ledger comparison preserved all 1,754 previously verified receipt events (with 1,757 events present in the inspected state) and all 87 prior reply post IDs. It found no new multiple-post logical effects in the inspected ledgers and no invalid critical ledgers. That supports preservation of the recorded history across the rollout; it does not certify every external platform effect ever attempted.

The part that remained open#

Collector durability acceptance was not full Post-Once production acceptance. The same closeout kept distinct unresolved work: 25 owner-review campaign cases, 647 permission-gated cases, one reply missing a post ID, external LinkedIn read permissions, 53 editorial requests, ten owner readiness attestations and unavailable learning evidence. The Workers AI editorial-generation path was returning HTTP 413 without enough retained safe metadata to prove a specific token or payload-size remedy.

Five vaults had source-linked publication proof; thirteen had unknown historical source lineage. Unknown lineage is neither evidence of a failed publication nor permission to resend one. Crash tests covered controlled process exits and restart identities; a real production reboot and disk-loss recovery were not demonstrated by this acceptance.

The public reproducibility limit is also material: Post-Once's main repository and many original provider/operational receipts are private. The numerical results above come from the recorded author/independent-review closeout, but an anonymous reader cannot reproduce the live provider checks from a public repository today. A future public reproduction should use sanitised fixtures, not real credentials or replays of uncertain effects.

The engineering lesson#

A useful reliability report has to separate what was attempted, what actually failed, what the policy skipped and what later completed. Calling thirteen deferrals thirteen lock collisions would have sent debugging effort into the wrong layer. Calling two healthy cycles complete production readiness would have hidden separate permission and editorial gates.

The strongest result here is not a slogan about autonomous publishing. It is a dated, exact-revision recovery: two observed lock refusals, eleven policy skips, a focused repair, a guarded no-stop deployment, two consecutive healthy natural cycles and preserved historical evidence. The distinction lets another engineer decide which part of the claim they can trust.

Sources and evidence boundary#

• Source of record (restricted repository): AyobamiH/post-once, PRs #380 (lock/root-cause repair), #381 (guarded release recovery), #384 (compatible reader bridge), #385 (two-cycle acceptance), exact deployed revision 7c14b5612b75a4a84811d849382f7640c5479142.
• Validation receipt digest: SHA-256 ab1891c43ccf77c77021c550cc9b33c06169bd5dd68a039899c9eb7b6c3fc2ee.
• Final independent closeout digest: SHA-256 1a881175c41a2fb14579331a02c972639c84e1583b2716c548c56a4cb8b0de99.
• Private evidence references: the author-controlled 29 October closeout includes immutable cycle snapshots, the independent observer result and a D1/SQLite consistency review. These references are deliberately not presented as public-downloadable files.
• Related reading: Why I kept the failed result; What counts as evidence that AI work is done?.

Authored as an engineering account of completed, bounded work with AI assistance. No peer review, universal exactly-once guarantee, independently reproduced public test suite or real-time status claim is implied.

Technical discussion#

I am inviting criticism of this specific evidence boundary: How should agents represent a possibly committed provider effect when an acknowledgement is lost and retries could duplicate the action? The cross-project conversation is open in Proof & State Discussion #44. No product integration has been agreed; counterexamples, simpler alternatives and reproducible failures are welcome.