When an agent tool reports the wrong success

Ayobami John Haastrup · Source-backed engineering case study · 25 October 2026

A command can exit successfully without performing the requested operation. A snapshot reader needs to distinguish an omitted scope from one it cannot resolve. A verifier can discourage a useful assertion, or execute a different condition from the one its description promised.

These are different bugs with a shared engineering question: does the result mean what the caller reasonably asked it to mean? For an AI agent, the distinction matters because the next decision may be based entirely on that result.

Four of my contributions to Reticle, merged on 19 October 2026, provide concrete examples. This article examines the accepted source, regression tests and review discussion. The wider lesson is an interpretation of those cases, not a measured claim about all agent systems.

A running daemon is not an opened browser tab#

Issue #1692 described a misleading success path. Starting reticle serve --drive <url> could open a driven tab when it started a daemon. If a daemon was already running, the command reported its presence and returned. The requested tab was never opened.

The existing-daemon branch proved a narrow fact: a suitable daemon answered on the selected port. It did not prove that the new --drive request had been honoured. Yet the successful process outcome left that distinction to the caller to discover.

The accepted #1749 implementation refuses that combination with exit code 1 and gives a command that can perform the intended action. Ordinary serve attachment remains successful. The patch stays in daemon lifecycle handling, where the existing-daemon decision is made; it does not add another browser driver or start a second daemon.

In simplified pseudocode, the boundary is:

if an existing daemon answers:
    validate any requested HTTP endpoint
    if a drive URL was supplied:
        report refusal and a usable drive command
        exit unsuccessfully
    report the existing daemon

The review exposed a second, subtler contract. A suggested repair must be executable using the actual CLI. The maintainer requested the real URL and a non-default port in the hint. Inspection showed that reticle drive rejects --port; the supported input is RETICLE_PORT. The final hint therefore uses the environment variable for a custom port and shell-quotes the URL so its query string is preserved. The approval explicitly accepted that adjustment.

The regression tests cover refusal, the URL in the hint, custom-port syntax and ordinary attachment. The docs now describe the same exit behaviour. One limitation remains in the review: the suggested command uses POSIX shell syntax and cannot be copied unchanged into PowerShell.

For agent-facing tools, a repair hint is operational output. Its argument syntax, quoting and environment assumptions deserve tests just as its error code does.

Blank scope and failed scope mean different things#

In issue #1679, an optional snapshot scope could contain an empty string or whitespace. The old reader attempted CSS selection and could throw. Malformed non-empty CSS also escaped as an exception.

A broad catch with a whole-page fallback would hide the exception, but would erase the caller's boundary. Asking for a particular subtree and failing to find it should not quietly become a request for every element.

PR #1747 normalises optional input at the browser snapshot reader. It trims the scope, resolves a non-empty reference or selector, and keeps a failed resolution distinct from an absent scope. The accepted behaviour is explicit:

The test file checks three blank inputs, malformed CSS, a missing selector and a real subtree. The last case contains both an inside and an outside button: checking that the outside text is absent protects the scope boundary, beyond merely checking that no exception occurred.

There is an important limit to the claim. The implementation still builds and returns status, using the body fallback when the selected element is missing. That status can include page-level information such as the document title. The fix establishes an empty tree on scope failure; it is not an authorisation boundary or proof that no page information is returned.

This is why a regression test's exact observation matters. “Does not return unrelated tree text” is supported. “Prevents every possible disclosure” would go beyond both the patch and the tests.

Better advice should not inflate the evidence grade#

Issue #1686 concerned reticle_assert. Its guidance classified an element-state assertion as a bare presence check. For example, after dismissing a modal, checking that focus returned to the opener tests a specific UI state. Advice telling the caller that this only checks existence misdescribes the assertion.

PR #1745 extends the existing advice helper to recognise focus, checked, expanded, pressed, disabled, enabled and in-viewport states. It leaves bare presence, visible and hidden checks in the existing advice category. This is a change to the interpretation used for guidance, not to the underlying predicate evaluator.

The distinction is pinned in the tests: each newly recognised positive state avoids the presence-only nudge, while gradeOfPredicate still returns HonestyGrade.PRESENCE. Removing inaccurate advice does not upgrade a DOM observation into evidence that an application-side business effect occurred.

Negation needs its own treatment. A positive assertion that a located control is focused says more than a bare locator. Its negation can be satisfied when the locator matches nothing at all. A wrong locator therefore makes an absence claim particularly easy to pass without demonstrating the intended behaviour. The existing walker preserves this distinction, and the added test keeps the advice for a negated focus check.

The engineering lesson is to keep two questions separate: is the advice describing this check accurately, and how strong is the evidence the check provides? A more useful explanation should not silently raise the confidence assigned to a result.

Compile the condition that the description promises#

The fourth case, issue #1674, involved a flow success-state containing console: { absent: true }. The description promised absence of console errors. The compiled predicate omitted level, and an omitted level matched ordinary logs and warnings too. A warning-only page could therefore fail a condition described as “no console errors”.

The #1739 change materialises the default at compilation: level: a.console.level ?? 'error'. An explicitly selected level remains unchanged. This repairs the translation from user annotation to executable predicate without changing the general meaning of an unfiltered console predicate elsewhere.

Two kinds of regression check matter here. The compilation test checks the stored predicate and preserves an explicit warning level. The replay test feeds the generated predicate into the existing replay path: warning output passes and error output fails.

That second test connects representation to behaviour. Asserting only the newly added property would establish the object shape; exercising the compiled predicate checks that the shape produces the intended distinction downstream.

What the verification record establishes#

GitHub metadata confirms all four merges. On 25 October, the exact recorded PR heads were checked again: unit-tests, verify and the aggregate gate are successful for #1749, #1747, #1745 and #1739. Optional jobs were skipped. That evidence should not be rewritten as every possible check having run.

The PR descriptions separately record failing-before/passing-after local checks: four new failures and 64 focused passes for snapshot scope; seven new failures and 36 focused passes for assertion advice; a failing default-level case and 28 focused passes for console compilation/replay. These counts are the contributor's recorded development evidence, not fresh executions performed for this article.

For the daemon change, the initial description reported 13 focused passes. The later review follow-up records three direct lifecycle tests passing after the hint changes, alongside an unrelated adjacent-suite timeout under simultaneous load. The final exact-head CI succeeded. Keeping those statements distinct preserves the chronology instead of replacing the failed local observation with a retrospectively perfect account.

The reviews for the scope and advice fixes also state that the new tests fail on the old implementation. That is useful evidence that the tests detect the defect. It does not measure field failure rates, deployment coverage or end-user outcomes.

A research direction, with a boundary#

Together, these cases suggest a practical review method: compare requested intent, normalised input, executable action or predicate, and reported result. Test the mismatch between adjacent stages, including negative cases and existing behaviour that must remain valid.

This case study supports that method with four accepted patches. A research paper would need a defined population of tools, a taxonomy applied consistently, seeded or independently identified faults, and measured detection results. The four examples alone establish neither prevalence nor a new verification theory.

For AI Work Accountability, the useful contribution is already concrete: show exactly what an agent-facing tool observed, preserve the boundary around that observation, and make the corrective route work. These fixes turn that principle into inspectable code and regression tests.

Disclosure: These contributions used AI-assisted development under my direction, as disclosed in their PRs. ChatGPT/Codex also assisted with source review and drafting this retrospective. No new execution of the upstream test suites or production behaviour measurement was performed for this publication.

The curated evidence record preserves the merge identities, source links, review references and CI observations used here.