OpenSidebar had spent sixteen turns working through an application
form. Then it called done().
It had typed into fields, selected options, uploaded a file, and checked boxes. The test’s field assertions disagreed with its conclusion. Required work remained.
A retry eventually completed the task, carrying forward the form state from the first attempt. The run passed, but describing it simply as a success would have hidden the most useful part of the result: the agent had stopped too early, and its completion checks had failed to catch that.
Field note 01 / A pass with a footnote
47 turns. Eventually passed.
The second
attempt did not start from an empty form.
That is the kind of problem I want an internal benchmark to expose.
It also creates an uncomfortable question. If I build both the agent and the benchmark, why should anyone trust the result?
I choose the tasks. I know the system’s weaknesses. I can change the software, the test, or the definition of success. Even without deliberately manipulating anything, I can build an evaluation that becomes increasingly comfortable for my own system.
I think an internal benchmark is still necessary. But its value depends on how willing I am to let it contradict me.
A passing run can conceal the wrong lesson
The application-form investigation contained several different kinds of failure.
In some attempts, the verifier rejected premature completion because a CV was missing or a phone field was empty. That was a useful distinction: the task failed, but the system recognized that it had failed.
In the sixteen-turn attempt, the external test caught unfinished work that the agent’s completion checks should have caught. A subsequent retry passed, but the combined run took 47 turns and inherited partially completed state. It could not fairly be compared with an earlier clean, single-attempt success.
Other attempts were interrupted by provider errors.
Collapsing those outcomes into one success rate would have made the report easier to read and harder to use. I needed to distinguish incomplete execution, incorrect completion judgment, recovery with carried-over state, and infrastructure failure. Each pointed toward a different engineering decision.
OpenSidebar’s harness—the software around the model that presents observations, executes actions, manages context, and checks completion—is responsible for much of this behavior. When I change it, I need evidence about what actually changed.
I’m not alone in running into this. AI Agents That Matter describes a similar gap between what model benchmarks measure and what developers need to know when building products. Accuracy alone can hide differences in cost and practical usefulness.
The form failure made that argument concrete for me. Knowing that an agent eventually passed was insufficient. I needed to understand how it got there and what it had incorrectly believed along the way.
How far does an improvement travel?
WorkArena provides a useful external test. Its ServiceNow workflows cover forms, lists, knowledge bases, service catalogs, and dashboards. It offers substantial depth within a real enterprise platform. WorkArena ↗
But suppose I improve OpenSidebar’s handling of a ServiceNow reference field. What have I learned about its ability to fill a similar field elsewhere?
Perhaps the fix captures a transferable behavior: enter a value, select a resolved result, and read back the selection. Perhaps it depends on something particular to that platform.
Both can be useful. They support different claims.
A strong WorkArena score does not establish transfer to unfamiliar applications on its own. Equally, its platform focus does not prove that improvements learned there fail to generalize. Transfer is something I have to test.
For OpenSidebar, I want the same workflow to appear in different implementations. A form might validate on blur, after submission, or through a delayed server response. A table might expose filters in a toolbar or a separate dialog.
If a change helps across those differences, I have stronger evidence that I improved a browser capability.
The live web changes the test
Testing across live websites introduces another problem: the environment can change between runs.
The Mind2Web family illustrates this. Original Mind2Web uses recorded website data. Its live counterparts must also maintain task validity as websites evolve.
When constructing Mind2Web-Live, researchers removed 96 of 780 candidate tasks because website changes had made them inapplicable. Another 142 were excluded for ambiguity or difficulties defining evaluation checkpoints. These were construction-time exclusions, not the current failure rate of the released suite. WebCanvas paper, §3 ↗
Separately, Online-Mind2Web’s maintainers reported updating 36 of 300 tasks in November 2025 because they were invalid or involved CAPTCHA verification. Their update history documents continued maintenance. Official update history ↗
My initial description was “broken fixtures.” Looking closer, I’d describe it differently: live tasks become stale, access conditions change, and evaluation requires maintenance.
A redesigned interface is a reasonable challenge for an agent. A requested product that no longer exists creates a different task. An evaluator expecting an obsolete element may need repair.
These distinctions matter when deciding what to fix. They also explain why I want controlled, reproducible cases alongside live evaluation.
An internal benchmark needs boundaries
For OpenSidebar, I would build around the behaviors these failures reveal: reading back form values, checking persistent state, recovering after interruptions, and recognizing when evidence is insufficient to claim completion.
Field note 02 / Three views of the same agent
Can I reproduce and fix this failure?
Watch for tidy, familiar worlds.Does the improvement travel?
Keep them out of daily tuning.Does it work as the world changes?
Recheck task validity.Controlled cases would let me reproduce specific conditions. Unfamiliar applications would test transfer. Public benchmarks would expose the system to tasks I did not select.
The internal suite would still be vulnerable to overfitting. I can tune prompts, tools, and recovery rules around familiar cases without ever changing model weights. Eventually, a benchmark I inspect every day becomes development material.
Recent research on automatic harness evolution reinforces this concern. In its Terminal-Bench experiments, one study found limited generalization and no consistent advantage over simpler search methods under comparable budgets. Its findings concern a particular optimization setting, but the warning is relevant: improvements on the tasks used to guide changes need independent testing. Rethinking the Evaluation of Harness Evolution for Agents ↗
That means separating regression cases from held-out evaluation. Claims about unfamiliar applications require holding out applications, not merely changing names and values in familiar forms.
It also means checking the judges. A validator should reject plausible near misses, including the application that looks complete until someone notices the missing attachment. Ambiguous verdicts need review. Declaring a task impossible must require evidence.
An internal benchmark gains relevance by representing users’ needs. It loses credibility when tasks are selected because the agent already handles them well.
The result I want to publish
The strongest evidence would be a small, inspectable sequence: a failure, a harness change, a successful regression test, and a comparable task in an unfamiliar application.
I would keep the model and execution conditions comparable, repeat trials, and report cost, interventions, and incorrect success claims alongside completion.
If the improvement fails to transfer, that belongs in the result. If success requires a retry with carried-over state, that belongs there too.
Owning an evaluation suite does not require inventing every task or keeping it private. Public components can be reused. Methods and traces can be shared. Its first job is to guide engineering; publishing its scores adds a responsibility to make the evidence understandable to others.
I still want public benchmarks. I also want tests close enough to OpenSidebar’s actual behavior to catch the sixteen-turn run that confidently stopped one step too early.
That failed completion claim told me something a green result would have concealed. A benchmark I can trust needs to keep telling me those things as the system improves—and as I become more invested in believing that it has.