Hill climbing the harness through evals

Evals are the unsexy part of building AI products, but they have a huge impact. They let you measure existing product quality more objectively, test new model releases quickly, and ship changes to the harness with more confidence.

More importantly, if done right, evals give you a way to hill-climb the harness: measure the present score, make a change, and check whether the score improved.

What should an eval measure?

In a typical agentic product, a user gives the agent a task. After doing some work, the agent responds with “Work Done”. We need a way to verify whether the work was actually done.

There are three common ways to do this:

1. Assert on the resulting state.
2. Assert on the tools the agent called.
3. Point the task and trace at another LLM and ask it to judge the result.

Assert on the resulting state

Suppose the user asks an agent to initiate a refund.

The strongest check is to call the Orders API and verify that the order state is now Refund Initiated. These checks are cheap to run, verify the actual outcome rather than what the agent claimed to do, and are usually robust once built.

The downside is that they are product-specific. You need to know which API to call, which state to expect, and what the business rules are. A central evals team cannot automate this without context from the product team, so these checks usually take longer to build.

When the outcome can be verified directly, this is usually the best option.

Assert on the tools called

Instead of checking the final state, we can inspect the agent trace for an initiateRefund tool call. Tool-call assertions are cheap to build and run. Another agent can inspect prior traces and help generate these assertions too.

These assertions can validate the tool name, inputs, and response. Their main limitation is that they encode an expected trajectory. Agents can take different trajectories to reach the same outcome. In a coding harness, one agent may use edit, while another may use bash to update the same file. Tool names, inputs, and outputs may also change without changing the product behaviour. The more of the trace you assert on, the more the eval gets coupled to the current implementation.

That said, tool-call assertions are useful when the tool choice itself is what you want to measure:

• Did the agent use web search before answering?
• Did it use Glob for pattern search instead of bash("grep -rn ...")?
• Did it read the refund policy before deciding whether to initiate a refund?

In these cases, checking only the final state would miss an important part of the agent’s behaviour.

Use another LLM as a Judge

Point the task, output, and trace at another LLM and ask it questions about correctness, truthfulness, response style, taste, or overall output quality. LLM judges are usually quick to build, and domain experts can write success criteria in natural language without translating every requirement into code.

They are especially useful when there is no clean final state to query. Coding, math, physics, and software products have relatively high levels of verifiability. Law, art, literature, and many open-ended research tasks have lower levels. LLM judges become more useful as direct verification becomes harder.

The trade-off is that they cost more to run than code-based checks, and their output can vary between runs. Results can also change based on the rubric, prompt wording, model, and the order in which evidence is shown. They are usually stronger signals for semantic or subjective output quality than for correctness that can be observed directly. Whenever correctness can be checked against the real state, prefer that over asking an LLM to infer it from the trace.

Avoid these mistakes while building LLM judges:

• Ask one rubric question per judge call. If a rubric has five questions, ask each question independently and combine the results outside the model. When all five questions share the same context, evidence used for one answer can affect the remaining answers.
• Prefer YES or NO to 1-10 ratings. A score like 6.3 is difficult to interpret. Did every run score around 6, or did half score 3 and the other half score 9? For binary criteria, 63 YES out of 100 runs is easier to reason about.
• Use pairwise comparisons for preferences. Show the judge two outputs and ask it to choose Output A, Output B, or a tie.

Hill-climbing the harness

This is the interesting part. We want to continuously improve our products using AI.

Run evals --> Find failures --> Improve the harness --> Run evals again

The harness here includes the instructions, tools, skills, context, and workflows around the model. The model may remain the same; we are improving the system around it.

There is also a second loop that keeps the eval set changing with real usage:

Look at real usage --> Find new failure modes --> Turn them into eval scenarios --> Add them to the eval set

Identify what you are optimizing for. Before changing the harness, decide what the hill-climbing agent should optimize for. Initially, the objective is usually product quality: improve the eval score without breaking existing behaviour. Once the score starts to saturate, you can add cost and latency to the objective by trying a cheaper model, reducing unnecessary model or tool turns, reducing the prompt, or building better tools without degrading task success.

Cost and quality do not always conflict. As models become more capable, instructions written for older models can become unnecessary or even constrain the newer model. Anthropic recently removed over 80% of Claude Code’s system prompt for its newer models with no measurable loss on its coding evals, after finding that parts of the existing context were over-constraining the models.

An objective such as “reduce the prompt to the absolute minimum without degrading quality” may reduce cost and latency. It may also uncover instructions that were hurting quality. The objective still needs to be explicit: if the hill-climbing agent is asked only to improve the eval score, it has no reason to care about cost or latency.

The catch: if AI can generate eval scenarios and improve the harness against them, every product should eventually score 100%, right?

A system will optimize for whatever you measure, which may not be what you intended. Suppose the eval set contains many failed customer-refund scenarios. A poorly designed harness optimizer may learn that eagerly issuing refunds improves the score.

Product owners, domain experts, and eval authors need to agree on what good behaviour looks like before the optimizer starts changing the harness. Agent-generated scenarios still need review.

Building and maintaining the eval set

It is not practical to author every scenario manually. A good place to start is real user conversations.

Start with real user conversations. Take a sample of conversations and annotate what went wrong:

• Why did the agent initiate a refund when the order was outside the refund window?
• Why did the agent not perform a web search here?
• Why did the agent say the task was complete without writing tests?

Then identify the likely cause:

• The agent did not read the refund policy, or the policy was missing an amendment.
• The agent relied too heavily on its existing knowledge.
• The system instructions did not require tests before considering the task complete.

Keep the failure and the cause separate. The eval should test the outcome:

The agent must not refund an ineligible order.

The harness change should address the cause:

Ensure the agent reads the refund policy before acting.

Involve domain experts while annotating these conversations. Record why something was a failure, what the correct behaviour should have been, and what the eval should measure. These annotations become the thought process behind your evals, not just a list of scenarios. The human-reviewed scenarios become your golden set.

Expand the eval set. Once the golden set is useful, agents can help expand it:

• Generate variations ranging from easy to complex.
• Review more production traces and identify new failure modes.
• Generate boundary cases around policies and product behaviour.
• Generate out-of-scope requests to test whether the agent refuses or redirects correctly.

Balance the scenarios too. If you test when the agent should use web search, also test when it should not. If you test when it should issue a refund, also test when it should refuse one.

Let an agent generate 100 or 1000 candidate scenarios, but maintain a high bar on what enters the eval set. A thousand scenarios that test the same capability are less useful than a smaller set with good coverage. As models and harnesses improve, some scenarios will become trivial; it is fine to remove them, reduce their weight, or replace them with harder ones.

Keep capability and regression evals separate. Capability evals contain hard scenarios where the harness still has room to improve and give the hill-climbing agent something to work on. Regression evals cover behaviour that already works and should continue to pass after every harness change. Run both before accepting a change.

Run each scenario more than once. Both the agent and the LLM judge can be non-deterministic, so a single pass or failure should not become the score. Three to five runs is a reasonable starting point for regression evals. Capability evals may need more runs because they operate closer to the edge of what the harness can do. Increase the number of runs as the scenario complexity or observed variance increases; there is no hard and fast rule.

Avoid overfitting to the eval set. If the harness optimizer repeatedly sees the same scenarios, it may overfit to them. Keep at least two sets:

• A working set used while improving the harness.
• A held-out set used to verify that the improvement generalizes.

Do not look only at the overall score. A change may improve easy scenarios while regressing on hard ones, or improve refunds while breaking cancellations. Track scores across the slices that matter to your product.

Before changing the harness because an eval failed, inspect the trace. The failure may be in the grader, task setup, tool, or sandbox rather than the agent. Production conversations provide another check: if the eval score improves but the same failures continue appearing in real usage, the eval set is missing something.

Also version the eval set. A score of 82% on one version is not directly comparable to 82% on another version if scenarios were added, removed, or changed.

Similarities to RL environments

Evals and RL environments look similar. Both run the agent in a controlled environment, define what success looks like, and produce a score or reward.

The main difference is what gets optimized. In RL, the reward is commonly used to update the model itself. In eval-driven harness optimization, the model can remain fixed; we use the score to improve the tools, instructions, skills, and context around it.

Not every company building agents needs an RL setup. Every agentic product needs a way to measure whether it got better or worse.