RRochak Shrestha

Build log · · Rochak Shrestha

The eval that stopped us switching to a cheaper model

A cheaper model looked fine in the demo. Then we built the harness, and the blind judge read every answer.

A cheaper model was proposed to run the coaching in the AI coach I built inside Livio, and in a demo it looked fine. The replies read well and nothing obviously broke. The reason to try it was cost, which matters in any subscription app. The easy thing would have been to ship it on the strength of that demo.

If you run an LLM feature for real users and want to know how to evaluate an AI agent before switching models, this is what we did instead: LLM-as-judge evals in production, built from a golden dataset of real cases and a judge that scores two models' answers blind. We agreed in advance that the score would overrule the demo. I wanted a number that would settle it before anyone argued.

A demo that looked fine

A demo is a handful of conversations someone chose. It tells you a model can be good, not that it is as good across the conversations your members actually have. On the day of the demo we had nothing else to go on. The harness did not exist yet.

That was the uncomfortable part, and it is the first thing I would change. We built the harness after the cheaper model was proposed, when it should have existed before the first prompt.

What the coach has to get right

The coach is the whole product experience of Livio, a subscription app for people losing weight in the United States. It is an autonomous Claude agent that streams its replies and calls more than 30 tools to read and change a member's food, weight, sleep and workout data. The member confirms every change it proposes with one tap. Every number, from calories to macros to trends, is computed in code, and the model only explains them.

Because the arithmetic lives in functions, a model switch raises a narrower question for evals: is the coaching as good, and does the agent still pick the right tools? Both can be measured. If the coaching got worse, the product got worse, because the coaching is the product.

Building the harness

The eval harness has two parts. The first is a golden dataset: a fixed set of cases with known good behaviour. The cases are built from the kinds of conversations members actually have, with anything personal stripped out. Each one checks both halves of the job: whether the agent called the right tools with the right fields, and whether the coaching around them was good. The set does not change between runs, so a score on Monday and a score on Friday mean the same thing.

The second part is a blind LLM-as-judge test of coaching quality. For each case, both models answer. A judge model reads the two answers side by side without being told which model wrote which, and says which is better coaching. Coaching quality cannot be scored by string matching, so a judge with clear criteria is a practical way to score it at volume.

Blinding matters because if the judge, or a person, knows which answer came from the expensive model, the result leans toward what they already believe.

Keeping the judge honest

A judge is another model, and it has its own biases. These are the ways our harness keeps it honest.

  • Blind pairs. Neither answer is labelled.
  • Position swapping. Judges tend to prefer whichever answer comes first, so each pair is scored twice with the order flipped, and a split verdict counts as a tie.
  • A written rubric. The judge scores against stated criteria for good coaching.
  • Human spot checks. A person reads a sample of the judge's verdicts and checks they agree with it.
  • A pass bar set in advance. The candidate has to match or beat the current model before anyone looks at the results.

The judge can be imperfect as long as it is consistent and pointed at what your members care about. With the bar written down, we ran the cheaper model through the golden set.

The score

The cheaper model scored worse in the blind test.

It had looked fine in the demo and it was worse across the golden set, so it did not ship. Without the harness, the switch would have been an argument between "it looked fine" and a feeling. With the harness, the score settled it.

The cost problem was still there

Blocking the switch did not make the cost go away. We worked on cost per message without swapping the model doing the coaching.

Simple requests are routed to a small model, and the main model handles the conversations where coaching quality is the point. A simple request is one like logging a meal or asking for today's totals, where the answer comes from a tool and there is little coaching to do.

The parts of the prompt that repeat across turns are cached, so we stop paying full price for them on every message.

We also fail over between two providers serving the same model, which is about reliability more than cost. With failover, an outage at one provider means a slow afternoon.

The routing decision itself should go through the same harness: a request that goes to the small model is a model switch for that request, and it deserves the same evidence.

What I would change

Build the eval harness before the first model prompt. Built early, it gets used for every prompt change and every model release.

Instrument cost per conversation from the first week. The routing and caching work would have started months earlier with that number in front of us.

And treat a demo as a hypothesis for the eval to test.

If you are making the same call

If you need to know whether a cheaper model is good enough for your product, this is the work I do in AI operations.