AI writing only impressed me once, the first time I tried ChatGPT and asked it to write a limerick. Guessing the next token is a neat trick but it doesn’t know what I want to say so it can’t say it for me.
But its coding ability is another thing entirely. I’ve been using AI models to help me code since late 2024 (I started with Claude Sonnet 3.5 and OpenAI GPT-4o in GitHub Copilot). Every couple of months something has blown my mind, from either a new technological breakthrough (new model, new harness) or simply a new-to-me use case. This post is about the latter.
Sometimes I tell a friend or coworker about the breakthrough; other times I just sit and stare at the computer, or go for a walk. This time I’ll try writing it. My purpose is two-fold:
- A diary that captures a moment in time. AI coding has changed so rapidly, and so completely since 2024, that maybe it’ll be nice to have an archaeological record of what will be olden times.
- To give non-coders who might read this a glimpse of what AI can do with programming these days.
Today I’m writing about “evals,” short for evaluation of AI products. I have built a few chatbots for internal use at work. At their simplest, they let users search “what is the maximum sick time paid out upon retirement?” and the most relevant chunks of policy text are fed to an AI model to synthesize. I could say more about these chatbots but that’s for another post.
When I built them, I tested out different models to see which should handle the task of synthesizing the answer. I pick from the choices on OpenRouter, which can route your call to any of hundreds of models. My testing was ad-hoc: I’d run the same query against different models and see what the cheapest model was that could give a good answer in a fast enough response. “Good answer” was just my gut opinion.
This month Anthropic released Claude Haiku 5.5, which scored high on benchmarks and was 90% cheaper than it’s predecessor Haiku 4.5. I wondered, if I swapped in Haiku 5.5 for the models currently powering my AI services (mostly Gemini 3.6 Flash), would I save money and get a better result? I decided that instead of hand-evaluating it myself on a few questions, I would direct Claude Code to do some evaluation.
My prompt was simple. I feel a little embarrassed sharing a prompt but I’m trying to provide a window into where these things are at, so:
Read more: AI coding in October 2026: AI independently evaluating AIOur different RAG services in this repo use different LLMs. At first we did no evals, then some informal / ad hoc testing. Today I’m interested in Claude Haiku 5.5, which is on OpenRouter for cheap and might outperform some of the models we’re using. Maybe this is a good occasion to formalize a set of evals? I’m new to eval practices for LLM work, so let’s maybe ease in with a single service. I was thinking you, Opus in Claude Code, would come up with evaluation tasks, then run the existing model vs Haiku 5.5 and score them. Store the results somewhere findable for future such work. Maybe it could capture qualitative notes too, e.g., if the responses have some characteristic or quirk, or if the prompts need to change to improve performance? Start by evaluating which model that we currently used is best suited for replacement with Haiku 5.5, based on its performance, class, release date, and cost on OpenRouter. Haiku 5.5 is currently $0.10 input and $0.50 output.
Based on that, Opus 5.5 did the following:
- Identify which AI chatbot service’s LLM was the best candidate for replacement with Haiku 5.5
- Pick 31 example user questions based on our logs of questions users actually asked
- Write its own answers to those and create a key for for what each answer must or must not include along with a scoring rubric
- Run those sample questions through the models of interest (Haiku 5.5 and Gemini Flash 3.6)
- Assign reviewer subagents to grade those outputs, blind to which model was which, and to choose which model had the better response to each
- Compute the average scores, response time, and cost (based on actual billing from OpenRouter), concluding that Haiku 5.5 beat the Gemini model at a quarter of the cost.
This was already a stunning result. I responded with:
Evaluate the prompt improvements and implement any that are actual improvements.
Another mediocre instruction from me and it was off to the races again. Opus tweaked the prompt that accompanies the user’s question, then ran the sample questions through the models again, blind-grading the outputs and double-checking the graded results from the reviewer subagents. It found an overall improvement, but noted, “the concision tweak on lookups shows mixed results (trading completeness for clarity), so I’ll test a v3 that keeps action-critical details while trimming only irrelevant provisions, then grade it against v2.”
It found that v3 was not an improvement. In the process of grading the v3 answers, it noticed some recurring defects in the answers, like the answers containing “the context you provided” – a confusing response from the LLM, given that the excerpts it was fed were search results that the user never sees. So Opus 5.5 created a v4 prompt that was a tweak of v2, ran it all one more time, and ended up with this:
Along the way it found some bugs in the overall AI chatbot service: answers were sometimes cut off (I needed to increase the maximum allowed token limit for the OpenRouter response) and that one of my input policy documents was not being ingested correctly into the knowledge corpus, causing the LLM to be fed garbled text.
The command /usage showed total cost $38.22, session runtime of 1h 10m. I used over 90% of my five-hour session limit, which I rarely hit. The main Opus agent context window was at 267k tokens (out of 1 million). I’ve just started using a Claude Code plugin called carbonlog to estimate the environmental damage caused by my personal Claude Code usage, but don’t have it installed on this work machine so can’t give a number here. Very much ballparking it, I’d wager 0.25-0.5 kg CO2 for a heavy Opus 5.5 session like this?
I didn’t pay that $38.22. I’m on a Team plan that costs my employer $20/mo for a fixed amount of weekly usage. $38.22 is the cost I would have paid using the Claude API or on an Enterprise plan. Pretty good deal for a single session!
But even $38 feels reasonable. This was an astounding session. I spent two minutes telling it to figure out if I should switch models and improve the prompt and I got an answer that was many times more rigorous than anything I could have feasibly done, in a sliver of the time.
I glanced briefly at the answer key it drew up, read its output in the Claude Code session, and deployed its changes. Like so many other coding things these days, what I know about evals now is the general concepts and that AI can perform them autonomously. Which feels like all I need to know to apply them in the future.
Is this post surprising or interesting or intelligible to non-programmers reading this? I wonder how this will age. At any rate, here’s a record of a facet of programming with AI that, in October 2026, impressed me a great deal. It’s a wild time to be coding.
