Evaluation is quietly becoming the bottleneck

Ask a team building an AI feature what's slowing them down and the answer is almost never "the model isn't good enough." It's some version of: we changed the prompt and we think it's better, but we can't prove it.

That's an evaluation problem, and it has a different shape from the testing problems software teams are used to.

Why the old tools don't transfer

Conventional tests assert exact outcomes. AI outputs are open-ended, and two very different strings can both be correct while a third, nearly identical to a correct one, can be badly wrong. Exact-match assertions are either useless or actively misleading.

So teams fall back on reading outputs by hand. That works at ten examples and collapses at a hundred, which means the feedback loop that should run on every change ends up running on none of them.

What seems to actually work

Nothing exotic, and all of it cheap:

• A small, real, hand-built set. Thirty to fifty genuine cases, including the ones that previously went wrong. Curated by someone who knows the domain, not sampled at random.
• Assertions on properties, not strings. Did it cite a source? Did it refuse when it should have? Did it stay under the length? Most real failures are property failures.
• A written record of every regression. Each production failure becomes a permanent case. This is the part that compounds.

The uncomfortable part

Building this is unglamorous and it competes directly with shipping features. It is also the thing that decides whether the fiftieth change to your system is an improvement or a coin flip — and that decision arrives whether or not you prepared for it.