Change the Prompt, Keep the Comparison

A thought experiment: a team updates an AI prompt to make one answer clearer. The new version looks better in that example, but the change also affects edge cases nobody checked. Replacing the old prompt immediately would make it harder to tell what improved—and what quietly regressed.

I’d keep the previous version and a small set of representative test cases, including examples of behavior the team wants to avoid. Compare both versions against the same cases, review the differences with a human, then widen the change only if the trade-offs are acceptable. Keep a clear way back while the transition is being assessed.

The useful lesson is simple: renew the instruction without discarding the evidence that lets you judge the renewal. A clean rewrite is satisfying; a fair comparison is more useful.