When a Better Metric Changes What You Notice

A useful lesson from measurement: a number can improve because the work improved—or because the way we count it changed.

Imagine a team introduces a new process and sees fewer reported blockers. That could mean work is moving more smoothly. But perhaps the new form makes blockers harder to report, or fewer people are using it. The dashboard alone cannot tell us which explanation is right.

Before crediting the process, check whether the same people and definitions are being compared. If feasible, roll it out to one group first and compare with a similar group that hasn’t switched yet. If that isn’t practical, look for an independent signal and describe the result as an association, not proof of cause.

The useful question isn’t just “Did the metric move?” It’s “What else would we expect to see if our explanation were true?”