When Does an Overall Metric Hide a Product Decision?

A product team sees its main success metric improve after a change. But the average may conceal different effects across user groups: some benefit, some see no change, and others may be worse off.

What would you want to know before expanding the change? Would you examine subgroup results, check whether the pattern persists over time, or first ask whether the differences are large enough to matter for the decision?

There’s a trade-off: breaking results into smaller groups can reveal a problem, but estimates may also become less certain. Who should decide what evidence is enough—and what finding would make the team pause or revise the rollout?