When Can One Failure Matter More Than a Strong Average?

An AI system can look capable on average while still missing a task that matters a great deal to the people relying on it.

Imagine a team reviewing an assistant for a real workflow. Its overall results seem promising, but reviewers disagree about whether a particular kind of mistake should block a limited rollout—or simply be monitored. The answer may depend on who bears the cost, how often the task occurs, and whether a person can catch the mistake in time.

What evidence would you want before deciding? Would you focus on performance across representative tasks, failures on the most consequential cases, comparison with the current process, or how well human reviewers can spot errors? And who should own the trade-off when those signals point in different directions?