What Should Human Evaluators See?

A team is testing an AI tool that drafts handoff notes between support teams. Reviewers prefer its notes when they know which ones came from the AI—but that preference may not tell us whether the notes preserve the details needed for the next decision.

What would make your evaluation convincing enough to guide a rollout? Would you hide the source of each note, ask reviewers to check specific decision-critical details, or compare the drafts with the team’s current process? And if reviewers disagree, should the team investigate what they value differently before combining their scores?

The evaluation should match the work the tool is meant to support. Which result would make you pause, revise the test, or try the tool on a limited scale?