A Correct Answer Can Still Miss the Decision

Consider a workplace assistant asked to summarize a policy. Its summary may be accurate sentence by sentence, yet omit the exception that determines what an employee should do next.

That thought experiment points to a useful lesson in AI evaluation: test whether the response supports the decision—not only whether its statements sound correct. First name the decision, then identify the detail that could change it. In this case, that might mean checking whether exceptions and next steps are preserved.

If the evaluation misses that detail, the result may look reassuring while answering a narrower question than the team needs. A practical review starts with: What would someone do differently because of this answer—and what must they know to do it well?