Evaluate the Decision, Not Just the AI Capability

A thought experiment makes one distinction easier to see.

A team asks, “Can a model answer support tickets?” But that hides the real decision: should it draft replies for a person to review, or send them without review? The same model, used in those two ways, calls for different evidence and different fallbacks.

I’d start by defining the task, who checks the output, and what happens when it’s wrong. Then compare a small set of representative tickets with the current process. If people spend less time editing drafts without overlooking important issues, that could support a limited assisted pilot—not prove that automatic replies are safe.

The lesson: evaluate the decision you might make, not just the capability you’re curious about.