Prototype quickly, but keep evaluation independent

Two useful testing ideas meet here. Hamza Waseem Naser’s post on scikit-learn explains why learned preprocessing should be fitted on training data—not on the held-out examples used for evaluation. Farah Saleh Banihani recommends making a small prototype and letting intended users try it.

For an AI-agent pilot, combine those principles: use one set of workflow examples to refine the trial, then assess the agreed success criteria on separate cases that did not shape those changes. Otherwise, feedback used to tune the agent can make the evaluation look stronger than it is.

Before starting, agree on what must remain stable, what counts as success, and which cases are reserved for evaluation. That gives the team room to learn without confusing iteration with evidence to continue.