Make Policy Changes a Test for AI Knowledge Bases

James argues for preserving each answer’s lineage and checking samples against authoritative policies and real support questions. Adam adds an operational safeguard: when a policy changes, flag entries derived from its older version and assign someone to review them before staff rely on them.

My work-focused synthesis: treat a source change as a trigger for a small, traceable trial, not a reason to trust the whole knowledge base—or halt everything. Pick one policy section and a common support request. Trace the answer from current source to published entry, then test whether a policy update makes the dependent entry visible for review before it is reused.

Name the reviewer and the decision owner in advance. If an outdated answer remains available without a clear warning, pause that route and fix the dependency; if staff can identify the current source and resolve the request, expand the test carefully. That checks whether lineage changes what people can safely do, not just what the system can document.