The retraining-cycle question your AI vendor cannot answer, and what that tells you.

There is one diligence question worth asking on every AI-vendor sales call. It filters the serious vendors from the rest more cleanly than any feature comparison, accuracy benchmark, or customer-reference list. The question is: how often do you retrain the model on new data, and what is your eval-deltas process for each retrain.

The vendor will answer in one of three patterns. Each pattern reveals more about the vendor's operational maturity than the rest of the diligence conversation typically does.

The first pattern is the confused-and-deflecting answer. The vendor's response is some version of, the model is continuously improving, our training data is constantly updating, our accuracy is increasing over time, with no specifics on cadence, no specifics on the eval-deltas process, and no specifics on how the customer would know whether the model has changed since their last evaluation. The vendor in this pattern does not have a defined retraining cycle, does not have an eval-deltas process, and is operating the model in a way that will produce drift the buyer cannot detect or manage. The diligence implication is to deprioritize the vendor or to require a contractual commitment to a retraining-and-eval cadence as a condition of purchase.

The second pattern is the honest-and-incomplete answer. The vendor describes a retraining cadence (quarterly, monthly, on-demand) and acknowledges that the eval-deltas process exists but is in development, with limited documentation available externally. The vendor in this pattern has thought about the question, has built some operational infrastructure, and is on a trajectory toward operational maturity. The vendor is more credible than the first pattern but is not yet at the bar that production-grade deployment requires. The diligence implication is to engage further with specific questions about the eval-deltas methodology, to ask for documentation, and to negotiate contract terms that protect the buyer against drift while the vendor continues to mature the process.

The third pattern is the documented-and-disciplined answer. The vendor provides a specific retraining cadence, a documented eval-deltas methodology, a specific commitment to performance reporting against the cadence, an audit-trail mechanism that lets the customer verify the model's behavior across retraining cycles, and a remediation process for cases where the eval-deltas show degradation against the buyer's specific use case. The vendor in this pattern has built the operational infrastructure that production-grade AI deployment actually requires. The diligence implication is to engage on commercial terms, because the operational risk has been substantially mitigated.

The three patterns are not equally distributed across the AI-vendor population. The first pattern is the most common in 2024-2025, particularly among vendors that have been building for under three years. The second pattern is increasingly common among vendors that have crossed two or three production-deployment milestones and have been forced to build the infrastructure. The third pattern is rare and concentrated among the vendors that have been deploying production AI for five-plus years and have institutional learning from past drift incidents.

The diligence value of the question is that it filters the population fast. Vendors in the first pattern can be passed over without further investment of evaluation time. Vendors in the second pattern can be evaluated further with realistic expectations. Vendors in the third pattern earn the deeper commercial conversation. The buyer who asks the question early in the sales cycle saves substantial time on vendors that would not survive the deeper diligence anyway.

The retraining-cycle question is the question. The other diligence questions are useful but secondary. The vendor's answer to this one tells the buyer most of what they need to know about whether the production deployment will work. Ask it on every call. Apply the three-pattern filter to the answer. Allocate the deeper evaluation time accordingly. The diligence efficiency this produces is meaningful and the operational outcomes follow from it.

—TJ