Evidence Before Enthusiasm: Five Questions Executives Should Ask of Any AI Claim
Vendors, consultants and internal teams all bring AI claims to the executive table. Five questions from research practice separate the claims worth funding from the ones worth a polite no.
Executive committees are now asked to approve AI investments almost every month. The proposals arrive with confident numbers: a 30% reduction in processing time, a 20% improvement in fraud detection, a doubling of productivity. Some of these claims are sound. Many are not. The difference is rarely visible in the slide deck.
Research has a simple discipline for testing claims. Executives do not need a statistics degree to use it, only five questions.
1. Compared with what?
Every improvement needs a baseline. “Detects 90% of fraud” means little without knowing what the current process detects. Ask for the comparison with today’s method, measured the same way, on the same kind of cases.
2. Measured where?
Results from a vendor’s showcase client, a different country or a laboratory dataset may not hold in your institution. Ask whether the result was measured on data that looks like yours: your customers, your languages, your transaction patterns. If not, ask for a small test on your own data before any large commitment.
3. What would have happened anyway?
Performance often improves after any change because of attention, new staff or seasonal patterns. The strongest evidence compares a group that used the AI with a similar group that did not, over the same period. Ask whether such a comparison exists.
4. Who is it wrong for?
An average can hide serious errors for particular groups. Ask for results broken down by customer segment, region and other groups that matter to your mandate, and for the cases where the system fails.
5. Has anyone else found the same?
A single result, however impressive, may be luck. Ask whether the claim has been replicated by an independent party or at another institution, and what happened there.
Using the questions well
These questions are not a way to say no. They are a way to say yes with confidence, and to size investments to the strength of the evidence. A claim with weak evidence can still deserve a small, well-designed test. A claim with strong evidence on comparable data can justify moving faster. Institutions that make these five questions routine will waste less money on AI and capture more of its value.