Buying the right AI product can be a high-reward investment for your business. The payoff — market competitiveness, business continuity and ROI — can't be understated.

But in the thick of the buying process, when you're battling overzealous salespeople and internal politics, that end goal can feel very far away. That's when many enterprises get distracted (and understandably so), losing sight of what they really need from AI.

Sticking to a few hard-hitting questions can help you weed out fake AI solutions and help you refine your own requirements.

In my previous article, I outlined four questions that non-technical buyers should ask AI vendors during a sales cycle and what to look for in their responses. After some positive feedback, I've decided to expand the series.

While buyer education was my main intention for writing this series, I also strive to hold AI vendors like myself accountable. Communicating with clarity and transparency is the only way to forge lasting relationships with your customers. Buyers deserve the language and knowledge to demand higher standards from the industry.

Here in part two, I'll explain the importance of accuracy, proprietary machine learning models, security and usability in AI business products.

‍

5. What's a good benchmark for AI accuracy?

The accuracy of AI models is always a trade-off between two metrics: recall and precision. Recall describes the ability of the model to find the right data without missing anything. Precision describes the ability of the model to return only the correct output.

Let's say you're using an AI-powered contract management platform to find the expiration date of a vendor agreement. The platform returns every date that appears in the document. One of those will be the expiration date, but you have to manually sift through the results to find the right one.

In the example above, the recall is very high — it didn't miss. On the other hand, the precision is very low because it returned a lot of false positives.

When you're buying an AI solution, you need to understand what metric is more important for the use case you're trying to solve.

If your main use case is search, then you're optimizing for recall. Maybe you're trying to find all of your organization's contracts that contain a certain clause. As long as it captures every contract with that clause, it's no big deal if the results contain an extra clause you weren't looking for.

If your use case is reporting, then you're optimizing for precision. You need to ensure that all the information returned is correct — you can't allow for false positives.

Ultimately, look for vendor answers that report at least 90% accuracy for both metrics. Platforms that dip below 90% will probably miss too much information and require too much review to give you good ROI.

That said, you should also be wary of pie-in-the-sky responses that gloss over technical and scientific limitations. An engine that's never seen your data won't work with 99% accuracy right out of the gate.

‍

6. What AI innovations are proprietary to your platform?

We're seeing a trend of vendors forming strategic partnerships with tech giants to incorporate the larger company's more established AI models into their own products.

Why is this a trend to watch?

There's a world of difference between a vendor who packages up pre-existing AI solutions built for broad applications and a vendor who builds proprietary models with state-of-the-art AI so you can get the best the market has to offer.

The third party or open-source models often scooped by vendors are usually optimized for general accuracy across a wide variety of use cases. That means they won't be a great fit for niche business use cases. To get any real ROI, you'll want to work with a vendor who has built proprietary models focused on solving your specific use case.

‍

7. What security and compliance certifications do you have?

You'll sometimes have to give the vendor your own data so they can train your models. It's critical to know whether you can trust the vendor with your data. In case of a breach, you want to ensure you can't trace the models back to the original end user's data.

Be sure to ask about the level of data protection they have in place. SOC 2 Type 2 and ISO 27001 are a few examples of security and compliance standards AI vendors might follow.

‍

8. Can everyone in my organization learn how to use the AI?

This question reflects less on the quality of the AI and more on usability. The product should be easy to learn and use — not just for one persona and not just for technical users — but for everyone who accesses the solution.

I often say that the best AI is hidden. You don't see the technical details of the AI unless you look for them. In other words, the best AI shouldn't make users feel that they're doing something very complicated. It should be a seamless part of their everyday work.

‍

Conclusion

The last piece of advice I give to new AI buyers? Trust your gut! If it sounds too good to be true, it probably is.

If you're not using AI on a daily basis, learning about precision and recall might feel pretty far out of your job description. But AI is seeping into every corner of how we do business: from how we manage our supply chains to how we reach customers and schedule meetings. You don't have to be an engineer to be able to spot the difference between a best-of-breed AI product and dressed-up rule-based system. You just have to know what to ask.

This article was written by Amine Anoun, co-founder and Chief Technology Officer at Evisort, and was first published by Forbes.