7 Red Flags When Evaluating AI Vendors for Your Business

Every week I talk to business owners who are either about to sign a contract with an AI vendor or who already signed one and regret it. After evaluating dozens of vendors across consulting engagements, I have seen the same warning signs come up again and again. Here are seven that should make you pause before writing that check.

1. They Cannot Produce Security Documentation on Request

If you ask an AI vendor for their SOC 2 report, data processing agreement, or BAA (if you are in healthcare), and the answer is "we can get that to you later" or "we are working on it," that is a problem.

A client in the healthcare space once asked a vendor for a BAA before signing. The vendor said they would "have one ready by implementation." Three months later, the client was already sending patient-adjacent data through the platform with no signed BAA. That is not a minor paperwork delay. That is a compliance liability.

Good vendors have these documents ready before the first sales call. If the paperwork does not exist, the security controls it describes probably do not exist either.

What to ask: "Can you send me your SOC 2 Type II report and a signed DPA before our next call?" If the answer is anything other than "yes," dig deeper.

2. They Cannot Explain How the Model Actually Works

You do not need a PhD-level explanation of transformer architecture. But you do need to understand, at a basic level, what the system is doing with your data and how it arrives at its outputs.

I worked with a client evaluating a "proprietary AI" for document classification. When I asked the vendor how the model was trained, what data it used, and how it handled edge cases, the answers were vague references to "deep learning" and "neural networks." No specifics about training data, no explanation of confidence scores, no discussion of failure modes.

It turned out the system was a fine-tuned open-source model with minimal customization. That is not inherently bad -- but the inability to explain it suggests the vendor either does not understand their own product or is deliberately obscuring it to justify a higher price.

What to ask: "Can you walk me through what happens to my data from the moment it enters your system to the moment I get a result?" If the answer is just buzzwords, be skeptical.

3. Pricing That Scales Unpredictably

Some AI vendors price per API call, per document processed, or per "AI credit" -- units that are impossible to forecast. Usage-based pricing is not always bad, but if you cannot model your costs at 2x and 10x your current volume with reasonable confidence, you have a budgeting problem.

A client signed up for a document processing tool priced per page. Seemed reasonable at $0.03 per page. What they did not account for was that the vendor counted each page of a multi-page document separately, re-processed pages that failed quality checks (also billed), and charged separately for the extraction and classification steps. Their projected $500/month bill turned into $2,800/month within the first quarter.

What to ask: "Give me a detailed cost estimate for my current volume, and show me exactly what happens to that cost at 5x and 10x volume." Also: "What counts as a billable unit, and are there any secondary charges?"

4. No Integration With Your Existing Tools

If the vendor's product lives in its own silo, you are buying a second job for your team. The value of most AI tools comes from embedding them into existing workflows -- your CRM, your ERP, your document management system, your communication tools.

I see this constantly with small and mid-size businesses. A client bought an AI-powered lead scoring tool that had no native integration with their CRM. The vendor said they had a "robust API." What they actually had was a basic REST endpoint with limited documentation and no pre-built connectors. The client ended up spending almost as much on custom integration work as they spent on the tool itself.

Before you evaluate the AI capabilities, evaluate the integration capabilities. Check for native connectors to your existing tools, a well-documented API, webhook support, and a track record of customers on your tool stack.

What to ask: "Show me a live integration with [your specific CRM/ERP/tool]. Not a slide -- a working demo."

5. The Demo Does Not Reflect Production Reality

This is the most common red flag and the hardest to spot. Vendor demos are carefully choreographed. The data is clean, the use case is straightforward, the edge cases have been pre-screened out.

A client was evaluating a natural language processing tool for customer support ticket routing. The demo was impressive -- the system accurately categorized a dozen sample tickets in real time. But when we ran a pilot with 500 actual tickets from the client's system, accuracy dropped from the demo's 95% to about 68%. The real tickets had typos, slang, abbreviations, multi-issue threads, and context that required knowledge of the client's specific products. The demo data had none of that.

Always insist on a pilot or proof of concept with your actual data before committing. Any vendor who resists this is telling you something important about their confidence in their own product.

What to ask: "Can we run a two-week pilot with our real data before signing a contract?" If the answer is no, or if they want to charge full price for a pilot, walk away.

6. It Is "AI" in Name Only

Not everything labeled AI is actually using machine learning. Some products are rule-based systems with a marketing layer of AI branding on top. Again, this is not always bad -- sometimes a well-built rules engine is exactly what you need. But if you are paying AI prices for a rules engine, you are overpaying.

I evaluated a tool for a client that claimed to use AI for invoice matching. When we dug into the technical documentation, the "AI" was a set of if-then rules that matched invoices based on amount, date range, and vendor name. No learning, no adaptation, no pattern recognition. Just conditional logic. You could replicate the entire thing in a spreadsheet with VLOOKUP and some filters.

The tell is usually in how the system handles new scenarios. A real machine learning system gets better or at least maintains performance as it encounters new patterns. A rules engine breaks the moment it hits a pattern that was not explicitly programmed.

What to ask: "How does your system handle scenarios it has not seen before? Can you show me an example of the system learning from new data?"

7. No Data Export or Portability

If you cannot get your data out of the system in a standard format, you are locked in. This is true for any software, but it is especially important with AI tools because you may be feeding the system months or years of proprietary data -- training data, annotations, configurations, custom models -- that have significant value.

A client used an AI-powered analytics platform for 18 months. When they decided to switch, they discovered that their custom dashboards, calculated metrics, and historical analysis were only accessible through the vendor's interface. The export function produced a basic CSV with raw numbers -- no formulas, no definitions, no model configurations. They lost 18 months of analytical context.

Before you sign, test the export function. Actually download your data and confirm you can use it independently. Check whether you own the model or configurations created using your data.

What to ask: "If we cancel our subscription tomorrow, what exactly do we get to take with us, and in what format?"

What Good Vendors Look Like

Good AI vendors are not unicorns. They exist, and they share some common traits:

• They lead with limitations. They tell you what their product cannot do before you have to discover it yourself. They define the use cases where they are strong and are honest about where they are not a fit.
• They welcome technical due diligence. They are happy to get on a call with your technical team (or your consultant) and answer detailed questions about architecture, data handling, and model performance.
• They offer real pilots. Not a sandbox with fake data, but a structured proof of concept with your actual data and clear success criteria defined upfront.
• They have transparent pricing. You can model your costs at any scale with confidence. There are no hidden multipliers or surprise charges.
• They publish integration documentation publicly. You do not have to sign an NDA or get on a sales call to see their API docs.
• They have a data exit plan. They can tell you exactly what you get if you leave, and they make it easy.

The AI market will mature, and the vendors who survive will be the ones who operate with this level of transparency. In the meantime, ask the hard questions early. It is far cheaper to spend two extra weeks on due diligence than six months unwinding a bad vendor relationship.