The Evaluation That Aimed at the Wrong Layer
An operations director selects an AI-enabled tool for client intake. She does the evaluation the way most organizations do it, and does it well: reviews the vendor’s security page, notes the certifications, reads the compliance language, checks two references, routes the agreement for signature. The tool works. Staff like it.
Fourteen months later, output quality changes despite nothing being reconfigured on her end. What changed was the model running underneath the feature which was swapped by a company she’s never heard of, working for the vendor she signed with.
Who Else Is in the Room
The AI capability inside a business application is often not built by the company selling it. It is a model licensed from a model provider, running on infrastructure owned by a cloud company, sometimes wrapped by a specialist vendor who assembled the feature. Four organizations can stand between a business and a working AI tool, and typically only one of them appears in the procurement conversation.
Each of the others has its own security posture, its own retention practices, and its own history. None of them were evaluated. None of them signed anything with the business.
This is not a theoretical concern raised by security researchers. It appears in the risk disclosures of companies large enough to have real leverage over their suppliers. McKesson, one of the largest healthcare distributors in the country, wrote in this public filing that: “reliance on third-party AI tools and solutions may expose us to risks that are outside of our control, including compliance gaps. Our AI policies and safeguards may not be sufficient to protect us against negative outcomes, such as the misuse or loss of data or the compromise of our intellectual property. that reliance on third-party AI tools may create risks outside its own control, including compliance gaps.”
That is a company with procurement resources most organizations will never have, telling its investors it cannot fully see the chain. The lesson here is not that the situation is hopeless. It’s that trust granted to one vendor quietly extends to companies that never asked for it.
The Tool You Bought Doesn’t Stay the Tool You Bought
Traditional business software changes on a release schedule. AI capability changes continuously, and often invisibly.
The model behind a feature can be replaced, upgraded, or retired by the provider. When it is, the tool’s behavior can shift → a different tone, different accuracy, different failure modes, and often with no update or announcement. A summarization feature that was reliable in March can be subtly less reliable in September for reasons that live three companies away.
For a business, this means AI-assisted work needs periodic re-checking rather than a one-time validation at rollout. The tool that earned trust in the pilot is not necessarily the tool in production a year later.
What Happens If the Vendor Doesn’t Make It
The AI vendor market is young, crowded, and consolidating. Some of these companies will be acquired. Some will discontinue the product. Some will not be operating in three years.
While it may be ordinary market behavior, it becomes an operational problem when a workflow is built around a tool without considering realistic possibilities:
- If this vendor is acquired, does the data handling commitment survive the change in ownership?
- If the product is discontinued, what happens to the information it holds, and in what format can it be retrieved?
- If the relationship ends badly, what does the business actually get back?
What’s most interesting is that organizations ask these questions of a payroll provider or other vendor without hesitation. But that same scrutiny is often skipped with AI tools because they arrive as features rather than as systems of record.
“Compliant” Describes a Claim, Not a Coverage Area
Certifications and compliance language describe what a vendor asserts about its own environment, audited at a point in time. They don’t automatically extend to a newly added AI feature, and they don’t cover the subprocessor delivering it.
The documents that create actual obligation are narrower and less decorative: a data processing agreement, or a business associate agreement where health information is involved. Those either exist, cover the AI feature specifically, and have been read (or they haven’t).
There’s a useful tell here. When a vendor is asked to produce the agreement and responds by describing its security investments instead, that is a different answer than the one requested. It may be entirely innocent. It’s still worth noticing.
Questions to Put to the Vendor
Earlier articles in this series suggested questions for leaders to ask themselves. These are for the vendor, and they belong in the evaluation before signature:
Who else is involved in delivering this AI capability? Name the companies. If the vendor can’t or won’t, that is itself information.
What happens if the model behind this feature changes? Will we be told, and how?
If you are acquired or discontinue this product, what happens to our data and how do we get it back? Ask for the format, not just the assurance.
Which of your compliance commitments cover this specific AI feature, and can you produce the underlying agreement?
How will we be notified when your terms or your subproviders change?
The last one is asked least and matters most. Over a three-year relationship, it is the difference between a vendor review that stays current and one that describes conditions that no longer exist.
Extending a Discipline You Already Have
Vendor review already exists in most organizations. AI doesn’t require inventing a new one, it requires extending an existing discipline and accepting that the review can’t be a single event at signature.
The practical shift is small.
- Ask who else is in the chain
- Ask what happens when things change
- Ask for the paper rather than the assurance
- Then ask again next year, because the answers will have moved
If your organization is evaluating AI-enabled tools, or has adopted some already, this is worth passing to whoever owns vendor relationships and contract review.
The next article in this series looks at what changes when AI tools stop drafting and start acting → sending messages, updating records, and triggering workflows on the organization’s behalf.
Source: This article draws on guidance from the OWASP Top 10 for LLM Applications, 2025 (LLM03: Supply Chain, LLM04: Data and Model Poisoning).
Leave a Reply