An AI product can look useful in a demonstration and still be a poor fit for real work. The vendor may retain prompts, use customer content for improvement, depend on unlisted model providers or change a model without giving buyers enough time to test the effect. A conventional software checklist can miss those questions.
Small businesses do not need an enterprise procurement department to conduct useful due diligence. They need a repeatable way to define the task, identify the data involved, request evidence, compare vendors and record the decision.
NIST’s AI Risk Management Framework organizes AI risk work around Govern, Map, Measure and Manage. Its Generative AI Profile adds risks and suggested actions specific to generative systems. CISA’s supply-chain templates and the UK National Cyber Security Centre’s secure-AI guidance reinforce a common principle: third-party technology should be assessed through evidence, responsibilities and continuing controls—not marketing language alone.
| Key takeaway: Do not ask only, “Does this AI work?” Ask what it may access, how it changes, what evidence supports its claims, who is accountable when it fails and how the business can leave. |
|---|
Why ordinary software due diligence is not enough
AI services share many risks with cloud software: outages, weak access controls, insecure integrations, subprocessors and difficult contract terms. They also introduce moving behavior.
A model can produce different outputs from similar inputs. Its provider may replace the underlying model. Retrieval sources can become stale. An integration can give an agent permission to take actions, not merely display information. A vendor can also combine customer inputs, feedback and product telemetry in ways that are not obvious from a product page.
That does not make AI unsuitable. It means the assessment must cover the system’s purpose, data, behavior and change process together.
Define the AI use case before contacting vendors
Write a one-page use-case card before sending a questionnaire. Include:
- the task and expected output;
- the people who will use or review it;
- the data types the system may receive;
- the systems it may connect to;
- actions it may take;
- the harm if it is wrong, unavailable or misused;
- required human approvals;
- the current manual or non-AI alternative; and
- the decision date and accountable owner.
This prevents a common mistake: buying a broad platform and only later deciding where it is safe or useful. It also makes vendor answers comparable. A control that is adequate for drafting internal meeting notes may be inadequate for approving employee access or sending customer-facing advice.
The TRUSTED due diligence framework
Use seven lenses to keep the review complete:
| Lens | Core question | Typical evidence |
|---|---|---|
| Task fit | Is the system suitable for this specific outcome and risk level? | Demonstration on representative cases, limitations, acceptance criteria |
| Rights and data | What data enters the service, and what can the provider do with it? | Data-use terms, retention schedule, deletion process, residency options |
| User controls | Can the business restrict access and consequential actions? | SSO, MFA, roles, audit logs, approval gates, admin documentation |
| Security | How does the vendor prevent, detect and respond to compromise? | Independent assurance, architecture summary, incident process, testing scope |
| Testing and change | How are quality, safety and model changes measured? | Evaluation results, release notes, monitoring, notice commitments |
| Exit | Can the business retrieve its records and leave cleanly? | Export format, deletion certificate, transition support, termination terms |
| Dependencies | Which models, clouds and subprocessors support the service? | Current subprocessor list, dependency map, notification process |
The framework is not a certification. It is a way to collect decision-relevant evidence and expose gaps.
question AI vendor checklist
Task fit and limits
- Which specific tasks and user groups is the product designed to support?
- Which tasks, data or decisions does the vendor say should not use it?
- Can the vendor demonstrate the product on representative, non-sensitive test cases supplied by the buyer?
- What measurable acceptance criteria does the vendor recommend for quality, latency and reliability?
- What happens when the system is uncertain, lacks evidence or encounters conflicting instructions?
Rights, data and privacy
- What customer data is collected in prompts, uploads, outputs, logs, feedback and support tickets?
- Is any customer data used to train, fine-tune or improve models, and is the restriction contractual rather than a settings preference?
- How long is each data category retained, and can the buyer select a shorter period?
- How are backups, caches and derived records deleted after a request or termination?
- Where is data processed and stored, and which transfers or residency options apply?
- Who owns prompts, outputs, configurations and buyer-created evaluation data?
- How does the service handle third-party intellectual property and requests involving personal or confidential information?
User and administrative controls
- Does the service support single sign-on, multifactor authentication and role-based access?
- Can administrators restrict connectors, external sharing, model choices and data sources by group?
- Which actions require human approval, and can an administrator make approval mandatory?
- Are prompt, file, connector, administrative and action logs available to the buyer?
- Can the business disable a user, connector, agent or workspace immediately without vendor support?
Security and incident response
- What current independent security assurance covers the actual service being purchased?
- How are customer environments separated, and how are secrets encrypted and managed?
- How does the vendor test for prompt injection, insecure tool use, data leakage and abuse of connected systems?
- What is the incident-notification commitment, including timing, content and buyer contacts?
- How are vulnerabilities reported, prioritized and remediated?
Testing, monitoring and change control
- Which evaluations measure accuracy, harmful behavior, robustness and task-specific failure modes?
- Can the buyer reproduce tests in its own environment and retain the results?
- How are model, policy and system-prompt changes documented, tested and communicated?
- Can the buyer pin a version, use a preview environment or delay a material change?
Exit and portability
- Which data, logs and configurations can be exported, in what format and at what cost?
- How long does the buyer have to export after termination, and how is deletion confirmed?
Dependencies and subprocessors
- Which model providers, cloud services and subprocessors can receive or process buyer data?
- How much notice is provided before adding or changing a material dependency, and can the buyer object or terminate?
Add questions only when the use case needs them. A high-risk workflow may require legal, privacy, safety or sector-specific review beyond this list.
Evidence-adjusted scoring method
Score every applicable question in two parts.
Answer rating
- 0 = unacceptable or unanswered
- 1 = partially adequate, unclear or dependent on a future feature
- 2 = adequate for the defined use case
Evidence factor
- 0 = no evidence
- 0.5 = sales statement, undated screenshot or non-binding description
- 1 = current contract term, technical record, independent report or buyer-verifiable control
Calculate the evidence-adjusted score:
Adjusted points = importance weight × answer rating × evidence factor
Then divide total adjusted points by the maximum possible points and multiply by 100.
Example: if incident notification has an importance weight of 3, the answer is adequate (2) and the commitment appears in the contract (1), it earns 6 points. If the same answer appears only in a sales email (0.5), it earns 3.
The number supports comparison; it does not make the decision. Record the date, reviewers, evidence links and open conditions beside the score.
Hard-stop conditions
A high average should never hide a critical failure. Set non-negotiable stops before comparing vendors. Examples include:
- confidential content may be used for model training contrary to policy;
- the vendor cannot provide enforceable deletion and retention terms;
- administrators cannot revoke access or disable consequential actions;
- a required subprocessor or data location violates an existing obligation;
- the service has no acceptable incident-notification commitment;
- the buyer cannot obtain needed logs or audit evidence;
- the vendor refuses to identify material AI or cloud dependencies; or
- the use case requires reliability or professional judgment the product cannot support.
Write each stop as a testable condition. “Strong security” is vague. “Contract commits to notify the named contact within the agreed period after confirming an incident affecting customer data” is testable.
A practical buying workflow
- Classify the use case. Identify data sensitivity, affected people, possible actions and failure impact.
- Shortlist by hard stops. Remove vendors that cannot meet non-negotiable requirements before a long demonstration cycle.
- Send one common questionnaire. Use the same wording, due date and requested evidence for every candidate.
- Verify the evidence. Confirm scope, date, service name and whether a report covers the environment being purchased.
- Run representative tests. Include ordinary cases, missing information, adversarial instructions and recovery from failure.
- Score and document gaps. Separate an accepted risk, a contract condition, a pilot condition and a rejection.
- Approve a limited pilot. Start with minimum access, named reviewers, cost limits and a stop date.
- Review before expansion. Compare accepted output, review time, incidents, cost and user behavior with the baseline.
Keep the questionnaire, evidence and decision record together. If the vendor changes the model, subprocessors, data terms or security posture, the business can identify which part of the approval needs review.
Contract and implementation handoff
Due diligence fails when procurement records and operational controls diverge. The person configuring the product should receive a short control sheet showing what the decision assumed.
Include approved users, permitted data, prohibited tasks, enabled connectors, logging requirements, review frequency, incident contacts, cost limits and renewal conditions. Translate promised features into configuration checks. A contract that says customer content will not be used for training is important; so is confirming that optional feedback or connected tools do not create a different path.
Assign a renewal owner before the first invoice. Review material changes, evidence expiry, open exceptions, actual usage and whether the exit plan still works. If the product becomes embedded in a workflow, switching will become harder even when the export feature exists.
FAQ
Is an AI vendor questionnaire enough?
No. It is a collection tool. Important answers should be verified with current evidence, tested in the buyer’s environment and translated into contract and configuration controls.
Should a small business require a security certification?
Independent assurance can be valuable, but scope and date matter. A report may cover only part of the company or a different service. Review what was tested and combine it with use-case controls, incident terms and buyer verification.
Can a vendor receive a high score with unresolved hard stops?
No. A hard stop overrides the average. The score compares acceptable candidates; it does not erase a critical data, security, legal or operational failure.
How often should AI vendors be reassessed?
Use a risk-based schedule and trigger a review when the model, terms, subprocessors, integrations, data use or task changes materially. High-impact use cases need closer monitoring than low-risk drafting tools.
Does this checklist replace legal or security advice?
No. It is a practical starting point. Regulated, sensitive or high-impact uses may require qualified legal, privacy, security, safety or sector specialists.
