The CISO's 12-Question Checklist for Evaluating AI Vendors

Enterprise AI vendor evaluations have a structural problem: the buyer is usually evaluating a capability they do not yet fully understand, against a vendor who has spent years learning how to present that capability favorably. The result is procurement decisions made on impressive demos and vague security assurances that collapse under operational scrutiny six months post-deployment.

This checklist is designed for CISOs and security teams conducting formal AI vendor evaluations. Each question has a specific answer you should expect from a vendor with a mature security program. Vague, deferred, or defensive responses are signals, not acceptable answers.

Data Handling and Residency

1. Where does my data go, and who can access it?

Acceptable answer: a specific list of data centers with jurisdictions, a clear statement that no human reviews your data without your explicit consent, and named access control mechanisms. Unacceptable: ‘your data is secure and private’ without specifics.

2. Is my data used to train or improve your models?

Acceptable answer: a clear opt-out mechanism (or default opt-out for enterprise tiers) and a contractual commitment in the DPA. Unacceptable: ‘we take privacy seriously’ without a contractual commitment.

3. What is your data retention and deletion policy?

Acceptable answer: specific retention periods for inputs, outputs, and logs; a documented deletion process; and confirmation that deletion is permanent (not archival). Unacceptable: ‘data is deleted in accordance with our privacy policy’, read the policy and verify.

Security Program Maturity

4. What certifications and audits have you completed?

Acceptable: SOC 2 Type II report available under NDA, ISO 27001 certification, and/or FedRAMP authorization for government-adjacent workloads. Unacceptable: SOC 2 Type I only (point-in-time, not continuous), or certifications in progress with no current report.

5. Have you conducted AI-specific penetration testing, including prompt injection and adversarial input testing?

This question separates vendors with genuine AI security programs from those applying standard AppSec to an AI product. Acceptable: yes, with a summary of findings and remediation. Unacceptable: ‘we conduct regular penetration testing’ without confirming AI-specific techniques were used.

6. How do you handle a model that begins producing harmful or off-policy outputs in production?

Acceptable: a documented incident response process specific to AI behavioral failures, with SLAs for detection and remediation. Unacceptable: escalation to a generic engineering ticket queue.

In a 2025 survey of enterprise AI vendors, fewer than 30% could produce an AI-specific incident response runbook when asked. Ask for it in writing before signing.

Model and Supply Chain Risk

7. What is the provenance of the base model you are using?

Acceptable: named base model, training data disclosure, and a statement on whether RLHF or instruction tuning was applied and by whom. Unacceptable: ‘proprietary model architecture’ with no further detail.

8. How do you manage dependencies in your AI supply chain, including third-party datasets and fine-tuning providers?

AI systems have software supply chain risk that extends to training data and model weights, not just code libraries. Acceptable: a software bill of materials equivalent for model components. Unacceptable: no documented inventory of model dependencies.

Contractual and Operational Protections

9. What liability do you accept for AI-generated outputs that cause harm?

Acceptable: clear contractual language on indemnification scope. Unacceptable: blanket disclaimer of all liability for outputs. This is legally significant when AI outputs affect regulated decisions.

10. What is your SLA for security vulnerability disclosure and patching?

Acceptable: specific timelines (e.g., critical vulnerabilities patched within 24–48 hours, disclosed to customers within 72 hours). Unacceptable: ‘we patch vulnerabilities promptly.’

11. Can I conduct my own security assessment or penetration test of your production environment?

Acceptable: yes, with a defined scope and rules of engagement. Unacceptable: no, without explanation. Vendors with mature security programs expect and facilitate customer security assessments.

12. What happens to my data and integrations if I terminate the contract?

Acceptable: a data export process, a specific deletion timeline post-termination (typically 30–90 days), and a written certification of deletion on request. Unacceptable: no defined offboarding process.

Using This Checklist

Send these questions in writing before the final evaluation stage. Vendors who deflect, answer vaguely, or require multiple follow-ups to provide basic documentation are showing you their security program’s maturity, or lack of it. The right vendor will have these answers documented and ready, because they have answered them before.