Twenty-three questions that separate vendors who can ship from vendors who demo. Print this. Take it into your next meeting. Watch the vendor's body language as you read down the list.
On the model and stack
- Which models do you support? Can we bring our own? What's your switching procedure?
- Where is inference run? Which jurisdictions? Can we choose?
- Is our data used to train any models? Is that contractually verifiable?
- How do you version prompts? Can we audit prompt changes?
- What does your eval harness look like? Can we run our own evals against your platform?
On security and compliance
- What certifications do you hold? (SOC 2 Type II, ISO 27001, HIPAA, GDPR DPA, EU AI Act readiness.)
- Will you sign a BAA / DPA?
- What's your incident response and notification policy?
- Penetration test cadence — and can we see the most recent report?
- What logging is retained, where, and for how long?
On reliability and operations
- What's your SLA? What's your historical uptime over the last 12 months?
- How do you handle model-provider outages?
- What's the support model? Where do humans live?
- How do you do change management on shared components?
- What's the escalation path during a P1?
On commercials and lock-in
- Pricing model — per-request, per-seat, per-token? Caps? Commitments?
- What's the data export procedure if we leave?
- What artifacts do we keep on termination — prompts, fine-tunes, evals?
- What's the price escalation clause?
On their actual experience
- Three reference customers in our industry, please. (Watch for hesitation.)
- When did you most recently lose a customer? Why?
- Show us an incident postmortem from the last 90 days.
- What is the largest production deployment you currently support?
WHY 23 QUESTIONS
Vendors who can answer 18+ confidently are usually serious. Vendors who hand-wave on more than 5 of them are not where they say they are.