How to evaluate the evidence behind an AI security vendor's prevention claims

What to pin down first

• Prevention of what, exactly? Blocking an attack, containing a partially successful one, detecting and responding, and deterring are different claims. Ask which one is being made.
• Under what conditions? Which attack categories were tested - direct prompt injection, indirect injection through retrieved content, tool abuse, data exfiltration - and who designed the tests?
• With what measured outcome? Detection rate, false positive rate, and the residual rate of attacks that still succeeded. A claim with no failure rate is incomplete by construction.

Evidence that holds up

1. 1
   A reproducible methodology.

   Named, public attack taxonomies - the OWASP Top 10 for LLM Applications is one reference point (genai.owasp.org/llmrisk/llm01-prompt-injection) - and test scenarios you can rerun yourself, in your own environment.

2. 2
   Realistic conditions.

   Tests against your models, your tools and your data flows - not only the vendor's demo setup.

3. 3
   Architectural clarity.

   Where does enforcement happen? A control enforced outside the model can be verified independently of model behavior. A control that depends on the model following instructions relies on instruction-following holding up under the attacks actually tested.

4. 4
   Scoped language.

   Vendors who state exactly what is covered, on which integration paths, with which known limits, are showing you the boundary of their evidence. Universal claims ("stops all prompt injection") have no evidence boundary and should be treated as unproven.

Red flags: percentage claims without a described test corpus, compliance claims inferred from product features, and case studies with no named attack scenario.