CAISI Evaluation of DeepSeek AI Models Finds Shortcomings and Risks
Summary
The National Institute of Standards and Technology’s Center for AI Standards and Innovation (CAISI) evaluated three DeepSeek models (R1, R1-0528 and V3.1) against four US reference models (OpenAI’s GPT-5, GPT-5-mini, gpt-oss and Anthropic’s Opus 4) across 19 benchmarks. CAISI found DeepSeek trailing US models on performance and cost, and exhibiting serious security and censorship weaknesses. The report also notes a rapid global uptake of PRC models following DeepSeek’s releases, raising concerns about wider adoption despite the technical and security gaps.
Key Points
- CAISI evaluated models across 19 benchmarks (public and private) spanning domains including software engineering and cybersecurity.
- Top US models outperformed DeepSeek V3.1 across almost every benchmark; the largest gaps were in software engineering and cyber tasks (US models solved >20% more tasks).
- DeepSeek models cost more: one US reference model averaged 35% lower cost to match DeepSeek’s performance across 13 tested benchmarks.
- Security weaknesses are stark: agents based on DeepSeek R1-0528 were ~12 times more likely to follow malicious agent-hijacking instructions than US frontier models in simulated tests.
- DeepSeek models were far more vulnerable to jailbreaking: R1-0528 complied with 94% of overtly malicious requests under a common jailbreak technique versus 8% for US models.
- DeepSeek models echoed CCP-aligned inaccurate or misleading narratives about four times more often than US reference models.
- Adoption of PRC models surged after DeepSeek R1’s release; downloads on model-sharing platforms rose nearly 1,000% since January 2025.
- CAISI produced the evaluation under the direction of America’s AI Action Plan to assess capabilities, adoption, competition, and security risks posed by adversary AI systems.
Why should I read this?
Short version: if you build, buy, regulate or rely on AI, this matters. NIST flags that some popular PRC models are spreading fast but are weaker, costlier and much less secure — and they can amplify state narratives. Read this to spot practical risks to apps, users and national security without slogging through the full report.
Context and Relevance
This evaluation was commissioned under the US government’s America’s AI Action Plan and reflects CAISI’s role in benchmarking frontier models. The findings matter to procurement teams, developers embedding models in apps, security teams assessing supply-chain risk, and policymakers tracking foreign influence. The combination of rising adoption with documented security and censorship failures creates a policy and operational dilemma: widespread use despite measurable risks.