China’s AI Trails West on Novel Reasoning Benchmarks

New Tests Suggest China’s AI Lags Behind Western Labs

Recent independent testing indicates that leading artificial intelligence models from China are not keeping pace with their Western counterparts, particularly in areas requiring novel problem-solving and complex reasoning. While Chinese AI labs have been perceived as rapidly closing the gap, new benchmarks designed to prevent ‘gaming’ the system reveal a significant performance difference.

The Challenge of Testing True AI Reasoning

Traditional AI benchmarks often test knowledge recall or patterns learned from vast datasets. However, these methods can be ‘gamed’ by models trained on similar data, making it difficult to assess genuine intelligence. To combat this, researchers have developed new tests that focus on reasoning abilities that cannot be easily replicated through data alone.

ARC AGI 2: A Benchmark for Novel Problem Solving

The Ark Prize’s AGI 2 benchmark is specifically designed to measure general, novel problem-solving skills. It prevents brute-forcing with more data or distilling knowledge from other models, requiring instead a deep, fundamental reasoning capability. Recent scores show that current top Chinese models like Kimi 2, Minmax 2.5, GLM 5, and Deepseek 3.2 perform below the level of Western ‘frontier’ models from July 2025. This suggests that even China’s most advanced models are effectively performing like Western models from eight months prior, indicating a generational gap rather than a narrow one.

Pencil Puzzle Benchmark: Testing Constraint Satisfaction

Another test, the Pencil Puzzle benchmark, evaluates large language models (LLMs) on constraint satisfaction problems. These puzzles are closely related to complex computational problems and feature deterministic step-by-step verification, making them resistant to manipulation. Early results from early 2025 showed initial capabilities, but performance dramatically increased with the GPT-5 family of models. Models released before 2024 scored zero percent, as they lacked the specific training data for these novel puzzle combinations. In contrast, U.S. closed models dominate this benchmark, with GPT 5.2 scoring 56%, Claude Opus 4.6 achieving 36.7%, and Gemini 3.1 Pro reaching 33%. Chinese models show a sharp decline, with Kimi 2 scoring 6%, Minmax 3.3%, Deepseek 2%, and GLM 5 and Quanshun 3.5 scoring below 1%. This benchmark specifically tests multi-step logical constraint reasoning without relying on prior knowledge.

Frontier Math and Humanities Last Exam: Uncompromised Knowledge and Reasoning

Standard benchmarks like GSM 8K and math tests often suffer from data contamination, where models have already been trained on the test questions. To address this, Frontier Math was created with entirely new, unpublished mathematical problems from advanced branches of mathematics. Solving these problems requires hours or even days of effort from human experts and cannot be gamed. Models are given access to coding tools, but even then, they struggle. Chinese models like Kimi 2, GLM 5, and Deepseek score very low, around 2-3%, appearing clustered at the bottom of the rankings, far behind models like Gemini.

Similarly, the Humanities Last Exam (HLE) tests deep and broad knowledge across various subjects using multimodal questions. It also includes a private, held-out set of questions removed from searchable databases. While Kimi 2 initially reported a 50% score on this benchmark, independent analysis found it to be 29.4%, an inflation of over 21 points. The use of tool-use features significantly boosted its reported score.

SWE Bench and SWE Rebench: Real-World Coding Challenges

In software engineering, the SWE bench evaluates how well AI models can fix bugs within complex codebases, a more realistic task than writing isolated functions. Initially, Chinese models like Minmax and Kimi appeared to perform comparably to Western frontier models such as Claude 4 Opus and GPT-5 on the original SWE bench. However, a newer, continuously evolving benchmark called SWE Rebench was introduced to address data contamination issues. This benchmark uses fresh GitHub tasks that models have not encountered during training.

On SWE Rebench, Chinese open-source models show a significant drop in performance, falling to ranks 11 and 17, while Western models remain higher up the list. This suggests that their strong performance on the original SWE bench may have been due to benchmark-specific optimizations rather than generalizable coding intelligence. The findings highlight the importance of running independent tests, as companies are incentivized to inflate benchmark scores for publicity.

Why This Matters

The consistent results across multiple, ungameable benchmarks suggest that while China is a major player in AI research, with a large pool of talented researchers, its leading models may not possess the same level of advanced reasoning and novel problem-solving capabilities as top Western models. This distinction is crucial for developing AI that can tackle complex, unforeseen challenges in science, medicine, and technology. The U.S. and other Western nations appear to hold an edge in developing AI that can truly think and innovate, rather than just recall and pattern-match. This could have significant implications for global AI development and competition.

Source: New Tests Reveal The Truth About China’s AI Progress… (YouTube)