AI’s Hidden Wiring Gets a Major Upgrade

AI’s Hidden Wiring Gets a Major Upgrade

A Chinese AI lab has introduced a significant improvement to the fundamental design of artificial intelligence models. This new technique, called attention residuals, could make AI systems more efficient and powerful. Even prominent figures like Elon Musk have taken notice, calling the work “impressive.” Most AI models you interact with, like ChatGPT, Claude, or Gemini, are built using similar core components. One crucial part of this internal design, however, has remained unchanged since 2015.

Researchers at Moonshot AI, the team behind the Kimi models, published a paper suggesting a flaw in this long-standing design. They argue that a key element, known as a residual connection, has a hidden problem. While it doesn’t break AI systems, it makes them perform slightly worse than they could. The new attention residuals approach aims to fix this by allowing AI models to better choose what information to focus on.

The Problem with Old AI Wiring

To understand the issue, think about how AI models process information in layers. Imagine you’re writing a long report, and a team of 50 editors reviews it. The first editor reads your draft and passes notes to the second. The second editor gets the original draft plus the first editor’s notes, adds their own, and passes everything to the third editor. By the time you reach the 50th editor, they have a massive stack of papers. It becomes very difficult to tell which notes are important and which are just noise. This is similar to how many current AI models work.

Large language models have dozens, sometimes hundreds, of these layers, like our editors. Each layer processes information and passes it along. The component that does this passing is the residual connection, first developed in 2015 for image recognition. Its purpose was simple: add the original input back at each step. This prevents information from being lost, allowing for much deeper networks. Without it, the signal weakens too much for the model to learn effectively. However, in very deep models, this constant addition creates that massive pile of information, burying important details.

The paper calls this effect “information dilution.” It’s like the pile of notes becomes so tall that no one can find what they need. The contributions from earlier layers get drowned out by the accumulated information from all the subsequent layers. Deeper layers have to work harder, essentially “shouting” to be heard over the noise.

The Clever Fix: Attention Residuals

The breakthrough comes from realizing that a solution to a similar problem already exists. Before the current AI models, there were recurrent neural networks (RNNs). RNNs processed text word by word, compressing all previous information into a single summary. This summary would get overloaded, losing information from earlier words. This sounds a lot like the problem with residual connections, but across time instead of across layers.

The transformer architecture, which powers most modern AI, solved the RNN problem with “attention.” Instead of one summary, attention allows the model to look back at all previous words and decide which ones are most important. It can focus on a relevant word from much earlier in the text and ignore less important ones.

Moonshot AI’s insight is that this attention mechanism can be applied to residual connections as well. Instead of blindly adding information from each layer, the model can use attention to look back at all previous layers. It can then choose which layers have the information it actually needs. This means each layer gets a custom blend of information, tailored to its specific needs, rather than a generic, averaged mix.

Does It Work? The Results

The researchers tested their attention residuals approach on five different model sizes. In every case, the new method outperformed the standard approach. The improvement was significant, equivalent to getting 25% more computing power for training without any extra cost. It’s like getting a performance boost for free.

They also tested it on their largest model, which has 48 billion parameters. The gains were consistent across all benchmarks. Reasoning abilities increased notably, math performance improved, and coding skills went up. On one specific reasoning test, the GPQA diamond, scores jumped from 36.9% to 44.4%. This is a substantial improvement stemming from a change in how information flows between layers.

Making It Practical: Block Attention Residuals

A potential concern is that the full attention residuals approach might use more memory and be more expensive to run. To address this, the team developed a practical version called block attention residuals. Instead of every layer looking back at all previous layers, they group layers into blocks of about eight. Within these blocks, the old system is used. However, between these blocks, the new attention-based system is employed.

This approach offers most of the benefits at a much lower cost. Training the model becomes less than 4% more expensive. And when the AI is actually generating text for users (inference), the slowdown is under 2%. This speed difference is practically unnoticeable, making it essentially free performance gain.

Why This Matters

This development is important because residual connections are a fundamental part of almost every transformer-based AI model. This includes chatbots, image generators, and coding assistants. The fact that this core component was overlooked for improvement for over a decade suggests there might be other areas in AI design waiting to be re-examined.

AI research often builds upon previous work, and sometimes these foundational choices are treated as fixed rules rather than adaptable designs. The attention residuals breakthrough shows that even seemingly basic parts of AI architecture can hold significant potential for improvement. Combining the 2015 residual connection idea with the 2017 attention mechanism led to a major leap forward in 2025.

However, it’s important to note that attention residuals might not be a universal solution for every AI task. Researcher Ziming Louie conducted experiments comparing structured data (with clear patterns) and random data. Attention residuals performed better on structured data, like human language and code, because they can learn to focus on the most useful information. For highly random or chaotic data, the older residual connection method might sometimes be more effective due to its more exhaustive, brute-force approach.

Source: China’s New AI Breakthrough – Attention Residuals Explained – (YouTube)