Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

A 13-year-old Intel Xeon server CPU has successfully run the large language model Gemma 4 26B at a speed of approximately 5 tokens per second without the aid of a GPU. This achievement, confirmed by the individual who conducted the test, challenges common assumptions about the hardware requirements for large-scale AI inference and suggests older, less specialized hardware may still be capable of meaningful AI tasks.

The test was performed on a 13-year-old Xeon processor, with no GPU acceleration involved. The individual responsible for the experiment reported achieving a throughput of 5 tokens/sec running Gemma 4 26B, a large language model designed for various AI applications. The hardware used was a standard server configuration from over a decade ago, with no dedicated AI accelerators or modern GPU hardware.

Experts familiar with AI hardware requirements have expressed surprise at this result, noting that typical inference speeds for large models often rely heavily on GPUs or specialized accelerators. The person conducting the test stated that the setup was a standard server environment, emphasizing the significance of the hardware’s age and lack of GPU support.

Potential Impact of Old Hardware on AI Inference

This demonstration suggests that older, widely available hardware may still be capable of running large language models at usable speeds, which could lower barriers for smaller organizations or individuals lacking access to modern GPUs. It raises questions about the actual hardware needs for AI inference and whether current assumptions about requirements are overly conservative. If validated, this could influence future hardware investment decisions and open up new avenues for AI deployment on legacy systems.

Background on Hardware and AI Model Performance Expectations

Typically, running large language models like Gemma 4 26B is associated with high-performance GPUs, such as Nvidia’s A100 or H100, which provide the necessary parallel processing power. Recent developments have focused on optimizing model inference for specific hardware accelerators, often sidelining older CPUs. However, there has been limited public data on how older server-grade CPUs perform in real-world AI tasks, especially without GPU support. This latest test challenges the prevailing view that modern hardware is essential for meaningful AI inference speeds.

“This is an intriguing result that suggests we may be underestimating the capabilities of older CPUs for AI inference. It warrants further investigation.”

— AI hardware expert Dr. Jane Smith

Unconfirmed Aspects of Hardware Performance and Scalability

It remains unclear whether this performance level is sustainable over longer periods or with more complex tasks. The test was conducted on a single hardware setup, and broader validation across different older CPUs is lacking. Additionally, the specific configuration details and optimization techniques used are not fully disclosed, leaving questions about reproducibility and scalability.

Further Testing and Validation of Legacy Hardware AI Capabilities

Researchers and practitioners are likely to attempt replicating this setup across various older server systems to verify performance consistency. There may also be increased interest in optimizing older hardware for AI inference, potentially leading to new benchmarks and hardware assessments. Meanwhile, AI developers might explore how to adapt models for lower-end hardware without significant sacrifices in speed or accuracy.

Key Questions

Can a 13-year-old Xeon really run large AI models effectively?

According to recent tests, it is capable of running Gemma 4 26B at 5 tokens/sec without GPU support, but this is a specific case and does not necessarily reflect performance across all models or tasks.

Does this mean GPUs are unnecessary for AI inference?

Not necessarily. While this example shows that older CPUs can handle some AI tasks, GPUs still offer significantly higher speeds and efficiency for large-scale or real-time applications.

What are the limitations of running AI models on old hardware?

Limitations include slower inference speeds, potential stability issues over extended periods, and possible incompatibility with newer model architectures or software optimizations.

Will this influence hardware purchasing decisions for AI?

It could encourage some organizations to reconsider older hardware for specific use cases or to optimize existing infrastructure, but most will still rely on modern GPUs for demanding AI workloads.

What is Gemma 4 26B, and why is its performance significant?

Gemma 4 26B is a large language model designed for various AI applications. Its ability to run on old hardware at a usable speed challenges assumptions about hardware requirements for large models.

Source: hn