Exploring The Limits Of MoE Inference With Transformers: 180B Parameters, 32GB RAM
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: Exploring The Limits Of MoE Inference With Transformers: 180B Parameters, 32GB RAM on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

TL;DR

A Hugging Face developer has proposed an experimental Transformers feature that streams selected mixture-of-experts weights from storage into a bounded device cache. The reported OLMoE test reached 34.6 tokens per second, near the 38.1-token baseline, but the source reports no completed test on a 180B model or a 32 GB Mac.

A Hugging Face developer has proposed an experimental way for Transformers to run mixture-of-experts models by loading selected expert weights from storage into a bounded device cache, as detailed in the original analysis. In a reported test using OLMoE, the method generated 34.6 tokens per second, compared with 38.1 for standard Transformers; a 180-billion-parameter model on a 32 GB Mac remains a target, not a reported benchmark.

The experiment addresses a memory constraint in mixture-of-experts (MoE) models: their expert weights can be too large to keep entirely in GPU or unified memory, although the routing mechanism uses only a subset of experts for each token. The author says experts account for more than 90% of the model weights, creating the possibility of keeping selected experts close to the processor while fetching others as needed.

The first implementation reportedly ran experts one by one in a Python loop. On OLMoE-1B-7B, it achieved 10.1 tokens per second even when experts were already cached, compared with about 32 for vanilla Transformers. The author then replaced that approach with a shared, preallocated device tensor and a table mapping expert IDs to cache slots. This allowed the existing grouped matrix multiplication kernel to process selected experts together, yielding the reported 34.6-token rate against a 38.1-token baseline.

The author says the change was submitted to Transformers’ from_pretrained flow as pull request #50717. The source also points to a Hugging Face Space, tarekziade/moe-expert-streaming, with project code. It describes work involving Transformers, a model-specific package and Metal kernels, and says AI tools were used in building the code. These are details of the developer’s account; the source does not report that the pull request has been accepted or released.

At a glance
reportWhen: Experimental work described in the sour…
The developmentA Hugging Face developer submitted an experimental SSD-backed expert-streaming option for Transformers, reporting near-baseline speed on OLMoE while leaving the target 180B-model, 32 GB Mac test unreported.
AI Inference · Briefing

Streaming MoE Experts With Limited Memory

An experimental Transformers change streams selected experts from SSD into a bounded cache. The reported OLMoE test ran close to vanilla speed, but the 180B model on a 32 GB Mac remains untested in the source.

34.6 tokens/s
OLMoE with expert streaming
38.1 tokens/s
Vanilla Transformers comparison
90%+
Model weights attributed to experts

How the cache works

The router selects experts for each token; the experiment fetches those experts into a shared cache.

1
Route each token

Choose a subset of experts for the token.

2
Load selected experts

Stream them from storage into available device cache slots.

3
Compute together

Use Transformers’ grouped matrix multiplication kernel.

What the test shows

On OLMoE-1B-7B, the shared cache kept the reported speed near the standard path.

Earlier approach
  • Python loop ran experts one at a time.
  • Reached 10.1 tokens per second, even with experts cached.
Shared cache
  • Preallocated tensor mapped expert IDs to cache slots.
  • Reached 34.6 tokens per second versus 38.1 baseline.

The larger device test is open

The 180B model on a 32 GB Mac is a motivating target, not a measured result in the supplied source.

Evidence

Promising engineering

The cache design brought the reported small-model test close to vanilla Transformers speed.

Open question

SSD performance

Storage traffic and cache misses could change speed when experts must be fetched repeatedly.

Next evidence

Measure the target setup

Report speed and memory use on the larger model, with storage, cache settings and workload specified.

StatusThe change was submitted as Transformers pull request #50717; the source does not say whether it was merged or released.
The takeaway
Expert streaming looks close to baseline on the reported OLMoE test, while performance on the 180B model and 32 GB Mac remains unproven.

Making Oversized MoE Models More Accessible

If the method works on models whose weights exceed available fast memory, it could give developers another way to run larger MoE models locally on a laptop or a single GPU. For those users, memory capacity can be a barrier even when inference needs only some experts for each token. The proposed cache is intended to reduce how many expert weights must stay in device memory at once.

The practical trade-off is speed. Fetching weights from SSD can save memory, but storage access and data movement may slow generation if requested experts are not already cached. The OLMoE result suggests the revised cache design can avoid much of the Python-loop penalty in that test. It does not show that SSD-backed inference on a 180B model will reach similar speeds, or that the method can meet the needs of a 32 GB Mac.

Products Worth Considering

Jackery HomePower 3600 Plus Portable Power Station, 3584Wh, 3600W

Jackery HomePower 3600 Plus Portable Power Station, 3584Wh, 3600W

Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Bluetooth 5.3, USB-C Charging, and IPX4 Water Resistance

Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Bluetooth 5.3, USB-C Charging, and IPX4 Water Resistance

Bose QuietComfort Ultra Earbuds (2nd Gen), Wireless Noise-Cancelling Earbuds, Black

Bose QuietComfort Ultra Earbuds (2nd Gen), Wireless Noise-Cancelling Earbuds, Black

As an affiliate, we earn on qualifying purchases.

How Expert Streaming Differs

In an MoE model, a routing mechanism selects a subset of expert networks for each token. The developer’s approach aims to load those selected experts into a shared, limited-size cache, rather than keeping every expert resident or handling experts individually in a Python loop. The design retains the grouped computation path described in the source.

The source also reports a separate comparison on Qwen3.6-35B-A3B using an A10G GPU with 24 GB of memory and ample system RAM. In that setup, Strata reportedly generated 82–88 tokens per second with Q8_0, while llama.cpp with Q8_0 and its fit option reached 63–64. Those results concern other systems and a specified setup; they do not establish how the proposed Transformers feature would perform on the same hardware or on a laptop.

The author characterizes existing offloading approaches as having different trade-offs: llama.cpp can place experts on the CPU, vLLM’s CPU offload reads offloaded weights over PCIe without a hot-expert cache, and Transformers with Accelerate moves an entire layer’s expert bank before processing it. These descriptions come from the source post and are not same-condition measurements of the new experiment.

The 32 GB Mac Benchmark Is Missing

The source does not provide a completed benchmark for the 180B model on a 32 GB Mac, the use case motivating the experiment. It also omits key details for the OLMoE comparison, including storage type, cache size, prompt conditions and repeated-run results. Without those details, readers cannot judge how the reported speed might vary with different workloads or how often storage access would be needed.

It is also unclear whether the cache will maintain performance when the router’s selected experts change rapidly, and how it compares with Strata on the same model and hardware. The source identifies pull request #50717 but does not say whether it was accepted, merged or released. The reported results support a narrow claim about the OLMoE test, not a general performance conclusion for large models.

Pull Request and Larger-Model Tests

The next project milestone is the status of pull request #50717 and whether the option becomes part of a published Transformers version. The linked Hugging Face Space offers a way to inspect the experimental code, according to the source. No release timetable is provided.

To evaluate the motivating use case, further testing would need to report generation speed and memory use on the intended 180B MoE model and 32 GB Mac, along with the storage device, cache settings, workload and run conditions. A comparison with Strata under matching conditions would help show how the approaches differ. Until such results are reported, the Mac scenario remains an open test rather than a demonstrated capability.

Key Questions

What does the Transformers experiment do?

It proposes loading selected MoE expert weights from storage into a bounded device cache, instead of requiring all experts to remain in fast memory.

What speed did the reported test achieve?

The source reports 34.6 tokens per second on OLMoE, compared with 38.1 for standard Transformers. It does not provide enough test conditions to generalize that result to other devices or models.

Has it been shown to run a 180B model on a 32 GB Mac?

No such benchmark is reported in the source. The 180B-model, 32 GB Mac case is described as a motivating target, not a completed result.

Is the feature available in Transformers now?

The developer says the work was submitted as pull request #50717. The source does not report whether it was accepted, merged or released.

What evidence would clarify the performance claim?

Tests identifying the model, device, storage, cache size and workload—plus repeated runs and a comparison on the same hardware—would help establish how the method performs when experts must be fetched from storage.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

What You Can Explore In Google Genie 4—And What Still Breaks

Google DeepMind’s Genie 4 demos show interactive worlds with sound. Access remains limited, and the scenes can drift after a few minutes.

How OpenAI’s 2026 Data Infrastructure Will Impact Business Intelligence

OpenAI’s 2026 enterprise strategy enhances data governance and AI integration, impacting how businesses leverage AI for internal insights and decision-making.

Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU

A 13-year-old Xeon CPU successfully runs Gemma 4 26B at 5 tokens/sec without a GPU, demonstrating unexpected hardware capabilities.

[Mimi] AI Overview. [220]

Mimi AI releases version 220, introducing significant updates to its AI platform. The update aims to enhance user experience and expand capabilities.