🔍 Read the full analysis: Exploring The Limits Of MoE Inference With Transformers: 180B Parameters, 32GB RAM on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A Hugging Face developer has proposed an experimental Transformers feature that streams selected mixture-of-experts weights from storage into a bounded device cache. The reported OLMoE test reached 34.6 tokens per second, near the 38.1-token baseline, but the source reports no completed test on a 180B model or a 32 GB Mac.
A Hugging Face developer has proposed an experimental way for Transformers to run mixture-of-experts models by loading selected expert weights from storage into a bounded device cache, as detailed in the original analysis. In a reported test using OLMoE, the method generated 34.6 tokens per second, compared with 38.1 for standard Transformers; a 180-billion-parameter model on a 32 GB Mac remains a target, not a reported benchmark.
The experiment addresses a memory constraint in mixture-of-experts (MoE) models: their expert weights can be too large to keep entirely in GPU or unified memory, although the routing mechanism uses only a subset of experts for each token. The author says experts account for more than 90% of the model weights, creating the possibility of keeping selected experts close to the processor while fetching others as needed.
The first implementation reportedly ran experts one by one in a Python loop. On OLMoE-1B-7B, it achieved 10.1 tokens per second even when experts were already cached, compared with about 32 for vanilla Transformers. The author then replaced that approach with a shared, preallocated device tensor and a table mapping expert IDs to cache slots. This allowed the existing grouped matrix multiplication kernel to process selected experts together, yielding the reported 34.6-token rate against a 38.1-token baseline.
The author says the change was submitted to Transformers’ from_pretrained flow as pull request #50717. The source also points to a Hugging Face Space, tarekziade/moe-expert-streaming, with project code. It describes work involving Transformers, a model-specific package and Metal kernels, and says AI tools were used in building the code. These are details of the developer’s account; the source does not report that the pull request has been accepted or released.
Streaming MoE Experts With Limited Memory
An experimental Transformers change streams selected experts from SSD into a bounded cache. The reported OLMoE test ran close to vanilla speed, but the 180B model on a 32 GB Mac remains untested in the source.
How the cache works
The router selects experts for each token; the experiment fetches those experts into a shared cache.
Choose a subset of experts for the token.
Stream them from storage into available device cache slots.
Use Transformers’ grouped matrix multiplication kernel.
What the test shows
On OLMoE-1B-7B, the shared cache kept the reported speed near the standard path.
- Python loop ran experts one at a time.
- Reached 10.1 tokens per second, even with experts cached.
- Preallocated tensor mapped expert IDs to cache slots.
- Reached 34.6 tokens per second versus 38.1 baseline.
The larger device test is open
The 180B model on a 32 GB Mac is a motivating target, not a measured result in the supplied source.
Promising engineering
The cache design brought the reported small-model test close to vanilla Transformers speed.
SSD performance
Storage traffic and cache misses could change speed when experts must be fetched repeatedly.
Measure the target setup
Report speed and memory use on the larger model, with storage, cache settings and workload specified.
Making Oversized MoE Models More Accessible
If the method works on models whose weights exceed available fast memory, it could give developers another way to run larger MoE models locally on a laptop or a single GPU. For those users, memory capacity can be a barrier even when inference needs only some experts for each token. The proposed cache is intended to reduce how many expert weights must stay in device memory at once.
The practical trade-off is speed. Fetching weights from SSD can save memory, but storage access and data movement may slow generation if requested experts are not already cached. The OLMoE result suggests the revised cache design can avoid much of the Python-loop penalty in that test. It does not show that SSD-backed inference on a 180B model will reach similar speeds, or that the method can meet the needs of a 32 GB Mac.
Products Worth Considering

Jackery HomePower 3600 Plus Portable Power Station, 3584Wh, 3600W

Bose QuietComfort True Wireless Earbuds with Active Noise Cancellation, Bluetooth 5.3, USB-C Charging, and IPX4 Water Resistance

Bose QuietComfort Ultra Earbuds (2nd Gen), Wireless Noise-Cancelling Earbuds, Black
As an affiliate, we earn on qualifying purchases.
How Expert Streaming Differs
In an MoE model, a routing mechanism selects a subset of expert networks for each token. The developer’s approach aims to load those selected experts into a shared, limited-size cache, rather than keeping every expert resident or handling experts individually in a Python loop. The design retains the grouped computation path described in the source.
The source also reports a separate comparison on Qwen3.6-35B-A3B using an A10G GPU with 24 GB of memory and ample system RAM. In that setup, Strata reportedly generated 82–88 tokens per second with Q8_0, while llama.cpp with Q8_0 and its fit option reached 63–64. Those results concern other systems and a specified setup; they do not establish how the proposed Transformers feature would perform on the same hardware or on a laptop.
The author characterizes existing offloading approaches as having different trade-offs: llama.cpp can place experts on the CPU, vLLM’s CPU offload reads offloaded weights over PCIe without a hot-expert cache, and Transformers with Accelerate moves an entire layer’s expert bank before processing it. These descriptions come from the source post and are not same-condition measurements of the new experiment.
The 32 GB Mac Benchmark Is Missing
The source does not provide a completed benchmark for the 180B model on a 32 GB Mac, the use case motivating the experiment. It also omits key details for the OLMoE comparison, including storage type, cache size, prompt conditions and repeated-run results. Without those details, readers cannot judge how the reported speed might vary with different workloads or how often storage access would be needed.
It is also unclear whether the cache will maintain performance when the router’s selected experts change rapidly, and how it compares with Strata on the same model and hardware. The source identifies pull request #50717 but does not say whether it was accepted, merged or released. The reported results support a narrow claim about the OLMoE test, not a general performance conclusion for large models.
Pull Request and Larger-Model Tests
The next project milestone is the status of pull request #50717 and whether the option becomes part of a published Transformers version. The linked Hugging Face Space offers a way to inspect the experimental code, according to the source. No release timetable is provided.
To evaluate the motivating use case, further testing would need to report generation speed and memory use on the intended 180B MoE model and 32 GB Mac, along with the storage device, cache settings, workload and run conditions. A comparison with Strata under matching conditions would help show how the approaches differ. Until such results are reported, the Mac scenario remains an open test rather than a demonstrated capability.
Key Questions
What does the Transformers experiment do?
It proposes loading selected MoE expert weights from storage into a bounded device cache, instead of requiring all experts to remain in fast memory.
What speed did the reported test achieve?
The source reports 34.6 tokens per second on OLMoE, compared with 38.1 for standard Transformers. It does not provide enough test conditions to generalize that result to other devices or models.
Has it been shown to run a 180B model on a 32 GB Mac?
No such benchmark is reported in the source. The 180B-model, 32 GB Mac case is described as a motivating target, not a completed result.
Is the feature available in Transformers now?
The developer says the work was submitted as pull request #50717. The source does not report whether it was accepted, merged or released.
What evidence would clarify the performance claim?
Tests identifying the model, device, storage, cache size and workload—plus repeated runs and a comparison on the same hardware—would help establish how the method performs when experts must be fetched from storage.
Primary source: Hugging Face · via ThorstenMeyerAI.com
