🔍 Read the full analysis: 180B Parameters, 32GB RAM: A Practical Guide To MoE Inference With Transformers on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A Hugging Face developer proposed an experimental Transformers option that loads selected mixture-of-experts weights into a bounded device cache. In a reported OLMoE test, it reached 34.6 tokens per second, close to the 38.1-token-per-second standard Transformers result; no completed benchmark is provided for a 180B model on a 32 GB Mac.
As detailed in the original analysis, a Hugging Face developer has proposed an experimental SSD-backed option for Transformers that loads selected mixture-of-experts (MoE) weights into a bounded device cache. In a reported test on OLMoE-1B-7B, the approach generated 34.6 tokens per second, compared with 38.1 for standard Transformers; the source does not report a completed test of the motivating 180B model on a 32 GB Mac.
The proposal targets MoE models, which route each token through only a subset of a model’s experts. The developer says experts account for more than 90% of model weights, creating the possibility of keeping selected experts close to the processor and retrieving others when needed. The experimental design uses a bounded, shared cache rather than requiring every expert to remain in fast memory.
The developer’s first implementation ran experts one at a time in a Python loop. On OLMoE, that version reached 10.1 tokens per second even when the experts were already cached; the reported standard Transformers result was about 32. The revised design stored experts in a preallocated device tensor and mapped expert IDs to cache slots, allowing Transformers’ grouped matrix multiplication kernel to process selected experts together. The developer reports 34.6 tokens per second for that version against 38.1 for the baseline.
The work was submitted to Transformers’ from_pretrained flow as pull request #50717. The source also points to a Hugging Face Space, tarekziade/moe-expert-streaming, with project code. It describes implementation work spanning Transformers, a model-specific package and Metal kernels, and says AI tools were used in building the code.
Streaming MoE Experts With Limited Memory
An experimental Transformers change streams selected experts from SSD into a bounded cache. The reported OLMoE test ran close to vanilla speed, but the 180B model on a 32 GB Mac remains untested in the source.
How the cache works
The router selects experts for each token; the experiment fetches those experts into a shared cache.
Choose a subset of experts for the token.
Stream them from storage into available device cache slots.
Use Transformers’ grouped matrix multiplication kernel.
What the test shows
On OLMoE-1B-7B, the shared cache kept the reported speed near the standard path.
- Python loop ran experts one at a time.
- Reached 10.1 tokens per second, even with experts cached.
- Preallocated tensor mapped expert IDs to cache slots.
- Reached 34.6 tokens per second versus 38.1 baseline.
The larger device test is open
The 180B model on a 32 GB Mac is a motivating target, not a measured result in the supplied source.
Promising engineering
The cache design brought the reported small-model test close to vanilla Transformers speed.
SSD performance
Storage traffic and cache misses could change speed when experts must be fetched repeatedly.
Measure the target setup
Report speed and memory use on the larger model, with storage, cache settings and workload specified.
Lower Memory Costs, Unproven Scale
If the approach works on models whose experts cannot all fit in memory, it could make some larger MoE systems more accessible on single GPUs or laptops. The potential benefit is not simply lower memory use: the system must retrieve the experts selected for each token quickly enough that storage and data movement do not overwhelm generation speed.
The OLMoE result is evidence for a narrower claim: the shared-cache design avoided the large slowdown seen in the developer’s initial Python-loop implementation and came close to the reported standard Transformers speed in that test. It does not establish performance when experts must be fetched from an SSD for a much larger model, or show that a 180B model can run usefully on 32 GB of unified memory.
Products Worth Considering

Amazon Kindle Scribe (16GB) with Premium Pen, Tungsten

DJI Mini 4K Drone Combo with 4K Camera, 3-Axis Gimbal, and Two Batteries

Jackery Explorer 300 Portable Power Station, 292Wh
As an affiliate, we earn on qualifying purchases.
How the Cache Differs
The proposal responds to a practical constraint in MoE inference: although a token uses only selected experts, those experts still need to be available when the router selects them. The developer’s approach aims to keep a limited set in a device cache and stream other experts from lower-capacity storage as required, while retaining Transformers’ grouped expert computation.
The source also reports comparisons on a different setup: Qwen3.6-35B-A3B running on an A10G GPU with 24 GB of memory and ample system RAM. In those tests, it says Strata reached 82–88 tokens per second with Q8_0, while llama.cpp with Q8_0 and its fit option reached 63–64. These figures apply only to the stated model and setup; they are not direct comparisons with the Transformers experiment or predictions for a Mac.
The post characterizes other methods as handling offloaded experts differently: llama.cpp can place experts on the CPU, while vLLM’s CPU offload reads weights over PCIe without a hot-expert cache. It says Transformers with Accelerate moves an entire layer’s expert bank before processing it. Those descriptions and performance figures are the author’s account, not results from a same-device comparison of all systems.
““choosing experiments, reading profiles and deciding what to retain””
— The developer, describing the implementation work
The 32 GB Mac Test Is Missing
The source gives no measured result for a 180B model running with this Transformers implementation on a 32 GB Mac. It also does not report the storage device, cache size, prompt conditions or repeated-run data for the OLMoE comparison, limiting assessment of how stable the reported speeds are across workloads.
The status of pull request #50717 is also unspecified: the source says it was submitted but does not say whether it was accepted, merged or released. It remains unclear how the cache performs when routing patterns change rapidly, how much SSD traffic the intended large-model workload would require, and how the proposal compares with Strata under matching hardware and model conditions.
PR Status and Large-Model Results
The next development to watch is whether pull request #50717 is accepted and made available in a Transformers release. The linked Hugging Face Space offers a way to inspect the experimental project, according to the source.
To test the proposal’s central motivation, future results would need to report generation speed and memory use on the intended large MoE model and 32 GB Mac, alongside the storage type, cache settings and workload. A comparison with Strata using the same model and hardware would also help establish whether the approaches have similar speed and memory trade-offs.
Key Questions
Did the developer show that a 180B model runs on a 32 GB Mac?
No. The source presents that setup as a motivating target but provides no completed benchmark for it.
What speed did the experimental Transformers option report?
The developer reports 34.6 tokens per second on OLMoE-1B-7B, compared with 38.1 for standard Transformers. The source does not provide enough test details to generalize that result to other models or devices.
What does the proposed cache do?
It aims to keep a bounded set of selected MoE experts in a shared device tensor and retrieve other experts as needed, while using Transformers’ grouped matrix multiplication kernel.
Is the change available in Transformers now?
The source says the work was submitted as pull request #50717, but does not state whether it was merged or released.
How does the OLMoE result compare with Strata?
The reported OLMoE speed is not a direct comparison with Strata. The source gives separate Strata and llama.cpp results for Qwen3.6-35B-A3B on an A10G GPU, a different model and setup.
Primary source: Hugging Face · via ThorstenMeyerAI.com
