180B Parameters, 32GB RAM: A Practical Guide To MoE Inference With Transformers
AIThis post was created with the assistance of artificial intelligence (AI).

🔍 Read the full analysis: 180B Parameters, 32GB RAM: A Practical Guide To MoE Inference With Transformers on ThorstenMeyerAI.com

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

TL;DR

A Hugging Face developer proposed an experimental Transformers option that loads selected mixture-of-experts weights into a bounded device cache. In a reported OLMoE test, it reached 34.6 tokens per second, close to the 38.1-token-per-second standard Transformers result; no completed benchmark is provided for a 180B model on a 32 GB Mac.

As detailed in the original analysis, a Hugging Face developer has proposed an experimental SSD-backed option for Transformers that loads selected mixture-of-experts (MoE) weights into a bounded device cache. In a reported test on OLMoE-1B-7B, the approach generated 34.6 tokens per second, compared with 38.1 for standard Transformers; the source does not report a completed test of the motivating 180B model on a 32 GB Mac.

The proposal targets MoE models, which route each token through only a subset of a model’s experts. The developer says experts account for more than 90% of model weights, creating the possibility of keeping selected experts close to the processor and retrieving others when needed. The experimental design uses a bounded, shared cache rather than requiring every expert to remain in fast memory.

The developer’s first implementation ran experts one at a time in a Python loop. On OLMoE, that version reached 10.1 tokens per second even when the experts were already cached; the reported standard Transformers result was about 32. The revised design stored experts in a preallocated device tensor and mapped expert IDs to cache slots, allowing Transformers’ grouped matrix multiplication kernel to process selected experts together. The developer reports 34.6 tokens per second for that version against 38.1 for the baseline.

The work was submitted to Transformers’ from_pretrained flow as pull request #50717. The source also points to a Hugging Face Space, tarekziade/moe-expert-streaming, with project code. It describes implementation work spanning Transformers, a model-specific package and Metal kernels, and says AI tools were used in building the code.

At a glance
reportWhen: Experimental proposal; pull request #50…
The developmentAn experimental change proposed for Hugging Face Transformers streams selected MoE experts into a bounded cache, with a reported near-baseline speed result on OLMoE.
AI Inference · Briefing

Streaming MoE Experts With Limited Memory

An experimental Transformers change streams selected experts from SSD into a bounded cache. The reported OLMoE test ran close to vanilla speed, but the 180B model on a 32 GB Mac remains untested in the source.

34.6 tokens/s
OLMoE with expert streaming
38.1 tokens/s
Vanilla Transformers comparison
90%+
Model weights attributed to experts

How the cache works

The router selects experts for each token; the experiment fetches those experts into a shared cache.

1
Route each token

Choose a subset of experts for the token.

2
Load selected experts

Stream them from storage into available device cache slots.

3
Compute together

Use Transformers’ grouped matrix multiplication kernel.

What the test shows

On OLMoE-1B-7B, the shared cache kept the reported speed near the standard path.

Earlier approach
  • Python loop ran experts one at a time.
  • Reached 10.1 tokens per second, even with experts cached.
Shared cache
  • Preallocated tensor mapped expert IDs to cache slots.
  • Reached 34.6 tokens per second versus 38.1 baseline.

The larger device test is open

The 180B model on a 32 GB Mac is a motivating target, not a measured result in the supplied source.

Evidence

Promising engineering

The cache design brought the reported small-model test close to vanilla Transformers speed.

Open question

SSD performance

Storage traffic and cache misses could change speed when experts must be fetched repeatedly.

Next evidence

Measure the target setup

Report speed and memory use on the larger model, with storage, cache settings and workload specified.

StatusThe change was submitted as Transformers pull request #50717; the source does not say whether it was merged or released.
The takeaway
Expert streaming looks close to baseline on the reported OLMoE test, while performance on the 180B model and 32 GB Mac remains unproven.

Lower Memory Costs, Unproven Scale

If the approach works on models whose experts cannot all fit in memory, it could make some larger MoE systems more accessible on single GPUs or laptops. The potential benefit is not simply lower memory use: the system must retrieve the experts selected for each token quickly enough that storage and data movement do not overwhelm generation speed.

The OLMoE result is evidence for a narrower claim: the shared-cache design avoided the large slowdown seen in the developer’s initial Python-loop implementation and came close to the reported standard Transformers speed in that test. It does not establish performance when experts must be fetched from an SSD for a much larger model, or show that a 180B model can run usefully on 32 GB of unified memory.

Products Worth Considering

DJI Mini 4K Drone Combo with 4K Camera, 3-Axis Gimbal, and Two Batteries

DJI Mini 4K Drone Combo with 4K Camera, 3-Axis Gimbal, and Two Batteries

As an affiliate, we earn on qualifying purchases.

How the Cache Differs

The proposal responds to a practical constraint in MoE inference: although a token uses only selected experts, those experts still need to be available when the router selects them. The developer’s approach aims to keep a limited set in a device cache and stream other experts from lower-capacity storage as required, while retaining Transformers’ grouped expert computation.

The source also reports comparisons on a different setup: Qwen3.6-35B-A3B running on an A10G GPU with 24 GB of memory and ample system RAM. In those tests, it says Strata reached 82–88 tokens per second with Q8_0, while llama.cpp with Q8_0 and its fit option reached 63–64. These figures apply only to the stated model and setup; they are not direct comparisons with the Transformers experiment or predictions for a Mac.

The post characterizes other methods as handling offloaded experts differently: llama.cpp can place experts on the CPU, while vLLM’s CPU offload reads weights over PCIe without a hot-expert cache. It says Transformers with Accelerate moves an entire layer’s expert bank before processing it. Those descriptions and performance figures are the author’s account, not results from a same-device comparison of all systems.

““choosing experiments, reading profiles and deciding what to retain””

— The developer, describing the implementation work

The 32 GB Mac Test Is Missing

The source gives no measured result for a 180B model running with this Transformers implementation on a 32 GB Mac. It also does not report the storage device, cache size, prompt conditions or repeated-run data for the OLMoE comparison, limiting assessment of how stable the reported speeds are across workloads.

The status of pull request #50717 is also unspecified: the source says it was submitted but does not say whether it was accepted, merged or released. It remains unclear how the cache performs when routing patterns change rapidly, how much SSD traffic the intended large-model workload would require, and how the proposal compares with Strata under matching hardware and model conditions.

PR Status and Large-Model Results

The next development to watch is whether pull request #50717 is accepted and made available in a Transformers release. The linked Hugging Face Space offers a way to inspect the experimental project, according to the source.

To test the proposal’s central motivation, future results would need to report generation speed and memory use on the intended large MoE model and 32 GB Mac, alongside the storage type, cache settings and workload. A comparison with Strata using the same model and hardware would also help establish whether the approaches have similar speed and memory trade-offs.

Key Questions

Did the developer show that a 180B model runs on a 32 GB Mac?

No. The source presents that setup as a motivating target but provides no completed benchmark for it.

What speed did the experimental Transformers option report?

The developer reports 34.6 tokens per second on OLMoE-1B-7B, compared with 38.1 for standard Transformers. The source does not provide enough test details to generalize that result to other models or devices.

What does the proposed cache do?

It aims to keep a bounded set of selected MoE experts in a shared device tensor and retrieve other experts as needed, while using Transformers’ grouped matrix multiplication kernel.

Is the change available in Transformers now?

The source says the work was submitted as pull request #50717, but does not state whether it was merged or released.

How does the OLMoE result compare with Strata?

The reported OLMoE speed is not a direct comparison with Strata. The source gives separate Strata and llama.cpp results for Qwen3.6-35B-A3B on an A10G GPU, a different model and setup.

Primary source: Hugging Face · via ThorstenMeyerAI.com

You May Also Like

How to Choose a NAS That Fits the Way You Actually Work

A well-chosen NAS can transform your workflow—discover how to select the perfect one tailored to your daily needs.

How AI Quantization To Four Bits Limits Model Capabilities

Exploring how quantizing AI models to four bits impacts their capabilities, revealing sharp performance drops beyond this threshold.

Mistral Forge: Owning the Model, Not Just Renting the API

Mistral’s Forge offers organizations the ability to build and operate proprietary AI models, moving beyond API rentals to full ownership and control.

Internet of Things: Building Smart Cities and Homes

Navigating the realm of the Internet of Things reveals how smart cities and homes can transform daily living, but what challenges lie ahead?