Hugging Face on November 18 published an experiment running Qwen3.8-Flash-Next, a mixture-of-experts model of about 180 billion parameters, through its Transformers library on an M5 MacBook with 32 GB of unified memory. A new “expert streaming” option reads only the experts the router picks from the SSD, and after tuning it decoded 13.9 tokens per second, about 6% behind the specialized Strata engine. The code is a pull request, not a released feature.
What expert streaming means
In a mixture-of-experts model, each layer holds many small “experts”, and a router sends every token to only a few. Experts are over 90% of a MoE model’s weights, the post notes. Qwen3.8-Flash-Next has 48 layers of 512 experts, 10 active per token, and its experts alone take 121 GB in FP8.
Expert streaming keeps the dense layers and a bounded cache of recently used experts in memory and leaves the rest on disk; a cache miss is read from the SSD. The author, Tarek Ziadé, says the key idea was turning the cache into one preallocated “slot pool” shared by all layers, so Transformers’ existing grouped matrix-multiply kernel runs on it unchanged. Eight I/O threads fetch misses, and the next two layers’ routers run early to prefetch likely experts.
The setup and the numbers as reported
Experts were quantized to 3 bits (49 GB on disk, 12 GiB cached on the GPU), dense layers to 8 bits (about 5.4 GB, costing 0.2% in perplexity against BF16 per the post), and a 51 GB n-gram embedding table stays on disk with rows read on demand. Conversion took about 5 hours. After loading, memory use is about 18 GiB, peaking at 25.4 GiB on a 6,000-token prompt, near the point where macOS starts swapping.
After custom Metal kernels, earlier GPU command commits and a switch to Strata’s GGUF weights (the first version managed about 10 tokens per second), the reported results on the same M5 were:
- Decode, 200 tokens: 13.9 tok/s for Transformers vs 14.8 for Strata 0.1.60
- Prefill, 4K prompt: 228 tok/s vs 220
- Decode after a 6K document: 12.4 tok/s vs 13.7
Both ran greedy, with thinking and speculative drafts off. The author blames most of the gap on about 10 ms per token of host round trips in PyTorch’s Apple GPU backend, and says 14 tokens per second is still too slow for his coding-agent work.
How to try it
The feature is in Transformers pull request #50717, installed with pip install "git+https://github.com/huggingface/transformers.git@refs/pull/50717/head". Experts go to disk via device_map (for example {"": "mps", "experts.*": "disk"}), and a new experts_offload argument to from_pretrained sets cache_bytes and an optional path to pre-quantized experts. Converted weights and a helper package are on the Hub. The post says the API is still being discussed with maintainers and names no PyTorch version.
How it compares with other ways to run big models locally
The post also benchmarked Qwen3.6-35B-A3B on one 24 GB A10G GPU with plenty of RAM. Strata decoded 82 to 88 tokens per second with Q8_0 weights, llama.cpp with Q8_0 and --fit on reached 63 to 64, vLLM with FP8 and experts-only CPU offload managed 20.1, and Transformers’ old Accelerate offloading crawled at 0.16, because it copies a layer’s whole expert bank to the GPU for every token.
llama.cpp, which reached v1.0 last week, memory-maps GGUF files and can compute experts on the CPU so only activations cross PCIe; the post calls it “hard to beat” when experts fit in RAM, but says the page cache thrashes when they do not. Ollama runs GGUF models too. Machines with large unified memory simply hold the whole model; on a Mac classic “offload to CPU” frees nothing, since CPU and GPU share one pool. Streaming from SSD covers the remaining case: a model far larger than memory, at the cost of speed.
What to watch next
- Whether pull request #50717 is merged and in which Transformers release
- The final shape of the
experts_offloadAPI - A PyTorch MPS fix for the per-layer synchronization the author names as the last bottleneck
- Whether the approach is extended to CUDA GPUs with limited VRAM
Key takeaways
- Transformers ran a roughly 180B-parameter MoE model on a 32 GB MacBook by streaming experts from SSD.
- Final speeds were 13.9 tok/s decode and 228 tok/s prefill, about 6% behind Strata on decode.
- It relies on 3-bit experts, 8-bit dense layers and a 5-hour conversion.
- llama.cpp stays strong when experts fit in RAM; streaming targets models that do not.
- The feature is a pull request whose API is still under discussion.
Frequently asked questions
What is MoE expert streaming?
It runs a mixture-of-experts model too big for memory by reading only the experts the router selects from disk into a bounded cache.
Can I run Qwen3.8-Flash-Next on a 32GB Mac with Transformers today?
Only experimentally, by installing Transformers from pull request #50717 and using the pre-converted weights from the Hub. The author reports memory use close to the 32 GB limit.
Is expert streaming faster than llama.cpp?
Not when the experts fit in RAM, where the post calls llama.cpp hard to beat. Streaming helps when the model is far larger than memory.




