How smolmaxxing runs an 80-billion-parameter model on a 2022 phone with 3 GB free

smolmaxxing is an open-source (MIT) patch on llama.cpp that runs language models far bigger than a device's RAM. On a OnePlus Nord CE 2 from 2022…

Loading
smolmaxxing (hyperionaidev)··PT2M16.5S

video loads only when you press play

smolmaxxing is an open-source (MIT) patch on llama.cpp that runs language models far bigger than a device's RAM. On a OnePlus Nord CE 2 from 2022…

// the short version

smolmaxxing is an open-source (MIT) patch on llama.cpp that runs language models far bigger than a device's RAM. On a OnePlus Nord CE 2 from 2022 (MediaTek Dimensity 900, 7.4 GB of RAM with about 3 GB free, Android 13, not rooted, driven over adb), it runs Qwen3-Next-80B-A3B, an 80-billion-parameter model whose file is 27.6 GB.

// what to take away

  • Qwen3-Next-80B-A3B is a mixture of experts: 48 layers of 512 experts, and a router in each layer runs only 10 of them (plus a shared one) per token, so about 3 billion of its 80 billion parameters are active. That is what lets most of the 27.6 GB file stay on the phone's flash storage.
  • smolmaxxing's expert pager keeps 56 experts per layer in RAM, least recently used out first, and reads a missed expert whole from flash before the layer finishes. It runs the router two layers ahead to start those reads early, and a small routing bias toward experts already in RAM cuts misses from 13.4% to 9.9%.
  • On a 2022 OnePlus Nord CE 2 with about 3 GB of RAM free, the project measures 3.1 tokens a second with the bias (perplexity +0.5%) and 2.6 without it. It is a lab notebook with benchmarks, not an app, and the corrected numbers replace earlier ones that came from a pager bug.

What smolmaxxing is, and what it is not

smolmaxxing is an open-source (MIT) set of patches to llama.cpp, the widely used C/C++ inference engine, aimed at one question: what is the biggest language model a small device can really run? It has been tried on three devices so far: a Xiaomi TV Box S (about 1 GB of usable RAM), a jailbroken 2013 iPad Air, and a OnePlus Nord CE 2 phone from 2022.

It is not an app. Everything on the phone runs from a computer over USB, as the Android adb shell user, without root. The repository is closer to a lab notebook: 166 numbered experiments, failures included, with the commands and logs behind every number.

The phone, and the model that should not fit

The OnePlus Nord CE 2 has a MediaTek Dimensity 900 (two Cortex-A78 cores and six Cortex-A55), 7.4 GB of LPDDR4X RAM with about 3 GB free while the phone's own apps run, and UFS 2.2 flash storage. The model is Qwen3-Next-80B-A3B: 80 billion parameters in total, about 3 billion active per token. The project's file, with 2-bit experts (Q2_K_S) and 4-bit everything else (Q4_K), is 27.6 GB, of which 1.5 GB is not experts.

So the file is about nine times the free RAM. Loading it the usual way is not an option.

Why a mixture-of-experts model can be paged

In a dense model, every weight in a layer takes part in every token. Qwen3-Next is a mixture of experts: each of its 48 layers has 512 experts, small networks of their own (here each is 2048 × 512 in shape), plus one shared expert. For every token, a router in each layer scores the 512 and runs the top 10, plus the shared one. Most of the model sits unused for any given token, which is what makes it possible to leave it on storage.

The expert pager, step by step

The pager lives in the patch (ggml-cpu/moe-pager-tv.inc) and works the same way on the TV box and the phone:

  • Experts live on flash. Each expert tensor is backed by empty memory. Right after a layer's router picks its experts, the ones that are missing are read from flash with parallel O_DIRECT reads straight into their own addresses, so the matrix kernels see an ordinary tensor.
  • A pool per layer in RAM. Each layer keeps a pool of recently used experts (least recently used out first), warmed at start from a list of each layer's most used experts. For the 80B on the phone the pool is 56 of 512 per layer.
  • Read whole, several at once. The phone's flash only reaches its ceiling of about 940 MB/s with two to four reads of 256 KB or more in flight; a single 4 KB read runs at 28 to 81 MB/s. So experts are read whole, by a pool of reader threads.
  • Pre-gated prefetch. Layer L+2's router is applied to layer L's hidden state, predicting what the router two layers ahead will pick, and those experts start loading early. On the TV box this loads about 60% of misses before they are needed; on the phone, 95% of Ling-mini-2.0's prefetches turned out to be used. The 80B runs prefetch 10 experts, two layers ahead.
  • Cache-aware routing. A small bonus added to the router's scores for experts already in RAM (GGML_MOE_BIAS). It changes which experts the model uses, so its quality cost is measured as perplexity before it is kept. Qwen's router is a softmax over its experts, so the bonus that suits it (0.005) is far smaller than the 0.25 used for Ling.

What the 2.6 and 3.1 tokens per second were measured on

Chat generation through llama-server with the server's repeat penalty, two prompts of 160 tokens per setting, two threads pinned to the A78 cores, with the screen off (which caps those cores at 1.3 GHz of their 2.4):

  • Exact routing, pool 56: 2.6 tokens/s, 13.4% of expert picks missed the pool, peak memory (RSS) 4.7 GB, perplexity 7.44.
  • Routing bias 0.005, pool 56: 3.1 tokens/s, 9.9% missed, peak memory 5.0 GB, perplexity 7.47 (+0.5%).

Perplexity was measured on the phone over 20 chunks of 128 tokens of an English evaluation text. Peak memory is above the 3 GB that is free at idle because Android hands back its caches when a process asks; every run sits under a memory guard so it cannot starve the system.

A correction the project published. The 80B was first posted at 2.8 and 3.2 tokens/s. The pager's routing hook assumed 8 experts per token (right for Ling and Qwen3.6), but Qwen3-Next routes through 10, so experts 9 and 10 were multiplied from empty pages unless they were already cached. With the fix (E166), perplexity barely moved and the honest speeds are 2.6 and 3.1.

Where the time goes

A profile of the 80B before the fix (bias 0.005, pool 56) split a token roughly into: 4-bit matrix multiplies for the non-expert weights 28%, waiting for flash 27%, 2-bit expert matrix multiplies 23%, the routers 13%. For the smaller Qwen3.6-35B-A3B, flash waits are down to about a fifth of the time, and the biggest cost is reading 1.1 GB of non-expert weights from RAM every token.

What the video left out

  • The other models on the phone. Qwen3.6-35B-A3B runs at 4.9 tokens/s exact and 5.7 at +0.7% perplexity, which the project calls the best model per answer on the phone. Ling-mini-2.0 (16B) runs at 11.1 tokens/s exact.
  • What did not help. More reader threads, readers on the slower cores, and overlapping reads with expert compute all moved the wait around without making tokens faster. Static pruning of rarely used experts raised perplexity by 34%.
  • The smaller devices. On the TV box, with about 1 GB usable, the same pager runs the 16B Ling-mini-2.0 at 5.1 tokens/s, and dense models streamed from flash are limited by its 278 MB/s storage (a dense 9B model runs at 0.27 tokens/s).

Terms

Mixture of experts (MoE)
A model design where each layer holds many small networks (experts) and a router runs only a few of them per token, so most of the weights are idle at any moment.
Router
The small layer that scores every expert for the current token and picks the top few to run.
Expert pager
smolmaxxing's code that keeps experts on flash and loads the ones the router picks into RAM on demand.
Pool
The experts per layer the pager keeps in RAM; 56 of 512 for the 80B on the phone.
Cache miss
A pick the pool doesn't hold, so the layer waits while the expert is read from flash.
Pre-gated prefetch
Running a later layer's router early, on the current layer's output, to start loading the experts it will probably pick.
Routing bias
A small bonus on the router's scores for experts already in RAM, trading a measured bit of quality for fewer misses.
Perplexity
A standard measure of how well a model predicts a text; lower is better, and +0.5% means slightly worse.
Peak RSS
The most physical memory the process held at once.
O_DIRECT
A Linux flag for reading a file straight into a buffer without going through the page cache.

Questions

Can a phone really run an 80-billion-parameter model?

This one can, slowly: a 2022 OnePlus Nord CE 2 with about 3 GB of RAM free runs Qwen3-Next-80B-A3B at 2.6 to 3.1 tokens per second in the project's tests, because the model is a mixture of experts and only the experts each token needs are read from flash.

Does the whole model have to fit in RAM?

No. The 27.6 GB file stays on flash. RAM holds the non-expert weights and a pool of 56 experts per layer, and peaks at about 5 GB.

Do I need root?

No. The phone is not rooted; everything runs as the adb shell user from a computer over USB.

How much quality does it cost?

With exact routing, none beyond the 2-bit compression of the experts. The routing bias that gets 3.1 tokens/s raised perplexity by 0.5% in the project's measurement.

Is it usable as a chat app?

Not yet. It is a research notebook driven from a computer, and 3 tokens a second is slow for chat. The 35B model on the same phone runs at about 5 to 6 tokens a second.

// further reading

// transcript

A phone from 2022 is running an 80-billion-parameter AI model. The model file is almost 28 GB, and the phone has 7.4 GB of RAM, with about three free once its own apps are running, and a model normally has to fit in memory, so this one is nine times too big. An open-source project gets around that by leaving most of the model on the phone's storage, and reading in only the pieces each word needs. Those pieces come from how the model is built. A language model writes one token at a time, a word or part of one, passing each through a stack of layers, and in this model, each of its 48 layers is split into 512 blocks of weights called experts. For every token, a router in each layer picks just ten of them, so only about three billion of its 80 billion parameters work on any given token, and everything else can wait somewhere slower. Slower means flash, where the whole model lives, while RAM holds a small shelf for each layer, 56 of its 512 experts, the ones used most recently. When the router picks an expert on the shelf, it runs right away. When it picks one that isn't, that's a cache miss, and the layer waits while it's read whole from flash, along with any other misses. To hide that wait, it guesses ahead, running the router of the layer two steps down early, on the current layer's output, and starts reading the experts it predicts before they're asked for. Even so, with the router left exactly as trained, about 13% of picks still miss, and the phone writes 2.6 tokens a second. The last dial is the router itself. The project gives it a small bonus for experts already on the shelf, so a near tie goes to the one in RAM. That bends the model's choices, so it was measured: misses fall to about one in ten, and its score on a test text gets half a percent worse. With the bonus on, the phone writes 3.1 tokens a second from an 80-billion-parameter model, using about 5 GB at its peak, on a phone that isn't rooted. That's slow, and it's a lab notebook rather than an app, but it turns out the model never had to fit on the phone, only the next word did.

// source

This explainer is based on smolmaxxing: run language models far bigger than your RAM by smolmaxxing (hyperionaidev) ↗. The original reporting and technical work belong to its publisher.