Strata is an open-source (MIT) inference engine that runs Qwen3.8-Flash-Next, a 125-billion-parameter language model, on a gaming PC with a 12 GB graphics…
// the short version
Strata is an open-source (MIT) inference engine that runs Qwen3.8-Flash-Next, a 125-billion-parameter language model, on a gaming PC with a 12 GB graphics card. Models like GPT and Claude run in their makers' data centers and their weights are not published; this model's weights can be downloaded, so it can run locally.
// what to take away
- Qwen3.8-Flash-Next is a mixture-of-experts model: 48 layers of 512 experts each, and a router in every layer runs only ten of them per token, about 2% of the 24,576.
- Strata uses graphics memory as a cache: the most-used experts sit on the card, all of them stay pinned in RAM, and the CPU computes the misses at the same time as the GPU works on the hits.
- The project measured 94 tokens a second on an RTX 5070 with 12 GB, at its most compressed setting and with 64 GB of RAM; it publishes no quality figure for that setting and answers one request at a time.
What Strata is, and what it is not
Strata is an open-source inference engine, released under the MIT license, that runs one specific large language model on an ordinary gaming PC. The model is Qwen3.8-Flash-Next, from Alibaba's Qwen team. Strata's documentation says models like it usually run on servers with hundreds of gigabytes of graphics memory, while a gaming card has 12 to 24 GB.
It is not a new model and it does not make the model smaller than the card. The model's weights are published separately, under the Qwen Community License rather than an open-source license, and you download them yourself (about 70 GB). Strata is the program that decides where each part of those weights lives and which processor works on it.
It is also not a hosted service. It serves a local address (http://127.0.0.1:8080) with a chat page and an OpenAI-compatible API, so the prompts stay on the machine. That is the practical difference from models such as GPT and Claude, which run in their makers' data centres and whose weights are not published. Whether those models use the design described below is not public, so nothing here is a claim about them.
How a mixture-of-experts model spends its weights
A language model produces one token at a time, and to choose each token it passes the text through a stack of layers. Each layer is a large set of numbers, the weights. In a dense model, every weight in a layer takes part in every token.
Qwen3.8-Flash-Next is a mixture-of-experts model. Its model card gives the layout:
- 48 layers
- 512 experts in each layer, so 48 × 512 = 24,576 experts in all
- for each token, 10 routed experts per layer, plus 1 shared expert that always runs
- 125 billion parameters in total, of which 6 billion are active for any one token
An expert is not a specialist in a subject. It is a block of a layer's weights that forms a small network of its own. A router at the front of each layer scores the experts for the current token and runs the ten with the best scores.
Strata's README shortens this to "each word needs only 10" of the 24,576 experts. Read with the model card, that is ten per layer: 480 routed experts per token, which is about 2% of them. The share of the whole model is closer to 5% (6 billion of 125 billion parameters), because the attention layers, routers and shared experts run for every token.
The expert cache, tier by tier
The idea Strata builds on is that the experts are not picked evenly. Some are routed to far more often than others, so the engine treats graphics memory as a cache for the popular ones.
Graphics card. It holds everything every token needs: the attention layers, the routers, the shared experts, the output head and the conversation's key-value cache. Whatever memory is left becomes the expert cache, filled with the most-used experts. The documentation says each extra gigabyte holds about 700 more experts, which is why it says more graphics memory matters more than a faster card.
RAM. All 24,576 experts are kept here, pinned so the operating system cannot page them out. When a layer routes to an expert the card does not hold, the CPU computes it in place, using AVX-512 or AVX2 code, at the same time as the card computes the cached ones. Neither waits for the other.
How it knows what is popular. Strata ships a profile that ranks every expert by how often it is used, and you can build one from your own prompts. On top of that, an adaptive tier keeps swapping experts between RAM and the card as the conversation changes subject (the --adapt-every setting, on by default).
The documentation does not publish a headline hit rate. One passage mentions 75 to 90% of expert lookups landing on the card for the single-card machines in a profiling run, and 99.7% for a two-card setup that holds nearly every expert. Treat those as examples from one test, not a specification.
What the video left out
Three parts of the engine were cut from the narration for length.
Guess, then check. The model has a small built-in draft layer (the model card calls it MTP, multi-token prediction) that guesses up to three tokens ahead. One pass through all 48 layers then checks every guess, keeps the ones the full model agrees with, and writes the next token itself. Strata reports 2.4 to 3.2 tokens per pass on average and answers arriving 1.6 to 1.8 times sooner, with the same output, because the full model still decides every token.
The lookup table on the SSD. Besides its experts, the model has an n-gram embedding table (51 billion parameters, by the model card). Strata keeps a 28.8 GB copy on the SSD and reads only a few rows of it per token.
Reading long prompts. A long prompt is processed in pieces of up to 8,192 tokens, and the next layer's experts are streamed to the card while the current layer runs. After the first message, only the new text is read.
What the 94 tokens per second was measured on
The headline figure is the project's own, and it comes with conditions:
- Machine: an RTX 5070 with 12 GB, a Ryzen 5 7600 and 64 GB of RAM.
- Setting: Q2_0, the most compressed size, at about two bits per weight. It needs 37.6 GB of RAM and graphics memory combined. The less compressed sizes ran at 53 to 79 tokens a second on the same machine.
- Engine version: the 94 was measured on engine 0.1.36; the other rows in the same table used 0.1.26.
- Workload: writing answers in a short chat. With a 128K-token context the same setting wrote 76 tokens a second.
- Quality: the documentation says smaller sizes are faster and larger ones "a bit smarter", and rates Q2_0 "good" without a number. The one quality figure it gives is for a different variant, the Coder, which keeps 256 of each layer's 512 experts and reaches 91.3% of the full model's SWE-bench Verified score according to that variant's authors.
A figure of 100 tokens a second on an RTX 4090 circulated with the project on Hacker News in early October 2026. The README does not measure a 4090. Its 100 to 140 tokens a second for an RTX 3090 is labelled as what the card "should" do, an estimate.
What it takes to run
- A graphics card with 12 GB of memory or more: NVIDIA RTX 20 series and later, or a listed AMD Radeon card.
- 32 GB of RAM as the stated minimum, which fits only the Coder variant. The other sizes need 48 GB or more, and 64 GB runs every size.
- About 80 GB of free disk, preferably an SSD.
- Windows 10 or 11, or Linux.
Two limits the README states plainly: the PC can be slow or stop responding for one to three minutes while the model loads 35 to 55 GB into RAM, and by default it answers one request at a time while others wait.
Terms
- Mixture of experts (MoE)
- A model design where each layer is split into many small networks and only a few run for each token.
- Expert
- One of those small networks: a block of a layer's weights. This model has 512 per layer.
- Router
- The small part at the front of a layer that scores the experts for the current token and picks which ones run.
- Token
- The unit a language model reads and writes, roughly three quarters of a word.
- VRAM
- A graphics card's own memory. Weights have to be in it for the card to compute with them.
- Quantization
- Storing each weight in fewer bits to make a model smaller, at some cost in quality. Q2_0 is about two bits per weight.
- Cache miss
- A lookup for something the fast tier does not hold. Here, an expert that is in RAM but not on the card, so the CPU computes it.
- KV cache
- The stored attention state for the conversation so far, which grows with the context length and competes with the expert cache for graphics memory.
- MTP (multi-token prediction)
- The model's built-in draft layer, which guesses the next few tokens for the full model to check in one pass.
- Pinned memory
- RAM the operating system is told never to move to disk.
Questions
Can I run a 125-billion-parameter model on a 12 GB graphics card?
With Strata and Qwen3.8-Flash-Next, the project's answer is yes, provided the PC also has enough RAM: 48 GB or more for the full model, or 32 GB for the cut-down Coder variant. The card holds only the most-used part of the model and the rest runs from RAM on the CPU.
Is this the same as running GPT or Claude at home?
No. Those models' weights are not published, so they cannot be downloaded. This is a different model whose weights are public, run at a heavily compressed setting, for one request at a time.
Does every expert have to fit in graphics memory?
No, and that is the point of the design. Every expert has to fit in RAM. The card holds a few thousand of the 24,576, and each extra gigabyte of graphics memory holds about 700 more.
How much quality does the compression cost?
The documentation gives no measurement for the fastest setting. It says larger sizes are somewhat smarter and that the largest, IQ3_S, matches the full model on the published tests.
Does Strata work with other models?
Every download the README offers is a version of Qwen3.8-Flash-Next: the same model at different compression levels, a coding variant with half the experts removed, and community fine-tunes. The documentation read for this page does not describe running any other model.
// further reading
- Niko1221/Strata ↗: the repository, with install steps and the measured speed tables
- How does Strata work? ↗: the project's own short explanation, with its two diagrams
- Qwen3.8-Flash-Next model card ↗: the layer, expert and parameter counts used above, and the model's license
// transcript
A server-sized AI model is running on a 12 GB gaming card. Models like GPT and Claude run in their makers' data centers, and their weights have never been published, so every prompt you write has to travel to someone else's graphics cards. This model's weights are public, all 125 billion of them, and an open-source engine called Strata runs it on a home PC, writing faster than you can read. A language model writes one token at a time, a token being a word or a piece of a word, and to choose each one it runs your text through a stack of layers, where every layer is a big grid of numbers called weights. That math only runs fast when the weights sit in a graphics card's memory, and even compressed to about two bits a weight, these are three times the size of the card. In the classic design, every weight in a layer is used for every token. This model is what's called a mixture of experts, where each of its 48 layers is cut into 512 separate blocks of weights, and each block is called an expert, which only means a small network of its own. For every token, a router at the front of each layer scores the experts and runs just ten of them, so any single token touches about 2% of the experts, and across a conversation the same experts keep getting picked. Strata treats that like any hot path, by putting a cache in front of it. Every expert stays in ordinary RAM, and the few thousand that get picked most are copied onto the card. When a layer picks an expert the card doesn't hold, that's a cache miss, and the CPU computes it straight out of RAM while the card works on the ones it does have, so neither waits for the other. And as the conversation changes subject, Strata keeps swapping experts so the card holds whatever's hot right now. On an RTX 5070 with 12 GB, that reaches 94 tokens a second, though that's the most compressed setting, on a machine with 64 GB of RAM, answering one request at a time. It isn't a data center, but it shows that a model never had to fit on the card, because no single word needs more than a sliver of it.
// source
This explainer is based on Strata: run a 125-billion-parameter AI model on your own gaming PC by Strata ↗. The original reporting and technical work belong to its publisher.