How to Run a 300B Model on a 128GB Mini PC

640 gigabytes of FP16 weights shrink to about 100 GB on disk, 8 of 256 experts activate per token, and one shared LPDDR5X pool feeds the GPU at 256 GB/s. That is how a 320B-parameter model on a 128GB mini PC writes at usable speed instead of crawling.

How a 300B Model on a 128GB Mini PC Is Possible

A 300B-class model runs on a 128GB mini PC because three levers work together: quantization cuts the weights to roughly 2.5-2.7 bits, a sparse mixture-of-experts design keeps only 15-18 billion parameters active per token, and one shared 256 GB/s memory pool feeds both CPU and GPU. Remove any one of them and the setup breaks. The speed figures come from the developers' own tables on a single AMD Ryzen AI Max+ 395 machine with 128GB of LPDDR5X, built on the ExLlamaV3 quantization format, and they have not been independently reproduced.

The arithmetic behind the headline is straightforward. At 16-bit precision each parameter takes two bytes, so GLM-5.3-Flash at 320 billion parameters, published by Z.ai, weighs about 640 GB, and Xiaomi's MiMo-V2.6-Flash at 309 billion weighs about 618 GB. Even the official FP8 releases sit near 320 GB, still two and a half times the machine's memory. Something besides the model size has to give, and the following sections trace exactly what gives and what it costs.

Quantization: 640 GB Becomes About 100 GB

Quantization is the first lever. Every weight keeps its place in the network but is stored with fewer bits and rounded, the way a library reprints every book as a pocket edition without removing titles from the shelves. The published packs for these two models land near 100 GB each: MiMo at 105 GB and GLM at 99.7 GB. Dividing file size by parameter count gives roughly 2.7 bits per weight for MiMo and 2.5 for GLM, above the two-bit marketing number because the squeeze is uneven across layers.

The layer-mixing detail

GLM's own model card explains the mix: the pack started from a 2.05-bit build and took expert layers 29 through 44 from a 3.05-bit build. That nuance matters when comparing quality, because not all parts of the network absorb the same rounding error. The transcript also confirms the alternative: a public four-bit build of GLM-5.3-Flash lists at about 153 GB on disk, which still does not fit in 128 GB. After loading a roughly 100-105 GB pack, only 23 to 28 GB remain for the operating system and the key-value cache, so the closet gets tight fast with long conversations.

Unified Memory: One LPDDR5X Pool at 256 GB/s

Unified memory is the second piece of the fit. On an ordinary desktop, system RAM and graphics VRAM are separate worlds, and the model gets hauled across the PCIe bus in pieces. The Ryzen AI Max+ 395 instead shares a single pool: AMD's spec listings put it at up to 128GB of LPDDR5X with about 256 GB/s of bandwidth available to both the CPU and the integrated GPU. That bandwidth figure sets a hard ceiling on decoding speed later in the analysis.

Assignment still matters. AMD's FAQ says the Windows graphics memory setting can hand the GPU up to 96 GB of that pool, but the tested path here is Linux with ROCm on GFX1151 hardware, where memory assignment is not documented the same way. Treat the Windows 96 GB number as a data point, not as proof of how the Linux setup partitions memory. The engine's README asks specifically for a 128GB GFX1151 machine, and that is the only configuration anyone has measured.

Sparse Experts: Why Only 15-18B Parameters Work Per Token

Sparse mixture-of-experts is the third lever, and it answers whether every token really needs all 309 billion parameters. It does not. MiMo is a sparse mixture of experts with 256 routed experts, of which eight are picked for each token, so only about 15 billion parameters do work on any given token. GLM follows the same design with 18 billion active parameters out of 320 billion.

The common misreading is to think the model is secretly small. It is not. All 309 billion stored parameters occupy memory at all times, because the next token might summon any of the 256 experts, so the memory saving comes entirely from quantization. What sparsity saves is memory read traffic: writing one token means reading only the eight chosen experts. Do not imagine thematic shelves in the expert library, either; neither model card documents what individual experts specialize in.

Speed: Where 26-44 Tokens Per Second Comes From

The measured speeds are real but developer-reported. A simple bandwidth estimate checks out against them: 15 billion active parameters at 2 bytes each means about 30 GB read per token, which at 256 GB/s caps 16-bit decoding near 8-9 tokens per second. At 2 to 2.7 bits the same reads shrink to 4-5 GB per token, raising the rough ceiling to 50-60 tokens per second before cache and attention overhead. Both models measure under that ceiling.

MiMo's plain decode runs 28.9 tokens per second, the median of six runs of 128 tokens at temperature zero. With speculative decoding on, using a separate 702 MB four-bit drafter model, generation climbs to 32.1 for prose, 34.8 for chat, and 44.3 tokens per second for code, gains of 1.11x, 1.21x and 1.53x. GLM's 99.7 GB pack writes 26-30 tokens per second with its multi-token prediction (MTP) speculation enabled, and a lighter 82 GB pack reports 32-39. Prompt reading, called prefill, runs at roughly 550-610 tokens per second, dropping to 200-240 on early requests while the engine tunes its GPU code.

The greedy decoding caveat

The MiMo speed table was measured at temperature zero, which is greedy decoding: always take the single most likely token. MiMo's own model card says greedy decoding is not recommended, and the README notes that greedy output under speculation is not token-identical to plain decode. So the 32-44 tokens-per-second range is a greedy-setting number, and the temperature-one GLM rows are the closer match to how you would actually run the model. For context, a llama.cpp row on a smaller 1.56-bit quant wrote 16.74 tokens per second, falling to 7.19 at 64,000 context, though the different format makes that not like-for-like.

Quality: 89-92% Agreement and No Task Scores

The quality evidence is the weak half of the story. Both model cards measure top-one agreement with the official FP8 weights: how often the quantized model's most likely next token matches the full model's. MiMo reports 91.99 percent overall, and 87.37 percent on French text. GLM reports 89.31 percent on a 30-row sample, 90.11 percent on the 126 longest of 129 rows, and 88.74 percent on four long rows. Subtract from 100 and the squeezed models pick a different top token about 8 to 11 percent of the time.

That number cannot tell you how many answers are wrong. A different top token might be a synonym, or it might be a factual error, and top-one agreement cannot distinguish the two. GLM's card says it plainly: task scores for this exact pack were not run, reasoning mode was not tested, and no same-size GGUF quality comparison was run. The repository does carry task numbers, but they belong to an earlier, smaller Nemo pack on different hardware. There is also a small unexplained gap between the MiMo card's 125 held-out rows and the README's 129. These agreement figures are, as of October 2026, the only quality numbers for these exact packs.

Hardware Economics: Strix Halo Mini PC vs DGX Spark

The price comparison decides whether the box is worth buying for this workload, and the arithmetic is unforgiving on the NVIDIA side. As of October 2026, one price aggregator lists 128GB Ryzen AI Max+ 395 mini PCs from $3,324 to $4,349, with prices moving daily. NVIDIA's DGX Spark sells at $4,999 for 64GB of unified memory, and a 100 GB model pack physically cannot fit in 64 GB. A 128GB Spark variant appears on secondary pages around $6,950, but neither pack has been shown running on a Spark; the model cards state that only AMD GFX1151 with ROCm is tested.

Option	Memory	Fit for a ~100 GB pack	Approx. price	Status of evidence
Ryzen AI Max+ 395 mini PC	128GB LPDDR5X	Fits, 23-28 GB left over	$3,324-$4,349	Developer-measured, one machine
DGX Spark (64GB)	64GB	Does not fit	$4,999	Arithmetic, untested
DGX Spark (128GB)	128GB	Possibly fits	~$6,950 (secondary pages)	Untested for these packs

The openness question deserves its own line: the engine and serving scripts are public, and so are the weights, but the quantization pipeline that produced these specific packs is not published. The repository was created October 2, 2026 and last pushed October 3, 2026 under the MIT license, so it has no track record, and whether the upstream ExLlamaV3 convert script reproduces these packs is not established. The sensible reading: if you already own a 128GB Strix Halo machine, try it with your own prompts and non-greedy settings; if you would buy hardware for this, wait for task scores on the exact packs.

FAQ

• Can a 128GB mini PC run a 300B model? Yes, at 26 to 44 tokens per second according to developer-published measurements on an AMD Ryzen AI Max+ 395 with 128GB of unified memory. The fit requires roughly 2.5-2.7-bit quantization, and only the Linux ROCm path on GFX1151 hardware has been tested.

• Why does a 320B model need only ~100 GB of RAM here? Quantization stores each weight at about 2.5-2.7 bits instead of 16, shrinking 640 GB of FP16 weights to roughly 100 GB. Sparse mixture-of-experts then keeps memory read traffic low, but all parameters still occupy RAM.

• What is the quality cost of 2-bit quantization on these models? The model cards report 89 to 92 percent top-one token agreement with the FP8 reference weights, meaning a different top token about 8 to 11 percent of the time. No task benchmarks have been published for these exact packs, so the practical impact is unmeasured.

• Is the 26-44 tokens per second number independently verified? No. The figures come from the developers' own README and model cards on one machine, measured as medians of six runs of 128 tokens at temperature zero. They are consistent with a bandwidth-based estimate, but no third party has reproduced them.

• Does this work on NVIDIA's DGX Spark? Not as demonstrated. The packs list only AMD GFX1151 with ROCm as tested, and the 64GB DGX Spark cannot hold a 100 GB model pack at any speed.

From Watched Video to Written Reference

This article exists because a 14-minute video held a full argument: the bandwidth math, the greedy-decoding caveat, the missing task scores. That kind of layered reasoning is hard to cite from a video timeline and easy to cite from a page. Projects like Crazystack typescript and independent builders such as Dev doido publish written technical references for exactly this reason: text is where claims get checked, linked and reused. If you keep insights like these inside your own recordings, Skala Blog turns a YouTube URL into a transcription and then into a structured written article, so the three levers and the caveats survive as searchable prose rather than a playhead position.

Source video