Open Source Special Topic · Running It on Your Own Machine

How Large a Model Can Your Computer Run

Everything so far has been concepts; this lesson is hands-on. Pick your GPU or your Mac model and read the answer straight off. By the end you will see that the bar for running models locally is lower than most people assume.

Run the Numbers First
Platform
Device
Precision
Every candidate model is Apache 2.0 weights you can actually download. The conversion produces estimates, meant to help you judge feasibility rather than replace real measurement.

The last row, Qwen3.8-Max, stays red no matter what you pick. That is not a bug in the list. It is the model from the third lesson of this chapter whose weights have been announced as coming but have not been released yet: run 2.4 trillion parameters through the formula above and you need 1560 GB, more than even the most expensive machine on the list can offer. It stays in the table because "the weights are open" and "you can run it" are two different things, and seeing that number lands better than reading the words "very large."

Where the Formula Comes From

The result above is not looked up in a table. It is one multiplication:

VRAM needed (GB) ≈ parameter count (B) × precision factor
Example: an 8B model at INT4, 8 × 0.65 ≈ 5.2 GB

The factor is the part that needs explaining. To load the weights alone, INT4 takes 0.5 bytes per parameter, so an 8B model needs only 4 GB. But a running model also needs extra room to hold the intermediate state that piles up during a conversation, the KV Cache covered in Part 2. That overhead is already baked into the factor, so do not multiply in another safety margin on top of it, or you will end up concluding that no machine can run anything.

Four Precision Levels

Quantization means storing each parameter with fewer bits. Fewer bits means a smaller footprint, and the price is lost precision.

FP32
× 4.0
Full precision. Used essentially only during training; nobody runs local inference this way.
FP16
× 2.6
Half precision. The original format models ship in, and the baseline for quality.
INT8
× 1.3
Half the size, with quality loss you usually cannot notice. A safe pick if you have the VRAM.
INT4
× 0.65
A quarter of the original size. On most everyday tasks the loss is noticeable but acceptable.
Most common locally

Switch the precision from FP16 to INT4 in the calculator above and watch which models become runnable. Quantization is the single most effective way to lower the bar for local deployment: an 8 GB card cannot handle an 8B model at FP16, but at INT4 it has room to spare.

Where the Lost Accuracy Actually Goes

Everyone can recite the line "fewer bits means some loss," but at which step does the loss actually happen? Quantization does exactly one thing: it rounds weights that used to be continuous onto a finite set of levels. The bit width decides how many levels there are to work with — 4 bits gives you 16, 8 bits gives you 256. The fewer the levels, the further each weight has to move to reach one.

Switch precision and watch where the same batch of weights ends up.
Precision

Two things are worth pausing on at INT4. First, the weights with the smallest magnitudes round straight onto 0, so those parameters stop doing anything at all in the quantized file; a model holds an enormous number of such small weights, and while none of them matters on its own, together they carry a lot of the fine detail. Second, there are fewer dots on the ruler — no dot went missing, several weights that used to be different got squeezed onto the same level. Sixteen levels cannot hold that many distinct values, so those weights have to share one number, and whatever set them apart is gone.

How a 6.7% Error Becomes a Visible Drop in Quality

Why should nudging the weights a little affect the quality of an answer? Because every word the model writes is the highest-scoring option out of a whole pool of candidates. Most of the time the front-runner is far enough ahead that a nudge changes nothing, but every so often the top two are almost tied, and then the smallest perturbation is enough to swap them.

Precision
Reasoning steps
Half of what the two labs above show is exact calculation and half is illustration, so here is which is which. The number of levels, the step size, the rounding error, and the value each weight lands on all follow directly from the definition of quantization, and you can check the arithmetic yourself. The candidate probabilities, the size of the perturbation, and the reasoning chain at the end are constructed to demonstrate the mechanism; they are not a measured quality drop for any real model. How the error actually travels from the weights to the output depends on the model architecture and on the specific quantization implementation (group size, whether sensitive layers are kept at higher precision, and so on), and there is no general formula for it. The chain figure also assumes the steps are independent, whereas in real inference every step depends on the ones before it, so use that number to understand why more steps mean more risk, never as an expected accuracy.
This is why quantization loss is not spread evenly. On everyday questions, summarizing, and rewriting, INT4 is generally good enough: those tasks finish in a step or two, and even if one word comes out as a near-synonym you still understand the answer. Tasks that need long reasoning chains or exact calculation are a different story, because one wrong step early on drags everything after it along. For demanding work, take a model one size down at INT8 rather than one size up at INT4.
Two Kinds of Hardware, Two Sets of Rules

NVIDIA GPU

  • VRAM is dedicated; you can use just about all of the rated capacity
  • High bandwidth and fast generation; at the same model size it feels noticeably smoother
  • Capacity is a hard ceiling — consumer cards currently top out around 32 GB
  • The most mature software ecosystem; almost any problem you hit already has a searchable fix

Apple Silicon

  • CPU and GPU share unified memory, and the system will not let you hand all of it to the GPU
  • The calculator assumes roughly 75% is assignable, which is a conservative estimate
  • A big capacity advantage; high-end configurations fit sizes consumer GPUs never reach
  • Bandwidth usually trails a similarly priced discrete GPU, so large models generate more slowly

Put simply: NVIDIA competes on speed, Apple on capacity. If you want to run models above 30B, a Mac with plenty of memory is often more realistic than a consumer GPU; if you are after response speed, a discrete GPU suits you better.

The share of unified memory available to the GPU can be adjusted on macOS through iogpu.wired_limit_max; 75% is a conservative estimate for the default configuration, not a hard ceiling.
What Makes MoE Models Different

Entries in the results list with an A in the name are MoE models, such as Qwen3-30B-A3B, meaning 30B parameters in total with 3B actually activated each pass. These come with a trap that is easy to fall into:

VRAM scales with total parameters; speed scales with activated parameters. For a 30B MoE model you need enough VRAM to hold 30B, but it runs at close to the speed of a 3B. Big footprint, fast execution.

That makes MoE a good fit when VRAM is plentiful but you want quick responses — a high-memory Mac, for example. The reverse also holds: if VRAM is tight, a smaller dense model gives you more for the same footprint.

A Few Caveats

Every number above is an estimate. Actual usage is also shaped by these factors:

So the calculator gives you a feasibility judgment, not a precise budget. When the verdict is "tight," plan as if it will not run.

Next Step

Now that you know what you can run, the next lesson gets it installed. Two tools — Ollama on the command line and LM Studio with a GUI — and ten minutes to a working setup.