A 30B Model Fit on My 16 GB Card. Eight Layers Changed Its Speed
On one 16 GB GPU, moving eight layers of a 30B model off the card cut short-prompt generation from 21.7 to 11.4 tokens per second while saving GPU memory. This measures speed and memory, not answer quality.
Bart Porcincula · Published
Finding
In three short-prompt timing runs per setting on one 16 GB RTX 5060 Ti, Muse Glimmer 30B generated 21.7 tokens per second with 52 layers on the GPU and 11.4 with 44; peak GPU memory was 15,873 and 13,650 MiB respectively. A separate 100,449-token synthetic prompt ran once at 9.4 tokens per second after prefill.
Glimmer partial-offload timing on one 16 GB GPU (October 2026)
Muse Glimmer 30B Q4_K_M with llama.cpp b11753, one parallel slot and a 2,000-token synthetic prompt; three repetitions per short-prompt offload setting. Long-prompt timings were single runs.
| Measurement | Value | Note |
|---|---|---|
| 52 GPU layers: decode / peak memory | 21.7 tokens/s / 15,873 MiB | 8K context; 3 repetitions |
| 44 GPU layers: decode / peak memory | 11.4 tokens/s / 13,650 MiB | 8K context; 3 repetitions |
| 32 GPU layers: decode / peak memory | 6.7 tokens/s / 10,352 MiB | 8K context; 3 repetitions |
| 100,449-token prompt | 138 s prefill; 9.4 tokens/s decode | one synthetic prompt; 48 GPU layers, q8_0 cache, peak 15,586 MiB |
The short version
A model can fit on a graphics card and still run at very different speeds depending on how much of it stays there. I ran a compressed 30B model on one 16 GB RTX 5060 Ti. With 52 layers on the GPU it generated 21.7 tokens per second; with 44, it generated 11.4. The latter setting used about 2,223 MiB less peak GPU memory.
These are speed and memory measurements, not answer-quality results. The model, compression, server, computer memory and hardware all affect the outcome.
What I changed
I used Muse Glimmer 30B in Q4_K_M form, served by llama.cpp with one request at a time. Offloading layers puts part of the model on the GPU and leaves the rest for the computer's main memory and processor. I varied the GPU layer count with the same 2,000-token synthetic prompt and 8K context setting, with three repetitions per setting.
| Layers on GPU | Generated tokens per second | Peak GPU memory |
|---|---|---|
| 52 | 21.7 | 15,873 MiB |
| 44 | 11.4 | 13,650 MiB |
| 40 | 9.4 | 12,594 MiB |
| 32 | 6.7 | 10,352 MiB |
| 24 | 5.0 | 8,180 MiB |
The first eight-layer change caused a much steeper drop than I expected. The test did not establish its cause, so I would not attribute it to a particular hardware bottleneck without another experiment.
A longer prompt
A different configuration with 48 GPU layers and a compressed q8_0 attention cache processed a 100,449-token synthetic prompt in 138 seconds, then generated at 9.4 tokens per second, with peak GPU memory of 15,586 MiB. That was one timing run close to the card's limit. A separate 52-layer configuration timed out on a roughly 69K-token prompt; it used a different cache setting, so it is not a clean layer-only comparison.
Getting a long prompt through is not the same as using its contents correctly. The recorded long prompt used synthetic filler, and the available Stage B quality results have not yet been written up. No long-context quality claim follows from these timings.
What this means when sizing a machine
Measure both speed and peak memory at the context length and settings you actually intend to use. Leave room for the operating system and other GPU use. These measurements describe one machine, one model and one serving build; RAM and connection speeds can change the trade-off elsewhere.
Cite this research
Bart Porcincula. "A 30B Model Fit on My 16 GB Card. Eight Layers Changed Its Speed." Published 2026-11-13; updated 2026-11-13. https://porsync.com/research/30b-model-16gb-layer-offload-speed-memory
Permanent article URL