ResearchPractical AIProjectsOpen SourceAbout
Research/A 30B Model Fit on My 16 GB Card. Eight Layers Changed Its Speed
Local AIGPU MemoryInferenceOriginal Research

A 30B Model Fit on My 16 GB Card. Eight Layers Changed Its Speed

On one 16 GB GPU, moving eight layers of a 30B model off the card cut short-prompt generation from 21.7 to 11.4 tokens per second while saving GPU memory. This measures speed and memory, not answer quality.

Bart Porcincula · Published

Finding

In three short-prompt timing runs per setting on one 16 GB RTX 5060 Ti, Muse Glimmer 30B generated 21.7 tokens per second with 52 layers on the GPU and 11.4 with 44; peak GPU memory was 15,873 and 13,650 MiB respectively. A separate 100,449-token synthetic prompt ran once at 9.4 tokens per second after prefill.

Glimmer partial-offload timing on one 16 GB GPU (October 2026)

Muse Glimmer 30B Q4_K_M with llama.cpp b11753, one parallel slot and a 2,000-token synthetic prompt; three repetitions per short-prompt offload setting. Long-prompt timings were single runs.

MeasurementValueNote
52 GPU layers: decode / peak memory21.7 tokens/s / 15,873 MiB8K context; 3 repetitions
44 GPU layers: decode / peak memory11.4 tokens/s / 13,650 MiB8K context; 3 repetitions
32 GPU layers: decode / peak memory6.7 tokens/s / 10,352 MiB8K context; 3 repetitions
100,449-token prompt138 s prefill; 9.4 tokens/s decodeone synthetic prompt; 48 GPU layers, q8_0 cache, peak 15,586 MiB

The short version

A model can fit on a graphics card and still run at very different speeds depending on how much of it stays there. I ran a compressed 30B model on one 16 GB RTX 5060 Ti. With 52 layers on the GPU it generated 21.7 tokens per second; with 44, it generated 11.4. The latter setting used about 2,223 MiB less peak GPU memory.

These are speed and memory measurements, not answer-quality results. The model, compression, server, computer memory and hardware all affect the outcome.

What I changed

I used Muse Glimmer 30B in Q4_K_M form, served by llama.cpp with one request at a time. Offloading layers puts part of the model on the GPU and leaves the rest for the computer's main memory and processor. I varied the GPU layer count with the same 2,000-token synthetic prompt and 8K context setting, with three repetitions per setting.

Layers on GPUGenerated tokens per secondPeak GPU memory
5221.715,873 MiB
4411.413,650 MiB
409.412,594 MiB
326.710,352 MiB
245.08,180 MiB

The first eight-layer change caused a much steeper drop than I expected. The test did not establish its cause, so I would not attribute it to a particular hardware bottleneck without another experiment.

A longer prompt

A different configuration with 48 GPU layers and a compressed q8_0 attention cache processed a 100,449-token synthetic prompt in 138 seconds, then generated at 9.4 tokens per second, with peak GPU memory of 15,586 MiB. That was one timing run close to the card's limit. A separate 52-layer configuration timed out on a roughly 69K-token prompt; it used a different cache setting, so it is not a clean layer-only comparison.

Getting a long prompt through is not the same as using its contents correctly. The recorded long prompt used synthetic filler, and the available Stage B quality results have not yet been written up. No long-context quality claim follows from these timings.

What this means when sizing a machine

Measure both speed and peak memory at the context length and settings you actually intend to use. Leave room for the operating system and other GPU use. These measurements describe one machine, one model and one serving build; RAM and connection speeds can change the trade-off elsewhere.

Cite this research

Bart Porcincula. "A 30B Model Fit on My 16 GB Card. Eight Layers Changed Its Speed." Published 2026-11-13; updated 2026-11-13. https://porsync.com/research/30b-model-16gb-layer-offload-speed-memory

Permanent article URL