ResearchPractical AIProjectsOpen SourceAbout
Research/A 74B Coding Model on a 16 GB Card: It Fit With Room for a 116K-Token Prompt
Local AIMixture of ExpertsCoding ModelsLong ContextOriginal Research

A 74B Coding Model on a 16 GB Card: It Fit With Room for a 116K-Token Prompt

I ran a community build of Nemotron Bolt on a 16 GB graphics card. It used 14.8 GB of video memory with 116,100 tokens in the prompt and generated about 50 tokens a second. Its coding and tool-call scores came within 2.5 points of the builder's reported scores. On 24 everyday questions, my smaller 9B model got five more right.

Bart Porcincula · Published

Finding

Nemotron Bolt REAP50 IQ2_XXS ran on an RTX 5060 Ti 16 GB with stock llama.cpp b11753. At a 128K context setting with 116.1K tokens actually in the prompt, it used 14.8 GB of video memory and decoded at 50.3 tokens per second. It scored 86.0% on HumanEval+, 69.6% on MBPP+ and 90.2% on BFCL simple call correctness. It answered 19 of 24 non-code questions correctly versus 24 of 24 for Qwen3.5-9B.

Nemotron Bolt fit, speed and benchmark reproduction on 16 GB (Oct 2026)

Single-slot uncached prompt-length sweep, three published benchmark comparisons and small non-code and tool-loop probes. Coding and tool-call benchmarks used four slots with 32K tokens per slot, with reasoning disabled.

MeasurementValueNote
Largest prompt tested116.1K tokens128K context setting, 14.8 GB video memory
Decode at 116.1K50.3 tok/s
HumanEval+ / MBPP+86.0% / 69.6%builder reported 83.5% / 68.8%
BFCL simple call correctness90.2%builder reported 90.8%
Everyday questions19/24 vs 24/24Bolt vs Qwen3.5-9B

The short version

I wanted to know how much room a community build of Nemotron Bolt would leave on an ordinary 16 GB card. The builder shrank the original model by removing some of its experts, the parts of a model that get called on for different inputs, and storing the remaining weights at low precision. I used the REAP50 IQ2_XXS build from the community model card and ran it on stock llama.cpp with a display attached.

At a 128K context setting, a 116,100-token prompt brought total video memory use to 14.8 GB. The model generated 50.3 tokens a second there, down from 57.4 with a 17,700-token prompt. I saw no sudden slowdown from running out of video memory across the lengths I tested. The long prompt was filler, so this measures how much text the model can load and process. How well it uses that much text is a separate question.

How well did the scores travel?

TestMy 16 GB cardBuilder's DGX Spark
HumanEval+ (164 coding problems)86.0%83.5%
MBPP+ (378 coding problems)69.6%68.8%
BFCL simple (400 tool calls)90.2% correct90.8% correct

These are single runs with different software builds. I made three compatibility patches to run EvalPlus on Windows, which removed its request timeout, memory cap and per-test timeout. My BFCL checker also differed from the upstream checker by about one point in an offline check. The scores are close enough for the practical question I had, which was whether the published results would carry over to my machine.

Everyday questions were a different test

Bolt got 19 of 24 right on a small set of arithmetic, facts, translation and text transformations. Qwen3.5-9B got all 24. Bolt also answered all six supported requests in my little tool loop, but both models struggled to return valid final JSON on the harder requests under a four-call cap. The probes are too small to rank the models broadly. I kept Bolt in the OpenCode comparison to see how it handled actual edits.

Cite this research

Bart Porcincula. "A 74B Coding Model on a 16 GB Card: It Fit With Room for a 116K-Token Prompt." Published 2026-11-07; updated 2026-11-07. https://porsync.com/research/nemotron-bolt-74b-on-a-16gb-graphics-card

Permanent article URL