ResearchPractical AIProjectsOpen SourceAbout
Research/One Long AI Session or a Fresh Start Each Step? What It Costs on a Single Graphics Card
CostLocal AIContext WindowPrompt CachePre-RegisteredOriginal Research

One Long AI Session or a Fresh Start Each Step? What It Costs on a Single Graphics Card

Part 1 of 5. If an AI helps with a long job, you can keep one growing conversation or start clean on every step and re-supply what matters. I timed both over 60 steps. On the 9B model, the long conversation was the cheapest, as long as the server remembers the start of the prompt. On the 27B, running at the memory edge of the card, starting fresh was cheaper.

Bart Porcincula · Published

Finding

Over 60 steps of a ~1,500-token note and ~150-token reply on one 16 GB card, Qwen3.5-9B finished in 179 s as one growing session with the prompt cache on, against 190 s starting fresh each step and 337 s starting fresh while re-reading five earlier notes; with the cache off the long session took 1,440 s (8 times slower). On Qwen3.6-27B at the memory edge the order flipped: 3,744 s fresh against 5,159 s for the long session.

Long session vs fresh start, 60-step cost curve (Oct 2026)

Cumulative wall-clock time over 60 steps (each a ~1,500-token note plus a ~150-token reply) on one RTX 5060 Ti 16 GB, llama.cpp b11408, one request at a time, context 131,072. Qwen3.5-9B Q4_K_M at two cache precisions (f16 and q8_0), 3 repetitions per arm; Qwen3.6-27B UD-IQ3_XXS at q8_0, 1 repetition. Arms: one growing session with the prompt cache on (L) or off (L-nc), a fresh start each step (F, F-nc), and a fresh start plus the last 2 or 5 notes re-read (F+r2, F+r5).

MeasurementValueNote
9B, one growing session, cache on (60 steps, f16)179 sq8_0: 208 s (stopped at step 59)
9B, one growing session, cache off1,440 s8 times slower than with the cache
9B, fresh each step190 sq8_0: 193 s
9B, fresh each step plus 2 earlier notes re-read254 s
9B, fresh each step plus 5 earlier notes re-read337 sabout 1.9 times the long session; never overtakes it
9B, per-step work at step 30 vs step 5 in the long session1.27x (f16)stays close to flat; the pre-set refutation line was 2x
27B, one growing session5,159 sper-step work at step 30 was 2.64 times step 5
27B, fresh each step3,744 scrosses below the long session at step 33; 27% cheaper at step 60
27B, fresh plus 5 earlier notes re-read10,636 s

The short version

A long job with an AI model has a design choice hiding in it. You can keep one conversation going and let it grow, or you can start clean at each step and hand the model only what it needs. The second sounds tidier and cheaper. I timed both. All five use the same setup: the open Qwen3.5-9B model compressed to 4 bits, run through llama.cpp on one 16 GB RTX 5060 Ti, at temperature 0 (the setting that makes a model as repeatable as it gets), with its optional "thinking" mode off. Each experiment's questions, thresholds and analysis were written down before it was run.

  • On the 9B, the single long session was the cheapest. 60 steps took 179 seconds, against 190 starting fresh each step and 337 starting fresh while re-reading five earlier notes.
  • That only holds if the server remembers the start of the prompt. Switch the prompt cache off and the same long session takes 1,440 seconds, eight times longer, because every step re-reads everything so far.
  • Re-reading earlier notes never paid off in time. Giving each fresh start the previous five notes cost about 1.9 times the long session at 60 steps, and never crossed below it.
  • The 27B model on the same card went the other way. It nearly filled the card's memory, ran far slower, and the long session fell behind a plain fresh start at step 33.

Takeaway: "keep it all in one conversation" is not automatically the expensive choice. It depends on whether the server reuses what it has already read, and on whether the model fits the card with room to spare. Check the cache before you design around cost.

Why a prompt cache matters

A language model has to read the whole conversation before it can write the next reply. A server can save that reading ("cache" it) and, on the next turn, only read what is new. When that works, a long conversation costs about the same per step as a short one. When it does not, each step re-reads the whole history, and the total time grows with the square of the number of steps: 60 steps cost far more than twice what 30 do. In my cache-off runs that curve fit a quadratic with R-squared 0.9998 against 0.935 for a straight line.

What I ran

ArmWhat the model sees at each step
Lone growing conversation, cache on
L-ncone growing conversation, cache off
Fa fresh start each step: just that step's note
F+r2 / F+r5a fresh start plus the previous 2 or 5 notes re-read

Each step is a note of about 1,500 tokens and a reply of about 150. I ran 60 steps with three repetitions per arm for the 9B, and one repetition for the 27B.

Results: the 9B

ArmTime to step 60 (f16 cache)q8_0 cache
Long session, cache on179 s208 s
Long session, cache off1,440 s1,456 s
Fresh each step190 s193 s
Fresh plus 2 notes254 s254 s
Fresh plus 5 notes337 s341 s

The three repetitions agreed to within about a second, so these gaps are not noise. The long session's per-step work grew only 1.27 times between step 5 and step 30.

One side effect: on the smaller cache precision (q8_0), plain "fresh each step" overtakes the long session at step 40. That is a memory-format effect on the long session, not a point about re-reading: the long session got slower at depth (208 s against 179 s) while fresh starts did not change.

Results: the 27B

ArmTime to step 60
Long session5,159 s
Fresh each step3,744 s
Fresh plus 2 notes6,389 s
Fresh plus 5 notes10,636 s

Here the long session's per-step work was 2.64 times larger at step 30 than at step 5, past the 2x line I had set for it. The card was almost full (about 15.9 of 16 GB) and prefill ran at about 56 tokens per second, which I think means the driver was spilling into shared memory, though I did not check. Treat this as "the 27B at the edge of this card", not as the 27B's best speed.

What both models share: re-reading earlier notes never won. What differs: on the 9B a warm cache keeps the long session nearly flat, and on the 27B at the memory edge it does not.

What I would not claim

  • Time only. Nothing here measures whether the answers are any good. Parts 2 to 5 take that up.
  • One server setup (one request at a time) and one build. A server handling many users at once behaves differently.
  • The 27B ran once, at one cache precision, on a card with almost no spare memory.
  • One detail of the Qwen3.5 family: its cache only reuses a prefix when the new prompt extends it. A fresh start reuses almost nothing, which is why I also ran a cache-off control.
  • A 60-step session here is about 100,000 tokens, well inside the model's 262,144 limit, so the memory limit itself was not reached.

Part 1 of 5 in the series on long sessions and memory. Next: where a small model stops using its context.

Cite this research

Bart Porcincula. "One Long AI Session or a Fresh Start Each Step? What It Costs on a Single Graphics Card." Published 2026-10-30; updated 2026-10-30. https://porsync.com/research/long-ai-session-vs-fresh-start-each-step-cost

Permanent article URL