What context length includes

A model's context window is the token budget available for the conversation it processes. It includes instructions, chat history, pasted documents, tool results and generated output. Reasoning output can consume that budget too.

Tokens are pieces of encoded text, not a fixed number of words. Leave room for the reply instead of filling the entire window with input. A conversation that approaches the limit may be truncated, rejected or handled with a runtime-specific shifting mechanism.

Three limits to check

There are three separate questions:

  • What context does the model publisher document?
  • What context does the downloaded artifact and runtime support and configure?
  • What context can your hardware hold at the selected quant and workload?

For example, Qwen3 8B documents 32,768 tokens natively and an extended 131,072 with YaRN. A higher slider value does not itself implement that scaling.

gpt-oss-20b also has a published long-context ceiling, but the local memory budget and runtime configuration still need to be checked. Local LLM Finder's catalogue limit is an estimate input, not proof that a particular app implements the full model limit.

Why the same file needs more memory

During autoregressive generation, attention uses a key/value cache so the model can reuse information from earlier tokens. The Hugging Face cache documentation describes cache strategies and their memory trade-offs.

For ordinary full-attention layers, a useful first approximation is that KV-cache storage increases with the number of cached tokens, layer count, key/value heads, head size and bytes per cache value. Grouped-query attention changes the head count that matters.

Models with sliding-window attention, compressed caches or other architectures can behave differently. Batching and simultaneous conversations also change the memory budget. A simple per-token estimate is therefore guidance, not a universal allocation formula.

The weights file stays the same size when you raise context. The cache and other runtime allocations are what change.

An illustrative memory budget

Consider a hypothetical setup with 5 GB of weights, 0.5 GB of other allocations and a full-attention cache that uses 0.75 GB at 4K tokens:

  • At 4K, the example totals 6.25 GB.
  • At 16K, four times as many cached tokens would use about 3 GB of cache.
  • The example then totals 8.5 GB, while the weights remain 5 GB.

These numbers explain the relationship; they are not measurements of Qwen3, gpt-oss or an RTX 3060. Real runtimes can allocate buffers differently or reserve the configured cache up front.

Weight quantization and cache quantization are separate

Choosing Q4_K_M instead of Q5_K_M primarily changes stored weights. It does not automatically lower the KV-cache precision. llama.cpp's server documentation exposes context and key/value cache controls separately.

Cache quantization or offload can reduce device-memory pressure in supported configurations, but may affect performance or output. Verify support for your model and backend before changing those controls.

Q4 vs Q5 covers the weight trade-off. Keeping a useful context can matter more to your task than spending the last available memory on a larger quant.

Set a useful context and verify it

For an initial local test, 4K is a manageable starting point when your task fits. Increase it for longer conversations or documents after checking memory, the model limit and the runtime's configuration.

In Ollama, the context-length documentation explains configuration and checking the allocated context with ollama ps. Other runtimes have their own logs and controls. Defaults can change, so check the allocation rather than assuming it matches the publisher's maximum.

  • Choose a representative prompt and reserve reply space.
  • Record quant, context and CPU/GPU offload.
  • Test a longer prompt, not only an empty conversation.
  • Reduce context or shorten the retained history if you exceed memory.
  • Compare time to a useful answer as well as whether the model loads.

On an RTX 3060 12 GB, this check can make the difference between a practical configuration and an allocation failure. Use the Finder to compare context estimates, then confirm them in your runtime.