Local AI Memory: Context and Concurrency

The model file size is a starting point for capacity planning. A running local assistant also needs memory for the conversation state, runtime buffers and concurrent work. A successful short prompt on an otherwise idle machine does not establish that the same setup will serve a team.

Define the workload first: typical and maximum input length, expected output length, number of simultaneous requests and acceptable waiting time. Those assumptions matter as much as the model name.

Split the memory budget

Think in four components: model weights, attention cache or other model state, temporary runtime allocations, and the rest of the system. Keep CPU memory and GPU memory separate where the hardware has distinct pools.

Simple arithmetic gives a rough lower bound for weights. Eight billion values stored at four bits each occupy about four billion bytes before scales, metadata, unquantised components and other overhead. This is an illustrative storage calculation, not a claim that every eight-billion-parameter model fits in four gigabytes of runtime memory.

Hugging Face's bitsandbytes guide documents quantised linear layers and different handling for other modules. It also notes that CPU-offloaded weights in its 8-bit offload path are stored in float32. A quantisation label therefore cannot describe every allocation in the running process.

Use a supported model, runtime and quantisation combination. A smaller download is not useful if the intended hardware backend cannot execute it correctly.

Context consumes capacity while the job runs

For transformer models using attention, the key-value cache stores state used during generation. Hugging Face's cache strategies documentation describes dynamic, static, offloaded and quantised approaches, with different memory and latency trade-offs. Some architectures also bound parts of the cache through sliding or chunked attention.

Avoid one universal formula based only on parameter count. Architecture, precision, context length and implementation affect the result. Measure the actual model at the context sizes the application permits, including the conversation history and retrieved material added by the wrapper.

A generous advertised context window is a capability limit, not a recommendation to allocate or fill it for every request. Choose a service limit that meets the task and fits the available capacity. Reject or deliberately shorten excess input instead of silently dropping an important instruction or document section.

Cache quantisation and offloading can provide alternatives to buying more memory, but measure their effect on completed-job latency and task quality. A configuration that avoids an out-of-memory error while making every request unacceptably slow has not met the service requirement.

Concurrency changes the result

The Ollama FAQ explains that parallel requests increase context allocation and exposes controls including OLLAMA_NUM_PARALLEL and OLLAMA_MAX_QUEUE. It also provides the read-only ollama ps command to show whether a loaded model is using CPU memory, GPU memory or both.

Use those observations alongside operating-system and GPU memory measurements. Do not assume that an apparently available GPU has its full advertised capacity free for the model. The display, other applications and another loaded model may be using part of it.

More parallel requests can improve throughput until memory or compute becomes the limiting factor. Beyond that point, queueing or rejection may give a more predictable service than attempting unlimited concurrency. Set a waiting-time limit and show users when work is queued.

Keep cold starts visible. Loading a model after idle eviction is a different experience from generating with it already resident. Measure both, especially if several models compete for the same machine.

Run a capacity ladder

Use synthetic or otherwise approved test data. Begin with one short request, then increase context length while keeping concurrency fixed. Next increase concurrency at a realistic context length. Finally test the largest permitted workload and the application's overload response.

For each step, record peak CPU and GPU memory, time to first output, completed-job time, errors and output acceptance. Watch for changes in placement or offloading that explain a sudden slowdown. Allow enough runs to observe sustained behaviour rather than only the first successful response.

Change one setting at a time and retain the previous configuration. If the machine becomes unstable, reduce the allowed workload or return to the last measured setting. Do not let an automatic fallback send private data to a cloud model unless that route is explicitly part of the approved design.

The local-versus-cloud privacy note addresses where processing happens. Capacity planning answers the next question: whether that chosen location can deliver the required work reliably within its actual limits.

References

• Hugging Face, Bitsandbytes quantisation. Accessed 17 October 2026.
• Hugging Face, Cache strategies. Accessed 17 October 2026.
• Ollama, FAQ: concurrency, queueing and loaded models. Accessed 17 October 2026.