Size model memory, quantisation, and context for your hardware

Start with the job, not a parameter count

A model’s size is a poor substitute for a workload specification. Write down the input length, expected output, number of simultaneous jobs, latency target, and acceptance criteria. A short classification request and a long technical-document comparison have different memory and compute demands even when they use the same weights.

Keep three quantities distinct: model parameters, weight-file size, and observed runtime memory. The last includes allocations that are not visible in the model download.

Estimate weights as a first approximation

For a dense model, a rough arithmetic estimate is:

weight bytes ≈ parameter count × bits per parameter ÷ 8

For illustration, four billion parameters at four bits occupy about two billion bytes before scales, metadata, embeddings, and implementation overhead. This calculation is a lower-level estimate, not a guarantee that the model fits in a 2 GB memory budget.

Quantisation formats differ. Group sizes, retained higher-precision layers, kernels, and file packaging affect both capacity and speed. A four-bit model from one runtime is not automatically equivalent to a four-bit model in another.

Reserve memory for the rest of the system

A practical budget includes:

Component	What to observe
Operating system and other applications	Available memory and pressure before loading
Model weights	Actual loaded configuration
Context cache	Growth with prompt length and generation
Runtime workspace	Temporary and persistent allocations
Concurrent requests	Additional active contexts and batching

On a Mac, these compete within unified memory. On an NVIDIA PC, distinguish GPU VRAM, host RAM, and transfer between them. Offload is an operating choice with a latency cost to measure.

Understand the context trade-off

A larger context window can hold more material, but it does not ensure the model will use every detail correctly. Retrieving a few relevant sections can be more practical than passing a whole document collection. Bound output length as well: uncontrolled generation consumes time and may retain the request’s cache longer.

Test short, typical, and worst-case inputs. Record the runtime’s configured context, actual input/output tokens, and peak memory. Increase one variable at a time. If a service must support several users, test their overlapping requests rather than multiplying a single-user speed figure.

Evaluate quality after quantisation

Run the same held-out tasks across the candidate formats. Check extraction fields, grounding, refusals, and missing-value behaviour. A model that fits comfortably but fails the business acceptance test is not the economical choice.

Use a prompted smaller model as the baseline before considering a larger model or adaptation. The delivery-note lab compares a small student’s original and adapted behaviour on independent synthetic test documents.

Choose an operating envelope

Document the maximum input size, output limit, queue length, and supported concurrency. Define how the application responds when those limits are exceeded. A controlled rejection or human handoff is preferable to a service that repeatedly exhausts memory.

Include spare capacity for routine system activity and updates. Repeat acceptance tests after changing weights, context settings, drivers, or runtime versions. Keep the known-good configuration available for rollback.

References

• Ollama memory and concurrency guidance
• MLX-LM model loading and quantisation
• llama.cpp supported execution paths