Benchmark a local AI workstation using real business tasks

Define the acceptance test first

A fast answer is useful only if it passes the business task. Choose representative work: extracting fields, finding a cited instruction, classifying an enquiry, or drafting an acknowledgement without inventing commitments. Write the acceptance rule before looking at model results.

Use independent ground truth where possible. For extraction, keep known records separate from the model’s predictions. For a drafted response, use a review rubric covering factual support, policy boundaries, and necessary handoff. Preserve failure cases rather than removing inconvenient examples.

Record a complete configuration

Every result should identify the exact hardware, operating system, runtime version, model revision, quantisation, prompt, context limit, output limit, and concurrency. Record whether the model was already loaded.

Measurement	Why it matters
Cold-start latency	First use after a restart or unload
Warm task latency	Normal repeated operation
Schema and task accuracy	Whether output can be accepted
Peak memory and pressure	Whether the configuration is stable
Rejection and retry rate	Hidden work and resource consumption
Review time	Human effort remaining in the process

A model weight size is not a runtime-memory measurement. An allocator peak is not the same as total system resident memory. State the instrument used for each figure.

Keep the comparison controlled

Run the same held-out inputs through each candidate. Use deterministic sampling for an initial comparison, while recognising that hardware/runtime changes can still affect exact outputs. Keep request order and model-loading behaviour documented.

Measure several runs if the conclusion depends on latency. Separate cold from warm results and report distributions when there are enough observations. A small demonstration can show the measurement method without establishing a production service-level guarantee.

Use the delivery-note experiment carefully

The model lab compares a teacher, the original student, and an adapted student on synthetic text. Training and test templates are disjoint. Teacher labels are checked against independent records before entering the training data.

Its report includes schema validity, exact-record match, top-level field accuracy, unsupported fields, missing-reference handling, elapsed time, and MLX peak allocation. Those results apply to the recorded M2 Pro environment and sample set. They do not establish Mac Studio or NVIDIA-PC performance.

Calculate accepted-task cost

An illustrative accounting model is:

monthly operating cost = hardware allocation + power + administration
                       + provider charges + review + failure handling
cost per accepted task = monthly operating cost / accepted tasks

Use explicit assumptions for utilisation, useful hardware life, staff time, and energy tariffs. Keep purchase cost distinct from already-owned hardware. Compare low-volume and sustained-volume scenarios. A local service can have a low marginal inference cost while retaining a meaningful operating burden.

Make the result actionable

Choose the least costly configuration that meets the agreed quality and latency requirements. If none passes, improve the data, workflow boundary, or validation before buying a larger machine. Record what remains untested and what observation would justify changing the decision.

Keep benchmark inputs and reports versioned. Re-run the acceptance set after changes to the model, prompt, context, runtime, or retrieval pipeline. The goal is a repeatable operating decision, not a one-off leaderboard.

References

• MLX-LM generation and fine-tuning tools
• Ollama runtime guidance
• Delivery-Note Model Lab