Delivery-Note Model Lab
Inspect a local response-distillation experiment: independent synthetic records, validated teacher labels, a small adapted student, and held-out evaluation.
A narrow model-adaptation experiment
This lab extracts supplier, order reference, delivery date, and item rows from synthetic text. It compares a four-bit DeepSeek-R1-Distill-Qwen-7B teacher with the original and adapted Qwen3-0.6B student. It does not train a foundation model or process production documents.
Keep the evidence independent
The generator creates known records, then renders disjoint template families for training, validation, and test. Teacher responses enter training only when they match those records exactly. Missing references and distracting document instructions are included. Incorrect labels remain rejected.
The test set comes from independent records, not teacher answers. The model manifest pins the downloaded snapshots and identifies their upstream models and licence references.
Reproduce the workflow
The archive includes a pinned Mac environment and a snapshot downloader. The canonical MLX path requires Apple silicon and Metal access. Allow approximately 5 GB for model snapshots plus experiment/cache space.
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -r labs/requirements-mac.lock
python -m labs.delivery_model.download_models
python -m labs.delivery_model.pipeline generate
python -m labs.delivery_model.pipeline label
python -m labs.delivery_model.pipeline train --iterations 30The README then runs independent evaluation for the teacher, original student, and adapted student. Models and adapters stay outside the source archive.
Inspect the actual experiment
experiment-report.json in the download records the measured local run, its environment, training settings, and all three evaluations. Read exact-record match, schema validity, field correctness, unsupported values, and missing-reference handling alongside latency. A faster answer that fails the task is not an improvement.
Initial hardware: Apple M2 Pro, 16 GB unified memory. Mac Studio and NVIDIA-PC examples elsewhere are reference configurations rather than measurements extrapolated from this machine. The small synthetic set does not establish production accuracy, OCR performance, or generalisation to arbitrary documents.
Serve the result locally
The README provides MLX fusion and loopback-server commands. Keep the same prompt/template and schema validator. MLX is the deployment path for this experiment; Qwen-to-Ollama GGUF conversion is not assumed.
Verification
Model-free tests verify deterministic generation, disjoint splits, rejection of incorrect labels, single application of the chat template, and evaluation accounting against independent ground truth. The separate experiment report documents actual training and inference. It is possible for the adapted student to perform worse; the report preserves that outcome.