local-model-workshop
A browser workshop that connects small models running on your own machine and compares their answers with fixed expected labels through the app’s own engine.
decision
A builder chooses a local extraction model after comparing expected fields and repeat runs. The current choice remains provisional until a quiet-machine timing run meets the wait budget.
baseline
The echo mock scored zero complete cases. It checks the harness, but is not a competitive extraction baseline. A rule parser was not compared, so these runs do not establish that a model beats rules.
review
I reviewed the schema and disputed labels as a pharmacist, split schedule from timing and as-needed use, then repeated the runs. A builder must rerun timing before depending on the model choice. None of the extracted output is approved for patient care.
data
The measured path sends synthetic instruction lines through the app engine and local Ollama adapter. The portfolio replays recorded results and does not start that service. Production input access, logging, retention and deletion were not evaluated.
platform
The reusable pieces are FlowRunner, its model adapter and a schema-constrained extraction step. The workshop consumes them in one-step flows so evaluation exercises application code. Schema v2 and exact field matching expose adapter and label errors. The 10-second p95 gate remains untested under fair load.
limits
One task, 25 synthetic lines, two runs per model and one shared machine. The echo mock cannot show better extraction than a useful rule parser. Duration and daily maximum are not scored.
Evaluation
Schema v2, two runs on 2026-10-30. Extract six fields from one synthetic medication instruction line: drug name, dose per administration (value and unit), frequency (the schedule), timing (time of day or meal anchor), whether the dose is as needed, and the as-needed reason if one is given. Each case runs as a one-step flow through the app's FlowRunner and Ollama adapter. Same prompt for every model, temperature 0, output constrained by a JSON Schema with closed vocabularies. One untimed warm-up per model, so load time is excluded. Each field is scored on its own, and a case counts as all-fields-correct only if all six match. Apple M1 Max (10 CPU cores), 64 GB unified memory, Ollama 0.40.6, Node 24.14.1. The machine was shared with other workloads. The 1-minute load average stayed between 121 and 361 on 10 cores while waiting to start, and ran from 138 to 468 during the model runs. Latencies are inflated and noisy, and the models did not run under equal load.
All fields correct and latency, n = 25 per run: Mock (echo baseline) | none | 0/25, 0/25 | 0/25 | n/a | n/a | 144 / 361
All fields correct and latency, n = 25 per run: gpt-oss:20b (20.9B, MXFP4, think=low) | 13.8 GB | 24/25, 24/25 | 25/25 | 6.30 s / 6.17 s | 11.65 s / 11.46 s | 138-217 / 320-352
All fields correct and latency, n = 25 per run: gemma4:31b (31.7B, NVFP4, think=off) | 19.4 GB | 25/25, 25/25 | 25/25 | 7.78 s / 14.14 s | 11.07 s / 20.64 s | 198-201 / 358-468
All fields correct and latency, n = 25 per run: qwen3.8:27b (27.8B, NVFP4, think=off) | 18.2 GB | 25/25, 25/25 | 25/25 | 6.59 s / 9.84 s | 12.59 s / 16.33 s | 206-247 / 340-363
Per-field accuracy (identical in both runs): Mock (echo baseline) | 0/25 | 0/25 | 0/25 | 0/25 | 0/25 | 0/25
Per-field accuracy (identical in both runs): gpt-oss:20b | 25/25 | 25/25 | 24/25 | 25/25 | 25/25 | 25/25
Per-field accuracy (identical in both runs): gemma4:31b | 25/25 | 25/25 | 25/25 | 25/25 | 25/25 | 25/25
Per-field accuracy (identical in both runs): qwen3.8:27b | 25/25 | 25/25 | 25/25 | 25/25 | 25/25 | 25/25
Tests: npm test | 23 files, 312 tests passed (Vitest)
Tests: npm run test:e2e | 67 tests passed (Playwright, Chromium)
Tests: npm run build | Typecheck and production build succeed
- One task, 25 synthetic lines, one prompt and two runs per model at temperature 0. With n = 25, one miss moves accuracy by 4 points, and p95 is effectively the second-slowest case. Read it as a model-selection screen, not an error-rate estimate.
- Latency was measured on one shared machine under heavy, uneven load. It is not a usable latency result, and the latency gate was not tested fairly.
- Schema and labels were reviewed by me as a pharmacist. That covers the test labels only. It is not a clinical validation of any model, and nothing here should be used for patient care.
- Three labels were judgment calls, and I confirmed each one as the pharmacist reviewer on 2026-10-30. The pick depends on one of them, and another reviewer could score them differently. Some information has no field and is not scored, such as treatment duration and daily maximums.
- All inputs are synthetic. Nothing here has been tested on real patient text.