Technical note2026Verification at scaleMeethealth Research
Assessment Gate certification
We ran a 45-rule deterministic contract over every record in our corpora: 9.59 million synthetic patients and 1.27 million generated plan records. Fewer than half of the plans came back with zero findings.
- 45
- deterministic rules, no LLM in the loop
- 47.9%
- of 1,012,105 Gemma 4 plans with zero findings
- 6,145
- plans verified per second on one 8-worker machine, fast enough to gate every record
Question
If every generated record had to pass a fixed, reproducible set of clinical and structural rules, how much of a large synthetic corpus would survive?
Data
All records on disk at certification time (2026-07-24): 9,589,659 synthetic patients, 1,012,105 Gemma 4 plans, 257,655 Llama 4 plan pairs, and 904 Claude Opus plans. Contract diet_plan@1.0.0, fingerprint 1a6c5cf6b9d37193.
Method
- Wrote a versioned YAML contract: 32 rules on patient-record coherence, 13 on generated plans (dose safety, contraindicated diets, language, output structure).
- Compiled the contract to a fingerprinted verifier and ran it over each corpus read-only, recording every verdict.
- Classified verdicts by severity; any critical finding blocks a record from training views.
Findings
- Gemma 4 plans: 47.9% had zero findings and 22.6% were blocked.
- Llama 4 plan pairs: 24.0% zero findings, 23.0% blocked.
- Synthetic patients: 80.4% zero findings, 12.2% blocked, mostly anthropometric incoherence such as implausible waist-to-hip ratios.
- The single most common plan finding was language: plans written in English for patients recorded in another language.
| Rule | What it checks | Share of plans |
|---|---|---|
| DIET-LNG-001 | Plan written in the patient's language | 31.7% |
| DIET-PRO-002 | Paediatric protein within safe upper limit | 10.1% |
| ANTH001 | Waist-to-hip ratio within the human range | 10.0% |
| NRG002 | Recall energy plausibly close to TDEE | 4.9% |
| DIET-CTX-001 | No ketogenic diet against a contraindication | 1.1% |
Failure examples
Patients recorded as speaking Arabic, Hindi, Japanese and a dozen other non-Latin-script languages received English plans with only their name localised. The cause was a prompt-builder fallback, not the models. It was invisible until a rule checked the script of every plan.
A plan can be flawless for the patient it was given and still be unusable if that patient is incoherent, such as a waist circumference of 124 cm on a BMI of 21. The gate checks both sides, which is why plan and patient findings are reported separately.
All patients are synthetic. Examples are drawn from the audited corpus.
Limitations
- Rules were written after seeing the corpus, so rates are in-sample.
- Rules check arithmetic, ranges, keywords, and categorical coherence. A record that passes is internally consistent, not clinically validated.
- Some rules are deliberately lenient bands; stricter clinical thresholds would flag more.
What we changed
A deterministic gate is cheap enough to run on every record and strict enough to change what ships. Every Meethealth delivery carries its contract fingerprint and per-record verdicts, so you can reproduce our numbers and set your own threshold.