Research
We audit our own generated corpora at scale, publish what breaks, and turn each finding into a rule that runs on every record we ship.
Reports
Working paper2026Numerical reasoning in nutrition
An audit of prescribed protein in 1.38 million AI-generated diet plans. Models lower protein for kidney disease, as they should, but not nearly enough: 81.4% of plans for stage 3–5 non-dialysis CKD exceed a lenient 0.8 g/kg ceiling.
Working paper2026Clinical constraint reliability
Given biologically impossible patient records, such as Turner syndrome in a male or co-coded type 1 and type 2 diabetes, models wrote fluent clinical plans and flagged none of them unless explicitly told to check.
Working paper2026Clinical constraint reliability
Food allergy is the one nutrition endpoint where correctness is binary. In 5.26% of plans for patients with a documented allergy, the model served the allergen, usually after listing it under “Avoid” in the same plan.
Technical note2026Verification at scale
We ran a 45-rule deterministic contract over every record in our corpora: 9.59 million synthetic patients and 1.27 million generated plan records. Fewer than half of the plans came back with zero findings.
Working summaryPreference and comparison data
Side-by-side comparison of candidate plans for the same synthetic patient, used to derive rubric-based preference labels and to study systematic differences between models.
All reports audit synthetic patients and model-generated plans. None has been peer reviewed yet, and each states its own limitations. Full manuscripts and analysis code are available to research partners on request.
Research areas
How do language models handle several interacting medical and dietary requirements?
When two or three constraints apply at once, models often satisfy the most salient one and quietly violate the others. We measure where that happens and represent constraints so violations are detectable.
Do models give directionally right but quantitatively wrong recommendations?
A recommendation can point the right way and still be wrong by a clinically meaningful margin. We look at weight-indexed doses, range adherence, and the gap between plausible and correct numbers.
What can deterministic rules catch that model judges miss, and what can't they?
Volume is cheap. We build versioned rule contracts that run on every record in microseconds, publish what they flag, and are explicit about what passing does not mean.
How can candidate comparisons expose model behaviour and give post-training signal?
Comparing candidates on explicit criteria reveals systematic tendencies that single-response scoring hides, and produces data post-training workflows can use.
How do language and local dietary practice change the usefulness of a plan?
Advice that ignores food availability, naming, or cultural practice is not useful advice. We treat locale as a task parameter, and we measure when models silently fall back to English.
We work with research groups, expert reviewers, and applied teams who care about measuring what their models actually do.
