Working paper2026Clinical constraint reliabilityMeethealth Research
Clinical Credulity
Given biologically impossible patient records, such as Turner syndrome in a male or co-coded type 1 and type 2 diabetes, models wrote fluent clinical plans and flagged none of them unless explicitly told to check.
- 0 / 128
- impossible records flagged without an explicit instruction (95% CI 0–2.9%)
- 81.3%
- flagged when instructed to screen (296 / 364, Fisher p ≈ 6×10⁻¹⁶)
- 56–100%
- range of instructed detection across the four configurations
Question
Before writing a plan, a clinician checks whether the referral makes sense. Do diet-generating models do the same when a patient record is impossible?
Data
1,527,415 plans from four model configurations (Gemma 4 26B and 31B, Llama 4 Maverick and Scout) over a shared population of synthetic patients. Impossible records occurred at 0.17–0.24%.
Method
- Built a high-precision impossibility detector covering sex–diagnosis contradictions and mutually exclusive conditions, validated by manual review.
- Detected whether each plan flagged the contradiction, using a window-based flag detector on English plans.
- Compared plans generated with and without a prompt instruction to screen for data contradictions (a natural experiment across batches).
Findings
- Without the instruction, all four configurations flagged 0 of 128 impossible-patient instances.
- With the instruction, detection rose to 81.3% (296 of 364), ranging from 100% to 56% by configuration.
- Residual failures concentrated in contradictions that need clinical knowledge to see, such as Klinefelter syndrome recorded on a female patient.
| Configuration | Detected |
|---|---|
| Gemma 4 31B | 100% |
| Gemma 4 26B | ≈96% |
| Llama 4 Maverick | 81.9% (86 / 105) |
| Llama 4 Scout | 56.2% (59 / 105) |
Failure examples
Uninstructed, the plan didn't just ignore the contradiction; in a quarter to three-quarters of cases it was organised around the impossible attribute (61.3% for Maverick, 74.2% for Scout).
A 15-year-old female record carrying Klinefelter syndrome, a male-only chromosomal condition, was served a full plan with 172 g of protein a day. A record co-coding type 1 and type 2 diabetes received a complete clinical anamnesis with no comment.
All patients are synthetic. Examples are drawn from the audited corpus.
Limitations
- Not randomised: instructed and uninstructed regimes correlate with generation batches.
- Maverick versus Scout confounds model with decoding temperature.
- Flag detection is English-only; the distinct-impossible sample is modest (492 profile-level observations).
- Covers two of eight contradiction classes at scale, and omits anthropometric incoherence, the most common defect class in the corpus.
What we changed
Credulity is a safety failure distinct from sycophancy, and the fix is upstream of generation. Our Assessment Gate now screens patient records before generation, so impossible profiles are stopped before they reach a model or a training set.