Systems Researcher

Epistemic Adequacy Consumption Probe

Epistemic Adequacy Consumption Probe

Beyond structured model access alone, does epistemic metadata change how an AI consumer answers derivation-style questions over an MBSE model? A first measured result, over the public Apollo 11 SysML v2 reconstruction.

Read the report · View the artefact

§ 1 Initial findings

§ 2 What this is

Apollo 11 lifting off from Launch Complex 39A on 16 July 1969, the Saturn V first stage at full thrust as the vehicle clears the tower.
Figure 1Apollo 11 leaves Launch Complex 39A, 16 July 1969. The S-IC first stage at full thrust is the derivation chain this probe interrogates. Image: NASA (6900540).

This is the first probe of a research programme set out in a companion paper published in the NIER track of MODELS 2026. We posed fifteen derivation-style questions about the Saturn V first-stage thrust chain to three frontier LLMs over the public Airbus Apollo 11 SysML v2 reconstruction, crossing two substrate arms with two instructions, plus a fictional out-of-distribution control vehicle. The arms were the bare model and the model plus a hand-authored epistemic-metadata sidecar. One instruction permitted reporting absence; the other pressed for a decisive answer under a simulated deadline. Across 120 judged answers, three results carry the probe.

§ 3 What we measured

Pressure breaks grounding.

Under a governed instruction the bare model produced unauthorised content in about 13% of answers; under deadline-style pressure this rose to 60% (9 of 15). The strongest model failed most fluently, every unauthorised claim was historically plausible.

Metadata holds the line.

A hand-authored epistemic-metadata sidecar held governed-instruction failures to 1 in 45, roughly halved the pressure-induced failures (to 33%), and turned the requirement-satisfaction verdict into one all three models answered from the record.

A live gap, in the artefact itself.

Assembling the probe surfaced a finding in the public model: four unreconciled liftoff-thrust figures, present since its first commit, with no construct to compute, flag, or reconcile them.

§ 4 Ungrounded answers, by cell

Each bar is the share of judged answers a cell could not authorise from its substrate, pooled across the three models. Pressure is the variable that separates a substrate that can refuse on the record's behalf from an instruction that merely asks the consumer to.

Apollo, governed

Model only 13.3%
+ EA sidecar 3.3%

Apollo, pressed

Model only 60.0%
+ EA sidecar 33.3%

Caldera control, governed

Model only 13.3%
+ EA sidecar 0.0%
  • Model only
  • With EA sidecar
Figure 2Ungrounded = either ungrounded verdict category, pooled across Haiku 4.5, Sonnet 4.6, and Opus 4.8. Denominators are per cell: n = 30 for each Apollo governed cell, n = 15 for each pressed and each Caldera cell, 120 in total. The control ran under the governed instruction only, so there is no Caldera pressed result to chart. Bars are scaled to the 60% maximum. The 2.2% quoted above pools the two governed +EA cells (1 of 45), the denominator the report uses for the governed-instruction headline. Single run per cell; see the caveats in the report. †

§ 5 The strongest model fails hardest

Reading the bare model, each consumer was asked the same five questions under both instructions. When reporting absence is permitted, capability protects: the strongest model was never ungrounded. When an answer is demanded, capability inverts, and the model with the most to draw on from training is the one that draws on it.

80% ungrounded 40% Governed Pressed Haiku 30% Sonnet 10% Opus 0% Haiku 4.5 · 40% Sonnet 4.6 · 60% Opus 4.8 · 80%
Figure 3Bare model (no sidecar), Apollo substrate. Governed n = 10 questions per model; pressed n = 5, the five hardest questions, which are a subset of the governed ten. Counts: Haiku 3/10 and 2/5, Sonnet 1/10 and 3/5, Opus 0/10 and 4/5. Single run per cell. †

Opus answered all ten governed questions from the record and none of them unauthorised. Pressed on the five hardest, four of five went ungrounded, and every one of those claims was historically true: content recovered from training rather than derived from the substrate, which is precisely the failure a reviewer cannot catch by reading the answer.

Haiku, the weakest consumer, moved least. It had less to confabulate with.

§ 6 Is the scoring sound?

The reported verdicts come from a judge in the same model family as the consumers, which is a fair thing to distrust. Every answer was therefore re-scored by an independent judge from a different family, five times over. The check bounds inter-model judge bias. It does not establish that any LLM judge tracks ground truth; that is what the per-question keys are for.

Recheck judge: DeepSeek-V4 Flash (served deepseek-v4-flash), run five times over all 120 answers because the judge is not bit-deterministic at temperature 0. Mean agreement is 79.7% on the grounded-versus-not binary (range 78.3–80.8%, Cohen's κ 0.46) and 73.5% on the full six-category verdict. On the Apollo governed cells that carry the substrate-arm contrast, the two judges agree on 52/60. Full distribution in Appendix A.1 of the report; replay it with harness/recheck_judge.py.

§ 7 What the models read

Three substrates, one instrument. The control exists to separate answers recovered from training from answers that are simply invented.

Real model
A verbatim excerpt of the public Airbus Apollo 11 SysML v2 reconstruction, pinned to a specific commit.
EA sidecar
A mock of the paper's epistemic-adequacy layer: closed status vocabulary, provenance, reachable evidence anchors, a derivation node, and explicit unresolved markers.
Out-of-distribution control
A fictional “Caldera-2” vehicle that no training corpus contains, separating recovered answers from fabricated ones.

§ 8 Four figures, no chain

Independent of the LLM runs, assembling the substrate exposed an epistemic-adequacy failure in the public model, present since its initial commit. Four liftoff-thrust figures coexist with nothing to reconcile them: no construct binds actualLiftoffThrust, no calculation connects engines to stage thrust, and CLR-R055 is absent from the model's own satisfy block.

Two of the four sit below the requirement minimum and one above it. The gap is not missing data. The data is present; the authorising chain over it is not.

Source Liftoff thrust
Per-engine rated arithmetic (5 × 6770 kN) 33.85 MN
S-IC document text (~7.7 million lb) ~34.25 MN
Requirement minimum (CLR-R055) 34.5 MN
Execution-package snapshot at liftoff 35.1 MN

Verified against the pinned commit. When asked directly, the stronger models found the discrepancy from the raw numbers; the model itself never surfaces it.

§ 9 Run it yourself

The harness re-runs the probe against the Anthropic API with the same prompts, schemas, and judge rubric. Hosted models are non-deterministic and single-run per cell, so cell-level rates track the reported figures rather than match them exactly.

pip install -r requirements.txt
export ANTHROPIC_API_KEY=...
python harness/run_probe.py \
  --models claude-haiku-4-5,claude-sonnet-4-6,claude-opus-4-8 \
  --out results/replication.json

§ 10 Documents & data

Cite this work

@inproceedings{gower2026models,
  author    = {Gower, Jason D. and Henshaw, Michael J. de C. and Ji, Siyuan},
  title     = {Models as Governed Interfaces for AI-Native {MBSE}:
               Read-Side Adequacy and Write-Side Admissibility},
  booktitle = {Proceedings of the ACM/IEEE 29th International Conference on
               Model Driven Engineering Languages and Systems (MODELS 2026),
               New Ideas and Emerging Results Track},
  year      = {2026},
  month     = oct,
  publisher = {Association for Computing Machinery},
  isbn      = {979-8-4007-2809-9},
  doi       = {10.1145/3822455.3838783}
}

Generated from CITATION.cff, which is the source of truth for citation metadata in this repository and also records the artefact itself as a citable software entry with ORCIDs.