Evaluating, qualifying and governing a system whose output informs consequential decisions

Why benchmark scores are not assurance, why the model cannot be certified, and what to build instead: containment, layered evaluation, calibrated abstention and an assurance case that survives challenge.

1. Executive summary

The two preceding papers in this sequence addressed construction: how a knowledge graph is built and maintained over time, and how it is bound to a document corpus so that the combined system can answer questions neither could address alone. This paper addresses the question that follows immediately and that determines whether any of it is usable: on what basis may an engineer rely on what the system produces?

The instinct is to answer with evaluation metrics. That instinct is not wrong, but it is insufficient by a wide margin. Evaluation measures whether a system performs well on a sample. Assurance establishes that a system is fit to be relied upon for a stated purpose, within stated limits, by stated people, with defined recourse when it is wrong. Engineering organisations have run assurance regimes for pressure equipment, protective systems and structural design for decades. Almost none of that discipline has been carried across to knowledge systems, and the gap is now the principal obstacle to deploying them where they would matter most.

This paper sets out four propositions. First, that the model cannot be certified and attempting to certify it is a category error — assurance must attach to the system in its context of use, and specifically to its architectural containment. Second, that evaluation must be layered, because a failure in the retrieval substrate silently invalidates every measurement taken above it. Third, that calibrated abstention is the single most important engineering property such a system can have, and that coverage should be treated as a tunable parameter set against a required accuracy rather than as a performance goal. Fourth, that the deliverable of an assurance programme is not a score but an assurance case: an explicit structure of claims, arguments and evidence that can be examined, challenged and refuted.

The paper closes with the parts of the problem that are least discussed and most likely to cause harm: automation bias in reviewers, the fluency premium that makes well-written wrong answers more persuasive than poorly-written right ones, and the surveillance regime required because these systems degrade silently and continuously.

2. Evaluation is not assurance

2.1 Two different questions

Evaluation asks: how well does this system perform on a distribution of tasks? Assurance asks: is this system fit to be relied upon for this purpose, in this setting, by these people, and how would we know if it stopped being fit? The questions are related but they are not substitutes, and the second is the one a plant manager, an inspection authority or an insurer will actually ask.

2.2 Why average performance is the wrong statistic

An accuracy figure aggregates over a distribution of questions. Engineering consequence does not distribute evenly over that same distribution. A system that is ninety-four per cent accurate overall may be ninety-nine per cent accurate on the routine lookups that constitute most of its traffic and sixty per cent accurate on the rare, high-consequence diagnostic questions that motivated its construction. The headline figure conceals exactly the behaviour that matters.

The correct treatment is to stratify by consequence before measuring anything, and to require performance evidence separately for each stratum. This is standard practice in equipment qualification and entirely absent from most AI evaluation practice.

2.3 The specific hazards of this system class

A knowledge-grounded system built on the architecture of the preceding paper introduces failure modes that generic AI evaluation does not probe:

•    Silent evidence exclusion. A chunk that was never anchored to the entity it discusses is invisible to frontier-scoped retrieval. The system produces a confident answer from the evidence it did find, with no indication that the decisive document was excluded before ranking began.

•    Temporal substitution. The system answers a question about 2019 using the configuration in force in 2026, and presents it with the same confidence and the same citation format as a correctly scoped answer.

•    Citation drift. Every claim carries a citation and every citation resolves to a real document, but the cited passage does not actually support the specific claim attached to it. This is the most dangerous single failure mode, because the presence of citations is precisely what reviewers use as a proxy for reliability.

•    Interpretation laundering. A conclusion from a prior investigation — which may have been wrong, as two of the three were in the worked case of the preceding paper — is retrieved, restated and presented as established fact rather than as a prior interpretation.

•    Confident extrapolation. The system answers a question that falls outside the domain its corpus and graph actually cover, with no signal that it has left the region where its evidence exists.

Each of these produces a fluent, well-formed, plausibly-cited answer. None of them is detected by answer relevance, helpfulness ratings or standard faithfulness scoring. Assurance for this system class is largely the discipline of building specific tests for specific hazards, rather than measuring general goodness.

3. Containment: what is actually being assured

The most consequential decision in an assurance programme for this system class is made before any measurement: deciding what the system is permitted to do, and therefore what must be assured about it.

3.1 The model cannot be certified

Functional safety practice assigns integrity levels on the basis of a quantified, demonstrable failure rate for a component whose behaviour is bounded and whose failure modes are enumerable. A large language model satisfies none of these preconditions. Its input space is unbounded, its failure modes are not enumerable, its behaviour changes with a vendor update outside the operator's control, and its failure probability on the tail of the input distribution — which is where consequential questions live — is not estimable from any feasible sample.

Attempts to work around this by fine-tuning, guardrails or ensembling do not change the situation. They may improve average behaviour; they do not produce a bounded, demonstrable failure rate. An organisation that claims otherwise has created a documentation artefact rather than an assurance argument, and the claim will not survive a competent challenge.

3.2 Assure the architecture, not the model

The productive response is to arrange the system so that the unassurable component is not load-bearing for safety or integrity. The machine produces evidence; a competent person decides; the engineering authority actuates. No machine-initiated path crosses into actuation.

This is not a hedge or a transitional arrangement pending better models. It is the correct architecture for the problem, for a reason that has nothing to do with current model capability: the questions this system exists to answer are exactly the ones where the evidence is incomplete, the mechanism is uncertain and the judgement is contested. Those are properly human decisions. The machine's contribution — assembling evidence that no human could assemble from four hundred thousand documents — is the valuable part, and it is assurable.

Assurance effort therefore concentrates on properties that are genuinely demonstrable: does the system find the material it should find, do its citations actually support its claims, is its stated confidence calibrated, does it decline when it should, and is every output traceable. These are measurable. Correctness of the ultimate engineering decision is not the system's claim to make.

3.3 Decision tiering

Not every question warrants the same rigour. Stratifying by consequence lets assurance effort be spent proportionately, and lets the system be deployed for lower tiers while higher-tier evidence is still being accumulated.

Two rules make tiering work. The tier is assigned by the use of the output, not by the difficulty of the question — an easy question whose answer feeds an integrity decision is T4. And the system must be able to recognise when a question falls above its qualified tier and route it accordingly, which is a design requirement, not a policy statement.

4. Layered evaluation

Evaluation of this system class must be organised in layers corresponding to the stages at which it can fail, because a failure at a lower layer invalidates every measurement taken above it. A system with poor anchor recall can post excellent faithfulness scores: it faithfully summarises the wrong evidence.

4.1 Layer 1 — Substrate

Can the right material be found at all? This layer is measured entirely without reference to generation, and it should be measured first and continuously. The measures were introduced in the preceding paper and are restated here as assurance obligations rather than engineering diagnostics: anchor recall and precision against a hand-labelled mention sample; frontier recall against known-relevant chunks; frontier efficiency at fixed recall; and temporal correctness — the proportion of historically-scoped questions retrieved against the state in force at the relevant time rather than the current state.

Layer 1 failures are silent by construction. Nothing in the output indicates that the decisive document was excluded. This is why the layer is measured directly rather than inferred from downstream quality, and why a labelled sample must be maintained even though maintaining it is tedious and unglamorous.

4.2 Layer 2 — Evidence

Does the cited source actually support the claim attached to it? Three measures matter.

•    Citation entailment. For each asserted claim, does the cited passage entail it, merely relate to it, or contradict it? This requires a judged sample; automated entailment checking is a useful screen but not a substitute at higher tiers. The failure this catches — real citation, real document, unsupported claim — is the one reviewers are least likely to catch unaided, because checking it requires opening every source.

•    Conjunction recall. For multi-source problems with expert-labelled evidence chains, what proportion of the required fragments does the system assemble? This is the direct measure of the capability the architecture exists to provide.

•    Trace completeness. What proportion of asserted causal links carry either a citation or a graph path? An uncited link in an otherwise well-cited answer is where unsupported inference hides.

4.3 Layer 3 — Reasoning

Is the conclusion sound, is the confidence calibrated, and are the limits honestly stated? This is the layer most published evaluation practice measures, and measuring it alone is what produces systems that score well and fail in service.

•    Falsification rate. The proportion of generated hypotheses rejected by graph adjudication. This is diagnostic rather than directional: a rate near zero almost always means the falsifier is inert rather than that generation is excellent, and it should be verified against an adversarial set of hypotheses known to be impossible.

•    Calibration. Does stated confidence track observed accuracy? Bin outputs by stated confidence and compare each bin's claimed confidence to its measured accuracy. Systematic overconfidence is the norm and must be corrected before any abstention policy can be set, because abstention thresholds are meaningless on uncalibrated confidence.

•    Specificity discipline. Does the system's explanation account for why the observed case and not the near-identical unaffected cases? This was the test the historical investigations in the preceding paper failed, and it generalises: an explanation that would apply equally to unaffected equipment is not an explanation.

4.4 Layer 4 — Decision

Did a competent person reach a better, faster or safer decision than they would have without the system? This is the only layer that measures value, and it is almost never measured.

•    Reviewer catch rate: given outputs with deliberately seeded defects, what proportion do reviewers detect? A low rate is a finding about the review process, not about the reviewers

•    Time to decision, measured against a matched baseline rather than against perception

•    Decision quality, assessed retrospectively by a panel blind to whether the system was used

•    Automation-bias probes: the same question presented with and without a system-generated answer, measuring how far the answer anchors the reviewer's conclusion

Layer 4 measurement is expensive and requires organisational cooperation. It is also the only defence against the most common deployment outcome, which is a system that scores well, is widely used, and quietly degrades the quality of decisions by anchoring reviewers to plausible wrong answers.

5. Abstention as an engineering property

A system that answers every question is not more useful than one that answers seventy per cent of them well and declines the rest. In an engineering context it is substantially less useful, because the thirty per cent it should have declined are not randomly distributed — they are concentrated among the novel, poorly-evidenced, high-consequence questions where a confident wrong answer does the most damage.

5.1 Coverage is a parameter, not a score

The right framing is selective prediction. The system is permitted to decline, and its performance is characterised by a curve relating coverage — the share of questions it chooses to answer — to selective accuracy, the accuracy on the answered subset. Higher coverage always costs accuracy, because the marginal question answered is the one the system was least sure about.

The operating point is set by the decision tier, not by a coverage ambition. A T4 question type requiring high selective accuracy will support a low coverage, and that is the correct outcome — the system answers the subset it can answer well and routes the rest to a human without an answer to anchor on. Presenting coverage as a headline performance figure inverts this and creates pressure in exactly the wrong direction.

5.2 What a good abstention looks like

An abstention that says only "I cannot answer this" wastes the work already done. A useful abstention is itself an evidence product:

•    What the system did establish, with citations, even though it is insufficient for the question asked

•    Which specific element is missing — an unanchored document class, an absent relationship in the graph, an unresolvable identity, a period with no coverage

•    What would resolve it: a document that should exist, a test that would discriminate, a person likely to know

•    The narrower question the system can answer, if there is one

This turns a refusal into a scoping output, and it has a second benefit: the accumulated record of abstentions is the highest-value input to corpus and ontology development, because it is a direct measurement of where the knowledge base is deficient relative to actual demand.

5.3 Abstention triggers

Abstention should be driven by explicit conditions rather than by a single confidence score, because different deficiencies warrant different responses:

6. The assurance case

The deliverable of an assurance programme is not a metrics dashboard. It is a written argument that the system is fit for its stated purpose, structured so that it can be examined and refuted. Safety engineering has used this form for decades; it transfers cleanly.

6.1 Structure

An assurance case decomposes a top-level claim into sub-claims, each supported by an argument and terminating in evidence. Its value lies in being explicit: a reader can see which claim rests on which evidence, and can attack either.

•    Top claim. The system is fit to produce evidence packages for T3 diagnostic questions concerning rotating equipment within the olefins unit, for review by a named competent person.

•    Sub-claim: the material can be found. Argued from anchor recall on a labelled sample of the relevant document classes, frontier recall on a known-answer set, and temporal correctness measurement.

•    Sub-claim: cited evidence supports the claims made. Argued from judged citation entailment on a sample, plus trace completeness measurement.

•    Sub-claim: impossible conclusions are rejected. Argued from falsifier performance against an adversarial set of physically and temporally impossible hypotheses.

•    Sub-claim: the system declines when it should. Argued from the coverage-accuracy characteristic, calibration measurement, and out-of-domain detection tests.

•    Sub-claim: reviewers detect residual defects. Argued from seeded-defect catch rate and automation-bias probes.

•    Sub-claim: fitness is maintained. Argued from the surveillance regime, the change-control regime and the revalidation triggers of Section 7.

Each sub-claim carries stated limitations and residual risk. A case that records no residual risk has not been written honestly, and that is itself the finding a reviewer should make.

6.2 The statement of intended use

Every assurance case rests on an explicit boundary statement, which should be short enough to be read and specific enough to be violated. It names the question types in scope, the entity classes and units in scope, the periods for which coverage exists, the decision tiers supported, the intended users and their required competence, and — importantly — the uses explicitly excluded.

The excluded list is what makes the boundary operable. Without it, the boundary is a description that everyone agrees with and nobody applies.

6.3 Independence

For higher tiers, the evidence supporting the assurance case should be produced or verified by people who did not build the system. This is uncomfortable and routinely resisted, and the argument for it is empirical: builders construct evaluation sets that reflect the cases they designed for. The gap between a builder's test set and a sceptic's test set is where the system's real limitations are found.

Independence need not mean an external body. A different team, a domain expert from operations, or an inspection function within the organisation is usually sufficient — provided the reviewer is competent in the domain rather than only in the technology, and provided they have the standing to record a negative finding.

7. Maintaining fitness

Assurance is a claim about the present. These systems change continuously, and most of the changes are outside the operator's control.

7.1 Everything is a configuration item

A change to any of the following invalidates the assurance case until revalidated: the language model version, the embedding model, the prompt or instruction set, the chunking strategy, the entity-linking model, the traversal typing rules, the reranker, the falsifier rules, the ontology, and the corpus itself. All should be under version control with a recorded configuration baseline, and the system should be able to state which baseline produced any given historical output.

This last requirement is not optional at higher tiers. When an output is challenged eighteen months later, the question will be what the system was when it produced that answer, and an organisation that cannot answer has no defence.

7.2 The regression harness

A fixed, versioned set of questions with known-good answers and labelled evidence chains runs on every configuration change. It should include the deliberately hard multi-source cases, the adversarial impossible hypotheses, the out-of-domain probes and the temporal-scoping tests. Aggregate scores are insufficient — per-case results must be compared, because a model change that improves the mean while breaking three hard cases is a regression in exactly the capability that justified the system.

7.3 Silent drift

These systems degrade without any change being made to them, and the degradation is invisible in normal use:

•    Corpus drift. New document classes arrive with different conventions and anchor poorly. Aggregate anchor quality declines while the metric, if measured on the original sample, stays constant.

•    Ontology drift. Relationship types are added or deprecated, changing traversal behaviour and therefore frontier composition.

•    Question drift. Users learn what the system does well and ask more of it, then extend to adjacent questions that were never in scope. Real usage migrates away from the qualified domain gradually and without announcement.

•    Write-back drift. The learning loop of the preceding paper changes the graph the system reasons over. This is intended, and it means the system that is running is not the system that was validated.

The response is periodic re-measurement on refreshed samples, monitoring of the live question distribution against the qualified domain, and an explicit revalidation trigger when either moves beyond a stated tolerance.

7.4 When it is wrong

A wrong output that reached a decision is an incident and should be handled as one: recorded, investigated to the layer at which the failure occurred, and closed with a corrective action. The layered structure of Section 4 is what makes this tractable — an investigation establishes whether the failure was an anchoring gap, a citation entailment failure, a falsifier gap or a review failure, and the corrective action differs completely in each case.

Two organisational practices matter more than the mechanics. Reporting a wrong output must be low-friction and free of blame, or reporting stops and the failure rate becomes unobservable. And the incident record must be reviewed in aggregate, because the pattern across incidents identifies the systematic weakness that individual investigations do not.

8. The human factors problem

The containment architecture of Section 3 places a competent human between the system and the decision. That control is only as good as the review it produces, and the evidence on human review of machine output is not encouraging.

8.1 The fluency premium

A well-written, well-structured, well-cited answer is more persuasive than a poorly-written one, independently of whether it is correct. The systems described in this sequence produce output that is uniformly well-written, well-structured and well-cited — including when it is wrong. The presentational cues reviewers have learned to use as quality signals over a career of reading human documents have been decoupled from quality.

The practical mitigations are unglamorous and effective:

•    Present uncertainty structurally rather than in prose, so it cannot be skimmed past — an explicit confidence and limitation block that is visually distinct

•    Require the reviewer to record which specific citations they verified, which converts passive reading into an active check

•    Surface the falsifier's rejected hypotheses alongside the accepted one, which restores the sense that alternatives existed

•    Present competing evidence where it exists rather than resolving it silently

8.2 Automation bias and its measurement

Reviewers converge on the machine's answer, and the effect strengthens as the system's reputation improves. This is rational individual behaviour that produces a poor collective outcome: the review control weakens precisely as reliance on it increases.

This is measurable and should be measured. Present matched reviewers with the same case, half with a system answer and half without, and compare conclusions. A large anchoring effect means the review is not functioning as an independent control and the assurance case must be adjusted to stop claiming that it does.

8.3 Competence

The assurance case names a competent person. Competence here has two components that are usually conflated. Domain competence — the ability to judge whether a proposed mechanism is physically plausible — is the one organisations think of. System competence — understanding what the system can and cannot see, what its abstentions mean, and which failure modes it is prone to — is the one they omit, and it is what allows a reviewer to challenge an output rather than merely read it.

Both should be defined, trained and recorded. A reviewer who does not know that the system cannot see documents outside its anchored corpus will not think to ask whether the decisive document was in it.

9. Building the assurance programme

Phase 1 — Boundary and tiering

Write the statement of intended use, including the exclusions, before building any evaluation. Assign decision tiers to the question types in scope. This costs a week and determines everything that follows; skipping it produces evaluation activity with no acceptance criteria.

Phase 2 — Labelled sets

Build the four sets the programme depends on: an anchor sample of several hundred labelled entity mentions across the document classes in scope; a known-answer question set spanning the tiers; an adversarial set of physically and temporally impossible hypotheses; and a conjunction set of multi-source problems with expert-labelled evidence chains. The last is the most expensive and the most important — it is the only direct measure of the capability that justifies the architecture.

Phase 3 — Layer 1 and 2 before anything else

Measure substrate and evidence quality before measuring reasoning. If anchor recall is weak, reasoning metrics are measuring a system that cannot see its own evidence, and improving them is wasted effort.

Phase 4 — Calibration and abstention

Establish the coverage-accuracy characteristic, correct systematic overconfidence, and set operating points per tier. Build the structured abstention output described in Section 5.2 at the same time; an abstention capability without a useful abstention product will be routed around by users.

Phase 5 — Layer 4 and the human control

Run seeded-defect and automation-bias studies with real reviewers before scaling deployment, not after. If the review control does not work, the containment argument does not hold, and this is better discovered with ten reviewers than with two hundred.

Phase 6 — Write the case, then defend it

Assemble the assurance case, then have it challenged by someone who did not build the system and whose job is to find the weakness. Record the residual risks. Publish the boundary to users in a form they will actually read.

Phase 7 — Surveillance

Stand up the regression harness, the drift monitoring, the incident process and the revalidation triggers. Assign the whole of it to a named owner. An assurance programme without an owner has a half-life of about two quarters. 

10. Closing observations

The three papers in this sequence describe an arc. The first concerns representing what an organisation knows, with identity and time handled honestly. The second concerns reasoning over that representation together with the far larger body of knowledge that was never structured. This third concerns the question that determines whether any of it reaches a decision: on what basis may it be relied upon?

The answer is not a benchmark score, and it is not a certification of the model, which is not achievable and should not be attempted. It is an architecture in which the unassurable component is contained and not load-bearing; an evaluation regime layered so that silent failures in the substrate cannot masquerade as good performance; a calibrated willingness to decline, treated as a first-class engineering property rather than a shortfall; and a written argument that can be attacked, with its residual risks stated rather than concealed.

There is a version of this that is much easier to build and much easier to sell. It answers every question, always confidently, always with citations, and scores extremely well on any evaluation that does not check whether the citations support the claims. It will be adopted quickly, it will anchor reviewers to plausible wrong answers, and the resulting decisions will be worse than the ones made without it. The difference between the two systems is invisible in a demonstration and obvious in an incident investigation.

The discipline that separates them is not technical sophistication. It is the willingness to measure the things that make the system look worse — anchor recall, citation entailment, coverage under an accuracy constraint, reviewer catch rate — and to publish the boundary honestly. An organisation prepared to do that can deploy these systems where they matter. An organisation that is not should keep them well away from anything consequential, and will find that it has built an expensive way of producing confident text.

About this paper

This is the third paper in a sequence with "Knowledge Graphs for the Industrial Enterprise" and "Beyond Either: Knowledge Graphs and Hybrid RAG as a Single System", and forms part of an ongoing whitepaper series on industrial digitalisation. Concepts are drawn from functional safety and equipment qualification practice by analogy only; no claim of conformance to any functional safety standard is made or implied. The paper is vendor-neutral: capabilities are described functionally and no specific platform, model or evaluation framework is recommended. Illustrative figures are schematic and do not represent measured performance of any system.