Proposition Audit Anthony Searle

Proposition Audit — AI Output Trust Verification

Profile

• Jurisdiction: England and Wales (the maintained domain-routing profile; other jurisdictions illustrated under Step 2).
• Practice area: Clinical negligence and healthcare law (the maintained example domain; the methodology generalises to any field where AI-drafted factual content needs structured verification).
• Intended user: A practitioner verifying AI-generated factual or interpretive content before professional reliance — publication, court use, formal external use, or internal reliance.

Purpose

AI-generated research, statistics, citations, and factual claims require structured verification before professional use. This skill provides a systematic post-hoc audit — classifying each claim by type and salience, searching domain-appropriate sources, scoring trustworthiness transparently, and flagging what needs attention.

This complements rather than replaces rigour during generation. It is an independent verification layer applied to completed output.

The Verification Process

Step 1: Agree the Threshold

Before beginning verification, ask what minimum standard applies to this use case. Offer these defaults if the user has no preference:

• Publication or formal external use (skeleton arguments, blog posts, published advice, court documents, regulatory submissions, external correspondence on file): 90%+ (empirical), Accurate (legal), Fair (rhetorical). Remove or independently verify anything below threshold.
• Internal working documents (drafts circulated for review, internal research notes, working files): 70%+. Flag claims below threshold but they may be retained with caveats.
• Exploratory research (early-stage research to identify avenues for further investigation): 50%+. The purpose is to identify which claims merit further investigation, not to produce a final verified output.

Step 2: Extract and Classify Claims

Extract each distinct factual or interpretive claim from the output. Break composite sentences into their constituent sub-claims — each gets verified independently, because a sentence can be half-right and half-wrong. Where a single sentence contains claims of different types (a citation plus an empirical figure, for example), split it; each sub-claim is classified and scored independently.

Classify each claim by type:

• Empirical fact — a verifiable statement about the world (a statistic, a date, a named event). Verify by source corroboration.
• Citation or reference — a claimed source, case, study, or authority. Verify by locating the primary source and confirming it says what is attributed to it.
• Legal or regulatory proposition — a statement about the law, regulation, or procedural rules. Verify by locating the primary legislative or judicial source and assessing whether the proposition accurately reflects it. Source-counting alone is insufficient here because legal accuracy depends on interpretive precision, not volume of agreement.
• Interpretive or analytical claim — a conclusion drawn from evidence ("this suggests X", "the trend indicates Y"). These cannot be verified by corroboration. Instead, assess whether the underlying evidence exists, whether the reasoning is sound, whether credible counter-interpretations exist, and whether the opposing position is presented fairly (see rhetorical fairness in Step 4). Flag rather than score.

Then assign each claim a salience level:

• Load-bearing — the surrounding argument turns on this claim. The argument fails or substantially weakens if the claim is wrong.
• Supporting — strengthens the argument but is not strictly necessary to it.
• Illustrative — used by way of example, colour, or background. The argument is unaffected by removal.

Salience determines proportionate remediation in Step 5. Load-bearing claims warrant Tier-1 corroboration; illustrative claims warrant a quick pass. Apply uniform scrutiny only if every claim is genuinely load-bearing, which is rare.

Step 3: Search for Sources

Search for independent corroboration using sources appropriate to the claim type. Do not verify from training data alone — every claim needs an active web search, because the whole point of this skill is to check what the model "knows" against external reality.

Source tiers:

• Tier 1 (Definitive):
  • Clinical: Cochrane reviews, NICE guidelines, SIGN guidelines, royal college guidance (RCOG, RCS, RCP, RCPCH), peer-reviewed journals in the relevant specialty.
  • Legal: Primary legislation (legislation.gov.uk), reported judicial decisions (caselaw.nationalarchives.gov.uk, BAILII), Civil Procedure Rules and Practice Directions, the relevant procedural rules of the forum.
  • Statistical: Office for National Statistics, official government data, primary research datasets.
  • General empirical: Peer-reviewed journals, regulatory bodies' official outputs.
• Tier 2 (Authoritative): Established news organisations (original reporting, not syndicated), university research centres, recognised professional and industry bodies, Law Commission reports, Hansard, MDU/MPS guidance.
• Tier 3 (Supporting): Reputable industry reports, recognised expert commentary, specialist trade publications, established practitioner blogs.

Domain-specific search routing:

Generic web search performs poorly for technical claims — it surfaces secondary reporting rather than primary sources. Route accordingly:

• Clinical or medical claims → PubMed, Cochrane Library, NICE, SIGN, and relevant specialty guidelines before general web search.
• Legal claims (England and Wales) → legislation.gov.uk, caselaw.nationalarchives.gov.uk, BAILII, and the relevant procedural rules before general web search.
• Legal claims (other jurisdictions, illustrative) → CourtListener and Westlaw for US, EUR-Lex for EU, the relevant primary-source service for other jurisdictions. The UK routing above is the maintained profile; others are illustrative and should be confirmed against the jurisdiction's actual primary-source register.
• Statistical claims → locate the primary dataset or study, not secondary reporting of it.
• Quantum or arithmetic claims (multiplier × multiplicand, periodical-payments calculations, life-expectancy adjustments, schedule-of-loss workings) → recompute deterministically against the inputs the source provided. Do not search; arithmetic is verifiable by recomputation, and web search performs poorly on numeric reasoning.

Step 4: Score Each Claim

Score empirical facts and citations on a 0–100% scale. Weight source quality above quantity — one definitive source is worth more than several weak ones, because a Cochrane review settling a clinical question is more reliable than three news articles paraphrasing each other. Two special verdicts (Unverifiable, Interpolated) sit outside the numerical scale and take precedence in their respective failure modes.

Important distinctions:

• "Not found" is not "contradicted." If web search cannot surface adequate sources, mark the claim Unverifiable rather than scoring it low. The distinction matters because a low score implies evidence was found and was weak, while Unverifiable means the audit itself has a gap. The user can then decide whether to seek verification through other means.
• For citation claims, score whether the source exists and says what is attributed to it. Partial accuracy (correct case name, wrong paragraph reference) sits at 50–69 with the discrepancy noted; invented sub-claims attached to a real citation sit at Interpolated.
• Where a claim is close to accurate but imprecise (a rounded statistic, a slightly misstated date), note the discrepancy rather than simply passing or failing it.

Legal and regulatory propositions receive a qualitative assessment instead of a numerical score:

• Accurate — Reflects the primary source precisely.
• Broadly accurate — Reflects the substance but with minor imprecision a careful reader would notice.
• Incomplete or misleading — Omits material qualification or context that changes the proposition's effect.
• Inaccurate — Misstates the primary source.

State the primary source consulted and any interpretive nuance. Numerical scoring is inappropriate here because legal accuracy is about interpretive precision, not corroboration volume.

Interpretive or analytical claims are not scored. Flag four dimensions:

• Evidence base — does the underlying evidence the claim relies on exist?
• Logical soundness — does the inference from evidence to conclusion hold?
• Counter-interpretations — do material counter-arguments or counter-interpretations exist that the claim does not engage with?
• Rhetorical fairness — how is the opposing position presented? Use one of:
  • Fair — the opposing position is represented as its strongest version.
  • Slanted — selection bias in presentation; only weaker parts of the opposing view are surfaced.
  • Strawman — a weakened or distorted version of the opposing view is attacked.
  • Uncharitable — the opposing view is presented in a form its proponents would not recognise.

For case-comment, analytical writing, and skeleton arguments, rhetorical fairness is often the substantive failure mode — the AI may be factually accurate but argumentatively unfair to the losing side. The fairness assessment matters at least as much as the factual scoring for this category.