What HMRC’s AI Fraud-Detection Pilot Tells You About Your Own AI Roadmap

Catching AI-generated forgeries is not the hard part. Explaining, months later, why you rejected someone’s claim is

A paper document and a screen side by side on a desk

In late September, PublicTechnology reported that HMRC is probing new technology to help its staff spot AI-generated forgery and fraud in tax compliance work. That is about all I know; I have no inside knowledge of what HMRC is building or how far along it is.

But the interesting question isn’t about one pilot. A news desk can tell you that a department is trialling AI to catch AI. It can’t tell you what that class of system needs architecturally, and detection turns out to be a much stranger engineering problem than generation.

Generation is graded by people. Detection is graded by tribunals

If a generative system writes you a poor first draft, you notice, you fix it, and nothing much happens. The person judging the output is the person using it, in the moment.

A detection system is different. Its output is an accusation, or at least a suspicion, and the person on the receiving end is not the user. They are a taxpayer with a claim that has been held up, and they have the right to ask why. The judgement of the system arrives much later, in an appeal or a tribunal, in front of someone who has no interest in your model at all.

That single difference drives almost everything else.

A confidence score is not a reason

Imagine the conversation. An officer has to tell someone their invoice has been flagged. What can they actually say?

“Our system rated this 0.91 on a scale where anything above 0.85 is likely to be synthetic” is not a reason. It’s a restatement of the decision with a decimal point attached. There is nothing in it for the other person to engage with, and nothing for your side to defend on appeal.

What survives scrutiny looks more like: this document’s metadata says it was created three days after the date printed on its face; the supplier’s registered address does not match the header; the typography is inconsistent between the line items and the total. Those are findings. A human being can look at them, agree, disagree, or produce an innocent explanation.

So the architectural requirement isn’t “explainable AI” as a box to tick. It is that the system must emit evidence a caseworker can read, repeat and defend — and that the officer, not the model, is the one making the decision.

The target keeps moving, so the detector decays

The second awkward property of detection is that someone is actively working against you. Forgery generation improves, detection improves in response, and then generation improves again. Nobody gets to stop.

The practical consequence is that a detection model’s accuracy decays by default. Not because anything broke, but because the world it was trained on has moved. That means three things have to be funded from day one, and not as a project:

Drift monitoring. You need to notice accuracy falling before an appeals backlog tells you. That usually means tracking the rate of flags, the rate of overturns at human review, and the outcome of appeals — a rising overturn rate is your early warning.
A retraining pipeline as a permanent running cost. Retraining has to be routine, cheap and safe to do. If it requires a procurement exercise each time, you will not do it.
A refresh cycle for the evaluation set. This is the one people miss. Your held-out test set ages too. A model that scores 98% against forgeries collected eighteen months ago tells you nothing reassuring about the ones being produced this week.

What good looks like

For this class of system, the architecture that matters is mostly the plumbing around the model rather than the model itself.

A document flowing through a versioned detection model, a human review gate and an audit log, with a feedback loop to retraining

Audit logs that record the whole picture. Not just the score. The input as received, the model version that saw it, the evidence produced, the human who reviewed it and what they decided.
Human review gates with real authority. A reviewer who cannot, in practice, overturn the model is a rubber stamp, and everyone will eventually work that out. Measure how often they do overturn it; if the answer is never, the gate isn’t a gate.
Model versioning and rollback. A new version will one day be worse in some way you didn’t test for, and going back should be routine rather than an incident.
Reconstruction. Six months later, someone should be able to take a case reference and rebuild exactly why that decision was made — same inputs, same model version, same evidence. If you can’t do that, you can’t defend it.

The architect and the developer

This is where I’d normally draw a line between architecture and development, and I want to be careful about it, because the line is fuzzy and often overdrawn. Plenty of mid-level developers make genuinely architectural decisions every week, and some of the best design work I’ve seen came from people whose job title said nothing about it.

The difference is more about span than seniority. A developer is usually asked whether the model performs well; an architect has to ask what happens in eighteen months when the forgeries have changed, the supplier has moved on, the evaluation set is stale and somebody wants to know why a decision was made in March. Both are technical questions. The second is just answered on a longer timescale, and is much harder to retrofit.

Pilots are cheap

None of this is an argument against trying. Departments should be testing this, and the people doing it are working on a real problem with real money behind it.

But a pilot proves that a model can spot something. It doesn’t prove the decision it supports can be explained to a taxpayer, defended at appeal, reproduced next year, or kept accurate against an opponent who is also improving. That second set of problems is the expensive one, and it never shows up in a proof of concept. It shows up about a year after go-live, which is exactly when it is most awkward to fix.

Source: PublicTechnology, “HMRC probes new tech to help beat AI-generated forgery and fraud”, 25 September 2026, https://www.publictechnology.net/2026/09/25/economics-and-finance/hmrc-probes-new-tech-to-help-beat-ai-generated-forgery-and-fraud/ (accessed 29 September 2026).