Facts & Figures
AI-generated document detection: facts and figures
How well frontier LLMs, open AI-image detectors and Holograin catch payslips and bank statements redrawn by an AI image generator, at 1% false positives, by channel and on unseen generators. CSV included.
By Holofin Research
Published Updated - AI-edited pages caught by Holograin at 1% false positives
- 89%
- Gemini 4 Argon at the same false-positive rate
- 69%
- genuine pages Gemini 4 Argon flagged as AI-edited
- 45 / 100
- caught on GPT-5.4 image, a generator kept out of training
- 95%
On this page
Summary
On the same 200 pages, 100 edited by AI image generators and 100 genuine, all through Holofin's upload pipeline, Holograin catches 89% of AI-edited pages while flagging 1 genuine page in 100. At the same false-positive rate, Gemini 4 Argon catches 69%, GPT-6.1 Sol Pro 40% and an off-the-shelf open AI-image detector 22%.
The two LLMs fail in opposite directions: Gemini 4 Argon caught 90 of the 100 fakes but flagged 45 of the 100 genuine pages; GPT-6.1 Sol Pro flagged 2 genuine pages and caught 40 fakes. By channel, Holograin catches 96% on the original file, 97% after a phone screenshot and 90% after upload. On GPT-5.4 image, kept out of training, it catches 95%.
Data
Four detectors on the same 200 uploaded pages (100 AI-edited, 100 genuine)
| Detector | AI-edited pages caught | Genuine pages flagged | Caught at 1% false positives |
|---|---|---|---|
| Gemini 4 Argon | 90 / 100 | 45 / 100 | 69% |
| GPT-6.1 Sol Pro | 40 / 100 | 2 / 100 | 40% |
| Open AI-image detector, off the shelf | 8 to 21 / 100 | 0 to 2 / 100 | 22% |
| Holograin | 84 / 100 | 1 / 100 | 89% |
The first two columns are each detector's own verdicts. The last column fixes the threshold so that 1 genuine page in 100 is flagged and reports the share of AI-edited pages still caught. The 100 AI-edited pages come from Nano Banana 2.1, Gemini 2.5 and 3.1 image, GPT-5 image mini and GPT-5.4 image; none of the 200 pages was used in training.
Download CSVAI-edited pages caught at 1% false positives, by how the document arrived
| Channel | Detector | AI-edited pages caught |
|---|---|---|
| Original file | Holograin | 96% |
| Phone screenshot | Holograin | 97% |
| Uploaded to Holofin | Holograin | 90% |
| Uploaded to Holofin | Open AI-image detector, tile by tile at full resolution | 20% |
Test documents never seen in training. Holograin has one threshold per channel, set on genuine documents outside the training set, so 1 genuine page in 100 is flagged whichever way a document arrives. Measured on a larger test set than the four-detector table, about 825 genuine pages per channel.
Download CSVImage generators never seen in training, caught after upload to Holofin
| Image model | AI-edited pages caught |
|---|---|
| GPT-5.4 image | 95% |
| Nano Banana Pro | 84% |
| Gemini 3.1 Flash-Lite Image | 82% |
| GPT-5 Image | Refused to edit financial documents |
One whole-page edit per test document, production threshold. GPT-5.4 image was kept out of training entirely. Measured coverage is Google's and OpenAI's image models; other families have not been measured.
Download CSVMethodology
Test set: 200 pages of financial documents, 100 edited by five AI image generators and 100 genuine, all converted by Holofin's upload pipeline. None was used in training.
Each LLM received the page with a fraud analyst's prompt and returned a verdict with a probability. Open AI-image detectors were run off the shelf, then tile by tile at full resolution. Holograin is the production model.
Every catch rate is reported at one operating point: the threshold at which 1 genuine page in 100 is flagged. A catch rate without its false-positive rate does not describe a detector, because analysts stop reading alerts from a tool that regularly flags honest applicants.
Holograin was neither trained nor measured on identity documents, and results cover Google's and OpenAI's image models only.
Spotted an error? Write to [email protected] and we will fix the row and note the change.
Sources
- Holofin, Holograin: Detecting Documents Redrawn by AI (full article and examples)(accessed Nov. 1, 2026)
Frequently asked questions
Can an LLM tell whether a document was generated by AI?
Not reliably. In this test Gemini 4 Argon caught 90 of 100 AI-edited pages but flagged 45 of 100 genuine ones, and GPT-6.1 Sol Pro flagged 2 genuine pages but missed 60 fakes. An LLM judges whether a document is plausible, and an AI-edited payslip is plausible by construction.
Why do off-the-shelf AI-image detectors miss documents?
They are trained on photographs. A scanned document is mostly flat white with sharp glyphs and compression artefacts, far outside what they have seen, and some shrink the page to a few hundred pixels before scoring it. Run tile by tile at full resolution, a leading open detector caught 20% of uploaded AI-edited pages at 1% false positives.
Does a screenshot or a recompression remove the traces?
No. Holograin catches 97% of AI-edited pages after a phone screenshot and 90% after upload and conversion, at 1% false positives.
Does detection hold on generators it was not trained on?
On the ones measured, yes: 95% on GPT-5.4 image, kept out of training, 84% on Nano Banana Pro and 82% on Gemini 3.1 Flash-Lite Image. Families that have not been measured are not claimed.
Can I reuse these figures?
Yes, under CC BY 4.0 with attribution to Holofin. Each table has a CSV download; the full write-up is in the linked article.
Related reading
Put Holofin on your own documents.
Every value linked to the page it came from, every PDF checked for tampering, rules checked across the whole file.