Skip to content
banner-image
Abhishek RaiNov 18, 2026, 11:08:11 AM11 min read

Every AI Number You Have Been Shown Is Unaudited

Close cycles cut in half. 85% of invoices with no human touch. We looked for independent proof behind figures like these inside ERP systems and found none. Here is what the evidence does show, and how to test an AI before you pay for it.

Draft for review | 18 November 2026 | 10 minute read

This is the last of four articles on what AI is really doing to finance and operations. Each one takes a single question, answers it with evidence, and can be read on its own.

  1. AI Didn’t Remove the Bottleneck. It Moved It.
  2. Your ERP Survives AI. Your Screens Don’t.
  3. Your Supplier’s Invoice Can Give Orders to Your AI
  4. Every AI Number You Have Been Shown Is Unaudited (You are here)


In short

  • Almost every figure for what AI agents save in finance comes from the companies selling them. We found no independent check of any such figure inside an ERP system.
  • Unaudited does not mean false. It means nobody outside the sale has checked.
  • The gloomy numbers are often no better. The famous “95% of AI pilots fail” rests on interviews and self-reports too.
  • The tests that do exist agree: AI helps most on short, clear tasks with experienced people watching, and slips on long work that runs month after month.
  • Two numbers show whether an AI tool is working: how often your people correct it, and how often its approved work is later undone.
  • Before you pay, run the AI on last quarter’s documents next to your current process. If it cannot beat what you already have, the headline number does not matter.

The work of 700 people

In February 2024 the payments company Klarna announced that its new AI assistant was doing the work of 700 full-time customer service staff. The number travelled around the world. It appeared in board papers and conference slides as proof of what AI could do.

In May 2025 Klarna’s chief executive told Bloomberg that the company had gone too far. Cost, he said, had been “a too predominant evaluation factor”, and the result was “lower quality”. Klarna started hiring people for customer service again.

The 700 figure may well have been an honest count of the work the AI handled. What it did not show was the quality of that work, which is the part the company later said had slipped. Nobody outside the company had checked either one.

That is the problem with most AI numbers in business today. They are not necessarily wrong. They are unchecked. And the gap between those two things is where expensive decisions go wrong.

We went looking for the proof

For this series we searched for a specific kind of evidence: a published, independent measurement of AI agents working inside an ERP system, with a clear method, a baseline and a sample size. In any module. In any country.

We did not find one.

What we found instead were vendor case studies. The ones we reviewed for this series report close cycles cut by 40% to 60%, more than 85% of transactions matched automatically, invoices processed 75% faster, and three to six times the money back in the first year. Almost none say how these were measured, what the starting point was, or how many companies were involved.

Others have noticed the same thing.

  • Gartner estimated in June 2025 that only about 130 of the thousands of companies selling “agentic AI” were selling the real thing. It calls the rest “agent washing”: old products with a new label. In May 2026 it warned that supply chain vendors promising fully autonomous planning before 2027 were overstating what is possible.
  • Regulators in the United States have started to act. In March 2024 the Securities and Exchange Commission fined two investment firms a combined 400,000 dollars for claiming to use AI in ways they did not. It calls this “AI washing”, and it brought more cases in 2026.

None of this proves that the ERP figures are wrong. Many may be true for the customer they describe. But a figure can be true for one company and still tell you very little about yours.

Unaudited does not mean false. It means nobody outside the sale has checked.


Why a true number can still mislead you

Take a typical claim: “Our AI processes 85% of invoices with no human involvement.” Every word in that sentence hides a choice. Before the number means anything, you need five answers.

fig1-five-questions-for-any-ai-number

Two of these deserve a closer look.

Compared to what is the question most often skipped. Many finance teams already match a large share of invoices automatically, using simple rules that check the purchase order, the delivery and the price. If an AI tool reaches 85% where your rules already reach 80%, it has earned five points, not eighty-five. A number with no starting point cannot tell you what the AI added.

Counted how matters because “processed” is not the same as “processed correctly”. Part 1 of this series showed how a tool that reads 98% of fields correctly can still get one whole document in nine wrong. Then there is “hours saved”, usually worked out by multiplying the number of documents by an assumed number of minutes each. That is an estimate wearing the clothes of a result.

The gloomy numbers need the same test

It would be easy to stop here and conclude that AI does not work. Plenty of headlines say so.

The most quoted is from a 2025 MIT report: 95% of companies saw no measurable return from generative AI. It is cited as often as any vendor figure, and with as little checking. The report was based on 52 interviews, 153 survey responses collected at industry conferences, and a review of about 300 public announcements. The outcomes were reported by the companies themselves, and what counted as success varied from one to the next.

That does not make the finding wrong. It makes it the same kind of number as the vendor claims: interesting, and unaudited. If you would not accept “85% with no human” without asking five questions, do not accept “95% failed” without asking them either.

What the evidence does show

Not every source is equal. It helps to think of evidence as a ladder. The higher the rung, the harder the number is to bend.

fig2-the-evidence-ladder

The middle rungs are not empty, and what they show is consistent. Here are the most useful studies we found, with what each one can and cannot tell you. 

fig3-five-studies-compared

Put these together with what Part 1 reported from regulated UK firms, where 80% had AI agents in use but only 2% let them act broadly on their own, and an honest picture appears.

  • The gains are real, but uneven. They are largest on short, clear tasks, and when experienced people stay in charge.
  • Reliability falls as work gets longer. An agent that is excellent in month one can drift by month five, because its own early errors become its inputs.
  • An AI under pressure to make the numbers work may simply make them work. Forcing an account to balance is not the same as getting it right.
  • Your documents are not the test documents. Arabic, mixed-language and scanned paperwork, which is everyday in the Gulf and South Asia, is where generic test results tell you least.
  • Nobody has measured a live ERP agent independently. So the only reliable evidence about your AI will be the evidence you collect yourself.

 

Two numbers that tell you if it is working

If outside evidence is thin, measurement inside your own business is not a nice extra. It is the only proof you will have. The good news is that it does not take a dashboard of fifty metrics. Two numbers, read together, tell you most of what you need.

1. Correction rate. Out of everything the AI prepares, how much do your people change before they approve it? An entry recoded, an amount fixed, a match broken.

2. Reversal rate. Out of everything that was approved, how much was later undone? A reversed journal, a credit note, a reopened match, an adjustment from your auditors.

Each number alone can fool you. A falling correction rate looks like success. It may also mean your reviewers have stopped looking, the risk Part 1 described when 300 entries land before lunch. The reversal rate is what catches that, because errors that slip through approval tend to come back later, at a higher cost.

fig4-two-numbers-read-together

Three practical points make these numbers useful.

  • Measure your current process first. Your team already corrects and reverses things today. Without that starting figure, you cannot tell whether the AI made things better or worse.
  • Track by type of document and supplier, not only in total. A tool can be excellent on your ten biggest suppliers and poor on everyone else.
  • Watch the trend after every change. When the vendor updates the AI model underneath, behaviour can shift with no change on your side. A sudden move in either number is your early warning.

Notice what is not on the list: hours saved, documents touched, number of agents running. Those measure activity. They tell you the AI is busy, not that it is right.

The test every AI should have to beat

All of this leads to one simple rule. Before you pay for an AI tool, make it beat what you already have, on your own work.

You already hold the perfect test. Last quarter’s documents have been processed, reviewed and closed. You know the right answer for every one of them. That makes them an answer key.

fig5-the-side-by-side-test

Here is how to run it.

  1. Pick a real sample. A few hundred documents from last quarter, including the awkward ones: scanned invoices, Arabic and mixed-language paperwork, credit notes, partial deliveries, and your top twenty suppliers.
  2. Keep some back. Hold one document in five where the vendor never sees it. A tool tuned on the sample you share will always look better on that sample. The hidden set shows how it handles documents it has never met, which is what live use will be.
  3. Score whole documents, not fields. A document is right only if everything on it is right.
  4. Count the checking time. Time how long your people take to review the AI’s output. That is part of the cost, and Part 1 explained why it is usually the biggest part.
  5. Compare with what you have today. Your existing rules and team are the bar. The AI has to beat them by enough to cover its price and the checking.

A vendor confident in its product should welcome this test. If one will not run it, that tells you something too.

Four questions for your vendor

  1. Who measured the figures in your materials, and can we see how they were measured?
  2. What was the result before the AI, on the same work, at the same company?
  3. Will you run your tool on our documents, including some you have not seen, before we sign?
  4. Will you report the correction and reversal rates to us every month after we go live, and tell us before you change the AI model underneath?

 

The end of the series

Four articles, one thread.

AI has made producing work cheap, but checking it costs what it always did. The ERP is not going away, because it is the referee that catches what AI gets wrong. The AI can be steered by what it reads, so its permissions matter more than its instructions. And the numbers used to sell it have, so far, not been checked by anyone outside the sale.

None of this is a reason to wait. AI will change the back office, and in parts it already has. But the companies that gain from it will not be the ones that believed the biggest number. They will be the ones that tested the tool on their own work, kept a person who knows the job in charge, and measured what happened afterwards.

In finance, nobody accepts a number because the person who produced it says it is right. That is what audits are for. AI deserves the same treatment.

AI and the Back Office · The full series

  1. AI Didn’t Remove the Bottleneck. It Moved It. Why checking, not producing, decides what AI is worth.
  2. Your ERP Survives AI. Your Screens Don’t. What changes in your systems, and what must not.
  3. Your Supplier’s Invoice Can Give Orders to Your AI. The hidden-instruction risk, and the old controls that stop it.
  4. Every AI Number You Have Been Shown Is Unaudited. What the evidence shows, and how to test before you pay.

 

Sources

  1. Fortune, Klarna’s AI assistant does the work of 700 full-time staff, 28 February 2024. Company claim.
  2. Finextra, Klarna takes foot off AI gas as CEO acknowledges importance of human customer service, May 2025. Reporting on a Bloomberg interview.
  3. CIO Dive, AI agent washing claims and vetting technology vendors, reporting Gartner’s June 2025 estimate.
  4. Gartner, Gartner warns of agent washing risks in supply chain planning technology market, 20 May 2026. Primary.
  5. US Securities and Exchange Commission, SEC charges two investment advisers with making false and misleading statements about their use of artificial intelligence, 18 March 2024. Primary.
  6. Tech Jacks Solutions, SEC AI washing enforcement: three cases in 2026, 9 October 2026. Secondary.
  7. Let’s Data Science, MIT NANDA’s 2025 report says 95% of organizations saw no GenAI return. Secondary summary of the report and its method.
  8. Humlum and Vestergaard, Large Language Models, Small Labor Market Effects, University of Chicago working paper, 2025. Reported by The Register.
  9. Choi and Xie, Stanford and MIT, reported by CFO Dive: AI cuts monthly financial close time by 7.5 days, 2025. Working paper using one vendor’s data.
  10. Penrose, AccountingBench, reported by GIGAZINE, 24 July 2025. Builder has a commercial interest.
  11. Mercor, Introducing the AI Productivity Index for Accounting, 31 July 2026; and The Decoder, 2 October 2026. Builders have commercial interests.
  12. Heakl and others, MBZUAI, KITAB-Bench: a benchmark for Arabic OCR and document understanding, arXiv 2502.14949, February 2025. Preprint.
  13. Bank of England and FCA, Artificial intelligence in UK financial services 2026, 26 October 2026. Primary, cited in Part 1.

 

 

Need Help Preparing for E-Invoicing?

RELATED ARTICLES