Why does AI hallucinate? And what to do about it

Anyone who wants to put AI to work in their business eventually hears the same objection: "But it hallucinates." True. It's still no argument against using AI. With solid engineering, the problem can be contained.

You have an idea for what AI could take over in your company. And sooner or later, someone says it: "AI hallucinates. You can't rely on it."

The short answer: yes, every language model hallucinates, because it calculates probabilities instead of looking up facts. Zero percent hallucination doesn't exist. But where an AI system can go wrong, and how often, depends on how it's built. Building systems you can rely on anyway is the real engineering task of our time.

So the objection is no reason to bury the idea. It's the starting point for the most important questions in any project: Where is a model allowed to be wrong, where isn't it, and how do you build a system that knows the difference? To answer that, you need to understand what a language model actually does when it answers.

What does a language model do when it answers?

A language model (an LLM, in industry jargon) doesn't know anything. It calculates. For every question, it works out which word is most likely to come next, then the next one, and so on. It has learned from billions of texts what answers sound like. It has never learned whether they're true.

That sounds like a weakness, but it's exactly why these models are so good. They can handle any kind of phrasing and any language. They summarize, translate, structure, explain. Anything that is language, they do better than any software before them.

There's just one thing they can't do on their own: know whether the number they're writing is the one you actually need. Ask "What was our revenue in March?" without giving the model your books, and it will most likely invent a fairly plausible figure. Not out of spite. It does exactly what it was built to do: write the most probable answer based on the information the LLM has. The number is presented like a real revenue figure. It isn't one.

Why does AI hallucinate by design?

You might assume this is a bug the next generation of models will fix. OpenAI's own researchers say it isn't. In a paper published in September 2025, they show that models hallucinate because training and testing reward guessing.

The logic is the same as an exam with no penalty for wrong answers. If you write something for every question, you'll outscore the person who honestly says "I don't know." Models are measured against exactly that kind of test, so they learn never to leave a blank. OpenAI shows this on its own models: one that declines to answer when unsure makes roughly a third as many errors as one that always answers, at roughly the same hit rate. The guessing model still tops the leaderboard.

For you, this means hallucination isn't a teething problem. It's a property of the technology, the way friction is a property of mechanics.

The holy grail: zero hallucinations guaranteed

And yet the sales decks keep saying it: "hallucination-free", "zero hallucinations guaranteed." In 2024, Stanford University put exactly those promises to the test, on legal AI tools from the biggest vendors in that industry, all of them backed by databases. The result: between 17 and 33 percent of answers contained invented or incorrect information. The tools were far better than a bare chatbot. But hallucination-free they were not.

It's the most important number in this article. Not because it's high, but because of what it shows: even good systems that pull their answers from real documents get things wrong sometimes. The difference between a good system and a bad one isn't zero versus something. It's this: Do you know where the errors sit, and have you made a deliberate decision about where errors are tolerated?

A lot has changed since 2024. Models have improved considerably. They guess less, and they're more willing to say when they don't know. None of that changes the underlying fact: a model that calculates probabilities can be wrong. Today it's wrong less often. It's still not never.

The right question isn't whether, but where

A distinction we try to settle in the very first conversation helps here. It starts with a word from the world of software: deterministic. A deterministic system returns the same result for the same input, every time. A database query for March revenue gives you the same number today, tomorrow, and a year from now. A language model is the opposite. It calculates probabilities, and the same question can produce two slightly different answers.

Roughly speaking, there are two kinds of knowledge in a company.

The first kind is exact. How many units did we ship last week? Which customer has which invoice outstanding? What was revenue in March? Each of these has exactly one answer. It lives in your systems, and it can't be off by a decimal point. A language model must never generate this kind of knowledge itself. It gets pulled deterministically (exact, identical every time) from your existing systems, the way every ERP report has worked for decades, and handed to the model ready to use.

The second kind is information that has to be interpreted. Why did a customer complain? What was in the last five emails with the supplier? Which reasons for lost deals keep coming up? These answers come in many phrasings, and any of them can be right. This is where the model shines. It might not reproduce the complaint word for word as it appears in the documents. But as long as the substance is correct, that's enough for the cases described above.

A well-built AI system knows this difference at every single step. Numbers and facts should always come from deterministic systems. The model should be used where it's best: processing language and interpreting information. The error everyone fears happens precisely when that line isn't drawn: when the model is asked to produce numbers it was never given. How we build systems like this is on our AI development page.

What gets lost in the hype

For two years, everyone has been talking about models. Which one is smartest, which one is newest, who won which benchmark. What gets lost is that the part of an AI system that makes the difference usually isn't AI at all.

Databases, interfaces, permissions, tests, monitoring in production. These have existed for decades, they're deterministic, and they work. An AI system that delivers real value in a business is, to a large extent, exactly that: proven software engineering, with a language model at the one point where language needs to be understood or produced.

This matches what Gartner, the market research firm, has been observing for two years. Gartner has spent decades analyzing which technologies take hold in business, and its assessments are a reference point for IT departments worldwide.

Its finding: in 2024, Gartner predicted that at least one in three generative AI projects would be abandoned after the pilot stage, naming data quality, missing controls, cost, and unclear value as the reasons. In 2026, Gartner followed up: at least half of all projects exceed their budget, because of poor architecture decisions and a lack of operational know-how. The model isn't on either list. Projects don't fail because the AI hallucinates. They fail because nobody built what goes around it. How to spot that early is something we covered in Why software projects fail.

How do you recognize a well-built system?

You don't need to understand how a system like this is built. But you can tell whether it's being built well. Two things are decisive.

First, the model should receive as little information as possible and as much as necessary. It's tempting to hand a model all your documents, emails, and spreadsheets. A measurement across 18 leading LLMs shows the opposite effect: the more text a model receives at once, the worse its answers get. A good system first gathers precisely what's relevant to the question, from your ERP, your CRM, or another data source, and gives the model only that. Experts call this context engineering.

The better that groundwork, the less the model hallucinates, the smaller and cheaper it can be, and the more easily it can run on your own infrastructure rather than in someone else's cloud.

Second, how often the system gets it wrong should be measured, before launch and above all throughout operation. There are systematic tests for this, known in the industry as evals. They show which kinds of questions or tasks the AI inside the system struggles with before a customer finds out. A solid test environment also makes sure that errors are avoided systematically and, as far as possible, for good.

A model that words a complaint slightly differently isn't a problem. A model that invents an invoice total is a problem. Good engineering makes the second case technically close to impossible. A vendor who can't tell you an error rate hasn't measured one.

Four questions before AI touches a critical process

If you want to use AI for anything where errors cost money or real trust, you should get four questions answered. By your IT team, by your vendor, and by us.

What exactly does the model receive? Which data, from which systems, in what form. Anyone who can't say that in one sentence isn't in control of the system.
How does my data flow? The more steps between your database and the model happen automatically, the more important it is that every one of them can be traced.
How do you test how often it's wrong? Before launch, continuously in operation, with every change, and above all when switching to a newer or different AI model.
What are the LLM costs? It's tempting to paper over weak software engineering or poor data quality with the most expensive frontier models.

Whoever can answer these four questions has a good chance that what was built is more than a demo.

Hallucination is no showstopper

That AI hallucinates is nothing new. It's a property that everyone who uses this technology seriously works with every day. Zero percent hallucination doesn't exist, and anyone who promises it hasn't understood the technology.

What does exist: systems where you know where the model can go wrong, built so that those spots cost nothing. That isn't a question of which model you pick. It's a question of how much solid engineering someone has built around it.

So the caution is justified. And your idea is most likely still feasible.

Sources

Insights · AI in Business2x10 · Vienna

Why AI hallucinates, and why that's no reason not to use it