AI hallucinations: why AI sometimes makes things up with full confidence

Because a language model predicts text instead of looking it up. It has no database of facts in which it checks something. It writes the most likely continuation, word by word, based on the enormous amount of text it was trained on. Usually that produces a good answer. But likely isn't the same as true.

That applies to ChatGPT, Copilot, Claude and every other language model. Ask about something that came up often in the texts the model was trained on, and the likely answer is usually also the right one. Ask about something rare, specific or recent, and the model has little to go on. It still writes a fluent answer, because that's what it does. It just fills the gaps with whatever looks plausible.

Three things make it worse:

• The model doesn't know you. Your delivery times, prices, customers and agreements aren't in its training data. Ask about them without providing them, and it guesses.
• It wants to be helpful. A model is designed to give a usable answer. 'I don't know' doesn't come out as easily as you'd like, unless you explicitly ask for it.
• It goes along with your question. Ask 'why is A better than B?', and you get reasons why A is better, even if it isn't.

Even an AI tool that searches the internet or your documents can still hallucinate. It can misread a source, mix up two sources or fill in the gap between two passages itself. Searching reduces the risk, it doesn't remove it.

The risk is highest with precise facts you didn't provide yourself: figures, sources and quotes, legal and medical details, and names, job titles and dates. When rewriting, organizing and summarizing text you provide yourself, the risk is much smaller. Not zero, but smaller.

High risk

• Figures and statistics. Market figures, percentages, amounts, numbers. It sounds precise, and is often made up.
• Sources and quotes. Titles of reports, links, things people said. Always check whether the source exists and whether it really says that.
• Legal details. Articles of law, deadlines, rulings. A wrong article number looks just as good as the right one.
• Medical details. Dosages, side effects, guidelines. Here AI should at most do groundwork for an expert.
• Names, job titles and dates. Who the director is, when something was founded, what someone's role is.
• Anything about your own company. If you don't provide it, the model doesn't know it.
• Calculations. A language model isn't a calculator. Let Excel or your system do the sums, or have the AI write a formula instead of an outcome.

Lower risk

• Making a text you wrote yourself shorter, clearer or more formal
• Drafting an email based on the facts you provide
• Coming up with ideas, adding structure, making an outline
• Summarizing a document you provide (do check the numbers and names)

The rule of thumb: use AI to write, organize and summarize what you provide. Get facts from your own sources and systems.

You can't prevent them completely. But you can make the risk so small that AI is usable for real work. Five things help most: provide the source, ask for sources and check them, keep the task small, give permission to say 'I don't know', and have a person check.

1. Provide the source

Don't let the model answer from memory, but from your own documents. Paste the text into the prompt, put the documents in your own assistant, or have a system look up the right passages first. That last option is called RAG: the AI first searches your documents and only then answers, with the source included. That way you can check every answer in ten seconds.

2. Ask for sources, and check them

For every fact, ask where it comes from: 'for each point, name the passage or page you're basing it on'. Then actually click through. If the source doesn't exist, or it doesn't say that, you know enough. A source you don't open isn't a check.

3. Make the task smaller and well defined

Not 'write a market analysis', but 'summarize these three reports in five points, based only on this text'. The smaller and clearer the task, the less room there is to fill things in. Break big jobs into steps, and check each step.

4. Give permission to say 'I don't know'

Put it literally in your prompt: 'If the answer isn't in the text below, say so and don't make anything up.' An AI that honestly says it doesn't know is worth much more than one that guesses. How to build an instruction like that properly is in prompting: how to write a good prompt.

5. Have a person check

Anything that goes to a customer or ends up in a system is checked by a person first. That's called human in the loop: the AI does the work, an employee checks it. With the source included, that takes seconds, not minutes.

One more useful trick: ask the same question in a new conversation, or ask 'which sentence are you basing this on?'. If the answer changes, or no clear sentence comes back, be careful.

By details you didn't provide and can't find again. A hallucination looks the same as a good answer, so you won't recognize it by the tone. You will by a few signals:

• Very precise details out of nowhere. A page number, a percentage with a decimal, an exact date you didn't mention.
• A source you can't find. The title sounds logical, but the link doesn't work or leads to something else.
• A perfect quote. Real people rarely phrase things as neatly as the quote the AI puts in their mouth.
• The answer changes. Ask the same question again and you get a different year or a different amount.
• It agrees with you too quickly. Say 'I don't think that's right' and the model turns around, even if the first answer was correct. Then it probably didn't know.

If in doubt, treat it as a tip, not a fact. Look it up in a source you trust.

In a chat you read the answer yourself, and you usually spot a strange figure. In a business process it's different. There, the AI's output lands in your ERP, in a quote or with a customer, often without anyone reading it first. A made-up order line is then a wrong delivery. A made-up condition in a customer email is a promise you have to keep.

That's why we build every process so that a person approves before anything goes out the door or is written to a system for good. And we build checks into the workflow itself:

• A fixed format. The AI fills in fields, and doesn't write free text where that isn't needed.
• Checks against your systems. Does this order number exist? Does this item belong to this customer? Does the total match the sum of the lines?
• The source included. With every answer, the passage it's based on, so the check goes quickly.
• Exceptions to a person. If the AI is in doubt, information is missing or a high amount is involved, it goes to an employee.

That's also how it works at one of our enterprise clients: for 80 percent of the 6,000 customer questions a month, a prepared answer is ready before the ticket opens, compared with 30 percent at first. An employee checks and sends it, always. At KplusV, AI assesses pitch decks against their own canvas and finds suitable funders, and a consultant always checks the result.

You see this in our calculations too: we never calculate with 100 percent saving, but with 90 percent per step at most, because checks and exceptions remain. How to keep that check light is in human in the loop: AI with a person who approves.

With a few fixed agreements, and by practicing on your own work. Banning it rarely helps: people often use AI anyway, just without agreements. It's better if everyone knows what AI is good at, where it goes wrong and who is responsible for what gets sent.

Agreements that belong in a good AI policy:

• AI writes, organizes and summarizes. You get facts from your own sources.
• What goes out is read by a person first. Whoever sends something stands behind it. The AI doesn't.
• A source you cite, you've opened yourself.
• You always check figures, names and legal details.

The fastest way to learn it is to see it happen, on your own work. In a half-day workshop we show when AI goes off track and how to prevent that, with your own emails and documents. That costs € 4,500 per half day, at your site, excluding VAT.