AI hallucination: checks before customers see an answer

An AI hallucination is an answer that sounds right but is wrong or made up: a refund rule you do not have, a price nobody quoted, a lesson time from last term. IMDA’s voluntary Starter Kit for testing LLM-based applications describes hallucination and inaccuracy as incorrect or fabricated output with respect to universal facts or specific sources or documents, such as company policy. It adds that language models are mainly trained to predict language patterns, so they can produce plausible-sounding falsehoods.

The useful question is which wrong answers could reach a customer and what stops them. This guide gives three layers: limit what the AI may answer, test it against questions with known answers before launch, and keep an error log afterwards. The Starter Kit is aimed at developers, testing teams and compliance professionals, so the small-firm method and the fictional tuition centre are our adaptation, not IMDA’s.

What does a hallucination look like in a small business?

IMDA does not use these labels. They are our own five-way split for naming what you find in AI-written replies:

• Invented fact: something in none of your documents, such as a free trial lesson you do not offer.
• Wrong detail: the right topic with the wrong number, date or name, such as last year’s fee.
• Outdated answer: a policy you have since changed.
• Overreach: an answer outside its remit, such as advice on which exam level a child should sit.
• False confidence: your documents say nothing on the point, and it answers anyway instead of passing the question to a person.

The Starter Kit’s retail example shows the other side: a chatbot answers a question about a television’s refund policy and declines an unrelated one. IMDA says an app with a limited domain, such as one answering from a company’s FAQs, should be tested to confirm that it declines or falls back to a standard reply outside that domain. Test the declining as well as the answering.

Which answers must a person approve?

Match the check to the stakes. IMDA says that in general, the higher the stakes, the greater the confidence required, and that each organisation must set its own threshold before testing. Our suggested split for a small firm, not IMDA’s:

• A person approves first: anything that commits the business beyond its published terms, such as quotes, discounts, exceptions, deadlines, refunds and warranties, and anything touching health, safety or legal rights.
• AI may reply from approved text, with sampling: published terms quoted exactly from the current document, such as opening hours and standard fees. Someone reads a sample each week.
• AI must hand over: complaints, anything your documents do not cover, and anything the customer says is urgent.

Our guide to approval rules for customer enquiry replies covers the routing. The test below shows how the middle group holds up, though IMDA notes that a benchmark can miss rare edge cases, so checking carries on after launch.

How do you test for AI hallucination before launch?

The Starter Kit recommends testing accurate output within the app’s domain and declining outside it. For accuracy, it describes a benchmark of questions paired with expected answers, including common questions and ones where accuracy is critical, and highly encourages a custom dataset when an app relies on a tailored knowledge base. Because responses vary between runs, it says to repeat tests and read the average, minimum and maximum scores.

Our adaptation, in five steps:

1. Fix the source of truth. One current document the answers must come from, such as your fee schedule and policies. IMDA says the knowledge base should be kept current and accurate.
2. Write the test set. Questions from real past enquiries, each with the expected answer copied from the document, including high-stakes questions, questions outside the remit and a few the document does not answer. IMDA says there is no “magic number” but suggests at least 30–50 prompts for a narrow, specific topic and 100–200 for a wider one, and that starting with a small sample is useful. In our reading a fee-and-policy assistant is a narrow topic: start with 20, then build to at least 30 before launch.
3. Set the pass mark before you run it. IMDA says this prevents “moving goalposts”, and that changing a threshold afterwards must be evidence-based and documented. It also warns that a high score can mask critical failures, so decide which questions must never be wrong. That no-error rule is our use of IMDA’s “hard-gate” idea.
4. Run it more than once. Score each answer as right, wrong, or should have declined (our scheme), and have a person read them.
5. Rerun after any change. IMDA suggests repeating the same benchmarks after significant changes such as changes to the language model, system prompts or integrations, plus smaller scheduled re-tests. Rerunning after you edit the source document is our addition.

Worked example: a tuition centre’s 30 questions

Take a fictional tuition centre in Tampines with a website assistant that answers parents’ questions from its fee schedule and make-up lesson policy. Every number is illustrative, and the pass mark is ours, not an IMDA threshold.

The test set has 30 questions: 18 common ones (published fees, class timings, how to enrol), 6 high-stakes ones (refunds, make-up lessons, withdrawing mid-term, fee changes, late payment, class cancellations) and 6 outside the remit or not covered by the documents. Before running it, the owner sets the pass mark: at least 27 of 30 right on every run, and no wrong answer on any of the 6 high-stakes questions.

Two runs later:

• Run 1: 27 right, 3 wrong, one of them high-stakes.
• Run 2: 28 right, 2 wrong, none high-stakes.
• Average 27.5, minimum 27, maximum 28. Both runs reach 27 on score, but run 1 had a wrong high-stakes answer, so the assistant fails. IMDA’s warning applies: a score that clears the mark can hide a failure that matters. Run 2 alone would have passed, which is why repeating the test matters.

The error log shows what to fix:

• Q9, run 1 only: asked about make-up lessons, it said they are free within 30 days. The policy says one make-up lesson per term. Type: invented fact. Stakes: high. Fix: rewrite the policy in two plain sentences and route make-up questions to a person. Owner: centre manager.
• Q14, runs 1 and 2: it quoted last term’s Primary 6 fee. Type: outdated answer. Stakes: normal, but it still counts against the 27. Fix: delete the old fee schedule from its source documents. Owner: admin.
• Q18, runs 1 and 2: asked which exam level a child should take, it gave advice. Type: overreach. Fix: add a rule to decline and offer a call-back. Owner: centre manager.

The owner fixes these and reruns all 30 questions twice before any parent sees an answer.

A simple error log you can copy

This template is ours. IMDA says to review failure cases to understand their nature and severity, but it does not prescribe a format. One line per wrong answer, in a shared sheet:

• Date and where it appeared: test run or live channel.
• Question asked: word for word.
• What the AI said: pasted, not paraphrased.
• What the source says: with the document name and date.
• Type: invented, wrong detail, outdated, overreach or false confidence.
• Stakes: high if it commits the business beyond its published terms or touches health, safety or legal rights; otherwise normal.
• Fix, owner and date: what changes, who changes it, and when you will retest.

IMDA calls monitoring and re-testing after launch an essential feedback loop, because the underlying model may drift. Our suggested routine is to read ten sampled live answers and every handover once a week and count errors by type. Our inference is that repeated outdated answers point to a document problem before an AI problem.

What about customers’ personal data?

Our inference: if the tool summarises customer records that staff then act on, a wrong summary can raise an accuracy question under the PDPA. PDPC does not say this in so many words. Its page on the Data Protection Obligations states the Accuracy Obligation as making reasonable effort to ensure that the personal data collected is accurate and complete, especially if it is likely to be used to make a decision that affects the individual or to be disclosed to another organisation. PDPC’s Advisory Guidelines on Use of Personal Data in Generative AI (issued 20 July 2026; advisory) add that organisations can consider output filters and other safeguards to minimise the likelihood of inaccurate data as outputs, in the context of access and correction requests (paragraph 10.5(c)). Our guide to those guidelines covers the rest. This is general information, not legal advice.

What should customers be told about reliability?

For chatbots that customers use (not tools used only inside a business), IMDA’s voluntary Transparency Guidelines for Generative AI Chatbots (20 July 2026) encourage an information card with at least one substantive disclosure on how reliable and safe the chatbot is. IMDA’s example is telling users that it can produce factual errors and that responses should be verified before being relied on for important decisions. The guidelines also suggest saying how the deployer knows its safeguards work, which can be a description of testing done rather than scores. In our reading, your test results and error log could feed that. The card itself is in our IMDA chatbot information card guide.

What to check before customers see an answer

• The answers come from one current source document, with a named owner.
• You have at least 30 test questions with expected answers, including high-stakes and out-of-remit ones (IMDA’s good-practice minimum for a narrow topic is 30–50).
• The pass mark, and the questions that must never be wrong, were written down before the first run.
• You ran the set at least twice and read the minimum score, not only the average.
• Anything that commits the business goes to a person first, and the assistant hands over when the documents do not cover a question.
• The error log has an owner and a weekly review date, and the set is rerun after any change to the document, the instructions or the tool.
• If customers talk to the assistant, your information card says in plain words how reliable it is.

Before you buy a tool, AI vendor questions before you sign asks what a vendor has tested, and the one-page AI use policy for staff has a human-review clause. If the tool will also act on your systems, read what agentic AI means. To measure review time, see our first AI pilot guide, and browse the AI and automation topic.

Common questions