AI Agent Testing: IMDA Starter Kit and Accredited AI Testers

AI testing for agents now has a Singapore playbook. On 14 October 2026 the Infocomm Media Development Authority (IMDA) launched a Starter Kit for Testing Agentic Systems for Safety and Reliability, and the AI Verify Foundation launched an AI Tester Accreditation Programme (AI TAP) with five accredited testing firms. The kit sets out four steps: identify risks such as wrong or unauthorised actions, design realistic tests, run them and read the logs, then keep monitoring and re-testing after launch. For an SME about to let an AI agent send messages, update records or book appointments, the practical upshot is a short, written test before go-live and a clear view of when an outside tester is worth paying for.

IMDA presents the kit as a practical reference for organisations, and nothing in its release makes testing mandatory. This article explains what it asks, what is new compared with testing a chatbot, a worked test plan for a small firm and how to judge an accredited tester.

What does IMDA's agentic testing kit ask organisations to do?

IMDA's release describes the four steps:

1. Identify risks such as erroneous or unauthorised actions.
2. Create realistic and robust tests that capture the spectrum of agent behaviour while minimising harmful impact.
3. Run the tests and analyse the logs.
4. Implement continuous monitoring and re-testing.

It also flags what is new for agents: testing the whole "trajectory", meaning the agent's reasoning and its actions, not just the final answer; agent-specific measures such as tool use accuracy, handoff accuracy and human-in-the-loop rates; gradually automating log analysis to spot failure signals; and adapting tests where the deployer or tester can only see the system as a black box, which is the usual position for an SME using a vendor's agent.

IMDA calls the kit a living document. On 17 October 2026 its release still showed "[Link]" placeholders where the kit and feedback form should be, and the one testing document it did link was the earlier Starter Kit for Testing LLM-Based Applications (PDF), version 1.0 of January 2026, which covers five risks: hallucination and inaccuracy, bias in decision making, undesirable content, data leakage and vulnerability to adversarial prompts. Until the agentic kit itself is published, the release and the framework below are what you can work from.

How is testing an agent different from testing a chatbot?

A chatbot can say something wrong; an agent can do something wrong. IMDA's Model AI Governance Framework for Agentic AI (PDF) lists what to test before deployment:

• Task execution: does the agent complete the task accurately?
• Policy compliance: does it follow your procedures and route to a person for approval when required?
• Tool calling: does it call the right tools, with the right permissions and inputs, in the right order?
• Robustness: how does it respond to errors and edge cases?

The framework adds that agent behaviour varies from run to run, so tests should be repeated and run across varied data; that testing should happen in an environment as close to real use as possible, while weighing the risk of letting an unproven agent touch real systems; and that results can be checked with deterministic tests for structured tool calls and human or AI review for reasoning.

Worked example: a 40-case test before go-live (illustrative)

A fictional aircon servicing company, run by Aisha, plans to let an AI agent answer WhatsApp enquiries, check the job calendar and book visits, with any refund, complaint or price outside the standard list handed to a person.

1. Write 40 scenarios from real enquiries: 25 routine bookings, 10 that must be handed to a person (complaints, refunds, unusual prices) and 5 awkward ones (a double booking, a customer changing their mind mid-chat, a message in mixed languages).
2. Run each scenario three times, because answers vary: 120 runs.
3. Score tool use accuracy: the agent picked the right tool, such as a calendar check or booking, with the right details in 112 of 120 runs, 93.3%.
4. Score handoff accuracy: of the 30 runs that should have reached a person, 27 did, 90%. The three misses are the finding that matters: one confirmed a refund it had no authority to give.
5. Record the human-in-the-loop rate: 33 of 120 runs paused for approval, 27.5%. Whether that is too high is a staffing question for Aisha, not a pass mark.
6. Set thresholds before you look: for example, any unauthorised action is a fail whatever the averages say, and tool use accuracy must reach an agreed level. Fix the prompts, permissions or handoff rules, then rerun all 120.

The percentages are invented for the example; the measures are the ones IMDA names. After launch, step 4 of the kit applies: sample real conversations weekly and rerun the 40 scenarios whenever the vendor updates the agent.

When should an SME pay for an accredited AI tester?

IMDA says accredited firms were evaluated for testing competency, track record, delivery capability and financial sustainability, so that businesses can engage them "with greater confidence". The release gives no prices. An outside test is easiest to justify when:

• the agent can move money, change customer records or affect people's jobs or health;
• you cannot see inside the vendor's system and need someone who can design black-box tests;
• a large customer or partner asks for independent evidence of testing.

For a low-stakes internal assistant, a written in-house test like Aisha's is usually proportionate. If you do engage a tester, check that its accreditation is current, since it lasts one year, and ask which of IMDA's measures it will report.

Checklist before your agent goes live

1. List every action the agent can take and mark the ones that cannot be undone.
2. Remove permissions it does not need for the first release.
3. Write 30 to 50 test scenarios from real cases, including ones that must reach a person.
4. Run each several times and score tool use, handoffs and approval pauses.
5. Set pass rules in advance, with unauthorised actions as an automatic fail.
6. Keep the logs and decide who reviews them weekly after launch.
7. Rerun the tests after every vendor update or change to your procedures.

Our guides to deciding what an AI agent may do and to IMDA's agentic AI framework cover the limits to set before testing, and our guide to testing a chatbot before accepting a quote shows how to use a shared test set with vendors. If you are trying an agent through one of IMDA's new programmes, see our comparison of the free AI trials announced on 13 October. More under AI.

Common questions