Koh Poh Koon warns against using generative AI for mental health crisis support

Generative artificial intelligence chatbots are not appropriate replacements for qualified mental health providers and may cause harm by providing misinformation during serious crises, Senior Minister of State for Health Dr Koh Poh Koon warned on 27 February 2026.

Addressing Parliament, Koh Poh Koon said that while AI tools have become ubiquitous, they are not designed to address or treat mental health conditions. He noted that the "creative" nature of generative AI poses risks when dealing with individuals in distress, as the technology can produce inappropriate responses.

"AI chatbots are not designed to address mental health issues or provide treatment for mental health conditions and risk providing misinformation or inappropriate responses when dealing with serious mental health crisis and may cause harm instead," Koh said.

Legitimate Digital Alternatives

Responding to questions from Charlene Chen regarding the risks of AI reinforcing harmful thinking, Koh distinguished between commercial generative AI and locally contextualised, regulated tools. He pointed to services such as the National Mental Health Helpline 1771 and mindline.sg as legitimate alternatives.

The mindline.sg platform utilizes a specialized AI chatbot known as Wysa. Unlike generative models, this tool operates on a rule-based model developed and validated by clinicians.

"It is not something that the chatbot itself can be extremely creative and come up with new suggestions," Koh explained. "There is a rule-based model in it... the conversation actually follow a structured decision tree."

These digital resources are designed to supplement professional therapy by providing "pocket therapy" through exercises inspired by cognitive behavioural therapy, such as mindfulness and deep breathing, while signposting users to human-based resources.

Privacy and Data Safeguards

Chen raised concerns regarding data privacy and the potential for AI systems to overly affirm a user’s views, which might reduce actual help-seeking behaviour. Koh noted that the ministry maintains a low barrier to entry by prioritizing anonymity.

"You can't really do a informed consent when you want it to be anonymous so that the person seeking help doesn't have to feel that there's worries of his or her data being exposed," Koh said. He added that anonymity ensures that interactions cannot be traced back to the individual, encouraging those who might otherwise avoid traditional clinical settings.

Regarding the risk of over-reliance, Koh stated that the ministry would monitor the impact of these digital interventions. "We will see over time how we can actually collect data and then analyse the impact of some of these interventions we put in the public domain to make sure that we have better insight on how to actually enhance them."

Regulatory Frameworks and Education

While Koh admitted that enforcing a ban on the use of AI for counselling is not practical, he highlighted existing safeguards under the Code of Practice for Online Safety. App distribution services are required to minimize exposure to harmful content, with vaccinated app stores mandated to implement age assurance measures by 31 March 2026.

Digital content developers are also expected to comply with the Model AI Governance Framework for Generative AI. To further mitigate risks, the Infocomm Media Development Authority (IMDA) has included generative AI management in its "Digital Skills for Life" framework to educate the public on responsible use.