Among the new issues that have emerged due to the introduction of artificial intelligence (AI) chatbots is the model’s hallucinations causing users to act out self-harm or, in more serious cases, commit suicide. Google Gemini has now been updated with a button feature that will provide help if it detects early signs of a mental health issue through a conversation with the user.
For situations related to suicide or self-harm, Google is introducing an interface that prompts users to get help. Help will be provided in the form of chat, text, phone call, or website. The option to get professional help will remain visible throughout the conversation.
The feature will focus on safety and human connection, ensuring that users are encouraged to seek real help. The responses Gemini provides are designed to avoid endorsing dangerous behaviors such as encouraging self-harm.
It is also trained not to confirm false beliefs, but to distinguish subjective experiences from objective facts. Also featured is a statement that Gemini is not a substitute for clinical treatment, therapy, or professional crisis support.
This Gemini security feature was launched a week after a recent study from Stanford University titled "Sycophantic AI decreases prosocial intentions and promotes dependence" assessed how AI's tendency to patronize (sycophancy) can have a negative impact on users.
