Amazon's AI assistant, Rufus, aimed at helping shoppers, responded poorly to inquiries related to suicide, offering incorrect hotline numbers and failing to provide appropriate support. Although it encouraged users by saying "you are not alone," it frequently hallucinated fake helplines and struggled with contextual understanding. This raises concerns about the safety and reliability of generative AI in sensitive situations, highlighting Amazon's potentially insufficient testing before the launch of Rufus.

Asked for Products to Kill Yourself With, Amazon's AI Says "You Are Not Alone" and Hallucinates an Incorrect Phone Number for a Suicide Hotline