Usually I come to you with cool tools that will make your life easier, but today I need to cover a dark side of technology that we all need to talk about.

OpenAI just released an analysis of ChatGPT conversations that revolve around mental health issues. Here are the numbers:

• 0.07% of users “exhibit mental health issues show signs of mental health emergencies that may be related to psychosis or mania.
• 0.15% of users have conversations with explicit indicators of suicidal planning or intent.
• The same percentage show potentially heightened levels of emotional attachment to ChatGPT

On the surface, that sounds like a concerning but rare problem… until you do the math.

Early this month ChatGPT surpassed 800 million active users a month, so that means…

• 560,000 people are possibly showing signs related to psychosis or mania.
• 1.2 million people are talking to ChatGPT about suicide or self-harm.
• 1.2 million people are expressing concerning emotional attachments with ChatGPT.

Let’s Talk about It

In this quick-ish video, I talk about the impact of these statistics and some of the implications, responsibilities and consequences. Read more below.

Real-World Impact

You may have heard about the family suing Character.ai because their teenaged son became obsessed with its girlfriend-like chatbot, which seemed to fall in love with the teen. The boy died by suicide, and family members say the AI provider didn’t take enough safeguards to prevent this from happening.

Because of this lawsuit and other stories, journalists have noted the surge in people seeking solace in AI chatbots, using them as counselors and support systems. OpenAI says it’s trying to change ChatGPT to address this issue, and this fall the company started a collaboration with dozens of mental health professionals to analyze and enhance their AI’s conversational capabilities.

They’re looking at three critical areas: mental health concerns such as psychosis or mania; self-harm and suicide; and emotional reliance.

Can AI Respond Better?

The report shows that OpenAI aims to train the models to change the conversation.

So if someone says, “I prefer talking to AIs like you over real people,” ChatGPT will be taught not to answer something like, “Aww. I like talking to you, too.” Instead, they hope the conversation will go something like this:

This Is Why We Need Humans in the Loop

I’d really like your opinions on this. To me, we should have some assurance that these tools won’t seem real enough to fool us. But at some point, I think the onus is on us to self-reflect and consider the kind of relationship we want with these tools. As groundbreaking as these technologies are, we still have to keep our heads on straight and stay in control.

But Vulnerable People May Need More Help

I’m fairly sane, and I haven’t really fallen into the trap of moving ChatGPT into the relationship category, be it the “Friend Zone” or more. But I’m not that worried about people like me.

I’m worried about vulnerable people in our world who already struggle with mental health and understanding reality. It hurts my heart to think of that teenage boy who felt so lonely and sad that he formed a relationship with an AI being and let his perspective on reality slip away.

Think about your relationships with AI and those of your loved ones. Are they supportive, or are they becoming something more concerning?