Hidden Use of ChatGPT in Online Mental Health Counseling Raises Ethical Concerns

“If you want to set back the use of AI in mental health, start exactly this way and offend as many practitioners and potential users as possible.”

So said medical ethicist, Art Caplan, of a digital mental health company’s use of the artificial intelligence program ChatGPT to generate responses to over 4,000 users. 

Rob Morris, the founder of a free therapy program named Koko, admitted on an extensive Twitter thread that his service used GPT-3 chatbots to help respond to more than 4,000 users seeking advice about their mental health. 

We provided mental health support to about 4,000 people — using GPT-3. Here’s what happened 👇

— Rob Morris (@RobertRMorris) January 6, 2023

Koko partners with online communities to find and treat at-risk individuals. In a video posted by Morris, the platform is described as “a place where you can get help from our network or help someone else.” 

ChatGPT, created by the company OpenAI, is a free artificial intelligence (AI) program that generates realistically conversational text based on the prompts it is fed.

Computer Vision Syndrome

Personality Traits and Online Pornography

Boredom Proneness, Loneliness, and Smartphone Addiction

In the Twitter thread, Morris said that his team took a “co-pilot” approach to using the tool by having humans supervise AI-composed messages. He claimed that these responses were rated significantly higher than those written by humans on their own and that the service’s response times went down 50 percent to well under a minute when the AI chatbot was employed.

Yet, when users were told they were being counseled by a computer, they didn’t like it, Morris admitted. “Simulated empathy feels weird, empty,” Morris tweeted. 

Once people learned the messages were co-created by a machine, it didn’t work. Simulated empathy feels weird, empty.

— Rob Morris (@RobertRMorris) January 6, 2023

It’s somewhat murky if the experiment was done without client consent or knowledge, said Caplan, who is the founding head of the Division of Medical Ethics at NYU Grossman School of Medicine in New York City. 

“If not, this was highly unethical,” he said. “Those with mental health issues require clear consent and in my view, a review by an independent committee to ensure all disclosures include a conflict of interest statement. Then there is privacy protection. And a notification that ChatGPT is experimental.  And so on,” he said. 

The backlash was swift with the majority of the more than 1200 people who responded to Morris’ original tweet siding with Caplan. Many called the experiment shameful and immoral.

You shouldn't be admitting you're willing to trial and error new products and skirt around informed consent rules where people's mental health is involved. This seems deeply unethical.

— Brennan Leffler (@brennanleffler) January 8, 2023

So you exploited a vulnerable population for profit…just to “conclude” what any human with basic empathy could have told you from the beginning? Your bs company deserves to be sued into nonexistence

— meredith (@meredith_2020) January 9, 2023

So, they conducted an experiment on vulnerable humans without informed consent and without ethics approval, and then shared the results on social media for clout? Ignorance and arrogance is a bad combo. https://t.co/OGnHXzZ3KZ

— Sabina Vohra-Miller (@SabiVM) January 8, 2023

Eventually, Morris attempted to clarify his original words. 

“We were not pairing people up to chat with GPT-3, without their knowledge. (in retrospect, I could have worded my first tweet to better reflect this),” he tweeted several days later. 

And in another tweet, he said:

“This feature was opt-in. Everyone knew about the feature when it was live for a few days.” This seemed to contradict his original statement about how people felt about “finding out” that they were receiving therapy from an AI bot. 

In a statement, Morris told Business Insider that, from his point of view, he didn’t need consent anyway. He referenced a clause that every user signs before they use the service, adding that, “If this were a university study (which it’s not, it was just a product feature explored), this would fall under an ‘exempt’ category of research.” 

Caplan disagreed, calling the use of  AI technology without informing users grossly unethical. Further, he explained, it is not standard of care nor has any professional organization tested its risks and benefits.

He also said that no matter how sophisticated ChatGPT seems, it is nowhere near ready for use in treating mental health issues.  

“ I can see its use in medicine in perhaps say, robotic surgery. AI can learn and be a responsive tool for health care. But in mental health, that day is not yet here. In fact it isn’t close,” Caplan said. 

Morris told Business Insider that his intention was to emphasize the importance of the human in the human-AI discussion. “I hope that doesn’t get lost here,” he said. 

He added that the tool imposed no further risk or deception to users and that Koko didn’t collect any personally identifiable information or personal health information during the test. 

Still, the Twitterverse did not appear to believe that Morris’ clarification let him off the hook.  One Twitter user summed the general consensus:

this is so incredibly unethical and you should be ashamed of yourself

— elena 🌸 (@cheer_wine) January 8, 2023