Love it or hate it, there’s no question that chatbots are here to stay. And while an increasing number of people have been turning to the likes of Claude and ChatGPT for personal help, there’s only been anecdotal evidence as to their value for mental health. That is, until now.
Dartmouth researchers have conducted the first-ever trial of its kind, using a generative AI chatbot built by the college to quantify how effective such new-tech models are for people with mental health challenges. In the study, 106 participants across the US, who had been diagnosed with major depressive disorder (MDD), generalized anxiety disorder (GAD) or an eating disorder, took part in a randomized clinical trial for four weeks. The control group then had access to the smartphone chatbot known as Therabot from week four through to week eight.
Unlike chatbots such as ChatGPT, Therabot would initiate engagement with the participants, prompting responses to questions about mood and emotions. This encouraged users to respond and then further engage them in conversation. At the end of the trial, Therabot users – of which around three quarters had no other mental health intervention at the time – were assessed based on symptom-relief measures.
Participants with depression reported that their symptoms had lessened by an average of 51%, with specific improvements when it came to mood and well-being in general. Meanwhile, the GAD cohort reported a mean 31% reduction in their symptoms, which for many moved their clinical anxiety levels from moderate to mild, or from mild to actually below the diagnostic threshold. And for participants with an eating disorder, there was an average 19% improvement when it came to body image – this was particularly impressive, given the difficulties in treating the condition via traditional methods.
“The improvements in symptoms we observed were comparable to what is reported for traditional outpatient therapy, suggesting this AI-assisted approach may offer clinically meaningful benefits,” said senior author Nicholas Jacobson, an associate professor at Dartmouth’s Geisel School of Medicine. “There is no replacement for in-person care, but there are nowhere near enough providers to go around.”
In our recent article on the use of chatbots for mental health support, we highlighted this, too: A lack of affordable and timely access to treatment has left many people, for a suite of reasons, with little or no help. The researchers add that in the US alone, demand far outweighs supply, with one therapist for around 1,600 people with clinical anxiety or depression in need of support. While chatbots are not trained for this kind of work (yet), they offer accessible and 24-7 access for people to reach out when they need to – and this real-time reliability builds trust for those turning to the technology.
“We would like to see generative AI help provide mental health support to the huge number of people outside the in-person care system,” Jacobson added. “I see the potential for person-to-person and software-based therapy to work together.”
If you’ve accessed a chatbot for mental health support, you’ll probably have found that it is helpful in that moment. And while the technology is improving at a rapid pace, healthcare professionals remain cautious about relying too heavily on its advice and feedback.
“While these results are very promising, no generative AI agent is ready to operate fully autonomously in mental health where there is a very wide range of high-risk scenarios it might encounter,” said first author Michael Heinz, an assistant professor and psychiatrist at the Dartmouth Hitchcock Medical Center. “We still need to better understand and quantify the risks associated with generative AI used in mental health contexts.”
Unlike other existing chatbots, Therabot has been developed through consultation with psychologists and psychiatrists specifically for use as a mental health support tool. As such, users would receive open-ended replies to prompts, encouraging them to talk further about their concerns and emotional state. During the trial, guardrails were in place to recognize any participants who were at risk of self-harm and provide clickable links to emergency care.
After the trial’s initial four weeks, users (as well as the control group) had another four-week period of access to Therabot, but this time without the app volunteering prompts. And participants continued to engage with the chatbot.
Overall, people spent around six hours throughout the trial communicating with Therabot, which is equivalent to about eight sessions with a therapist. And there was an uptick in use at times when symptoms are often most prominent, such as late at night. And the continued use beyond the prompt-stage of the trial suggested that users had formed somewhat of a bond with – and trust in – the tool.
“We did not expect that people would almost treat the software like a friend,” Jacobson said. “It says to me that they were actually forming relationships with Therabot.
“My sense is that people also felt comfortable talking to a bot because it won’t judge them,” he added.
Participants reported that they felt more willing to be candid and open up to Therabot, but the study doesn’t assess how this benefit alone helped with symptom relief. And while the researchers are quick to warn about the unstudied risks of relying too heavily on this technology, the study shows that there’s both demand and benefit there.
“Our results are comparable to what we would see for people with access to gold-standard cognitive therapy with outpatient providers,” Jacobson said. “We’re talking about potentially giving people the equivalent of the best treatment you can get in the care system over shorter periods of time.
“There are a lot of folks rushing into this space since the release of ChatGPT, and it’s easy to put out a proof of concept that looks great at first glance, but the safety and efficacy is not well established,” he added. “This is one of those cases where diligent oversight is needed, and providing that really sets us apart in this space.”
The study was published in the New England Journal of Medicine: NEJM AI
Source: Dartmouth