According to a new study published in the Journal of Medical Internet Research, researchers evaluated the accuracy of the ChatGPT chatbot's response to symptoms of five common bone diseases, including carpal tunnel syndrome, cervical myelopathy, and hip osteoarthritis.
Could it be used for medical diagnosis?
Over a five-day trial, researchers in the new study presented the same questions to ChatGPT, describing the symptoms and asking the chatbot about the disease.
The study found that ChatGPT was 100% accurate in diagnosing carpal tunnel syndrome, but its accuracy dropped to only 4% for cervical myelopathy. Worse still, the chatbot's answers regarding the same condition varied from person to person and were often inconsistent, which constitutes a flaw in the diagnostic method's validity.
The researchers also found that while nearly 80% of its responses recommended medical consultation, only 12.8% of those responses included a strong recommendation in direct language.
According to a statement released by Tokyo Medical and Dental University, it is crucial for any diagnostic tool to include clear language to encourage patients to seek expert medical opinions to confirm their diagnosis.
The Future of Medicine: According to a research article published in Nature Medicine last September, both doctors and researchers now agree that allowing chatbots to interact with patients in general is a reckless move that should be considered with caution. ANKARA, TURKIYE - SEPTEMBER 05: In this photo illustration, the ChatGPT logo is displayed on a mobile phone screen in front of a computer screen displaying pharmaceutical oral tablets, in Ankara, Turkey on September 5, 2023. (Photo by Harun Ozalp/Anadolu Agency via Getty Images)
Some companies are now seeking to design chatbots that go beyond diagnosis to treatment (Anadolu Agency).
According to this report, those working in the technology industry itself share the same concerns. Last March, more than 1,000 technology industry leaders issued an open letter calling for a temporary halt in the development of artificial intelligence, partly due to these risks.
However, specialists in these fields do not say that chatbots will be permanently ineffective. Rather, they require significant development and medical testing, the results of which should be published in peer-reviewed journals, to confirm their suitability for medical diagnosis.
In addition, some companies are now working to design chatbots that go beyond diagnosis to treatment, particularly in the realm of non-pharmacological psychotherapy. Chatbots can support patients with therapeutic guidance, but even in these cases, medical supervision remains essential.
All of this comes at a crucial time: 300 billion Google searches are conducted annually on medical topics. People often obtain their medical information and self-diagnose from search engines, which is already causing significant problems and making chatbots a new disruptive force in medicine.
What is ChatGPT?
ChatGPT is a large language model that enables general-purpose language understanding and generation. It acquires these capabilities by using massive amounts of data to learn in the same way humans learn language, but with a significant advantage for humans so far.
ChatGBT was launched in November 2022 as a chatbot built on the OpenAI family of large language models, and it quickly gained significant attention for its detailed responses in many areas of knowledge.