Artificial Intelligence Fails in Disease Diagnosis Through Patient Conversations

Despite the success of AI chatbots in medical professional exams, they still face challenges in one of the most critical tasks of physicians, which is diagnosing diseases through conversations with patients. New research shows that the accuracy of these models significantly decreases when interacting with simulated patients. According to a study conducted by researchers at Harvard University, AI models like GPT-4 from OpenAI performed notably well in multiple-choice medical exams with an accuracy of 82%, but their accuracy dropped to 26% when diagnosing diseases through conversations with simulated patients. Researchers used 2,000 medical cases, mostly extracted from the American Medical Board exams, to evaluate these models. In this process, the GPT-4 model played the role of the simulated patient and conversed with other AI models acting as doctors. The results of these conversations were also reviewed by medical experts. The findings showed that AI models were not only incapable of fully gathering the patient's medical information but also could not always provide correct diagnoses even when complete information was received. For instance, the GPT-4 model succeeded in gathering complete information in only 71% of conversations. Pranav Rajpurkar, the senior researcher of this study, stated, 'Real-world medical practice is much more complex and involves factors like managing multiple patients, coordinating with treatment teams, and understanding social and systemic factors.' According to him, AI can be an effective auxiliary tool in medicine but will not replace the comprehensive judgment of physicians.