Artificial Intelligence (AI) continues to make waves in various industries, and healthcare is no exception. A recent study set in Iran in 2024-2025 examines the prowess of AI Large Language Models (LLMs) such as Bing AI Copilot and Gemini in comparison to human experts, focusing on healthcare policymaking. This study reveals AI’s potential to enhance decision-making processes, presenting data that juxtaposes the performance of AI with that of seasoned human professionals in a series of context-specific inquiries.
Methods of Analysis
The research incorporates a mixed-methods cross-sectional study design. Using confusion matrix analysis, researchers assessed responses from AI LLMs, including Bing AI Copilot and Gemini, against answers from 15 experienced human healthcare policy experts. This methodology enabled the calculation of multiple evaluation metrics, such as sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and overall accuracy, exposing comparative performance levels.
Performance Metrics
In terms of sensitivity, Bing AI Copilot boasted a figure of 0.867 with a specificity of 0, while its PPV and accuracy were 0.722 and 0.65, respectively. Gemini showed a sensitivity of 0.733 and a slightly better specificity of 0.4; it also exhibited a PPV of 0.786 and matched Copilot’s accuracy at 0.65. Human experts exhibited lower sensitivity (0.5808), specificity (0.2571), and accuracy (0.5050), depicting AI’s superior processing capability in the study context.
– AI’s analytical abilities in healthcare policymaking exceeded expectations.
– Bing AI Copilot and Gemini competently addressed complex, context-specific questions compared to human peers.
– Human experts still play a critical role, but AI serves as a strong complementary asset.
These findings emphasize the advantages AI LLMs offer in the realm of healthcare policy, highlighting their potential as indispensable tools when paired with human expertise. AI does not replace the nuanced insights of human professionals but augments their capacity to process vast and intricate data points efficiently. As AI technology continues to advance, its role is crucial not only in enhancing accuracy but also in crafting nuanced policy interventions that optimize healthcare outcomes globally.
This article has been prepared with the assistance of AI and reviewed by an editor. For more details, please refer to our Terms and Conditions. We do not accept any responsibility or liability for the accuracy, content, images, videos, licenses, completeness, legality, or reliability of the information contained in this article. If you have any complaints or copyright issues related to this article, kindly contact the author.
