A pioneering AI-assisted platform has been introduced to streamline the process of conducting systematic literature reviews for health technology assessment (HTA) submissions. This innovative system leverages a large language model (LLM) to assist researchers in various stages of the review process, potentially transforming how evidence is generated for HTA.
System Development and Methodology
The developed system comprises five distinct modules that utilize abstracts sourced from PubMed. These modules include setting up the literature search query, establishing study protocols based on PICOs criteria, assisting with abstract screening, extracting relevant data, and summarizing the collected information. The incorporation of a human-in-the-loop design allows researchers to adjust PICOs criteria in real-time by analyzing disagreements between the AI and human reviewers, thereby refining the inclusion and exclusion decisions dynamically.
Performance Evaluation
To assess the system’s efficacy, four evaluation sets focusing on relapsed and refractory multiple myeloma (RRMM) and advanced melanoma were utilized. The system demonstrated impressive performance metrics, achieving an average sensitivity of 90%, an F1 score of 82, and an accuracy of 89% during abstract screening. Additionally, it showed high reliability in providing exclusion rationales and data extraction, with F1 scores reaching up to 98 for certain categories.
- Enhances accuracy in abstract screening with high sensitivity and precision.
- Facilitates real-time adjustments to research criteria, improving review quality.
- Reduces the time and cost associated with manual literature reviews.
- Minimizes human errors, ensuring more reliable evidence generation.
The integration of GPT-4’s contextual learning capabilities eliminates the need for manually annotated training data, making the system more efficient and adaptable. Subject matter experts benefit from greater control over the review process through prompt adjustments and immediate feedback, allowing for iterative refinement based on performance metrics.
By automating key aspects of systematic literature reviews, this AI-driven system holds the promise of significantly reducing the resources required for HTA submissions. Researchers can achieve more comprehensive and accurate reviews in a fraction of the time traditionally needed, thereby accelerating the pace at which health technologies are assessed and implemented.
This advancement not only optimizes the review process but also enhances the overall quality of evidence generated for healthcare decisions. As the system continues to evolve, it is poised to become an indispensable tool for researchers and policymakers involved in health technology assessments, ultimately contributing to more informed and effective healthcare strategies.
This article has been prepared with the assistance of AI and reviewed by an editor. For more details, please refer to our Terms and Conditions. We do not accept any responsibility or liability for the accuracy, content, images, videos, licenses, completeness, legality, or reliability of the information contained in this article. If you have any complaints or copyright issues related to this article, kindly contact the author.
