The Sentinel System, a pivotal element of the US Food and Drug Administration’s (FDA) postmarketing safety surveillance, is crucial for evaluating drug safety post-approval. However, the observational data traditionally used in this system often face scrutiny due to inherent limitations. Recent advancements in large language models (LLMs) offer promising solutions to overcome these challenges. By leveraging LLMs, the FDA can potentially improve its methodologies for drug safety assessments, thereby enhancing market access and ensuring public health safety.
New Avenues for Signal-Identification
LLMs provide innovative methods for identifying novel adverse event signals from the narrative text found in electronic health records. This capability can be instrumental in supporting epidemiologic investigations to ascertain the causal links between medical product exposure and adverse events. The probabilistic phenotyping of health outcomes and extraction of critical confounding factors are areas where LLMs can significantly contribute, thus streamlining the process of market access by offering more reliable safety evaluations.
Challenges and Considerations
Despite the potential benefits, deploying LLMs in postmarket surveillance is not without challenges. Prompt engineering is vital to ensure the accuracy and specificity of LLM-derived associations. Moreover, the extensive infrastructure required for the effective use of LLMs is a barrier for many healthcare systems. This disparity can affect diversity, equity, and inclusion, potentially leading to overlooked adverse event patterns in less represented populations. Additionally, the propensity of LLMs to generate nonfactual statements poses a risk of false positives, necessitating rigorous downstream evaluation activities, which could incur substantial costs and affect market access strategies.
Concrete Inferences
Implications for Future Use:
- LLMs can enhance the precision of adverse event detection, thus improving the reliability of postmarket surveillance data.
- Infrastructure disparities must be addressed to ensure equitable access to LLM technologies across various healthcare systems.
- Careful prompt engineering is crucial to minimize the risk of false positive signals, ensuring cost-effective regulatory processes.
LLMs represent a significant paradigm shift in the realm of postmarket surveillance, offering the potential to generate unprecedented insights into drug safety. However, for LLMs to be effectively integrated into the FDA’s surveillance activities, substantial work is required to mitigate risks such as false positives and to ensure fair and equitable usage across diverse healthcare settings. By addressing these challenges, LLMs can play a pivotal role in enhancing the rigor of signal detection and supporting robust regulatory decisions, ultimately improving market access for safe and effective medical products.
Original Article:
JAMA Netw Open. 2024 Aug 1;7(8):e2428276. doi: 10.1001/jamanetworkopen.2024.28276.
ABSTRACT
IMPORTANCE: The Sentinel System is a key component of the US Food and Drug Administration (FDA) postmarketing safety surveillance commitment and uses clinical health care data to conduct analyses to inform drug labeling and safety communications, FDA advisory committee meetings, and other regulatory decisions. However, observational data are frequently deemed insufficient for reliable evaluation of safety concerns owing to limitations in underlying data or methodology. Advances in large language models (LLMs) provide new opportunities to address some of these limitations. However, careful consideration is necessary for how and where LLMs can be effectively deployed for these purposes.
OBSERVATIONS: LLMs may provide new avenues to support signal-identification activities to identify novel adverse event signals from narrative text of electronic health records. These algorithms may be used to support epidemiologic investigations examining the causal relationship between exposure to a medical product and an adverse event through development of probabilistic phenotyping of health outcomes of interest and extraction of information related to important confounding factors. LLMs may perform like traditional natural language processing tools by annotating text with controlled vocabularies with additional tailored training activities. LLMs offer opportunities for enhancing information extraction from adverse event reports, medical literature, and other biomedical knowledge sources. There are several challenges that must be considered when leveraging LLMs for postmarket surveillance. Prompt engineering is needed to ensure that LLM-extracted associations are accurate and specific. LLMs require extensive infrastructure to use, which many health care systems lack, and this can impact diversity, equity, and inclusion, and result in obscuring significant adverse event patterns in some populations. LLMs are known to generate nonfactual statements, which could lead to false positive signals and downstream evaluation activities by the FDA and other entities, incurring substantial cost.
CONCLUSIONS AND RELEVANCE: LLMs represent a novel paradigm that may facilitate generation of information to support medical product postmarket surveillance activities that have not been possible. However, additional work is required to ensure LLMs can be used in a fair and equitable manner, minimize false positive findings, and support the necessary rigor of signal detection needed for regulatory activities.
PMID:39150707 | DOI:10.1001/jamanetworkopen.2024.28276
This article has been prepared with the assistance of AI and reviewed by an editor. For more details, please refer to our Terms and Conditions. We do not accept any responsibility or liability for the accuracy, content, images, videos, licenses, completeness, legality, or reliability of the information contained in this article. If you have any complaints or copyright issues related to this article, kindly contact the author.
