Abstract

HALLUCINATIONS IN LARGE LANGUAGE MODELS: A CRITICAL THREAT TO MEDICATION SAFETY AND EVIDENCEBASED PHARMACY PRACTICE

Mr. Ishaan Pathak*, Mr. Om Giri, Ms. Kalyani Chande

Abstract

Background: LLMs have recently received considerable attention as potential applications within the realm of healthcare, notably pharmacy practice and databases of medications information. However, the utility of LLMs in a clinical context is severely limited due to the possible generation of factually untrue or invented output that is highly risky to patients' safety. Objectives: In this paper, there is a comprehensive analysis of the typology of hallucinations that are generated by LLMs, dangers associated with hallucinations in relation to medications, and the impact of hallucinations on evidence-based pharmacological data sets. Methods: Literature narrative synthesis has been performed on the subject, emphasizing the frequency of hallucinations, the methodology of their assessment for clinical safety, and risk mitigation strategies in AI-enabled clinical apps. Results: LLM hallucinations manifest themselves as facts-dissonant fabrications, inappropriate clinical advice, and significant omissions. It has been proven that LLMs generate fabricated medication instructions and disregard important aspects of patients' medical data, such as having an allergy to prescribed medication, when engaging in clinical reasoning and planning processes. Additionally, LLMs demonstrate serious deficiencies when synthesizing scientific evidence, making up references, and providing unsubstantiated therapeutic suggestions. These issues raise concerns over the ability of LLMs to make evidence-based pharmacological recommendations. Conclusion: The unsupervised application of LLMs within the field of pharmacy practice is highly dangerous from a clinical standpoint. To ensure the safe application of LLMs in pharmacy practice, a controlled approach needs to be employed.

Keywords: Large Language Models (LLMs), AI hallucinations, Clinical decision support, Medication errors, LLM Hallucinations.