Case Overview:

To address the frequent underdiagnosis and delayed recognition of hospital delirium—a condition linked to increased morbidity, mortality, and prolonged hospitalizations—our team at Mount Sinai Hospital developed and deployed an AI-powered risk stratification tool targeting adults aged 60 and older admitted to non-ICU settings.

We built a multimodal machine learning model that integrates structured electronic health record (EHR) data with natural language processing (NLP) of clinical notes to identify patients at high risk for developing delirium. The model generates daily risk scores, embedded within the EHR, to support clinical teams in prioritizing assessments. In live clinical practice, the model achieved an AUROC of 0.94 and drove a fourfold increase in delirium detection rates.

Importantly, this work contributed to a reduction in the prescribing of high-risk, deliriogenic medications—such as benzodiazepines and antipsychotics—by enabling more targeted, evidence-based interventions. By embedding AI into routine workflows, we enhanced early detection, optimized resource allocation, and improved medication safety for a vulnerable hospitalized population.

Tool and Project Details:

The tool is a multimodal machine learning model developed in-house at Mount Sinai Hospital to stratify delirium risk in real time for patients aged 60 and older in non-ICU settings. It fuses structured electronic health record (EHR) data—such as demographics, vital signs, labs, and medication history—with unstructured clinical notes processed through natural language processing (NLP). The model uses a vertically integrated approach, incorporating feedback from clinical, nursing, and data science teams throughout its development, deployment, and refinement.

The model was built and validated using data from over 32,000 hospital admissions and integrated directly into the Epic EHR system. Risk scores are displayed in daily patient lists with visual cues (e.g., color coding), and clinicians can access contextual information by hovering over the score. This functionality supports real-time decision-making and prioritization of Confusion Assessment Method (CAM) screenings.

No third-party vendors were used—this was a fully internal initiative leveraging Mount Sinai’s clinical data science and informatics infrastructure. The tool aligns with existing delirium services and care pathways, enhancing screening efficiency, supporting clinical decision-making, and reducing inappropriate prescribing of deliriogenic medications by guiding more targeted assessments and interventions.

Key Elements of Success:

This initiative succeeded through a vertically integrated, multidisciplinary approach that embedded AI directly into clinical workflows. Key elements included the co-development of the model by clinicians, nurses, data scientists, and informaticists, with strong support from hospital leadership and IT. The model was built to align with existing clinical priorities, specifically improving the early identification and treatment of delirium.

The project followed a rigorous development and evaluation process. Data from over 25,000 admissions were used to train and test the model, which was then prospectively validated in live clinical practice. The Confusion Assessment Method (CAM) was used as the diagnostic reference standard, and successive model iterations incorporated stakeholder feedback to improve performance and usability.

Stakeholders included clinical nursing teams (who performed CAM assessments), physicians, data scientists, digital strategy leadership, pharmacy, IT, and nursing informatics. No external vendors were involved. While no separate AI governance board was formed, continuous collaboration between operational leaders and clinical data science teams ensured responsible oversight. The project was approved by the Mount Sinai Institutional Review Board (IRB) as a quality improvement initiative, with a waiver of informed consent due to minimal risk. This collaborative model enabled rapid development, regulatory alignment, and sustainable deployment.

Impact on Outcomes:

Outcomes were substantial: delirium detection rates increased more than fourfold (from 4.4% to 17.2%), and daily doses of high-risk deliriogenic medications—including benzodiazepines and antipsychotics—were significantly reduced. These results reflect the power of AI-enabled decision support to improve patient safety and optimize care delivery. The JAMA publication (May 2025) included details both the methodology and results.

Role of the Pharmacy and Pharmacists:

Pharmacy played a critical role in the development and implementation of the delirium risk stratification model by guiding medication safety strategies and informing model features related to pharmacologic risk factors. Pharmacists collaborated with the clinical and data science teams to identify high-risk deliriogenic medications—such as benzodiazepines and antipsychotics—that were incorporated into both the model’s inputs and outcome measures.

Budget & Resource Allocation:

This initiative was internally funded and integrated as part of a broader institutional strategy to advance clinical decision support and patient safety through AI. It was not grant-funded, and no external vendors were used. Instead, existing resources within Mount Sinai’s clinical data science, pharmacy, nursing, and IT teams were leveraged to design, build, and operationalize the tool.

Return on investment (ROI) was justified through its demonstrated impact on clinical quality: a greater than fourfold increase in delirium detection rates and significant reductions in the use of high-risk medications such as benzodiazepines and antipsychotics. These improvements are expected to reduce downstream complications, such as prolonged hospital stays and readmissions, which carry high financial and patient safety burdens. By aligning with institutional goals around quality, safety, and efficiency, the program gained strong leadership support without requiring new capital investment.

Lessons Learned:

A major lesson from this initiative was recognizing the time and complexity involved in scaling AI tools in clinical practice. Integrating a machine learning model into real-world workflows—across diverse teams, systems, and clinical units—requires sustained effort, iterative refinement, and close coordination. While technical development can be rapid, true implementation success depends on deep alignment with clinical operations and cultural readiness.

The most critical success factor was our co-design approach. From the outset, nurses, pharmacists, physicians, data scientists, and informaticists worked together to build a tool that reflected real clinical needs and workflows. This ensured usability, trust, and long-term adoption.

We considered commercial AI tools but chose to build internally to retain flexibility and ensure seamless Epic integration. Going forward, we would avoid developing tools in isolation from operational planning. Our strongest champions were nursing and pharmacy leaders, as well as the delirium service team and digital transformation leadership, who prioritized patient safety and medication stewardship.

Biggest champions included nursing leadership, our delirium service team, pharmacy leaders, and institutional digital strategy stakeholders, all of whom advocated for the tool’s impact on patient care and medication safety.

Future Goals & Ongoing Monitoring:

Our next goal is to expand the delirium risk stratification model to additional hospitals within the Mount Sinai Health System. We are also exploring integration with additional data sources and enhancements to the model’s interpretability to further support clinical decision-making.

Ongoing monitoring is supported through a real-time dashboard that tracks model performance, user adoption, and clinical outcomes such as delirium detection rates and medication usage. We continue to review these metrics in partnership with clinical and operational leaders to ensure the model remains accurate, actionable, and aligned with care delivery goals.

Maintenance is overseen by our clinical data science and informatics teams, with workflows in place to support model retraining, stakeholder feedback integration, and EHR updates. Nurse assessors, pharmacists, and IT analysts are core to this infrastructure.

The findings have been published in JAMA Network Open and presented internally, with plans to share at national forums focused on geriatrics, pharmacy, and applied clinical AI. This initiative serves as a model for how AI can be responsibly deployed to improve patient safety and reduce medication-related harm.