<< Back to the abstract archive
Automating Clinical Guideline Generation with Artificial Intelligence: RAG-LLM Model Proof-of-Concept Study in Melanoma Surgery
Berk B Ozmen, MD; Nishant Singh, MSc; Kavach Shah, MSc; Ibrahim Berber, MSc; Joseph D Quick; Eugene Pinsky, PhD; Graham S Schwarz, MD, MSE, FACS
Cleveland Clinic Department of Plastic Surgery
2025-01-10
Presenter: Berk B Ozmen, MD
Affidavit:
Yes
Director Name: Graham S Schwarz, MD, MSE, FACS
Author Category: Fellow Plastic Surgery
Presentation Category: Clinical
Abstract Category: General Reconstruction
Background: Melanoma management requires intricate clinical decision-making informed by evolving guidelines. Current approaches to clinical guideline updates are labor-intensive and can lag behind new research insights. Retrieval-augmented generation (RAG) with large language models (LLMs) represents an innovative approach, offering potential for automated, evidence-based guideline generation directly from medical literature.
Methods: We developed a RAG-enhanced LLM, integrating open-access full-text articles referenced in the latest NCCN melanoma guidelines. A hierarchical document processing pipeline (RAPTOR) enabled the organization of literature into a ChromaDB vector store. The system was tasked with generating clinical guideline content for melanoma by answering 10 core queries on surgical margins, SLNB indications, molecular testing, and treatment of metastatic disease, then assessed for alignment with NCCN guidelines.
Results: The model effectively generated responses across all queries, demonstrating accuracy in synthesizing evidence-based, guideline-like recommendations. Recommendations closely aligned with NCCN's protocols, covering aspects such as SLNB criteria, treatment options for metastatic melanoma, and structured follow-up care.
Conclusion: This study demonstrates the feasibility of using RAG-enhanced LLMs to automate the generation of clinical guidelines from referenced literature. Such models could streamline guideline creation, providing timely updates directly from peer-reviewed literature.
