Subject: Medical Education, Knowledge Assessment; Retrieval Augmented Generation; Mul>ple Choice ,Ques>ons; Medical Knowledge Base; Healthcare AI; Theore>cal Medicine
Year: 2025
Type: Journal Article
Title: RAGCare-QA: A Benchmark Dataset for Evaluating Retrieval-Augmented Generation Pipelines in Theoretical Medical Knowledge
Author: Dobreva, Jovana
Author: Karasmanakis, Ivana
Author: Ivanisevic, Filip
Author: Horvat, Tadej
Author: Gams, Matjaz
Author: Mishev, Kostadin
Author: Simjanoska Misheva, Monika
Abstract: The paper introduces RAGCare-QA, an extensive dataset of 420 theoretical medical knowledge questions for assessing Retrieval-Augmented Generation (RAG) pipelines in medical education and evaluation settings. The dataset includes one-choice-only questions from six medical specialties (Cardiology, Endocrinology, Gastroenterology, Family Medicine, Oncology, and Neurology) with three levels of complexity (Basic, Intermediate, and Advanced). Each question is accompanied by the best fit of RAG implementation complexity level, such as Basic RAG (315 questions, 75.0%), Multi-vector RAG (82 questions, 19.5%), and Graph-enhanced RAG (23 questions, 5.5%). The questions emphasize theoretical medical knowledge on fundamental concepts, pathophysiology, diagnostic criteria, and treatment principles important in medical education. The dataset is a useful tool for the assessment of RAG- based medical education systems, allowing researchers to fine-tune retrieval methods for various categories of theoretical medical knowledge questions.
Publisher: Cold Spring Harbor Laboratory Press
Relation: medRxiv
Identifier: oai:repository.ukim.mk:20.500.12188/34028
Identifier: http://hdl.handle.net/20.500.12188/34028
