RS23 - Large language models accurately extract aortic information from abdominal imaging reports in a large, real-world database

Objectives: Maintaining robust surveillance programs for abdominal aortic aneurysms (AAA) is important, but these programs are expensive and labor-intensive, typically requiring manual data review by trained healthcare professionals. Studies have shown that natural language processing software can assist in these functions, but each task-specific algorithm requires human-directed training before use. Our objective was to evaluate the use of an open-source large language model (LLM) to extract AAA related data using generalized artificial intelligence, negating the need for task-based training.

Methods: This study examined ultrasound (US) and cross-sectional (CT,MR) abdominal imaging reports randomly selected for human review from a prospectively-maintained AAA surveillance registry within a large, integrated health system from 2008-2018. Llama 3.1 70B was utilized on a local Ollama server behind health system firewalls without external telemetry. Thus, there was no ability for data exchange outside of this system, protecting both sensitive patient information and health system intellectual property. The model extracted the maximal abdominal aortic diameter from each radiology report, or, if no diameter was stated, interpreted descriptive terms to determine whether a report was positive or negative for AAA, or if the aneurysmal status could not be determined (unknown). These data were compared to results extracted by independent expert reviewers for accuracy, sensitivity (recall), positive predictive value (PPV, precision), and F1-score.

Results: Among 8,131 human-adjudicated abdominal imaging studies (3,156 [39%] ultrasounds and 4,975 [61%] cross-sectional studies), the accuracy, sensitivity, precision, and F1-score of the LLM model were 0.90, 0.96, 0.94, and 0.95, respectively (Table I and Table II). 51.9% of reports included a discrete, numeric aortic diameter. At maximal AAA diameters between 3 and 7 cm, model sensitivity (recall) was 0.97-0.99. LLM accuracy was higher when analyzing US reports (0.94) vs. cross-sectional imaging (0.87, p< 0.001). Mean processing time was 27 seconds/US and 53 sec/cross-sectional imaging report.

Conclusions: LLM extraction of aortic information from abdominal imaging reports is exceptionally reliable without the need for additional human-directed training. Llama is open-source with no transactional processing costs, making this a cost-effective strategy to decrease the administrative burden of running complex AAA surveillance registries, with the added opportunity to improve the quality and efficiency of clinical research in the field. LLMs allow for flexible and efficient data mining at minimal cost or human effort. The potential impact of these technologies is immense.