Vascular Surgeon UC San Diego Health La Jolla, California
Objectives: Early diagnosis of peripheral artery disease (PAD) is crucial for preventing major adverse cardiovascular and limb events and optimizing disease management. This study evaluates generalizability of a machine learning PAD detection model trained on EHR data and tested across five health systems.
Methods: A total of 59,672 patients diagnosed with PAD after 2014, aged ≥40, and observed >30 days in the University of California (UC) health system were included in this retrospective study. Differences in gender, age, observation time, and comorbidities were analyzed. Kruskal-Wallis test was used to identify significant differences (p < 0.01) in continuous variables, followed by Dunn’s test (p < 0.01) for pairwise comparisons between facilities. Significant differences in comorbidities were assessed using Chi-square test (p < 0.01).
We performed case/control propensity matching to ensure comparable distributions for age, gender, diagnosis date, observation time, and lab value sparsity, with a 1:1 case-to-control ratio. Prediction date was defined 60 days prior to diagnosis. Features included demographics, lab values, comorbidities, counts of medication, diagnoses, and procedure codes across four observation periods (≤30, 30–90, 90–180, 180–360 days) prior to prediction date, resulting in a dataset with 13,514 features. A LightGBM classifier was trained and validated on data from 22,219 patients treated at UCSD, and performance was validated at four other UCs.
Results: Demographic differences across institutions are summarized in Table 1. Overall, hypertension was the most prevalent comorbidity (~80%). Using UCSD as reference, there were significant demographic and clinical differences across the UCs. UCSD had the highest prevalence of PAD patients (1.1%), and the oldest patient population (70±11 years) (p < 0.01). UCLA had the lowest proportion of white patients (54%) compared to UCSD (63%) (p < 0.01) and all other UCs. UC Davis and UCSF had populations with significantly lower prevalence of diabetes (45% and 43%, respectively) than UCSD (49%) (p < 0.01). UCLA had a significantly longer period of observation for patients (85 days more on average) and UC Davis had a significantly shorter observation period (294 days less, on average).
Our classifier developed at UCSD achieved an AUC of 0.97, F1 score of 0.92, and accuracy of 91%, evaluated at the optimal threshold of 0.53. The model demonstrated excellent generalizability when applied to the four other UC systems (Figure 1) with AUCs ranging from 0.93 (UC Davis) to 0.96 (UCLA).
Conclusions: This study highlights significant demographic differences across UC institutions and demonstrates that a PAD prediction model trained on UCSD data is generalizable.
