Evaluating the Performance and External Validity of Machine Learning-Based Prediction Models in Liver Transplantation: An International Study
1University of Toronto - University Health Network, Toronto, ON, Canada, 2The Centre for Computational Medicine, The Hospital for Sick Children, Toronto, ON, Canada, 3Multi-organ transplant program, University Health Network, Toronto, ON, Canada, 4Department of Health Services Research and policy, London School of Hygiene and Tropical Medicine, London, United Kingdom, 5Division of Surgical Transplantation, University of Texas Southwestern Medical Center, Dallas, TX, 6Multi-organ transplant program, University of Toronto - University Health Network, Toronto, ON, Canada, 7Department of Health Services Research and Policy, London School of Hygiene and Tropical Medicine, London, United Kingdom, 8The Centre of Computational Medicine, The Hospital for Sick Children, Toronto, ON, Canada
Meeting: 2022 American Transplant Congress
Abstract number: 1799
Keywords: Liver transplantation, Prediction models
Topic: Clinical Science » Organ Inclusive » 72 - Machine Learning, Artificial Intelligence and Social Media in Transplantation
Session Information
Session Name: Machine Learning, Artificial Intelligence and Social Media in Transplantation
Session Type: Poster Abstract
Date: Tuesday, June 7, 2022
Session Time: 7:00pm-8:00pm
Presentation Time: 7:00pm-8:00pm
Location: Hynes Halls C & D
*Purpose: National liver transplant (LT) registries are curated in many countries. We compared data from three national registries and developed machine learning algorithm (MLA)-based models to predict post-LT mortality within and across different countries. Predictive performance and external validity of each model was assessed to contextualize the applicability of MLA in LT.
*Methods: We studied adult primary LTs between Jan-2008 and Dec-2018 from the US (n=59,558), the UK (n=5,287), and Canada (n=1,214). MLA models for 90-day post-LT mortality were built on each individual registry (based on variables inherent to the individual registry) and then using all 3 registries (based on harmonized variables). The models’ predictive abilities were evaluated across countries using area-under-the-receiver-operator-curve (AUROC) and area-under-the-precision-recall-curve (AUPRC).
*Results: ElasticNet was the best performing MLA model across both individual and harmonized registries. Model performance diminished from individualized to harmonized registries, especially in the UK (individualized ElasticNet:AUROC:0.54;Range:0.52-0.56;harmonized AUROC:0.48;Range:0.48-0.50) and the US (individualized ElasticNet:AUROC:0.70;Range:0.70-0.71;harmonized AUROC:0.65;Range:0.64-0.65). Model performance after external validation across countries was poor (Table 1).
*Conclusions: MLA-based models can be constructed using international LT registries, with independent ElasticNet models having optimal predictive performance. While MLA-based models yield fair discriminatory potential when used within individual registries, the external validity is poor when applied to different registries across countries.
To cite this abstract in AMA style:
Ivanics T, So D, Claasen MP, Wallace D, Patel M, Gravely A, Walker K, Cowling T, Erdman L, Sapisochin G. Evaluating the Performance and External Validity of Machine Learning-Based Prediction Models in Liver Transplantation: An International Study [abstract]. Am J Transplant. 2022; 22 (suppl 3). https://atcmeetingabstracts.com/abstract/evaluating-the-performance-and-external-validity-of-machine-learning-based-prediction-models-in-liver-transplantation-an-international-study/. Accessed November 21, 2024.« Back to 2022 American Transplant Congress