Rates of False Flagging Due to Statistical Artifact in CMS Evaluations of Transplant Programs
Surgery, Johns Hopkins School of Medicine, Baltimore, MD
Meeting: 2013 American Transplant Congress
Abstract number: 306
The Center for Medicare/Medicaid Services (CMS) evaluates transplant centers based on risk-adjusted models produced by the Scientific Registry for Transplant Recipients (SRTR). The accuracy of these models in identifying the poor-performing centers is unknown.
In this Monte Carlo experiment, one-year mortality outcomes were simulated in virtual transplant centers, and used to flag centers according to the methods used by the CMS, evaluating nine overlapping 2.5-year periods of simulated data.
In a simulation where all centers had the same underlying risk, 10.2% were falsely flagged at least once during the 4.5 years of simulated evaluations. The probability of false-positive flagging was lowest in low-volume centers (2.5%) and highest in high-volume centers (16.2%).
In another simulation where 5% of centers were assigned 2-fold risk ("poor-performing centers"), only 32% of poor-performing centers were correctly flagged, and only 44% of flagged centers were actually poor-performing centers.
In a final simulation where each center was assigned a unique mortality risk, 94% of flagged centers had greater-than-median risk, but only 32% of flagged centers were among the 5% with highest risk.
Even after disregarding known covariate limitations to the risk adjustment models, statistical noise leads to spurious flagging of adequately-performing transplant centers, yet the methods used by CMS fail to flag most centers with actual elevated risk.
To cite this abstract in AMA style:
Massie A, Segev D. Rates of False Flagging Due to Statistical Artifact in CMS Evaluations of Transplant Programs [abstract]. Am J Transplant. 2013; 13 (suppl 5). https://atcmeetingabstracts.com/abstract/rates-of-false-flagging-due-to-statistical-artifact-in-cms-evaluations-of-transplant-programs/. Accessed October 10, 2024.« Back to 2013 American Transplant Congress