The Center for Medicare/Medicaid Services (CMS) evaluates transplant centers based on risk-adjusted models produced by the Scientific Registry for Transplant Recipients (SRTR). The accuracy of these models in identifying the poor-performing centers is unknown.

In this Monte Carlo experiment, one-year mortality outcomes were simulated in virtual transplant centers, and used to flag centers according to the methods used by the CMS, evaluating nine overlapping 2.5-year periods of simulated data.

In a simulation where all centers had the same underlying risk, 10.2% were falsely flagged at least once during the 4.5 years of simulated evaluations. The probability of false-positive flagging was lowest in low-volume centers (2.5%) and highest in high-volume centers (16.2%).

In another simulation where 5% of centers were assigned 2-fold risk ("poor-performing centers"), only 32% of poor-performing centers were correctly flagged, and only 44% of flagged centers were actually poor-performing centers.

In a final simulation where each center was assigned a unique mortality risk, 94% of flagged centers had greater-than-median risk, but only 32% of flagged centers were among the 5% with highest risk.

Even after disregarding known covariate limitations to the risk adjustment models, statistical noise leads to spurious flagging of adequately-performing transplant centers, yet the methods used by CMS fail to flag most centers with actual elevated risk.

« Back to 2013 American Transplant Congress