View Abstract

A6063 - Understanding the Reproducibility of Randomized Controlled Trials in Critical Care: A Systematic Review
Author Block: D. Niven1, J. McCormick2, T. Barnes1, K. Fiest1, S. Straus3, B. Hemmelgarn4, L. Jeffs5, H. T. Stelfox1; 1Department of Critical Care Medicine, University of Calgary, Calgary, AB, Canada, 2Department of Anesthesiology and Pain Medicine, University of Ottawa, Ottawa, ON, Canada, 3Department of Medicine, University of Toronto, Toronto, ON, Canada, 4Department of Medicine, University of Calgary, Calgary, AB, Canada, 5Li Ka Shing Knowledge Institute of St. Michael’s Hospital, University of Toronto, Toronto, ON, Canada.
Rationale: Recent randomized controlled trials (RCTs) suggest that the ability to reproduce clinical experiments may be a problem in critical care research. This study systematically examined the reproducibility of high-profile, late-phase (III or IV) RCTs in critical care to determine factors that predict whether an RCT will be reproduced and whether the results will be consistent with the original RCT. Methods: Systematic review methodology defined the target cohort of RCTs. To identify relevant clinical practices, we searched the New England Journal of Medicine, The Lancet, and JAMA to April 2016. To identify a comprehensive set of studies for these practices, included articles informed secondary searches within other high-impact clinical journals. We included late-phase RCTs examining a diagnostic or therapeutic clinical practice in adults admitted to an intensive care unit. Included articles were classified using a reproducibility framework; an original RCT was the first to evaluate a clinical practice, and a reproduction attempt re-evaluated that practice in a new set of participants. RCTs were assessed using the Cochrane Risk of bias tool for RCTs. Multivariable logistic regression was used to determine predictors of reproducibility. Results: We identified 294 RCTs that examined 167 clinical critical care practices. A reproduction attempt was identified for 70 practices (42%, 95% CI 34%-50%). Less than half of clinical practices with a reproduction attempt demonstrated effects consistent with the original RCT (43%, 95% CI 31%-55%). Multivariable predictors of a reproduction attempt included time elapsed since original RCT publication, benefit reported for a secondary outcome, Web of Science citations, and several study methodology features. Predictors of a reproduction attempt reporting effects consistent with the original RCT included an interaction between characteristics of the original RCT, namely sample size and primary outcome result (efficacy vs lack of efficacy/harm). For original RCTs with under 200 patients, the effect reported for the primary outcome did not effect the likelihood of a consistent reproduction attempt. For original RCTs with more than 200 patients, a consistent reproduction attempt was more likely if the original RCT reported lack of efficacy/harm (odds ratio 34.9, 95% CI 1.5-792.1, p=0.03), and less likely if the original RCT reported efficacy (odds ratio 0.003, 95% CI 0.00003-0.34, p=0.02). Conclusions: Less than one-quarter of clinical critical care practices examined in high-profile RCTs have reproducible effects. Combinations of study characteristics can predict whether a reproduction attempt will take place, and whether the results will be consistent with the original RCT.