Large-scale external validation and comparison of prognostic models: an application to chronic obstructive pulmonary disease
Guerra B (1), Haile SR (1), Lamprecht B (2,3), Ramírez AS (4), Martinez-Camblor P (5), Kaiser B (6), Alfageme I (7), Almagro P (8), Casanova C (9), Esteban-González C (10), Soler-Cataluña JJ (11), de-Torres JP (12), Miravitlles M (13), Celli BR (14), Marin JM (15), Ter Riet G (16), Sobradillo P (17), Lange P (18), Garcia-Aymerich J (19), Antó JM (20), Turner AM (21), Han MK (22), Langhammer A (23), Leivseth L (24), Bakke P (25), Johannessen A (26), Oga T (27), Cosio B (28), Ancochea-Bermúdez J (29), Echazarreta A (30), Roche N (31), Burgel PR (32), Sin DD (33), Soriano JB (34,35), Puhan MA (36,37); 3CIA collaboration.
(1) Epidemiology, Biostatistics and Prevention Institute, University of Zurich, Zurich, Switzerland.
(2) Department of Pulmonary Medicine, Kepler Universitatsklinikum GmbH, Linz, Austria.
(3) Faculty of Medicine, Johannes Kepler Universitat Linz, Linz, Austria.
(4) Facultad de Medicina UASLP, Universidad Autonoma de San Luis Potosi, San Luis Potosi, Mexico.
(5) Dartmouth College Geisel School of Medicine, Dartmouth, NH, USA.
(6) Department of Pulmonary Medicine, Paracelsus Medizinische Privatuniversitat, Salzburg, Austria.
(7) Hospital Universitario de Valme, Sevilla, Spain.
(8) Internal Medicine, Hospital Universitario Mutua de Terrassa, Terrassa, Spain.
(9) Pulmonary Department and Research Unit, Hospital Universitario NS La Candelaria, Tenerife, Spain.
(10) Network and Health Services Research Chronic Diseases (REDISSEC), Hospital Galdakao, Bizkaia, Spain.
(11) Servicio de Neumología, Hospital Universitari Arnau de Vilanova, Lleida, Spain.
(12) Pulmonary Department, Clinica Universidad de Navarra, Pamplona, Spain.
(13) European Respiratory Society (ERS) Guidelines Director, Barcelona, Spain.
(14) Pulmonary and Critical Care Medicine, Brigham and Women's Hospital, Boston, MA, USA.
(15) IISAragón and CIBERES, Hospital Universitario Miguel Servet, Zaragoza, Spain.
(16) Department of General Practice, Academic Medical Center, University of Amsterdam, Amsterdam, The Netherlands.
(17) Hospital Univarsitario de Cruces, Barakaldo, Vizcaya, Spain.
(18) Department of Public Health, Section of Social Medicine, University of Copenhagen, Copenhagen, Denmark.
(19 ISGlobal, CIBER Epidemiología y Salud Pública (CIBERESP), Universitat Pompeu Fabra (UPF), Barcelona, Spain.
(20) ISGlobal, Centre for Research in Environmental Epidemiology (CREAL), IMIM (Hospital del Mar Medical Research Institute, Universitat Pompeu Fabra (UPF), CIBER Epidemiología y Salud Pública (CIBERESP), Barcelona, Spain.
(21) Institute of Applied Health Research, University of Birmingham, Birmingham, UK.
(22) Division of Pulmonary and Critical Care, University of Michigan, Ann Arbor, MI, USA.
(23 Department of Public Health and Nursing, Norvegian University of Science and Technology, Trondheim, Norway.
(24) Centre for Clinical Documentation and Evaluation, Northern Norway Regional Health Authority, Bodø, Norway.
(25) University of Bergen, Haukeland University Hospital, Bergen, Norway.
(26) Department of Global Public Health and Primary Care, University of Bergen, Bergen, Norway.
(27) Department of Respiratory Care and Sleep Control Medicine, Graduate School of Medicine, Kyoto University, Kyoto, Japan.
(28) Department of Respiratory Medicine, Hospital Son Espases-IdISBa-CIBERES, Palma de Mallorca, Spain.
(29) Instituto de Investigación Sanitaria Princesa (IISP)-Servicio de Neumología- Hospital Universitario de la Princesa, Universidad Autónoma de Madrid, Madrid, Spain.
(30) Universidad Nacional de la Plata, Hospital San Juan de Dios de La Plata, Buenos Aires, Argentina.
(31) Hopitaux Universitaires Paris Centre, Service de Pneumologie AP-HP, Paris, France.
(32) Hopital Cochin; Universite Paris Descartes, Paris, France.
(33) University of British Columbia, James Hogg Research Centre, Vancouver, Canada.
(34) Instituto de Investigación del Hospital Universitario de la Princesa (IISP), Universidad Autónoma de Madrid, Servicio de Neumología, Madrid, Spain.
(35) Scientific and Methodological Consultant of SEPAR www.separ.es, Barcelona, Spain.
(36) Epidemiology, Biostatistics and Prevention Institute, University of Zurich, Hirschengraben 84, Room HRS G29, CH -8001, Zurich, Switzerland.
(37) Epidemiology & Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Baltimore, MD, USA.
External validations and comparisons of prognostic models or scores are a prerequisite for their use in routine clinical care but are lacking in most medical fields including chronic obstructive pulmonary disease (COPD). Our aim was to externally validate and concurrently compare prognostic scores for 3-year all-cause mortality in mostly multimorbid patients with COPD.
We relied on 24 cohort studies of the COPD Cohorts Collaborative International Assessment consortium, corresponding to primary, secondary, and tertiary care in Europe, the Americas, and Japan. These studies include globally 15,762 patients with COPD (1871 deaths and 42,203 person years of follow-up).
We used network meta-analysis adapted to multiple score comparison (MSC), following a frequentist two-stage approach; thus, we were able to compare all scores in a single analytical framework accounting for correlations among scores within cohorts. We assessed transitivity, heterogeneity, and inconsistency and provided a performance ranking of the prognostic scores.
Depending on data availability, between two and nine prognostic scores could be calculated for each cohort. The BODE score (body mass index, airflow obstruction, dyspnea, and exercise capacity) had a median area under the curve (AUC) of 0.679 [1st quartile-3rd quartile = 0.655-0.733] across cohorts.
The ADO score (age, dyspnea, and airflow obstruction) showed the best performance for predicting mortality (difference AUCADO - AUCBODE = 0.015 [95% confidence interval (CI) = -0.002 to 0.032]; p = 0.08) followed by the updated BODE (AUCBODE updated - AUCBODE = 0.008 [95% CI = -0.005 to +0.022]; p = 0.23). The assumption of transitivity was not violated. Heterogeneity across direct comparisons was small, and we did not identify any local or global inconsistency.
Our analyses showed best discriminatory performance for the ADO and updated BODE scores in patients with COPD. A limitation to be addressed in future studies is the extension of MSC network meta-analysis to measures of calibration. MSC network meta-analysis can be applied to prognostic scores in any medical field to identify the best scores, possibly paving the way for stratified medicine, public health, and research.
CITATION BMC Med. 2018 Mar 2;16(1):33. doi: 10.1186/s12916-018-1013-y