A critical analysis of Discovery Health’s claims-based risk adjustment of mortality rates in South African private sector hospitals
DOI:
https://doi.org/10.7196/SAMJ.2023.v113i1.16768Keywords:
AUROSCORE, Myocardial infarction, populationAbstract
In 2019, Discovery Health published a risk adjustment model to determine standardised mortality rates across South African private
hospital systems, with the aim of contributing towards quality improvement in the private healthcare sector. However, the model suffers from limitations due to its design and its reliance on administrative data. The publication’s aim of facilitating transparency is unfortunately undermined by shortcomings in reporting. When designing a risk prediction model, patient-proximate variables with a sound theoretical or proven association with the outcome of interest should be used. The addition of key condition-specific clinical data points at the time of hospital admission will dramatically improve model performance. Performance could be further improved by using summary risk prediction scores such as the EUROSCORE II for coronary artery bypass graft surgery or the GRACE risk score for acute coronary syndrome. In general, model reporting should conform to published reporting standards, and attempts should be made to test model validity by using sensitivity analyses. In particular, the limitations of machine learning prediction models should be understood, and these models should be appropriately developed, evaluated and reported.
References
Moodley Naidoo R, Timothy GA, Steenkamp L, Collie S, Greyling MJ. Measuring quality outcomes across hospital systems: Using a claims data model for risk adjustment of mortality rates. S Afr Med J 2019;109(5):299-305. https://doi.org/10.7196/SAMJ.2019.v109i5.13775
Biccard BM, Rodseth RN. Utility of clinical risk predictors for preoperative cardiovascular risk prediction. Br J Anaesth 2011;107(2):133-143. https://doi.org/10.1093/bja/aer194
Rodseth RN, Biccard BM, Le Manach Y, et al. The prognostic value of pre-operative and post-operative B-type natriuretic peptides in patients undergoing noncardiac surgery: B-type natriuretic peptide and N-terminal fragment of pro-B-type natriuretic peptide: A systematic review and individual patient data meta-analysis. J Am Coll Cardiol 2014;63(2):170-180. https://doi.org/10.1016/j.jacc.2013.08.1630
Leisman DE, Harhay MO, Lederer DJ, et al. Development and reporting of prediction models: Guidance for authors from editors of respiratory, sleep, and critical care journals. Crit Care Med 2020;48(5):623-633. https://doi.org/10.1097/CCM.0000000000004246
Collins GS, Reitsma JB, Altman DG, Moons KGM; members of the TRIPOD group. Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD Statement. Eur Urol 2015;67(6):1142-1151. https://doi.org/10.1016/j.eururo.2014.11.025
Shahian DM, Silverstein T, Lovett AF, Wolf RE, Normand SL. Comparison of clinical and administrative data sources for hospital coronary artery bypass graft surgery report cards. Circulation 2007;115(12):1518-1527. https://doi.org/10.1161/CIRCULATIONAHA.106.633008
Subramanian MP, Hu Y, Puri V, Kozower BD. Administrative versus clinical databases. J Thorac Cardiovasc Surg 2021;162(4):1173-1176. https://doi.org/10.1016/j.jtcvs.2020.03.183
Hanchate AD, Stolzmann KL, Rosen AK, et al. Does adding clinical data to administrative data improve agreement among hospital quality measures? Healthc (Amst) 2017;5(3):112-118. https://doi. org/10.1016/j.hjdsi.2016.10.001
Rhee C, Wang R, Song Y, et al. Risk adjustment for sepsis mortality to facilitate hospital comparisons using Centers for Disease Control and Prevention’s Adult Sepsis Event criteria and routine electronic clinical data. Crit Care Explor 2019;1(10):e0049. https://doi.org/10.1097/CCE.0000000000000049
Pope GC, Kautter J, Ingber MJ, Freeman S, Sekar R, Newhart CN. Evaluation of the CMS-HCC Risk Adjustment Model. https://www.cms.gov/Medicare/Health-Plans/MedicareAdvtgSpecRateStats/ downloads/evaluation_risk_adj_model_2011.pdf (accessed 5 August 2022).
Heathcote K, Devlin A, McKie E, et al. Rural and urban patterns of severe injuries and hospital mortality in Australia: An analysis of the Australia New Zealand Trauma Registry: 2015 - 2019. Injury 2022;53(6):1893-1903. https://doi.org/10.1016/j.injury.2022.03.044
FleetR,BussieresS,TounkaraFK,etal.Ruralversusurbanacademichospitalmortalityfollowingstroke in Canada. PLoS ONE 2018;13(1):e0191151. https://doi.org/10.1371/journal.pone.0191151
Ben-TovimD,WoodmanR,HarrisonJE,PointerS,HakendorfP.Measuringandreportingmortalityin hospital patients. Cat. No. HSE 69. Canberra: Australian Institute of Health and Welfare, 2009. https:// www.safetyandquality.gov.au/sites/default/files/migrated/Measuring-and-reporting-hospital-mortality- in-patients.pdf (accessed 5 August 2022).
Austin SR, Wong YN, Uzzo RG, Beck JR, Egleston BL. Why summary comorbidity measures such as the Charlson Comorbidity Index and Elixhauser score work. Med Care 2015;53(9):e65-e72. https://doi. org/10.1097/MLR.0b013e318297429c
Charlson ME, Carrozzino D, Guidi J, Patierno C. Charlson Comorbidity Index: A critical review of clinimetric properties. Psychother Psychosom 2022;91(1):8-35. https://doi.org/10.1159/000521288
SharmaN,SchwendimannR,EndrichO,AusserhoferD,SimonM.ComparingCharlsonandElixhauser
comorbidity indices with different weightings to predict in-hospital mortality: An analysis of national
inpatient data. BMC Health Serv Res 2021;21:13. https://doi.org/10.1186/s12913-020-05999-5
Chang HJ, Chen PC, Yang CC, Su YC, Lee CC. Comparison of Elixhauser and Charlson methods for predicting oral cancer survival. Medicine (Baltimore) 2016;95(7):e2861. https://doi.org/10.1097/
MD.0000000000002861
Zhang F, Chiu Y, Ensor J, Mohamed MO, Peat G, Mamas MA. Elixhauser outperformed Charlson comorbidity index in prognostic value after ACS: Insights from a national registry. J Clin Epidemiol 2022;141:26-35. https://doi.org/10.1016/j.jclinepi.2021.08.025
Johnston TC, Coory MD, Scott I, Duckett S. Should we add clinical variables to administrative data? The case of risk-adjusted case fatality rates after admission for acute myocardial infarction. Med Care 2007;45(12):1180-1185. https://doi.org/10.1097/MLR.0b013e318148477c
Aoyama D, Morishita T, Uzui H, et al. Sequential organ failure assessment score on admission predicts long-term mortality in acute heart failure patients. ESC Heart Fail 2020;7(1):244-252. https://doi. org/10.1002/ehf2.12563
Hernandez-Boussard T, Bozkurt S, Ioannidis JPA, Shah NH. MINIMAR (MINimum Information for Medical AI Reporting): Developing reporting standards for artificial intelligence in health care. J Am Med Inform Assoc 2020;27(12):2011-2015. https://doi.org/10.1093/jamia/ocaa088
Stevens LM, Mortazavi BJ, Deo RC, Curtis L, Kao DP. Recommendations for reporting machine learning analyses in clinical research. Circ Cardiovasc Qual Outcomes 2020;13(10):e006556. https://doi. org/10.1161/CIRCOUTCOMES.120.006556
Collins GS, Moons KGM. Reporting of artificial intelligence prediction models. Lancet 2019;393(10181):1577-1579. https://doi.org/10.1016/S0140-6736(19)30037-6
Hung PS, Lin PR, Hsu HH, Huang YC, Wu SH, Kor CT. Explainable machine learning-based risk prediction model for in-hospital mortality after continuous renal replacement therapy initiation. Diagnostics (Basel) 2022;12(6):1496. https://doi.org/10.3390/diagnostics12061496
Downloads
Published
Issue
Section
License
Copyright (c) 2022 R N Rodseth, D Smith, C Maslo, A Laubscher, L Thabane
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Licensing Information
The SAMJ is published under an Attribution-Non Commercial International Creative Commons Attribution (CC-BY-NC 4.0) License. Under this license, authors agree to make articles available to users, without permission or fees, for any lawful, non-commercial purpose. Users may read, copy, or re-use published content as long as the author and original place of publication are properly cited.
Exceptions to this license model is allowed for UKRI and research funded by organisations requiring that research be published open-access without embargo, under a CC-BY licence. As per the journals archiving policy, authors are permitted to self-archive the author-accepted manuscript (AAM) in a repository.
Publishing Rights
Authors grant the Publisher the exclusive right to publish, display, reproduce and/or distribute the Work in print and electronic format and in any medium known or hereafter developed, including for commercial use. The Author also agrees that the Publisher may retain in print or electronic format more than one copy of the Work for the purpose of preservation, security and back-up.
