Published benefits of ivermectin use in Itajaí, Brazil for COVID-19 infection, hospitalisation, and mortality are entirely explained by statistical artefacts
Robin Mills, Ana Carolina Peçanha Antonio, Greg Tucker-Kellogg
doi:10.1101/2023.08.10.23293924
Background Two recent publications by Kerr et al. (Cureus 14(1):e21272; Cureus 14(8): e28624) reported dramatic effects of prophylactic ivermectin use for both prevention of COVID-19 and reduction of COVID-19-related hospitalisation and mortality, including a dose-dependent effect of ivermectin prophylaxis. These papers have gained an unusually large public influence: they were incorporated into debates around COVID-19 policies and may have contributed to decreased trust in vaccine efficacy and public health authorities more broadly. Both studies were based on retrospective observational analysis of city-wide registry data from the city of Itajaí, Brazil from July-December 2020.
Methods Starting with initially identified sources of error, we conducted a revised statistical analysis of available data, including data made available with the original papers and public data from the Brazil Ministry of Health. We identified additional uncorrected sources of bias and errors from the original analysis, including incorrect subject exclusion and missing subjects, an enrolment time bias, and multiple sources of immortal time bias. In models assuming no actual effect from ivermectin use, we conducted Monte Carlo simulations to estimate the contribution of these biases to any observed effect.
Results Untreated statistical artefacts and methodological errors alone lead to dramatic apparent risk reduction associated with Ivermectin use in both studies. The magnitude of apparent risk reduction from these artefacts is comparable to the results reported by the studies themselves, including apparent protection from infection, hospitalisation, and death, and including the reported apparent dose-response relationship.
Conclusions The inference of ivermectin efficacy reported in both papers is unsupported, as the observed effects are entirely explained by untreated statistical artefacts and methodological errors. Our re-analysis calls for caution in interpreting highly publicised observational studies and highlights the importance of common sources of bias in clinical research.
Author contributions RM conducted the research required to uncover all fallacies mentioned in the manuscript (delayed registrations, missing data, incorrect inclusion of prior infections, all biases). GTK independently uncovered the immortal time bias, enrolment bias, and attrition bias. RM and GTK wrote the simulation and analysis code. GTK in particular, RM, and ACPA contributed to writing and reviewing the manuscript. ACPA acquired the data from the Brazilian Health Ministry. ACPA has been in frequent contact with Itajaí City Hall in an attempt to get access to missing raw data. RM has been in frequent contact with the KC22 and KB22 authors in an attempt to discuss the issues in their work before publishing this manuscript.
References
Adam, Journal says ivermectin study met standard for 'credible science'. Retraction Watch
Alein, Haro-Ramos, Frontline work and racial disparities in social and economic pandemic stressors during the first COVID-19 surge, Health Services Research
Aragon, epitools: Epidemiology tools. manual
Atif, Effects of glutamate and ivermectin on single glutamate-gated chloride channels of the parasitic nematode H. contortus, PLoS Pathogens
Bandt, Boen, A prevalent misconception about sample size, statistical significance, and clinical importance, Journal of Periodontology
Bordalo, Gennaioli, Shleifer, Salience Theory of Choice Under Risk, The Quarterly Journal of Economics
Bordalo, Gennaioli, Shleifer, Salience and consumer choice, Journal of Political Economy
Bramante, Randomized Trial of Metformin, Ivermectin, and Fluvoxamine for Covid-19, The New England Journal of Medicine
Buffet, Mendoza-Sassi, Fysekidis, Inaccurate Real-World Data Does Not Provide Real-World Answers, American Journal of Therapeutics
Caly, The FDA-approved drug ivermectin inhibits the replication of SARS-CoV-2 in vitro, Antiviral Research
Chee, Lim, Efficacy of Ivermectin Treatment on Disease Progression Among Adults With Mild to Moderate COVID-19 and Comorbidities: The I-TECH Randomized Clinical Trial, JAMA Internal Medicine
Da Silva, Factors Contributing to Exacerbating Vulnerabilities in Global Clinical Trials, Frontiers in Pharmacology
Daniel, Lackland, Impact of COVID-19 on Clinical Research and Inclusion of Diverse Populations, Ethnicity & Disease
Dornels Freire De Souza, Feliciano Do Carmo, Machado, The burden of COVID-19 in Brazil is greater in areas with high social deprivation, Journal of Travel Medicine
Drummond, Da, Influência da desigualdade socioeconômica na distribuição das internações e dos óbitos por covid-19 em municípios brasileiros, 2020: um estudo ecológico Influência da desigualdade socioeconômica na distribuição das internações e dos óbitos por covid-19 em municípios brasileiros, 2020: um estudo ecológico, Epidemiologia e servicos de saude : revista do Sistema Unico de Saude do Brasil
Flaxman, Estimating the effects of non-pharmaceutical interventions on COVID-19 in Europe, Nature
Furlan, Caramelli, The regrettable story of the "Covid Kit" and the "Early Treatment of Covid-19" in Brazil
Gelman, Loken, The statistical crisis in science, American Scientist
González, The Pharmacokinetics and Interactions of Ivermectin in Humans-A Mini-review, The AAPS Journal
Grasselli, Hospital-Acquired Infections in Critically Ill Patients With COVID-19, Chest
Greenland, Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations, European Journal of Epidemiology
Hill, Retraction to: Meta-analysis of Randomized Trials of Ivermectin to Treat SARS-CoV-2 Infection, Open Forum Infectious Diseases
Jaramillo, Clinical Trials Show Ivermectin Does Not Benefit COVID-19 Patients, Contrary to Social Media Claims
Keegan, Crown, Joseph, Diversity, Equity, and Inclusion in Clinical Trials, Surgical Oncology Clinics of North America
Kerr, Ivermectin Prophylaxis Used for COVID-19: A Citywide, Prospective, Observational Study of 223,128 Subjects Using Propensity Score Matching, Cureus
Kerr, Regular Use of Ivermectin as Prophylaxis for COVID-19 Led Up to a 92% Reduction in COVID-19 Mortality Rate in a Dose-Response Manner: Results of a Prospective Observational Study of a Strictly Controlled Population of 88,012 Subjects, Cureus
Kory, Review of the Emerging Evidence Demonstrating the Efficacy of Ivermectin in the Prophylaxis and Treatment of COVID-19, American Journal of Therapeutics
Laplante, Enrollment, retention, and strategies for including disadvantaged populations in randomized controlled trials: a systematic review protocol, Systematic Reviews
Lawrence, The lesson of ivermectin: meta-analyses based on summary data alone are inherently unreliable, Nature Medicine
Lee, Nunan, Immortal time bias
Leppink, Winston, Sullivan, Statistical significance does not imply a real effect, Perspectives on Medical Education
Long, Bui, Estimation of the incubation period of COVID-19 in Vietnam, PloS one
Martins-Filho, COVID-19 fatality rates related to social inequality in Northeast Brazil: a neighbourhood-level analysis, Journal of Travel Medicine
Men, Estimate the incubation period of coronavirus 2019 (COVID-19), Computers in biology and medicine
Meyerowitz-Katz, Unethical studies of ivermectin for covid-19, British Medical Journal Publishing Group Section: Editorial
Mueser, Granberg, The Monty Hall Dilemma Revisited: Understanding the Interaction of Problem Definition and Decision Making. Experimental
Naggie, Effect of Ivermectin vs Placebo on Time to Sustained Recovery in Outpatients With Mild to Moderate COVID-19: A Randomized Clinical Trial, JAMA
Nilima, Kumar, Moxidectin and Ivermectin Inhibit SARS-CoV-2 Replication in Vero E6 Cells but Not in Human Primary Bronchial Epithelial Cells, Antimicrobial Agents and Chemotherapy
Patrick, Walker, The impact of COVID-19 and strategies for mitigation and suppression in low-and middle-income countries, Science
Pennycook, Shifting attention to accuracy can reduce misinformation online, Nature
Peña-Silva, Pharmacokinetic considerations on the repurposing of ivermectin for treatment of COVID-19, British Journal of Clinical Pharmacology,
doi:10.1111/bcp.14476
Popp, Ivermectin for preventing and treating COVID-19, Cochrane Database of Systematic Reviews
Reardon, Flawed ivermectin preprint highlights challenges of COVID drug studies
Reis, Effect of Early Treatment with Ivermectin among Patients with Covid-19, Publisher: Massachusetts Medical Society,
doi:10.1056/NEJMoa2115869,null
Sackett, Bias in analytic research, Journal of Chronic Diseases
Saenen, Why Humans Fail in Solving the Monty Hall Dilemma: A Systematic Review
Sander Van Der Linden, Misinformation: susceptibility, spread, and interventions to immunize the public, Nature Medicine
Sheldrick, Meyerowitz-Katz, Tucker-Kellogg, Plausibility of Claimed Covid-19 Vaccine Efficacies by Age: A Simulation Study, American Journal of Therapeutics
Spencer, Heneghan, Chronological bias
Stacy, Uybico, Pavel, Gross, Recruiting Vulnerable Populations into Research: A Systematic Review of Recruitment Interventions, Journal of General Internal Medicine
Sundy, Yang, The broad spectrum antiviral ivermectin targets the host nuclear transport importin α/β1 heterodimer, Antiviral Research
Turner, Schaeffer ; William, Campbell, Mode of Action of Ivermectin
Tversky, Kahneman, Judgment under uncertainty: Heuristics and biases, Science
Wagstaff, Ivermectin is a specific inhibitor of importin α/β-mediated nuclear import able to inhibit replication of HIV-1 and dengue virus, Biochemical Journal