The uses and abuses of systematic reviews
Analysis of defects in the Popp et al. meta analysis.
Fordham et al., 7 Oct 2021, preprint, 4 authors.
Abstract: The uses and abuses of systematic reviews:
the case of ivermectin in Covid-19
*Edmund J Fordham PhD FInstP EurIng
Theresa A Lawrie MBBCh PhD
Katherine MacGilchrist MSc
EbMCsquared, a Community Interest Company
Northgate House, Upper Borough Walls, Bath BA1 1RG, UK
Andrew Bryant MSc
Population Health Sciences Institute, Newcastle University
Baddiley-Clark Building, Richardson Road, Newcastle upon Tyne NE2 4AX, UK
Address: EbMCsquared CIC
Northgate House, Upper Borough Walls,
Bath BA1 1RG, UK
2839 words excl Refs
2 Tables; 2 Figures
A recent Cochrane review1 of ivermectin for the treatment and prevention of Covid-19
stands in contrast to the world-first PRISMA guidelines2 systematic review and meta-analysis
by Bryant et al.3 (hereinafter “Bryant”) , which is criticised in the review with further
commentary elsewhere4. We review here the approach of Popp et al.1 (hereinafter “Popp”).
Ivermectin5, a WHO Essential Medicine6 of negligible cost7 and outstanding safety
record8-13 has attracted significant attention for its potential use both in treatment and in
the prevention of Covid-19, from its known anti-viral effect over a wide class of RNA
viruses14 and specific action15 in vitro against SARS-CoV-2. These initial reports were
bolstered by multiple reports16 of efficacy at all stages of the Covid-19 illness (from
prophylaxis through to critical care17), in vitro18 and in silico19,20,21 studies of its likely modes
of action, clinical experience reports from multiple contexts22, population-level case
studies23, retrospective observational studies24, through to randomised controlled clinical
trials3 (RCTs). These reports occupy all levels of the so-called “Evidence Quality Pyramid” in
the doctrines of Evidence-Based Medicine.
Systematic Reviews (SRs) and Meta-Analyses (MAs) occupy, in theory, the summit of the
pyramid, and are considered the premier quality of evidence by Health Regulatory
Authorities (HRAs) and for clinical practice guidelines by national and international health
services. However, though SRs and MAs are carried out to identify, qualify or avoid the
worst aspects of reporting bias, excluding all other evidence can become itself a powerful
source of bias.
The Cochrane organisation25 is dedicated to the development of a library of SRs,
confined to the evidence. Cochrane SRs are traditionally conducted by volunteer reviewers,
reducing the potential for financial conflicts of interest. Advocacy is not properly part of a SR
but of processes such as DECIDE26 in which other stakeholders including public health
professionals and patients should be involved. Cochrane has a tradition of accuracy,
objectivity and attention to reliability through systematised but necessarily subjective
criteria. Similar criteria are more generally available through the guidelines of the PRISMA
Working Group2, followed in Bryant3.
However, SR methodologies are capable of abuse in respect of (i) choice of inclusion
criteria (ii) choice of outcome measures, comparisons and analyses (iii) the subjectivity
inevitable in Risk of Bias assessments, even when codified procedures and criteria are
observed. We analyse these choices in Popp1.
The Cochrane Review of Popp1: synopsis
The stated objectives are to assess efficacy and safety of ivermectin compared to: no
treatment, placebo, “standard of care” (SoC) or any “proven” intervention. The analyses
compare hospitalised patients and..
Please send us corrections, updates, or comments. Vaccines and
treatments are complementary. All practical, effective, and safe means should
be used based on risk/benefit analysis. No treatment, vaccine, or intervention
is 100% available and effective for all current and future variants. We do not
provide medical advice. Before taking any medication, consult a qualified
physician who can provide personalized advice and details of risks and
benefits based on your medical history and situation. FLCCC
provide treatment protocols.