Detailed analysis due to inaccurate consequential press
RCT low-risk outpatients with very late treatment (median 6
days, 25% ≥8 days) in the USA, showing 98% probability of efficacy for
clinical progression at day 14, a treatment delay-response relationship, and
significant efficacy for patients with severe symptoms at baseline. The
posterior probability ivermectin is effective was 99%, 98%, 97% for mean time
unwell and clinical progression @14 and 7 days. All exceed the pre-specified
threshold for superiority
Note that the clinical progression results exceeding the superiority
threshold in the preprint
in the journal version, with no explanation for over 48 days).
There are many critical issues as below. Design, presentation,
and analysis shows a strong negative bias. While authors recommend and are
performing further study, they appear to be continuing with almost all of the
critical flaws below. Submit
Updates or Corrections
Superiority found, not reported.
Day 7 and day 14 clinical progression results and mean time unwell show superiority of ivermectin (note: preprint version [medrxiv.org]
, clinical progression results were changed without explanation in the journal version). The protocol indicates superiority for OR < 0.9 and posterior probability > 0.95 [fnih.org]
. In the presentation [rethinkingclinicaltrials.org]
, author shows a slide containing these results while stating "this was, um, not statistically significant"
(@22:36). These results were seen despite 107 patients having no symptoms at baseline and the use of the skeptical prior [doyourownresearch.substack.com]
Death reported in mITT, however participant not in mITT, did not receive study drug.
Authors report one death in the ivermectin arm in the mITT population (817 patients that received the drug within 7 days), however in the presentation [rethinkingclinicaltrials.org]
(@21:15) author reports that the patient that died did not receive the drug because they were admitted prior to receipt. The reported mITT death is therefore incorrect. Similarly, at least one and potentially many or all hospitalizations may have occurred before receiving the drug.
Clinical progression results changed.
The preprint [medrxiv.org]
and journal version show very different clinical progression results, with no acknowledgement or explanation.
Primary outcome not reported, closest reported outcome shows superiority.
The protocol shows the primary symptom outcome using a longitudinal statistical model with an ordinal variable based on symptom count and hospitalization/death measured daily until day 14 [fnih.org]
(see section 10.6.1). This outcome is not reported. The closest reported outcome is clinical progression at 14 days, which shows superiority of ivermectin, OR 0.73 [0.52-0.98], posterior probability of efficacy 98%, which exceeds the pre-specified threshold for superiority (note: changed without explanation as above).
Pre-specified primary 14 day outcomes not reported, clinical status shows 30% benefit.
The pre-specified primary 14 day outcomes [fnih.org]
are still not reported in the journal version. However, authors now show the clinical status graphs in eFigure 2, which shows 30% benefit for ivermectin for limited activity at 14 days.
Results show 10 and 9 events for hospitalization/death, however eFigure 1A shows 4 and 3.
90 day followup results not provided.
Authors do not provide the PROMIS-29 results, stating that this is due to the 90 day followup. The 90 day followup period ended 169 days before publication (38 days before the preprint publication). The protocol also specifies 7, 14, and 28 day PROMIS-29 results.
Very late treatment.
Patients were treated a median of 6 days late, with 25+% 8+ days late. Extensive research for COVID-19 and other viral diseases show that early antiviral treatment is critical. While authors recommend (and are performing) further study, they do not recommend or perform the obvious step of doing an early treatment trial, as is done for NIH recommended treatments like Paxlovid, suggesting a strong negative bias with a goal of maintaining late treatment and obtaining poor results.
Key clinical question consistent with unreported pre-specified primary outcome but not the reported outcome.
Authors report the key clinical questions for ACTIV-6 as being "How to help someone feel better faster with newly diagnosed mild-moderate COVID-19?" and "How to prevent hospitalizations or death in someone with newly diagnosed mild-moderate COVID-19?" [rethinkingclinicaltrials.org]
. The pre-specified but unreported primary symptom outcome provides a measure for feeling better faster, however the reported post-hoc primary outcome is very poorly matched. For example, a treatment that resolves serious symptoms 10x faster, but does not speed up 100% resolution of cough for three consecutive days, would show zero benefit in the post-hoc primary outcome. Cough may persist long after viral clearance [bmjopenrespres.bmj.com]
. Note also that the trial does not address "newly diagnosed" patients, but rather very late treatment a median of 6 days after symptoms. The very late treatment also minimizes the chance of preventing the initiation of viral cough.
Patients with symptoms >7 days included.
The trial specifies symptoms ≤7 days, however subgroup results show symptoms ≤9, 11, and 13 days, and the Q3 for the ivermectin arm was 8 days, indicating 25% of patients with a treatment delay of ≥8 days. The difference is likely due to the authors not considering receipt of medication or treatment time in inclusion, i.e., due to shipping delays. However, ≤7 days treatment delay already makes the results inapplicable to real-world usage where antivirals are used early.
Data unavailable over 179 days from publication.
Data for the study is unavailable over 179 days after publication.
Outcomes reported do not match protocol.
The reported outcomes are very different to the trial registration [clinicaltrials.gov]
and the pre-specified protocol [fnih.org]
. The trial registration shows three primary outcomes, of which zero are reported in the paper. The pre-specified protocol shows the primary outcome using a longitudinal statistical model with an ordinal variable based on symptom count and hospitalization/death measured daily until day 14.
Primary outcomes changed after publication.
The primary outcomes were changed from day 14 to day 28 on June 25, 2022, after publication [clinicaltrials.gov (B)]
. Two of the three primary outcomes were changed to match what was reported, while the third remains unreported, and none of the pre-specified primary outcomes have been reported to date.
Different hospitalization/urgent care numbers between paper and presentation.
The paper and the later presentation show different numbers for hospitalization and urgent care/ER. In the presentation, a hospitalization was moved from the placebo arm to the ivermectin arm [rethinkingclinicaltrials.org]
(@21:44). The HRs did not change.
Post-hoc primary outcome measured on day 3.
The new primary outcome of sustained 100% recovery for 3 days is measured on day 3 rather than day 1 [rethinkingclinicaltrials.org]
(@19:10). We are unaware of any reason to use day 3 rather than the day of 100% recovery, other than to reduce the observed efficacy. Both the pre-specified [fnih.org]
and post-hoc protocols include a secondary outcome of "symptom resolution, defined as first of at least three consecutive days without symptoms"
A post-hoc protocol is included with the paper published October 21, 2022 which differs significantly from the pre-specified protocol [fnih.org]
. The post-hoc protocol is dated December 20, 2021, after both interim analyses. No confirmation of existence prior to October 21, 2022 has been found. Metadata shows the author for both protocol files to be Jenny Jackman.
Dose below 400μg/kg.
The abstract states that patients received 400μg/kg. This is incorrect, section 16.3.3 in the protocol shows that the actual dose was always below 400μg/kg, unless patients weighed exactly 35kg or 70kg, as shown below [doyourownresearch.substack.com]
. Dosage was as low as 269μg/kg for 52kg. When considering administration as below, the average dose administered is equivalent to ~130μg/kg as used in practice. Authors state: "ivermectin was dosed by weight to achieve a goal dose of 400μg/kg, but the maximum dose of ivermectin provided by the study was 35mg. While almost 42% of participants had a weight of more than 88kg and thus did not achieve the goal dose, more than 75% of participants had a weight of less than 100kg and so received at least 90% of the target dose"
. This is incorrect - the goal dose varied between ~300-400μg/kg, and the percentage that received 90+% of 400μg/kg is likely < 40%.
Effective dose ~130μg/kg, administration on empty stomach.
Authors instructed patients to take ivermectin on an empty stomach (not done for fluvoxamine). [Guzzo]
show that the plasma concentration of ivermectin is much higher when administered with food (geometric mean AUC 2.6 times higher). "Ivermectin should be taken on an empty stomach with water"
(protocol section 16.3.3). This is not mentioned in the paper or the supplementary appendix, only in the protocol. This makes the average effective dose administered equivalent to ~130μg/kg for administration with a meal (as used in clinical protocols).
Clinical progression details for fluticasone/fluvoxamine but not ivermectin.
Authors provide clinical progression details for fluticasone (Table S3, Figure S5) and fluvoxamine (eFigure 3), but not for ivermectin. This may be related to authors not reporting any of the pre-specified primary outcomes — the same table would reveal 2 of the 3 pre-specifed primary outcome results.
COVID-19 mortality/hospitalization not reported.
Authors only report all-cause mortality and hospitalization. Notably, the baseline hospitalization and mortality rate for non-COVID-19 causes may account for the death and many of the hospitalizations. This may also explain why authors report only 28 day mortality/hospitalization in violation of the protocol where the primary outcomes specify 14 days [clinicaltrials.gov]
. Additionally, adverse events show only one case of aggravated COVID-19 pneumonia for ivermectin, versus 3 for placebo.
Many pre-specified outcomes missing.
Authors do not report [fnih.org]
•OR describing the overall difference in symptoms and clinical events over 14 days (primary outcome)
•Overall clinical progression OR (only specific day 7, 14, 28 values are provided)
•Time to first urgent care, emergency care, hospitalization or death
•Risk and time to event for each component of the composite
•Mean and median time to symptom freedom
•Overall QOL OR
•Day 7, 14, 28, 90 QOL OR
•Mean difference in QOL scores at day 7, 14, 28, 90
•Mean and median time to symptom resolution (only a new sustained resolution measure is reported, which is not in the protocol)
•Day 90 mean and median symptom count
Full protocol unavailable before October 2022.
The pre-specified protocol [fnih.org]
is missing the appendix which includes contraindications, exclusions, formulation, appearance, packaging, dispensing, dosing, and dose rationale.
IDMC not independent.
The IDMC vice chair was reportedly on the NIH panel that did not recommend treatment despite strong evidence, and provided no quantitative analysis, no reference to the majority of the research, and no updates for new research for a very long period [ivmmeta.com]
Reported primary outcome low relevance.
The reported primary outcome (which matches neither the trial registration or the protocol) is of relatively low relevance being based on sustained absence of all symptoms, where symptoms includes many things that may be found after viral clearance and may be unrelated to COVID-19, including fatigue, headache, and cough (which may remain for some time). Authors may have searched for the outcome that shows the least benefit. The 3-day sustained definition further adds two days for all participants, reducing efficacy. Authors should report data for more significant symptoms such as dyspnea, fever, and loss of sense of taste/smell.
Shipping and PCR delays largely enforce late treatment.
Authors required positive PCR before randomization, and shipped medication to participants. The delay before PCR results become positive, delay in receiving PCR results, and the shipping delay largely ensure that patients will not be treated early.
Very slow shipping.
While one day or faster shipping should have been possible (~$11,000 funding per patient [doyourownresearch.substack.com (B)]
), the shipping delays in this trial appear to be very long based on the ≤7 days inclusion criterion and subgroup analysis up to 13 days. One participant in the ivm-600 arm shared their experience showing 6 days from signing up until arrival of the medication, resulting in a total of 11 days treatment delay [doyourownresearch.substack.com (C)]
. COVID-19 is an acute disease (which may or may not be mild). Participants cannot be expected to wait 1-2 days or longer for treatment. Chances are that patients feel better by the time medication arrives and do not take the medication, which may explain why adherence is not reported, or their condition became worse and they found alternative immediate care elsewhere.
The placebo arm included multiple regimens matching different treatment arms, hence some participants will know they are not in the ivermectin arm, and others will know that there is a higher probability of them being in the ivermectin arm than the placebo arm. This may be more important given the politicization in the study country. The fluticasone arm and matching placebo use an inhaler, the fluvoxamine arm uses 10 days treatment. Matched placebo analysis should be provided.
Extreme conflicts of interest.
This trial has extreme conflicts of interest, being funded by an organization that chose not to recommend treatment while providing no quantitative analysis, no reference to the majority of the research, and no updates for new research for a very long period [ivmmeta.com]
. Further, a majority of the panel providing the recommendation has major conflicts of interest [ivmmeta.com]
. Also see [trialsitenews.com (B), trialsitenews.com (C)]
. The ACTIV executive committee was chaired by employees of J&J and NIH, and is now chaired by employees of Pfizer and NIH. Other members of the committee are from NIAID (Dr. Fauci), FDA, and Pfizer [nih.gov, vimeo.com]
Treatment delay-response relationship.
Subgroup results for treatment delays 13, 11, 9, 7, and 5 show monotonically improving results (less than 1% probability due to chance). ≤3 days may have very few patients, and is within confidence limits for monotonically improving results. Improved efficacy for earlier treatment matches extensive results for ivermectin and other COVID-19 treatments [c19early.org]
, however authors ignore this trend, claiming only a lack of statistical significance for one specific binary threshold (which may have few patients on one side), and authors have not initiated an early treatment trial.
Asymptomatic patients included.
Study inclusion required >2 symptoms, however the subgroup analysis includes 109 patients with no symptoms, where results favored placebo. The primary outcome may reach statistical significance without these patients.
The conclusion states that treatment did not lower mortality of hospitalization, however it is impossible to lower zero mortality. While authors do not indicate COVID-19 versus other hospitalization, statistically significant reduction in hospitalization would require at minimum 79% efficacy, but for COVID-19 hospitalization it is likely impossible based on expected non-COVID-19 hospitalizations. The trial is underpowered by design due to selection of a low-risk population. Note that among the 90 severe cases, statistically significant efficacy is reported.
Significant missing data, not mentioned in paper.
The paper does not mention missing data, however in the presentation [rethinkingclinicaltrials.org]
(@44:20) authors report close to 10% missing survey data. One author indicates there was less then 10% missing survey data through day 14. However, the presentation also shows clinical progression graphs (@22:10) that contradict this, showing 650/614 patients at day 14, which is over 20% missing data.
Statistically significant efficacy for severe patients removed.
The statistically significant HR 1.86 [1.10-3.16] efficacy for severe patients at baseline (using the post-hoc primary outcome) was noted in the text of the preprint [medrxiv.org]
, but has been deleted in the journal version (only seen in the appendix eFigure 3).
Statistical analysis plan dated after trial end.
The statistical analysis plan included with the journal paper is dated after the end of the trial.
31% more severe cases in the ivermectin arm.
There were 31% more severe cases in the ivermectin arm at baseline (39 control vs. 51 ivermectin).
The visual abstract reports the population as patients experiencing two or more symptoms for 7 days or less. This is incorrect and refers to the time of enrollment, not the time of intervention. The study actually includes a subgroup of patients 13 days from onset, and 107 patients that had no symptoms at baseline (eFigure 3).
Early treatment incorrect.
The abstract and paper claim to study "treatment of early mild to moderate COVID-19."
This is incorrect, treatment was very late, median 6 days, 25+% 8+ days, and with subgroup results up to 13 days. For influenza and oseltamivir or baloxavir, treatment is typically considered early within 24-48 hours [c19ivm.org]
Author claims results from 569 researchers should be censored for false information.
58 studies by 569 scientists report statistically significant positive results for ivermectin treatment of COVID-19 [ivmmeta.com (B)]
. One author claimed that a report of positive results is "disinformation" and distributed a request to report and censor the author [, , twitter.com (C)]
. While discussion is warranted for all studies, a call for censorship of results is extreme and raises questions. Author provides no basis for the results of the 569 scientists being wrong and warranting of censorship, and there is no indication that author has even read most of the studies. Author cherry-picked two of 93 studies, (COVID-OUT and ACTIV-6 [Bramante, Naggie]
, both very high COI studies with an extensive list of issues and very delayed treatment) and claimed that "no benefit of ivermectin was observed" 
. In addition to ignoring the 58 studies reporting statistically significant positive results, ACTIV-6 [c19ivm.org (B)]
reported a posterior probability that ivermectin is effective of 99%, 98%, and 97% for mean time unwell, clinical progression @14 days, and clinical progression @7 days (even though none of the pre-specified primary outcomes were reported, and noting that these preprint results were changed without explanation), and COVID-OUT showed 61% lower hospitalization with ivermectin vs. placebo (not including metformin), although this was not reported.
The treatment and control groups were drawn from different populations. Patients were allowed to select which medication they would like to test, while the control group contains patients assigned to other medications, some of which specifically requested that medication [doyourownresearch.substack.com (D)]
. Additionally, drug-specific exclusions further modify the populations.
Low risk patients.
Authors focus on patients at low risk of COVID-19 severe outcomes, which ensures an underpowered trial, with only one death which may not be due to COVID-19. All-cause mortality and hospitalization become less meaningful, with a significant contribution from non-COVID-19 causes.
No adherence data or per-protocol analysis.
Authors provide no adherence data. Non-adherence may de-power the trial and may harm randomization.
Subject to participant fraud.
The self-reported design, partial blinding, and absence of professional medical examination opens the trial to participant fraud, which may be significant due to extreme politicization in the study country.
Skeptical prior not justified.
The skeptical prior, which reduces the observed efficacy in the post-hoc primary outcome, is not justified based on the studies to date. The skeptical prior was pre-specified. Authors may argue that the prior is justified because the trial was designed to avoid showing efficacy.
Not enough tablets provided.
Participants were supplied 15 7mg tablets and instructed to take the number of tablets to approximate 400μg/kg, however not enough tablets were provided for patients with higher weights, indicating that higher risk patients received lower dosage. 41% of patients had BMI >30 and subgroups include BMI 50. In the journal version authors confirm that 42% of patients exceeded 88kg and did not receive the intended dose.
Monotherapy with no SOC for most patients.
Authors perform monotherapy and the standard of care for most patients in the study country included no active treatments. Other treatments were very rare - remdesivir 0.3%, monoclonal antibodies 3%, and paxlovid 0.1%. However, extensive and growing research shows greater and synergistic benefits from polytherapy. Many studies use polytherapy and/or the standard of care includes multiple active treatments.
Over 2x greater severe dyspnea at baseline for ivermectin.
There was over 2x greater severe dyspnea in the ivermectin arm at baseline (1.65% vs. 0.71%), which may be very important for analyzing mortality and hospitalization. Notably, the opposite is the case for fluticasone. The ivermectin placebo arm has less severe dyspnea than fluticasone, despite being larger.
Safety conclusion removed, suggests bias.
Authors included the conclusion "Ivermectin at 400 μg/kg was safe and without serious adverse events as compared with placebo"
in the abstract [medrxiv.org]
. This was deleted in the JAMA paper [twitter.com (E)]
Authors suggest high-income country healthcare is better, however almost all patients received no active SOC.
Authors suggest the operation in a high-income country with an associated healthcare system is a notable strength, however the study country provided no active treatment for almost all patients in the study, in contrast to many lower income countries that provide multiple treatments. Remdesivir, monoclonal antibodies, and paxlovid are very difficult to obtain and rarely used for outpatients in the study country. High income countries also may have significantly higher conflicts of interest.
Authors do not specify placebo details, only that packaging was identical. If the tablets were not identical, this would be an additional reason for blinding failure.
Patient choice supports potential participant fraud.
Patients were allowed to specify treatments that they accept or decline to be a part of. One author indicates this was most commonly used by patients to specify only ivermectin . Author also acknowledges that they were aware of politicization in the study country at design time . This facilitates potential participant fraud. For example, simply signing up, specifying ivermectin only, and reporting no symptoms upon medication receipt will guarantee biasing toward null results. Analysis may be done with the subgroup of patients that specified restrictions if the data is made available in the future.
No breakdown of severe outcomes.
Notably, no details are provided for the hospitalization and mortality events, which may have been more likely among patients with extremely late treatment, or influenced by the higher baseline severity in the ivermectin arm. No severe outcome results are provided for (relatively) early treatment.
Missing subgroup counts.
No subgroup counts are provided for several subgroups including treatment delay, while they are provided for baseline symptoms and vaccination status. The number of patients with symptoms ≤3 days may have been very small given the design of the trial.
Overlapping fluticasone placebo shows very different event numbers.
The ivermectin and fluticasone arms have 79% overlap in time (Jun 23, 2021 - Feb 4, 2022 vs. Aug 10, 2021 - Feb 12, 2022). The ivermectin placebo arm is 20% larger, suggesting approximately 20% more events. However, hospitalization is 3x larger (9 vs. 3), and combined hospitalization, urgent care, ER, and death is 2.2x larger (28 vs. 13).
Overlapping fluticasone placebo shows unexpected baseline numbers.
The ivermectin and fluticasone arms have 79% overlap in time (Jun 23, 2021 - Feb 4, 2022 vs. Aug 10, 2021 - Feb 12, 2022). The ivermectin placebo arm is 20% larger, suggesting approximately 20% more patients for each characteristic. However, ivermectin placebo has less Latino patients than fluticasone and over 2x COPD patients.
Inconsistent calendar time subgroups.
The calendar time subgroups for ivermectin and fluticasone are identical, from Oct 15, 2021 to Feb 1, 2022, however these do not match the reported recruitment periods.
Outcome graph presented does not match either medication tested.
In the ACTIV-6 presentation [rethinkingclinicaltrials.org]
(@9:22) an outcome graph is shown, however there is no indication what treatment it is for. The deaths and hospitalizations do not match those reported for either ivermectin or fluticasone.
Efficacy was higher over calendar time, which may reflect
higher efficacy with more recent variants. Efficacy was higher for vaccinated
Posterior probability ivermectin is effective:
Mean time unwell: 99%
Clinical progression @14 days: 98%
Clinical progression @7 days: 97%
All exceed the pre-specified threshold for superiority
(Clinical progression results showing superiority in the preprint
been changed without explanation)
“No differences were observed in relief of mild-to-moderate COVID-19 symptoms”
“No evidence of improvement in time to recovery”
"The posterior probability for treatment benefit did not meet prespecified
thresholds for clinical events or on the COVID Clinical Progression Scale" (in the preprint)
What can be done better? This long list of issues
details the flaws prohibiting any negative conclusion about early treatment.
In fact, the results are extremely positive given the conditions. Despite
extreme and obvious measures used to avoid showing efficacy, efficacy was
still found. Running a better trial is a simple matter of avoiding the issues
above. How do you ensure early treatment with high-risk patients? One example
would be pre-enrolling nursing home patients, providing treatment packages in
advance, and instructing local medical staff to initiate randomization,
treatment, and monitoring immediately on symptoms. This would likely be
cheaper to run, and easily extended to also study prophylaxis.