No evidence of important difference in summary treatment effects between COVID-19 preprints and peer-reviewed publications: a meta-epidemiological study
Mauricia Davidson, Theodoros Evrenoglou, Carolina Graña, Anna Chaimani, Isabelle Boutron
Journal of Clinical Epidemiology, doi:10.1016/j.jclinepi.2023.08.011
Objectives: Preprints became a major source of research communication during the COVID-19 pandemic. We aimed to evaluate whether summary treatment effect estimates differ between preprint and peer-reviewed journal trials. Study Design and Setting: A meta-epidemiological study. Data were derived from the COVID-NMA living systematic review (covid-nma.com) up to July 20, 2022. We identified all meta-analyses evaluating pharmacological treatments vs. standard of care or placebo for patients with COVID-19 that included at least one preprint and one peer-reviewed journal article. Difference in effect estimates between preprint and peer-reviewed journal trials were estimated by the ratio of odds ratio (ROR); ROR !1 indicated larger effects in preprint trials. Results: Thirty-seven meta-analyses including 114 trials (44 preprints and 70 peer-reviewed publications) were selected. The median number of randomized controlled trials (RCTs) per meta-analysis was 2 (interquartile range [IQR], 2e4; maximum, 11), median sample size of RCTs was 199 (IQR, 99e478). Overall, there was no statistically significant difference in summary effect estimates between preprint and peer-reviewed journal trials (ROR, 0.88; 95% CI, 0.71e1.09; I 2 5 17.8%; t 2 5 0.06).
Conclusion: We did not find an important difference between summary treatment effects of preprints and summary treatment effects of peer-reviewed publications. Systematic reviewers and guideline developers should assess preprint inclusion individually, accounting for risk of bias and completeness of reporting.
Supplementary data Supplementary data to this article can be found online at https://doi.org/10.1016/j.jclinepi.2023.08.011.
References
Bartoszko, Siemieniuk, Kum, Qasim, Zeraatkar et al., Prophylaxis against covid-19: living systematic review and network meta-analysis, BMJ
Bero, Lawrence, Leslie, Chiu, Mcdonald et al., Cross-sectional study of preprints and final journal publications from COVID-19 studies: discrepancies in results reporting and spin in interpretation, BMJ Open
Boutron, Chaimani, Devane, Meerpohl, Rada et al., Interventions for the prevention and treatment of COVID-19: a living mapping of research and living network metaanalysis, Cochrane Database Syst Rev
Boutron, Chaimani, Meerpohl, Hr Objartsson A, Devane, Rada, The COVID-NMA project: building an evidence ecosystem for the COVID-19 pandemic, Ann Intern Med
Core, R: a language and environment for statistical computing
Davidson, Chaimani, Boutron, Comparing numerical results between preprints and peer-reviewed publications of COVID-19 trials
Flanagin, Fontanarosa, Bauchner, Preprints involving medical researchddo the benefits outweigh the challenges?, JAMA
Horby, Why preprints are good for patients, Nat Med
Kapp, Esmail, Ghosn, Ravaud, Boutron, Transparency and reporting characteristics of COVID-19 randomized controlled trials, BMC Med
Kirkham, Penfold, Murphy, Boutron, Ioannidis et al., Systematic examination of preprint platforms for use in the medical and biomedical sciences setting, BMJ Open
Kwon, How swamped preprint servers are blocking bad coronavirus research, Nature
Lawrence, Why was a major study on ivermectin for COVID-19 just retracted?, Grftr News
Marshall, Murthy, Diaz, Adhikari, Angus et al., A minimal common outcome measure set for COVID-19 clinical research, Lancet Infect Dis
Medrxiv, org -the preprint server for health sciences
Sever, Roeder, Hindle, Sussman, Black et al., bioRxiv: the preprint server for biology, bioRxiv
Siemieniuk, Bartoszko, Zeraatkar, Kum, Qasim et al., Drug treatments for covid-19: living systematic review and network meta-analysis, BMJ
Sterne, Savovi C, Page, Elbers, Blencowe et al., RoB 2: a revised tool for assessing risk of bias in randomised trials, BMJ
Sterne, Schulz, Altman, Bartlett, Egger, Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological' research, Stat Med
Zeraatkar, Pitre, Leung, Cusano, Agarwal et al., Consistency of covid-19 trial preprints with published reports and impact for decision making: retrospective review, BMJ Med
{ 'indexed': {'date-parts': [[2024, 1, 11]], 'date-time': '2024-01-11T11:41:43Z', 'timestamp': 1704973303254},
'reference-count': 19,
'publisher': 'Elsevier BV',
'license': [ { 'start': { 'date-parts': [[2023, 10, 1]],
'date-time': '2023-10-01T00:00:00Z',
'timestamp': 1696118400000},
'content-version': 'tdm',
'delay-in-days': 0,
'URL': 'https://www.elsevier.com/tdm/userlicense/1.0/'},
{ 'start': { 'date-parts': [[2023, 9, 28]],
'date-time': '2023-09-28T00:00:00Z',
'timestamp': 1695859200000},
'content-version': 'vor',
'delay-in-days': 0,
'URL': 'http://creativecommons.org/licenses/by-nc-nd/4.0/'}],
'funder': [ { 'DOI': '10.13039/501100004562',
'name': "Ministère de l'Education Nationale, de l'Enseignement Superieur et de la Recherche",
'doi-asserted-by': 'publisher'},
{ 'DOI': '10.13039/501100001665',
'name': 'Agence Nationale de la Recherche',
'doi-asserted-by': 'publisher'},
{'DOI': '10.13039/501100021716', 'name': 'Université de Paris', 'doi-asserted-by': 'publisher'},
{ 'DOI': '10.13039/501100001677',
'name': 'Institut National de la Santé et de la Recherche Médicale',
'doi-asserted-by': 'publisher'},
{ 'DOI': '10.13039/100004423',
'name': 'World Health Organization',
'doi-asserted-by': 'publisher'},
{'DOI': '10.13039/100009647', 'name': 'Ministry of Health', 'doi-asserted-by': 'publisher'},
{ 'DOI': '10.13039/501100002738',
'name': 'Assistance Publique - Hôpitaux de Paris',
'doi-asserted-by': 'publisher'}],
'content-domain': { 'domain': [ 'clinicalkey.fr', 'clinicalkey.jp', 'clinicalkey.es', 'clinicalkey.com.au',
'clinicalkey.com', 'jclinepi.com', 'elsevier.com', 'sciencedirect.com'],
'crossmark-restriction': True},
'published-print': {'date-parts': [[2023, 10]]},
'DOI': '10.1016/j.jclinepi.2023.08.011',
'type': 'journal-article',
'created': {'date-parts': [[2023, 8, 25]], 'date-time': '2023-08-25T15:50:54Z', 'timestamp': 1692978654000},
'page': '90-97',
'update-policy': 'http://dx.doi.org/10.1016/elsevier_cm_policy',
'source': 'Crossref',
'is-referenced-by-count': 1,
'title': 'No evidence of important difference in summary treatment effects between COVID-19 preprints and '
'peer-reviewed publications: a meta-epidemiological study',
'prefix': '10.1016',
'volume': '162',
'author': [ { 'ORCID': 'http://orcid.org/0000-0002-9991-8035',
'authenticated-orcid': False,
'given': 'Mauricia',
'family': 'Davidson',
'sequence': 'first',
'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0003-3336-8058',
'authenticated-orcid': False,
'given': 'Theodoros',
'family': 'Evrenoglou',
'sequence': 'additional',
'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0002-3707-3006',
'authenticated-orcid': False,
'given': 'Carolina',
'family': 'Graña',
'sequence': 'additional',
'affiliation': []},
{'given': 'Anna', 'family': 'Chaimani', 'sequence': 'additional', 'affiliation': []},
{ 'ORCID': 'http://orcid.org/0000-0002-5263-6241',
'authenticated-orcid': False,
'given': 'Isabelle',
'family': 'Boutron',
'sequence': 'additional',
'affiliation': []}],
'member': '78',
'reference': [ { 'key': '10.1016/j.jclinepi.2023.08.011_bib2',
'first-page': '833400',
'article-title': 'bioRxiv: the preprint server for biology',
'author': 'Sever',
'year': '2019',
'journal-title': 'bioRxiv'},
{ 'issue': '12',
'key': '10.1016/j.jclinepi.2023.08.011_bib3',
'doi-asserted-by': 'crossref',
'DOI': '10.1136/bmjopen-2020-041849',
'article-title': 'Systematic examination of preprint platforms for use in the medical and '
'biomedical sciences setting',
'volume': '10',
'author': 'Kirkham',
'year': '2020',
'journal-title': 'BMJ Open'},
{ 'key': '10.1016/j.jclinepi.2023.08.011_bib4',
'doi-asserted-by': 'crossref',
'first-page': '130',
'DOI': '10.1038/d41586-020-01394-6',
'article-title': 'How swamped preprint servers are blocking bad coronavirus research',
'volume': '581',
'author': 'Kwon',
'year': '2020',
'journal-title': 'Nature'},
{ 'key': '10.1016/j.jclinepi.2023.08.011_bib5',
'doi-asserted-by': 'crossref',
'first-page': '1109',
'DOI': '10.1038/s41591-022-01812-4',
'article-title': 'Why preprints are good for patients',
'volume': '28',
'author': 'Horby',
'year': '2022',
'journal-title': 'Nat Med'},
{ 'key': '10.1016/j.jclinepi.2023.08.011_bib7',
'series-title': 'Why was a major study on ivermectin for COVID-19 just retracted?',
'author': 'Lawrence',
'year': '2021'},
{ 'key': '10.1016/j.jclinepi.2023.08.011_bib8',
'doi-asserted-by': 'crossref',
'first-page': '1840',
'DOI': '10.1001/jama.2020.20674',
'article-title': 'Preprints involving medical research—do the benefits outweigh the '
'challenges?',
'volume': '324',
'author': 'Flanagin',
'year': '2020',
'journal-title': 'JAMA'},
{ 'key': '10.1016/j.jclinepi.2023.08.011_bib9',
'doi-asserted-by': 'crossref',
'first-page': '1513',
'DOI': '10.1002/sim.1184',
'article-title': 'Statistical methods for assessing the influence of study '
'characteristics on treatment effects in ‘meta-epidemiological’ research',
'volume': '21',
'author': 'Sterne',
'year': '2002',
'journal-title': 'Stat Med'},
{ 'key': '10.1016/j.jclinepi.2023.08.011_bib10',
'doi-asserted-by': 'crossref',
'first-page': '1015',
'DOI': '10.7326/M20-5261',
'article-title': 'The COVID-NMA project: building an evidence ecosystem for the COVID-19 '
'pandemic',
'volume': '173',
'author': 'Boutron',
'year': '2020',
'journal-title': 'Ann Intern Med'},
{ 'key': '10.1016/j.jclinepi.2023.08.011_bib12',
'doi-asserted-by': 'crossref',
'first-page': 'e192',
'DOI': '10.1016/S1473-3099(20)30483-7',
'article-title': 'A minimal common outcome measure set for COVID-19 clinical research',
'volume': '20',
'author': 'Marshall',
'year': '2020',
'journal-title': 'Lancet Infect Dis'},
{ 'key': '10.1016/j.jclinepi.2023.08.011_bib13',
'series-title': 'R: a language and environment for statistical computing',
'year': '2022'},
{ 'key': '10.1016/j.jclinepi.2023.08.011_bib14',
'series-title': 'Cochrane handbook for systematic reviews of interventions. version 6.3 '
'(updated February 2022)',
'article-title': 'Chapter 10: analysing data and undertaking meta-analyses',
'year': '2022'},
{ 'key': '10.1016/j.jclinepi.2023.08.011_bib15',
'article-title': 'Interventions for the prevention and treatment of COVID-19: a living '
'mapping of research and living network meta-analysis',
'author': 'Boutron',
'year': '2020',
'journal-title': 'Cochrane Database Syst Rev'},
{ 'key': '10.1016/j.jclinepi.2023.08.011_bib17',
'doi-asserted-by': 'crossref',
'first-page': 'l4898',
'DOI': '10.1136/bmj.l4898',
'article-title': 'RoB 2: a revised tool for assessing risk of bias in randomised trials',
'volume': '366',
'author': 'Sterne',
'year': '2019',
'journal-title': 'BMJ'},
{ 'issue': '1',
'key': '10.1016/j.jclinepi.2023.08.011_bib18',
'doi-asserted-by': 'crossref',
'DOI': '10.1136/bmjmed-2022-000309',
'article-title': 'Consistency of covid-19 trial preprints with published reports and '
'impact for decision making: retrospective review',
'volume': '1',
'author': 'Zeraatkar',
'year': '2022',
'journal-title': 'BMJ Med'},
{ 'key': '10.1016/j.jclinepi.2023.08.011_bib19',
'doi-asserted-by': 'crossref',
'first-page': 'm2980',
'DOI': '10.1136/bmj.m2980',
'article-title': 'Drug treatments for covid-19: living systematic review and network '
'meta-analysis',
'volume': '370',
'author': 'Siemieniuk',
'year': '2020',
'journal-title': 'BMJ'},
{ 'key': '10.1016/j.jclinepi.2023.08.011_bib20',
'doi-asserted-by': 'crossref',
'first-page': 'n949',
'DOI': '10.1136/bmj.n949',
'article-title': 'Prophylaxis against covid-19: living systematic review and network '
'meta-analysis',
'volume': '373',
'author': 'Bartoszko',
'year': '2021',
'journal-title': 'BMJ'},
{ 'key': '10.1016/j.jclinepi.2023.08.011_bib21',
'series-title': 'Comparing numerical results between preprints and peer-reviewed '
'publications of COVID-19 trials [conference presentation abstract]. '
'Ninth International Congress on Peer Review and Scientific Publication',
'author': 'Davidson',
'year': '2022'},
{ 'issue': '1',
'key': '10.1016/j.jclinepi.2023.08.011_bib22',
'doi-asserted-by': 'crossref',
'first-page': '363',
'DOI': '10.1186/s12916-022-02567-y',
'article-title': 'Transparency and reporting characteristics of COVID-19 randomized '
'controlled trials',
'volume': '20',
'author': 'Kapp',
'year': '2022',
'journal-title': 'BMC Med'},
{ 'issue': '7',
'key': '10.1016/j.jclinepi.2023.08.011_bib23',
'doi-asserted-by': 'crossref',
'DOI': '10.1136/bmjopen-2021-051821',
'article-title': 'Cross-sectional study of preprints and final journal publications from '
'COVID-19 studies: discrepancies in results reporting and spin in '
'interpretation',
'volume': '11',
'author': 'Bero',
'year': '2021',
'journal-title': 'BMJ Open'}],
'container-title': 'Journal of Clinical Epidemiology',
'original-title': [],
'language': 'en',
'link': [ { 'URL': 'https://api.elsevier.com/content/article/PII:S0895435623002160?httpAccept=text/xml',
'content-type': 'text/xml',
'content-version': 'vor',
'intended-application': 'text-mining'},
{ 'URL': 'https://api.elsevier.com/content/article/PII:S0895435623002160?httpAccept=text/plain',
'content-type': 'text/plain',
'content-version': 'vor',
'intended-application': 'text-mining'}],
'deposited': { 'date-parts': [[2023, 12, 12]],
'date-time': '2023-12-12T19:12:28Z',
'timestamp': 1702408348000},
'score': 1,
'resource': {'primary': {'URL': 'https://linkinghub.elsevier.com/retrieve/pii/S0895435623002160'}},
'subtitle': [],
'short-title': [],
'issued': {'date-parts': [[2023, 10]]},
'references-count': 19,
'alternative-id': ['S0895435623002160'],
'URL': 'http://dx.doi.org/10.1016/j.jclinepi.2023.08.011',
'relation': {},
'ISSN': ['0895-4356'],
'subject': ['Epidemiology'],
'container-title-short': 'Journal of Clinical Epidemiology',
'published': {'date-parts': [[2023, 10]]},
'assertion': [ {'value': 'Elsevier', 'name': 'publisher', 'label': 'This article is maintained by'},
{ 'value': 'No evidence of important difference in summary treatment effects between '
'COVID-19 preprints and peer-reviewed publications: a meta-epidemiological study',
'name': 'articletitle',
'label': 'Article Title'},
{ 'value': 'Journal of Clinical Epidemiology',
'name': 'journaltitle',
'label': 'Journal Title'},
{ 'value': 'https://doi.org/10.1016/j.jclinepi.2023.08.011',
'name': 'articlelink',
'label': 'CrossRef DOI link to publisher maintained version'},
{'value': 'article', 'name': 'content_type', 'label': 'Content Type'},
{ 'value': '© 2023 The Authors. Published by Elsevier Inc.',
'name': 'copyright',
'label': 'Copyright'}]}