Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study
Helene Moustgaard, Gemma L Clayton, Hayley E Jones, Isabelle Boutron, Lars Jørgensen, David R T Laursen, Mette F Olsen, Asger Paludan-Müller, Philippe Ravaud, Jelena Savović, Jonathan A C Sterne, Julian P T Higgins, Asbjørn Hróbjartsson
BMJ, doi:10.1136/bmj.l6802
Objectives To study the impact of blinding on estimated treatment effects, and their variation between trials; differentiating between blinding of patients, healthcare providers, and observers; detection bias and performance bias; and types of outcome (the MetaBLIND study).
Design Meta-epidemiological study.
Data sOurce Cochrane Database of Systematic Reviews (2013-14). eligibility criteria fOr selecting stuDies Meta-analyses with both blinded and non-blinded trials on any topic.
review methODs Blinding status was retrieved from trial publications and authors, and results retrieved automatically from the Cochrane Database of Systematic Reviews. Bayesian hierarchical models estimated the average ratio of odds ratios (ROR), and estimated the increases in heterogeneity between trials, for non-blinded trials (or of unclear status) versus blinded trials. Secondary analyses adjusted for adequacy of concealment of allocation, attrition, and trial size, and explored the association between outcome subjectivity (high, moderate, low) and average bias. An ROR lower than 1 indicated exaggerated effect estimates in trials without blinding.
results The study included 142 meta-analyses (1153 trials). The ROR for lack of blinding of patients was 0.91 (95% credible interval 0.61 to 1.34) in 18 metaanalyses with patient reported outcomes, and 0.98 (0.69 to 1.39) in 14 meta-analyses with outcomes reported by blinded observers. The ROR for lack of blinding of healthcare providers was 1.01 (0.84 to 1.19) in 29 meta-analyses with healthcare provider decision outcomes (eg, readmissions), and 0.97 (0.64 to 1.45) in 13 meta-analyses with outcomes reported by blinded patients or observers. The ROR for lack of blinding of observers was 1.01 (0.86 to 1.18) in 46 meta-analyses with subjective observer reported outcomes, with no clear impact of degree of subjectivity. Information was insufficient to determine whether lack of blinding was associated with increased heterogeneity between trials. The ROR for trials not reported as double blind versus those that were double blind was 1.02 (0.90 to 1.13) in 74 meta-analyses. cOnclusiOn No evidence was found for an average difference in estimated treatment effect between trials with and without blinded patients, healthcare providers, or outcome assessors. These results could reflect that blinding is less important than often believed or metaepidemiological study limitations, such as residual confounding or imprecision. At this stage, replication of this study is suggested and blinding should remain a methodological safeguard in trials.
Contributors: AH and HM conceived and organised the study, interpreted the results, and drafted the manuscript. HM also extracted data. GLC analysed the data, interpreted results, and drafted the manuscript. HEJ, JS, IB, PR, JPTH, and JACS conceived the study, interpreted the results, and drafted the manuscript. LJ, DRTL, AP-M, and MFO extracted data and drafted the manuscript. HM is guarantor. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted. Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work. Ethical approval: Not required. Data sharing: Dataset available from the corresponding author after a post-publication period of 1 year allowing time for follow-up projects. The lead author affirms that this manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that any discrepancies from the study as planned have been explained. Dissemination to participants and related patient and public..
References
Akl, Sun, Busse, Specific instructions for estimating unclearly reported blinding status in randomized trials were reliable and valid, J Clin Epidemiol,
doi:10.1016/j.jclinepi.2011.04.015Armijo-Olivo, Fuentes, Da Costa, Saltaji, Ha et al., Blinding in physical therapy trials and its association with treatment effects: a meta-epidemiological study, Am J Phys Med Rehabil,
doi:10.1097/PHM.0000000000000521Bello, Moustgaard, Hróbjartsson, Unreported formal assessment of unblinding occurred in 4 of 10 randomized clinical trials, unreported loss of blinding in 1 of 10 trials, J Clin Epidemiol,
doi:10.1016/j.jclinepi.2016.08.002Claessen, Mellema, Stoop, Lubberts, Ring et al., Influence of priming on patient-reported outcome measures: a randomized controlled trial, Psychosomatics,
doi:10.1016/j.psym.2015.09.005Dechartres, Trinquart, Faber, Ravaud, Empirical evaluation of which trial characteristics are associated with treatment effect estimates, J Clin Epidemiol,
doi:10.1016/j.jclinepi.2016.04.005Devereaux, Manns, Ghali, Physician interpretations and textbook definitions of blinding terminology in randomized controlled trials, JAMA,
doi:10.1001/jama.285.15.2000Ghimire, Hasegawa, Kalyal, Hurwitz, Ashkan, Patient-reported outcome measures in neurosurgery: a review of the current literature, Neurosurgery,
doi:10.1093/neuros/nyx547Giraudeau, Higgins, Tavernier, Trinquart, Sample size calculation for meta-epidemiological studies, Stat Med,
doi:10.1002/sim.6627Haahr, Hróbjartsson, Who is blinded in randomized clinical trials? A study of 200 trials and a survey of authors, Clin Trials,
doi:10.1177/1740774506069153Higgins, Altman, Gøtzsche, The Cochrane Collaboration's tool for assessing risk of bias in randomised trials, BMJ,
doi:10.1136/bmj.d5928Higgins, Ramsay, Reeves, Issues relating to study design and risk of bias when including non-randomized studies in systematic reviews on the effects of interventions, Res Synth Methods,
doi:10.1002/jrsm.1056Hróbjartsson, Emanuelsson, Thomsen, Hilden, Brorson, Bias due to lack of patient blinding in clinical trials. A systematic review of trials randomizing patients to blind and nonblind substudies, Int J Epidemiol,
doi:10.1093/ije/dyu115Hróbjartsson, Thomsen, Emanuelsson, Observer bias in randomised clinical trials with binary outcomes: systematic review of trials with both blinded and non-blinded outcome assessors, BMJ,
doi:10.1136/bmj.e1119Hróbjartsson, Thomsen, Emanuelsson, Observer bias in randomized clinical trials with measurement scale outcomes: a systematic review of trials with both blinded and nonblinded assessors, CMAJ,
doi:10.1503/cmaj.120744Hróbjartsson, Thomsen, Emanuelsson, Observer bias in randomized clinical trials with time-to-event outcomes: systematic review of trials with both blinded and non-blinded outcome assessors, Int J Epidemiol,
doi:10.1093/ije/dyt270Kaptchuk, Intentional ignorance: a history of blind assessment and placebo controls in medicine, Bull Hist Med,
doi:10.1353/bhm.1998.0159Moustgaard, Bello, Miller, Hróbjartsson, Subjective and objective outcomes in randomized clinical trials: definitions differed in methods publications and were often absent from trial reports, J Clin Epidemiol,
doi:10.1016/j.jclinepi.2014.06.020Moustgaard, Jones, Savović, Ten questions to consider when interpreting results of a meta-epidemiological studythe MetaBLIND study as a case, Res Synth Meth,
doi:10.1002/jrsm.1392Nüesch, Reichenbach, Trelle, The importance of allocation concealment and patient blinding in osteoarthritis trials: a meta-epidemiologic study, Arthritis Rheum,
doi:10.1002/art.24894Page, Higgins, Clayton, Sterne, Hróbjartsson et al., Empirical evidence of study design biases in randomized trials: systematic review of meta-epidemiological studies, PLoS One,
doi:10.1371/journal.pone.0159267Rhodes, Mawdsley, Turner, Jones, Savović et al., Label-invariant models for the analysis of meta-epidemiological data, Stat Med,
doi:10.1002/sim.7491Rosenthal, On the social psychology of the psychological experiment: the experimenter's hypothesis as unintended determinant of experimental results, Am Sci
Saltaji, Armijo-Olivo, Cummings, Amin, Da Costa et al., Influence of blinding on treatment effect size estimate in randomized controlled trials of oral health interventions, BMC Med Res Methodol,
doi:10.1186/s12874-018-0491-0on6Savović, Turner, Mawdsley, Association between risk-of-bias assessments and results of randomized trials in Cochrane reviews: the ROBES meta-epidemiologic study, Am J Epidemiol,
doi:10.1093/aje/kwx344Schulz, Chalmers, Hayes, Altman, Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials, JAMA,
doi:10.1001/jama.1995.03520290060030Sterne, Jüni, Schulz, Altman, Bartlett et al., Statistical methods for assessing the influence of study characteristics on treatment effects in 'meta-epidemiological' research, Stat Med,
doi:10.1002/sim.1184Sterne, Savović, Page, RoB 2: a revised tool for assessing risk of bias in randomised trials, BMJ,
doi:10.1136/bmj.l4898Welton, Ades, Carlin, Altman, Sterne, Models for potentially biased evidence in meta-analysis using empirically based priors, J R Stat Soc Ser A Stat Soc,
doi:10.1111/j.1467-985X.2008.00548.x{ 'indexed': {'date-parts': [[2024, 2, 29]], 'date-time': '2024-02-29T17:39:23Z', 'timestamp': 1709228363694},
'reference-count': 34,
'publisher': 'BMJ',
'license': [ { 'start': { 'date-parts': [[2020, 1, 21]],
'date-time': '2020-01-21T00:00:00Z',
'timestamp': 1579564800000},
'content-version': 'unspecified',
'delay-in-days': 0,
'URL': 'http://creativecommons.org/licenses/by-nc/4.0/'}],
'content-domain': {'domain': ['bmj.com'], 'crossmark-restriction': True},
'abstract': '<jats:title>Abstract</jats:title>\n'
' <jats:sec>\n'
' <jats:title>Objectives</jats:title>\n'
' <jats:p>To study the impact of blinding on estimated treatment effects, and their '
'variation between trials; differentiating between blinding of patients, healthcare providers, '
'and observers; detection bias and performance bias; and types of outcome (the MetaBLIND '
'study).</jats:p>\n'
' </jats:sec>\n'
' <jats:sec>\n'
' <jats:title>Design</jats:title>\n'
' <jats:p>Meta-epidemiological study.</jats:p>\n'
' </jats:sec>\n'
' <jats:sec>\n'
' <jats:title>Data source</jats:title>\n'
' <jats:p>Cochrane Database of Systematic Reviews (2013-14).</jats:p>\n'
' </jats:sec>\n'
' <jats:sec>\n'
' <jats:title>Eligibility criteria for selecting studies</jats:title>\n'
' <jats:p>Meta-analyses with both blinded and non-blinded trials on any '
'topic.</jats:p>\n'
' </jats:sec>\n'
' <jats:sec>\n'
' <jats:title>Review methods</jats:title>\n'
' <jats:p>Blinding status was retrieved from trial publications and authors, and '
'results retrieved automatically from the Cochrane Database of Systematic Reviews. Bayesian '
'hierarchical models estimated the average ratio of odds ratios (ROR), and estimated the '
'increases in heterogeneity between trials, for non-blinded trials (or of unclear status) '
'versus blinded trials. Secondary analyses adjusted for adequacy of concealment of allocation, '
'attrition, and trial size, and explored the association between outcome subjectivity (high, '
'moderate, low) and average bias. An ROR lower than 1 indicated exaggerated effect estimates '
'in trials without blinding.</jats:p>\n'
' </jats:sec>\n'
' <jats:sec>\n'
' <jats:title>Results</jats:title>\n'
' <jats:p>The study included 142 meta-analyses (1153 trials). The ROR for lack of '
'blinding of patients was 0.91 (95% credible interval 0.61 to 1.34) in 18 meta-analyses with '
'patient reported outcomes, and 0.98 (0.69 to 1.39) in 14 meta-analyses with outcomes reported '
'by blinded observers. The ROR for lack of blinding of healthcare providers was 1.01 (0.84 to '
'1.19) in 29 meta-analyses with healthcare provider decision outcomes (eg, readmissions), and '
'0.97 (0.64 to 1.45) in 13 meta-analyses with outcomes reported by blinded patients or '
'observers. The ROR for lack of blinding of observers was 1.01 (0.86 to 1.18) in 46 '
'meta-analyses with subjective observer reported outcomes, with no clear impact of degree of '
'subjectivity. Information was insufficient to determine whether lack of blinding was '
'associated with increased heterogeneity between trials. The ROR for trials not reported as '
'double blind versus those that were double blind was 1.02 (0.90 to 1.13) in 74 '
'meta-analyses.</jats:p>\n'
' </jats:sec>\n'
' <jats:sec>\n'
' <jats:title>Conclusion</jats:title>\n'
' <jats:p>No evidence was found for an average difference in estimated treatment '
'effect between trials with and without blinded patients, healthcare providers, or outcome '
'assessors. These results could reflect that blinding is less important than often believed or '
'meta-epidemiological study limitations, such as residual confounding or imprecision. At this '
'stage, replication of this study is suggested and blinding should remain a methodological '
'safeguard in trials.</jats:p>\n'
' </jats:sec>',
'DOI': '10.1136/bmj.l6802',
'type': 'journal-article',
'created': {'date-parts': [[2020, 1, 21]], 'date-time': '2020-01-21T16:35:15Z', 'timestamp': 1579624515000},
'page': 'l6802',
'update-policy': 'http://dx.doi.org/10.1136/crossmarkpolicy',
'source': 'Crossref',
'is-referenced-by-count': 125,
'title': 'Impact of blinding on estimated treatment effects in randomised clinical trials: '
'meta-epidemiological study',
'prefix': '10.1136',
'author': [ { 'ORCID': 'http://orcid.org/0000-0002-7057-5251',
'authenticated-orcid': False,
'given': 'Helene',
'family': 'Moustgaard',
'sequence': 'first',
'affiliation': []},
{'given': 'Gemma L', 'family': 'Clayton', 'sequence': 'additional', 'affiliation': []},
{'given': 'Hayley E', 'family': 'Jones', 'sequence': 'additional', 'affiliation': []},
{'given': 'Isabelle', 'family': 'Boutron', 'sequence': 'additional', 'affiliation': []},
{'given': 'Lars', 'family': 'Jørgensen', 'sequence': 'additional', 'affiliation': []},
{'given': 'David R T', 'family': 'Laursen', 'sequence': 'additional', 'affiliation': []},
{'given': 'Mette F', 'family': 'Olsen', 'sequence': 'additional', 'affiliation': []},
{'given': 'Asger', 'family': 'Paludan-Müller', 'sequence': 'additional', 'affiliation': []},
{'given': 'Philippe', 'family': 'Ravaud', 'sequence': 'additional', 'affiliation': []},
{'given': 'Jelena', 'family': 'Savović', 'sequence': 'additional', 'affiliation': []},
{'given': 'Jonathan A C', 'family': 'Sterne', 'sequence': 'additional', 'affiliation': []},
{'given': 'Julian P T', 'family': 'Higgins', 'sequence': 'additional', 'affiliation': []},
{'given': 'Asbjørn', 'family': 'Hróbjartsson', 'sequence': 'additional', 'affiliation': []}],
'member': '239',
'published-online': {'date-parts': [[2020, 1, 21]]},
'reference': [ { 'key': 'key20200122000045_ref1',
'doi-asserted-by': 'publisher',
'DOI': '10.1001/jama.2014.8167'},
{ 'key': 'key20200122000045_ref2',
'doi-asserted-by': 'publisher',
'DOI': '10.1136/bmj.d5928'},
{ 'key': 'key20200122000045_ref3',
'doi-asserted-by': 'publisher',
'DOI': '10.1016/S0140-6736(05)71879-1'},
{ 'key': 'key20200122000045_ref4',
'doi-asserted-by': 'publisher',
'DOI': '10.1353/bhm.1998.0159'},
{ 'key': 'key20200122000045_ref5',
'doi-asserted-by': 'publisher',
'DOI': '10.1001/jama.1995.03520290060030'},
{'key': 'key20200122000045_ref6', 'doi-asserted-by': 'publisher', 'DOI': '10.1002/sim.1184'},
{ 'key': 'key20200122000045_ref7',
'doi-asserted-by': 'publisher',
'DOI': '10.1371/journal.pone.0159267'},
{ 'key': 'key20200122000045_ref8',
'doi-asserted-by': 'publisher',
'DOI': '10.1177/1740774506069153'},
{ 'key': 'key20200122000045_ref9',
'doi-asserted-by': 'publisher',
'DOI': '10.1001/jama.285.15.2000'},
{ 'key': 'key20200122000045_ref10',
'doi-asserted-by': 'publisher',
'DOI': '10.1002/art.24894'},
{ 'key': 'key20200122000045_ref11',
'doi-asserted-by': 'publisher',
'DOI': '10.1016/j.jclinepi.2011.04.015'},
{ 'key': 'key20200122000045_ref12',
'doi-asserted-by': 'publisher',
'DOI': '10.7326/0003-4819-157-6-201209180-00537'},
{ 'key': 'key20200122000045_ref13',
'doi-asserted-by': 'publisher',
'DOI': '10.1093/ije/dyu115'},
{ 'key': 'key20200122000045_ref14',
'doi-asserted-by': 'publisher',
'DOI': '10.1111/j.1467-985X.2008.00548.x'},
{ 'key': 'key20200122000045_ref15',
'doi-asserted-by': 'publisher',
'DOI': '10.1002/1097-0258(20001130)19:22<3127::AID-SIM784>3.0.CO;2-M'},
{ 'key': 'key20200122000045_ref16',
'doi-asserted-by': 'publisher',
'DOI': '10.1002/sim.7491'},
{ 'key': 'key20200122000045_ref17',
'doi-asserted-by': 'publisher',
'DOI': '10.1136/bmj.e1119'},
{ 'key': 'key20200122000045_ref18',
'doi-asserted-by': 'publisher',
'DOI': '10.1503/cmaj.120744'},
{ 'key': 'key20200122000045_ref19',
'doi-asserted-by': 'publisher',
'DOI': '10.1093/ije/dyt270'},
{ 'key': 'key20200122000045_ref20',
'doi-asserted-by': 'publisher',
'DOI': '10.1002/sim.6627'},
{ 'key': 'key20200122000045_ref21',
'doi-asserted-by': 'publisher',
'DOI': '10.1016/j.jclinepi.2016.08.002'},
{ 'key': 'key20200122000045_ref22',
'doi-asserted-by': 'publisher',
'DOI': '10.1093/aje/kwx344'},
{ 'key': 'key20200122000045_ref23',
'doi-asserted-by': 'publisher',
'DOI': '10.1016/j.jclinepi.2016.04.005'},
{ 'key': 'key20200122000045_ref24',
'doi-asserted-by': 'publisher',
'DOI': '10.1097/PHM.0000000000000521'},
{ 'key': 'key20200122000045_ref25',
'doi-asserted-by': 'publisher',
'DOI': '10.1186/s12874-018-0491-0'},
{ 'key': 'key20200122000045_ref26',
'doi-asserted-by': 'publisher',
'DOI': '10.1016/j.jclinepi.2014.06.020'},
{ 'key': 'key20200122000045_ref27',
'doi-asserted-by': 'publisher',
'DOI': '10.1093/neuros/nyx547'},
{ 'key': 'key20200122000045_ref28',
'doi-asserted-by': 'publisher',
'DOI': '10.1016/j.psym.2015.09.005'},
{ 'key': 'key20200122000045_ref29',
'doi-asserted-by': 'publisher',
'DOI': '10.1056/NEJMp1103502'},
{ 'key': 'key20200122000045_ref30',
'doi-asserted-by': 'publisher',
'DOI': '10.1037/1089-2680.2.2.175'},
{ 'key': 'key20200122000045_ref31',
'first-page': '268',
'article-title': 'On the social psychology of the psychological experiment: the '
'experimenter’s hypothesis as unintended determinant of experimental '
'results',
'volume': '51',
'author': 'Rosenthal',
'year': '1963',
'journal-title': 'Am Sci'},
{ 'key': 'key20200122000045_ref32',
'doi-asserted-by': 'publisher',
'DOI': '10.1002/jrsm.1056'},
{ 'key': 'key20200122000045_ref33',
'doi-asserted-by': 'publisher',
'DOI': '10.1002/jrsm.1392'},
{ 'key': 'key20200122000045_ref34',
'doi-asserted-by': 'publisher',
'DOI': '10.1136/bmj.l4898'}],
'container-title': 'BMJ',
'original-title': [],
'language': 'en',
'link': [ { 'URL': 'http://data.bmj.org/tdm/10.1136/bmj.l6802',
'content-type': 'unspecified',
'content-version': 'vor',
'intended-application': 'text-mining'},
{ 'URL': 'https://syndication.highwire.org/content/doi/10.1136/bmj.l6802',
'content-type': 'unspecified',
'content-version': 'vor',
'intended-application': 'similarity-checking'}],
'deposited': { 'date-parts': [[2020, 9, 26]],
'date-time': '2020-09-26T03:54:33Z',
'timestamp': 1601092473000},
'score': 1,
'resource': {'primary': {'URL': 'https://www.bmj.com/lookup/doi/10.1136/bmj.l6802'}},
'subtitle': [],
'short-title': [],
'issued': {'date-parts': [[2020, 1, 21]]},
'references-count': 34,
'alternative-id': ['10.1136/bmj.l6802'],
'URL': 'http://dx.doi.org/10.1136/bmj.l6802',
'relation': {},
'ISSN': ['1756-1833'],
'subject': ['General Engineering'],
'container-title-short': 'BMJ',
'published': {'date-parts': [[2020, 1, 21]]}}
