Randomized Controlled Trials (RCTs)
Results restricted to Randomized Controlled Trials (RCTs) are shown in Figure
13,
14,
15, and
16,
Table 1, and
Table 2. The
supplementary data contains RCT results after exclusions.
RCTs have many potential biases.
Bias in clinical research may be defined as something that tends to make conclusions differ systematically from the truth. RCTs help to make study groups more similar and can provide a higher level of evidence, however they are subject to many biases
Jadad, and analysis of double-blind RCTs has identified extreme levels of bias
Gøtzsche. For COVID-19, the overhead may delay treatment, dramatically compromising efficacy; they may encourage monotherapy for simplicity at the cost of efficacy which may rely on combined or synergistic effects; the participants that sign up may not reflect real world usage or the population that benefits most in terms of age, comorbidities, severity of illness, or other factors; standard of care may be compromised and unable to evolve quickly based on emerging research for new diseases; errors may be made in randomization and medication delivery; and investigators may have hidden agendas or vested interests influencing design, operation, analysis, and the potential for fraud. All of these biases have been observed with COVID-19 RCTs. There is no guarantee that a specific RCT provides a higher level of evidence.
Conflicts of interest for COVID-19 RCTs.
RCTs are expensive and many RCTs are funded by pharmaceutical companies or interests closely aligned with pharmaceutical companies. For COVID-19, this creates an incentive to show efficacy for patented commercial products, and an incentive to show a lack of efficacy for inexpensive treatments. The bias is expected to be significant, for example
Als-Nielsen et al. analyzed 370 RCTs from Cochrane reviews, showing that trials funded by for-profit organizations were 5 times more likely to recommend the experimental drug compared with those funded by nonprofit organizations. For COVID-19, some major philanthropic organizations are largely funded by investments with extreme conflicts of interest for and against specific COVID-19 interventions.
RCTs for novel acute diseases requiring rapid treatment.
High quality RCTs for novel acute diseases are more challenging, with increased ethical issues due to the urgency of treatment, increased risk due to enrollment delays, and more difficult design with a rapidly evolving evidence base. For COVID-19, the most common site of initial infection is the upper respiratory tract. Immediate treatment is likely to be most successful and may prevent or slow progression to other parts of the body. For a non-prophylaxis RCT, it makes sense to provide treatment in advance and instruct patients to use it immediately on symptoms, just as some governments have done by providing medication kits in advance. Unfortunately, no RCTs have been done in this way. Every treatment RCT to date involves delayed treatment. Among the 66 treatments we have analyzed, 63% of RCTs involve very late treatment 5+ days after onset. No non-prophylaxis COVID-19 RCTs match the potential real-world use of early treatments (they may more accurately represent results for treatments that require visiting a medical facility, e.g., those requiring intravenous administration).
RCT bias for widely available treatments.
RCTs have a bias against finding an effect for interventions that are widely available — patients that believe they need the intervention are more likely to decline participation and take the intervention. RCTs for ivermectin are more likely to enroll low-risk participants that do not need treatment to recover, making the results less applicable to clinical practice. This bias is likely to be greater for widely known treatments, and may be greater when the risk of a serious outcome is overstated. This bias does not apply to the typical pharmaceutical trial of a new drug that is otherwise unavailable.
Non-RCT studies have been shown to be reliable.
Evidence shows that non-RCT trials can also provide reliable results.
Concato et al. found that well-designed observational studies do not systematically overestimate the magnitude of the effects of treatment compared to RCTs.
Anglemyer et al. summarized reviews comparing RCTs to observational studies and found little evidence for significant differences in effect estimates.
Lee et al. showed that only 14% of the guidelines of the Infectious Diseases Society of America were based on RCTs. Evaluation of studies relies on an understanding of the study and potential biases. Limitations in an RCT can outweigh the benefits, for example excessive dosages, excessive treatment delays, or Internet survey bias could have a greater effect on results. Ethical issues may also prevent running RCTs for known effective treatments. For more on issues with RCTs see
Deaton,
Nichol.
Using all studies identifies efficacy 5.7+ months faster for COVID-19.
Currently, 44 of the treatments we analyze show statistically significant efficacy or harm, defined as ≥10% decreased risk or >0% increased risk from ≥3 studies. Of the 44 treatments with statistically significant efficacy/harm, 28 have been confirmed in RCTs, with a mean delay of 5.7 months. When considering only low cost treatments, 23 have been confirmed with a delay of 6.9 months. For the 16 unconfirmed treatments, 3 have zero RCTs to date. The point estimates for the remaining 13 are all consistent with the overall results (benefit or harm), with 10 showing >20%. The only treatments showing >10% efficacy for all studies, but <10% for RCTs are sotrovimab and aspirin.
Summary.
We need to evaluate each trial on its own merits. RCTs for a given medication and disease may be more reliable, however they may also be less reliable. For off-patent medications, very high conflict of interest trials may be more likely to be RCTs, and more likely to be large trials that dominate meta analyses.