Covid-19: Sputnik vaccine rockets, thanks to Lancet boost
- Christoffer van Tulleken, honorary associate professor
Journals risk being used in place of regulators when they publish studies of novel vaccines that have not yet been authorised by a major regulator. Chris van Tulleken argues that peer review is inadequate to decide the risk-benefit ratio of new drugs
In August 2020 President Vladimir Putin announced Sputnik V, a vaccine developed by Russia’s Gamaleya National Center of Epidemiology and Microbiology. The president’s claim that it had gone through “all the necessary trials”1 did not seem to be backed up by the information on the Russian language registration certificate, which said that just 38 participants had received the vaccine.2
International responses ranged from concern to derision. By granting approval to a vaccine before results from large phase III randomised trials were available, the Russian government seemed to be taking two immense risks. The first was a risk of direct harm to large numbers of people. Bad vaccines don’t just fail to protect, they might have serious adverse effects including making subsequent infection more dangerous through antibody associated disease enhancement, a phenomenon previously seen with SARS and MERS coronaviruses.3 Second, if people were harmed, public confidence in the vaccination programme and future investment in covid-19 vaccine development and uptake might be jeopardised. Trust in vaccines is easily bruised and recovers slowly.
In September 2020, the first peer reviewed Sputnik V data were published in the Lancet: two non-randomised, open label studies, each of 38 people. No serious adverse events were reported, and the vaccine seemed to induce robust immune responses in participants.4
Phase I and II trial data concerns
Enrico Bucci was one of the first people to spot inconsistencies in this paper. Bucci runs an Italian research integrity company, and just three days after the publication he posted an open letter expressing concern that participants seemed to have identical values for different variables.5 He also noted identical repeating patterns of data points in separate groups of participants.
These findings seemed important, so I signed Bucci’s open letter, with more than 40 other scientists. Then, with 15 others, we wrote a letter, which the Lancet published, requesting access to the data from which the figures were generated.6
The Sputnik team responded, saying that the patterns in the data were “coincidences associated with the discreteness of the data, as well as with the small number of participants in the groups” and confirmed that individual participant data would be made available on request to the letter’s lead author and that “after approval of a proposal, data can be shared through a secure online platform.”7
Initially, this was reassuring. The Lancet is enthusiastic about open data. Its website says, “We envision a global research community in which sharing de-identified data becomes the norm,” and a September 2020 editorial said that “authors must endeavour to validate their conclusions with data that are accessible to readers, so that analyses can be reproduced. The Lancet journals will continue to hold authors and editors accountable for the data published in our pages, and we encourage our readers to do the same.”8
But, despite these assertions, neither the Lancet nor the Sputnik team have provided either the data from which the figures were generated or individual patient data. The Lancet declined to respond specifically to questions about whether they would uphold the data sharing agreement or whether they had even requested more data from the Sputnik team. It did publish a brief correction amending the paper without explanation,9 which seemed to be in response to the letters but made no reference to them.
Phase III data concerns
After a flurry of stories, press interest in the concerns around Sputnik V waned until the publication of an interim report on the phase III study on 2 February 2021, again in the Lancet.10
The paper reported an efficacy of over 90% in 14 964 vaccinated adults and was followed by favourable editorials in the journal. One announced: “Sputnik V covid-19 vaccine candidate appears safe and effective” and said that “another vaccine can now join the fight.”11 A second editorial applauded Russia “for their efforts in making their vaccine available and affordable to countries across the globe.”12
But once again, the paper was followed by an open letter of concern from Bucci, who drew attention to a large number of minor errors that would not be expected in a study of such importance.13 On a Kaplan-Meier plot, for example, hundreds of people whose data were available at day 20 were not included in the analysis at day 10. In another data table in the appendix, the numbers didn’t add up to the reported total. Later, in an online response to The BMJ, Bucci and a group of international authors pointed out the improbable consistency of vaccine efficacy values reported at interim analyses.14
Bucci was not alone. Vasiliy Vlassov, a professor at the National Research University Higher School of Economics in St Petersburg, also wrote an open letter referencing the Lancet’s brief correction of the previous paper, which, he said “has exacerbated distrust.”15 He pointed out that, unlike vaccines authorised by a major regulator, the Sputnik team was unique in being the only major vaccine developer not to release its full trial protocol.
The Lancet subsequently issued a correction for the phase III trial, again amending some of the anomalies.16 The overall impression is that of inadequate peer review and editorial processes that failed to detect what seem to be obvious errors in reporting the results of this high profile research.
The Lancet as cheerleader
These obvious errors and the uncritically glowing editorials would be worrying enough under normal circumstances. But they are particularly concerning given that, at the time of publication of the phase III trial, no major regulator had even received an application for marketing. By comparison, every other covid-19 vaccine with a phase III trial published in a high impact journal had already been submitted to or authorised by a major regulatory authority. The European Medicines Agency only began reviewing the Sputnik team’s application on 4 March, a month after the Lancet publication.
Meanwhile, the publications in the Lancet are being used to great effect by the Sputnik V marketing team and its primary investor, Russian Direct Investment Fund. The Lancet paper has been cited on Sputnik’s popular Twitter17 and Instagram feeds, in every press release,18 and in multiple interviews.19 The vaccine website claims: “Sputnik V’s efficacy and safety results are validated by internationally peer reviewed data published in the Lancet” (https://sputnikvaccine.com/).
And the Lancet paper seems to have given other countries confidence. Before publication of the phase III trial, 16 countries had authorised Sputnik V for use—now, over 40 have authorised it.18 Hundreds of millions of doses have been ordered and, according to the Sputnik team, millions of doses have been given.20
Most of the 40 countries using the vaccine ahead of EMA authorisation are low and middle income countries without well resourced, independent regulators. Understandably, because of the desperate global shortage of vaccines approved by a major regulator, they may have had no choice but to rely on the Lancet’s vetting of the science. But despite its international reputation, is the Lancet’s peer review process adequate for this?
Peer reviewing the pandemic
Clinicians and researchers are trained to trust in the power of peer review. Such faith was on display when Pfizer released interim trial results in a press release, ahead of journal publication.
Richard Horton, editor in chief of the Lancet expressed the same sentiment: “On the Pfizer covid-19 vaccine: publishing interim results through a press release is neither good scientific practice nor does it help to build public trust in vaccines. An announcement should come with full publication of a peer-reviewed research paper in a scientific journal.”21
Horton’s statement indicates a misunderstanding of the reasons for press releasing important results—companies around the world are legally obliged to disclose major new developments to investors without delay—as well as a misplaced confidence in journal peer review. The Lancet might be expected to exercise extra caution when it comes to papers on covid-19 or vaccines.
In summer 2020 both the Lancet and the New England Journal of Medicine published—and then retracted—major covid-19 studies based on a fraudulent dataset. And the Lancet made a similar mistake on a 1998 paper linking the MMR vaccine to autism, which remained in the literature for 12 years and contributed to the resurgence of measles around the world.
The widely recognised inability of traditional journal peer review to detect errors and fraud is hardly surprising.22 After a manuscript is selected for external review beyond the editorial team, it will typically be seen by two or three reviewers and a statistician. Turnaround times might be quick, and reviewers are almost always unpaid and, being international experts, busy. Usually a few hours are spent reviewing a paper, and occasionally post docs are roped in for additional scrutiny. At well resourced journals, further internal review and detailed editing will be done before acceptance and publication, but fast tracking topical and important findings might mean that corners are cut. After the retraction in summer 2020, Horton talked to the New Yorker about the stresses of publishing in the pandemic. “I don’t think we’ve had the capacity easily to deal with it [the increase in submissions], and that has stretched all of us,” Horton said. “Inevitably, in moments like that, you get very, very anxious about mistakes.”
The editors and peer reviewers of the Sputnik V paper are likely to have had only the 20 or so pages of PDF documents that were ultimately published. The Lancet’s website makes it clear that, like many journals, it does not have access to “raw data related to research studies.”23
Such limitations affect trust in journal publications generally but are most concerning when published data on important public health interventions, such as vaccines, have not yet been scrutinised by a stringent regulator.
The regulatory process
The regulatory process has its flaws and critics.242526 It is fundamentally similar to peer review in its purpose—to scrutinise a received submission—but the scope and scale is orders of magnitude larger.
At most journals, peer review is undertaken over a few hours, by two or three anonymous, unpaid experts, without publicly declared interests and without access to underlying data. By contrast, the EMA and other major regulators typically use named teams of in-house and external experts, all with declared interests and expertise in different critical aspects of a new product. The regulator also has unlimited access to all the non-clinical, clinical, and manufacturing data. They frequently audit the sponsor and inspect research and manufacturing sites. If they choose to exercise it, they have the power to look at individual patient charts to verify data.
The application for regulatory authorisation is not simply a vast data dump. It’s an orchestrated, collaborative process, governed by hundreds of pages of law and guidelines. Typically, meetings will begin before a submission, and drug developers can then be given formal guidance about every aspect of study design.
In terms of transparency, the EMA has published public assessment reports for the covid-19 vaccines they have authorised. These are over 150 pages long and detail the logic leading to authorisation. They include legal obligations on the sponsor to resolve any data discrepancies. In addition, the EMA has published thousands of pages of data from the submissions.
Compared with this extensive and well documented process, peer reviewed publication, even in a highly reputed publication such as the Lancet, is a relatively low bar to clear. Yet, since the extraordinary initial announcement, everything about Sputnik V has seemed worthy of detailed scrutiny by a journal.
The vaccine was developed at an institution in a country with no substantial track record of vaccine development and was intensively marketed without being submitted for authorisation to a major regulator. These things alone might have raised the need for exceptional caution in publishing the results of a phase III vaccine trial. Yet the Lancet chose to accompany publication with favourable editorials that made no mention of the need for regulatory scrutiny. After publication, credible concerns around the data were raised, and the Lancet has been unable to enforce the data sharing commitments made by the authors.
It is unclear exactly when the EMA will render its judgment on Sputnik V, especially considering the concerns about clotting problems that have since emerged with vaccines using similar adenovirus vector platforms. If it is authorised, Sputnik V will be a boost to global health, an idea which the Lancet, under Richard Horton, has championed with a radical approach. Perhaps their early endorsement of Sputnik is consistent with this, but, just as this episode raises questions about the Lancet’s commitment to open data, it also raises questions about the depth of the other commitments that they place under the banner “the best science for better lives.”
If Sputnik is not authorised, much more serious questions will surface, about the avoidable harm driven by overconfidence in journal peer review and the more far reaching damage to the public’s fragile confidence in other vaccines that are truly safe and effective.
Response from the Lancet
Prior to publication, The BMJ provided the Lancet with a list of allegations contained in this article regarding the Lancet’s publication of the Sputnik V trials. We received the following response from Emily Head, media relations manager:
“This research was independently peer reviewed by international experts on covid-19 and vaccines, including a statistical reviewer. At the Lancet journals, our editors treat communication with authors as confidential, and details of peer review including dates and peer review comments are not shared publicly.
All publicly available information for Lancet articles is published with the article, in the Supplementary Materials or Linked Articles sections on the article webpage. In addition, explanations of any errors that have been corrected within an article are provided in the Department of Error notice.
Our policies on peer review, data access, and corrections are available here: https://www.thelancet.com/publishing-excellence.”
Christoffer van Tulleken is an infection doctor at University College London Hospital and an honorary associate professor at UCL in the division of infection and immunity. He also works as a broadcaster covering science and health for children and adults on the BBC (including the double Bafta winning Operation Ouch). Much of his academic and broadcasting work focuses on conflicts of interest and research integrity.
Provenance and peer review: Commissioned; externally peer reviewed.
Competing interests: I have read and understood BMJ policy and have the following relevant interests to declare: none.
This article is made freely available for use in accordance with BMJ’s website terms and conditions for the duration of the covid-19 pandemic or until otherwise determined by BMJ. You may use, download and print the article for any lawful, non-commercial purpose (including text and data mining) provided that all copyright notices and trade marks are retained.https://bmj.com/coronavirus/usage