That Florida “analysis” on COVID vaccines is—you guessed it—total garbage
Epidemiologists and public health experts spent the past weekend collectively shaking their heads at the latest harmful pronouncement from Florida’s provocative surgeon general, Joseph Ladapo, who on Friday announced that he was recommending against mRNA-based COVID-19 vaccines for men ages 18 to 39.
Ladapo based his recommendation on a dubious analysis, which was posted online by the Florida Department of Health. According to a misleading press release from the department, the analysis found “an 84% increase in the relative incidence of cardiac-related death among males 18-39 years old within 28 days following mRNA vaccination.”
The press release says the analysis was carried out by the department, but it was posted as a simple PDF without the health department’s official letterhead, and—most strikingly—no authors are listed, which is highly unusual. It has clearly not been peer-reviewed, published in a scientific journal, or even thoroughly edited.
“Studying the safety and efficacy of any medications, including vaccines, is an important component of public health,” Ladapo said in the press release. “Far less attention has been paid to safety and the concerns of many individuals have been dismissed—these are important findings that should be communicated to Floridians.”
Experts roundly dubbed the analysis “utter rubbish,” “extremely misleading,” and “comically bad.” Some called the analysis method “terrible,” and one epidemiologist called it “the absolute most batshit study design & analysis plan I have ever seen.” Others noted that the conclusion “smells of p-hacking” and data cherry-picking.
These reactions are likely unsurprising to anyone who has followed Ladapo’s work as Florida’s top doctor. Brought in by Governor Ron DeSantis in 2021, Ladapo has spent the pandemic downplaying the threat of COVID-19, fighting public health measures, surrounding himself with prominent anti-vaccine voices, and attacking life-saving COVID-19 vaccines.
But fearmongering and discouraging use of COVID-19 vaccines amid the national fall booster campaign and ahead of an anticipated winter wave has the potential to do further harm to public and individual health. The US booster rates are already abysmal. Less than half of people who received a primary series have gotten a single booster. Only about 38 percent of people age 50 and older, who are more vulnerable to severe disease, have gotten a second booster. And so far, only 11.5 million Americans—roughly 3.5 percent of the population—have gotten the updated bivalent booster during this fall campaign, according to the latest figures from the Centers for Disease Control and Prevention.
COVID-19 vaccines and boosters have proven remarkably safe and highly effective at preventing severe disease and death. But, that hasn’t stopped some right-wing politicians and like-minded anti-vaccine voices and COVID-19 downplayers, such as Ladapo, from attacking them. This new “analysis” is a prime example.
As the reactions from experts suggest, there are a lot of things wrong with the analysis—too much to delve into in a single story. But, below are some of the highlights of just how bad this analysis really is.
The “batshit” study design
The analysis uses an unusual “alternative” epidemiological study design called a self-controlled case series (SCCS). This is a study design that was initially developed to assess vaccine safety, though not necessarily vaccine-related deaths, as was used for Ladapo’s analysis. The study design is legitimate—in that it is an established method that has been used before for similar purposes. But, as experts have pointed out, that doesn’t make it the best design, or even a good one.
To assess a vaccine’s safety and effectiveness, you might expect a study to compare a group of vaccinated people to a similar group of people who were not vaccinated, or perhaps received a placebo. Then you could compare the outcomes of people in the two groups, possibly attributing differences—the number of infections or possible side effects—to the obvious variable: the vaccine. But in a case-series design, there is no control or comparator group. The design uses each individual as their own comparator in fixed windows of time, comparing a risk period to a later control period.
By design, the study must only include people who experience both a specific intervention—such as a vaccination—and a specific health event after the intervention—such as a possible vaccine side effect. The study then looks at the timing of the health event in relation to the intervention, assessing if people appear more likely to experience the event in the risk period soon after the intervention compared with a later control period. If so, it suggests that the intervention played a role in the event.
For example, a 2004 study published in the New England Journal of Medicine used a case-series design to see if people are more likely to have a stroke or heart attack after routine vaccination or an acute infection. The study concluded that acute infections did transiently increase risk of stroke and heart attack, while vaccination did not.
Deadly outcomes
But the case-series design seems a bit unnatural when the health event is death—people often survive things like a stroke or heart attack, meaning they’re alive in both the risk period and the control period. But once people are dead, they’re dead. For example, people in a study could collectively have a high relative risk of a non-lethal stroke in the first few weeks after an infection, but risk could return to baseline in later weeks. On the other hand, if death is the sought-after event and a person dies in the first few weeks, their risk of death does not return to baseline later on—they are already dead. Still, one of the ways biostatisticians handle deaths in this case is by continuing to count people who died during the risk period as if they were alive during the control period—essentially calculating the risk of death among ghosts, which some epidemiologists say is a strange way to assess risk of death.
In Ladapo’s analysis, the unnamed authors wanted to assess risk of any death (by all causes) as well as cardiac-related deaths after a COVID-19 vaccine. They looked only at people who had received a COVID-19 vaccine and died within 25 weeks of a vaccination. The analysis was set up to look at whether the deaths unexpectedly clustered in the first four weeks after the vaccination—the risk period—with weeks 5 through 25 acting as the control period. The authors note that participants were not “censored” after death, but rather “followed” for the full 25 weeks.
While unusual, this is not the first study like this. In March, researchers at the UK’s Office of National Statistics posted a study on a pre-print server that was extremely similar. The study used a self-controlled case-series design to look at the risk of all-cause and cardiac deaths in people ages 12 to 29 who died within 12 weeks of either COVID-19 vaccination or SARS-CoV-2 infection. They set their risk period as the first six weeks after either vaccination or infection, with weeks seven through 12 acting as the control period. They also performed sub-group analyses, looking at risk based on age, sex, vaccine type, and last dose.
They concluded: “There is no evidence of an association between COVID-19 vaccination and an increased risk of death in young people. By contrast, SARS-CoV-2 infection was associated with substantially higher risk of cardiac related death and all-cause death.”
Shady specifics
The analysis in Florida went differently. First, it didn’t look at risk of death from SARS-CoV-2 infections. In fact, the unnamed authors write in the data-source section of their analysis that they tried to exclude anyone who had a documented SARS-CoV-2 infection, died specifically of COVID-19, or received a vaccine booster dose. While mountains of non-Florida analyses have found that COVID-19 vaccines reduce the risk of death from COVID-19, this study actively tried to avoid confirming that result.
Still, in the limitations section of the analysis, the authors suggest that their exclusions are moot, writing: “This study cannot determine the causative nature of a participant’s death. We used death certificate data and not medical records. COVID testing status was unknown for those who did not die of/with COVID.”
Notably, the analysis also didn’t include data on which vaccine people received as their intervention—whether it was the first or second dose in the primary series. Thus, if people died of undocumented COVID-19 cases shortly after their first dose—which does not offer full protection—that could easily inflate risk of death during the study, which spanned the height of the pandemic. The study’s risk period is 28 days, which is, or is close to, the interval between first and second doses of the two mRNA-based vaccines.
Heartfelt wonkiness
After trying—likely unsuccessfully—to filter out COVID-19 deaths, the authors-who-must-not be-named tried to home in on cardiac-related deaths. They likely did this poorly, too.
They note in their limitations section that:
“Cardiac-related deaths were ascertained if an [Automated Classification of Medical Entities] code of I3-I52 were on their death certificate, thus, the underlying cause of death may not be cardiac-related.”
Earlier in the analysis, the authors said they looked at codes 130-152, so it’s unclear which cardiac-related codes were actually counted, in addition to the fact that death certificate data doesn’t show a complete picture of why people died.
A whiff of cherries
When the authors ran the numbers for all-cause mortality, they found no increased risk of death from vaccination. In fact, they noted a statistically significant decrease in the risk of death among people age 60 and above. This, of course, was not mentioned in Ladapo’s press release.
When the authors plugged in the numbers for cardiac-related deaths, they found a slight increased risk of death among people ages 25 to 39 and those 60 and above. For the 25 to 39 group, the numbers were small, 29 deaths in the risk period compared with 75 in the control period. The authors report this as being twice the relative incidence of cardiac-related deaths in the risk period compared with the control. But, the 95 percent confidence interval is large, spanning from 1.35 to 3.47.
For the 60 and above group, the increased relative incidence was an underwhelming 5 percent, with confidence intervals of 1.01 to 1.10. The authors’ cutoff for “statistically significant” was a 95 percent confidence that didn’t include 1.00.
Yet, it seems even the authors weren’t convinced of their results. They write in the limitations section:
“Confounding by age may be present in the 60 years or older age group, which may explain the slight elevated risk for cardiac-related deaths following vaccination. This may also explain the increased risk for the entire vaccination analysis group for cardiac-related deaths since this group comprises the vast majority of deaths. Removing those aged 60 years or older yielded non-significant results for cardiac-related deaths following vaccination (RI = 1.15, 95% CI = 0.99 – 1.34), mRNA vaccination (RI = 1.17, 95% CI = 1.00 – 1.37), and males with mRNA vaccination (RI = 1.09, 95% CI = 0.89 – 1.34).”
Data dredging
Still, the authors dove down into their murky data further—which is a well-established red flag for people fishing for a result they want to find.
The authors sorted each age group by sex, then by type of vaccine a person got, though they didn’t have good data on this. The classifications were mRNA versus “not mRNA/unknown.” They glommed onto males, ages 18 to 39, who had received an mRNA vaccine as having a higher risk of cardiac-related deaths after a vaccine dose.
The number of deaths in this specific group during the risk period was 20, with 52 in the control period. The authors report this as representing an 84 percent increase in the relative incidence of cardiac-related deaths after vaccination—forming the basis of Ladapo’s press release. But, again, the confidence interval does not instill confidence, ranging from 1.05 to 3.21. If just a few of the deaths had a different cause than whatever cardiac-related code was counted, the calculation would easily slip this group into the not-statistically significant realm.
It’s also suspicious that this analysis of those ages 18 to 39 combined two age groups used in the first analysis, 18 to 24 and 25 to 39. Experts speculated that combination could have netted them the finding they wanted, while keeping them separate didn’t.
But even in the unlikely situation that this dodgy finding represents a real increase in cardiac-related deaths among young men, the analysis doesn’t even attempt to assess the cost-benefit ratio of vaccination. That is, the number of young men’s lives saved from COVID-19 vaccination undoubtedly outweighs whatever it is Ladapo is touting here—which is probably nothing.
