Vaccine safety signals – What are they and what to do when you find one

Helen Petousis-Harris, BSc, PGDipSci (Dist), PhD

9 May 2023

A vaccine safety signal is an alert that indicates an adverse health outcome may be occurring more frequently following administration of a vaccine. Signals can be generated from a variety of sources, including healthcare providers and spontaneous vaccine adverse event reporting systems. However, a signal cannot be interpreted as evidence for a causal relationship between the vaccine and the adverse event until confirmatory studies are conducted.
Most signals turn out to be false alarms. Therefore, careful examination of the results needs to be conducted before jumping to conclusions about a safety signal.
It is crucial to communicate the signal to relevant stakeholders, including regulatory agencies, the WHO Global Advisory Committee on Vaccine Safety (GACVS), the relevant national immunisation programme leaders, and the funding source for the study. Additionally, appropriately designed association studies should be conducted to investigate the relationship between the vaccine and the signal.

Vaccine safety signals are essential to stimulate questions about the potential relationship between a vaccine and an adverse event. However, it is crucial to follow a sequence of steps and conduct confirmatory studies before concluding a causal relationship between the vaccine and the adverse event exists. Good communication and appropriately designed association studies are critical steps in validating a safety signal.

A vaccine safety signal is an alert that an adverse health outcome may be occurring more often following administration of a vaccine, usually a newly introduced vaccine. Signals can be generated in a number of ways. They are hypothesis generating in that they stimulate a question about the potential relationship between a vaccine and adverse event. A signal can never be interpreted as indicating an association of causal relationship between the vaccine and the adverse event until confirmatory studies are conducted.

What is a signal?

In vaccine safety monitoring and pharmacoepidemiology, the level of statistical significance that constitutes a signal can vary depending on the specific context and the type of data being analysed. Generally, a signal could be considered to be any statistically significant association between a vaccine and an adverse event that is not likely to have occurred by chance, once the questions above have been considered.

The same threshold may not always be appropriate for each signal, and other factors such as the strength and consistency of the association, the biological plausibility of the relationship, and the potential impact of the adverse event on public health should also be considered when interpreting signals. For example, a relatively minor and transient condition may not warrant urgent investigation, but a more serious condition would.

It is important to reiterate that the detection of a signal does not necessarily mean that a causal relationship between the vaccine and the adverse event exists or has been established. Further investigation, such as conducting additional studies and analysing other data sources, is often needed to confirm or refute the signal and to assess the potential risks and benefits of the vaccine.

Gontran-Isnard-3-fuFf4gPNY-unsplash — Photo by Gontran Isnard on Unsplash

Where do signals come from?

Signals can come from a variety of sources. They might come from health care providers who notice a change in number of cases they are seeing; they might come from the spontaneous vaccine adverse event reporting systems such as VAERS in the U.S.; and they might come from actively searching for a change in pattern through systems such as the Vaccine Safety Datalink in the U.S.

What is the next step if a vaccine safety signal arises?

When a safety signal arises it first needs to be verified. The key question is “are we seeing more cases of a condition that would normally be expected?”. To answer this question an Observed versus Expected Study can be conducted. Essentially this involves comparing the rates of the condition observed after the vaccine started to be used with the rates prior to its use.

Does an increase in cases appearing after the introduction of a vaccine mean the vaccine is causing these events?

No. In fact most turn out to be false alarms. Experience in the U.S. with the VSD showed that from ten signals generated only one was confirmed, this was the risk for febrile seizures following administration of the MMRV vaccine. The other nine signals were ultimately eliminated after additional examination that found: Temporal changes in coding, confounding by age, inappropriate comparison group, miscoding in the electronic record, bias due to uncertainty of background rates, and chance.

When a signal appears, it must be interrogated before a decision is made to conduct an association study, for example a self-controlled case series study.

What to check about the safety signal

There are questions and a series of steps that need to be considered before jumping to conclusions about the results of an observed over expected study and a safety signal. These include:

Are there possibly data quality problems or errors in the background incidence?
Are the background rates (expected rates) rates consistent with what is published in the literature?
Is there a problem with the background rates? These can be uncertain or imprecise for a number of reasons. For example, a change in coding practices at the hospital or higher risk people receiving vaccine first therefore skewing the observed rates.
Examine the descriptive statistics, for example tabulate variables such as age, gender, and study site.
Double check the computer code. Has the same code been used at each study site?
Look for patterns in time from vaccine exposure to the onset of the outcome, these are called risk windows.
If there are enough cases in the study then do some statistical adjustments for additional confounders, for example age or other characteristics.
Conduct a review of medical records to exclude erroneously coded cases.
Compare results for similar outcomes if practical. For example, if there is an increase in febrile seizures then one might expect an increase in fever over the same period.
Compare results with other existing data – such as clinical trials, passive/spontaneous reporting, is this an anomaly and unexpected or do other sources of evidence concur? If there is little other evidence, then collect more data and/or conduct new study.

What do you do when a safety signal has been confirmed?

Once an observed versus expected study has identified a safety signal there are two critical actions that should ideally occur.

1. Communication. Good communication is an essential step to ensure all the key stakeholders are aware of the signal. At this stage these stakeholders include regulatory agencies, such as the FDA and EMA, the WHO Global Advisory Committee on Vaccine Safety, the relevant national immunisation programme (NIP) leaders, and the funding source for the study.

2. Conduct an appropriately designed association study that is hypothesis driven to validate the signal and determine the relationship between the vaccine and the adverse event. In this study you essentially ask the question: “Are people who receive the vaccine at greater risk of a particular adverse event?”

Blogs