What We Learned From the Philly Vax Sweepstakes

What impact did the Philly Vax Sweepstakes — a large-scale, high-payoff vaccine regret lottery program — have on Philadelphia’s COVID-19 vaccination rates?

The Philly Vax Sweepstakes was a vaccine incentive program launched in the summer of 2021 by the Center for Health Incentives & Behavioral Economics (CHIBE), Wharton’s Behavior Change for Good Initiative (BCFG), the City of Philadelphia, and the Black Doctors COVID-19 Consortium. Read CHIBE’s original story on the sweepstakes to learn more about the design of this program.

Wharton Doctoral Candidate Linnea Gandhi, the lead author on a recent SSRN pre-print on this topic, spoke with CHIBE about what we learned from the vaccine lottery sweepstakes in Philadelphia.

What did you find in this analysis of the Philly Vax Sweepstakes?

To understand the results of the recent Philadelphia vaccine lottery, it’s important to understand the setup. It shares many of the features of the vaccine lotteries we’ve seen in other regions (e.g., Ohio) but includes a design twist that, to our knowledge, is unique and provides a bit more precision when evaluating the lottery’s benefits.

All other vaccine lotteries have been rolled out as a uniform policy, where all vaccinated individuals in a geography have an equal chance to win. This means that to evaluate a lottery’s impact, researchers need to compare the target geography (e.g., Ohio) to other geographies where no lottery exists (e.g., states surrounding Ohio) to infer what kind of impact the lottery had. We do this as well, comparing Philadelphia to adjacent counties (Bucks, Delaware, and Montgomery) and to Allegheny County in a differences-in-difference analysis. But this type of comparison can be problematic. Different regions have different vaccination policies and accessibility, not to mention other socioeconomic or political differences that muddy how clear that comparison can really be. Accordingly, results from early evaluations of the Ohio vaccine lottery are mixed, and our own quasi-experimental results for the Philadelphia vaccine lottery are similarly equivocal.

Ideally, we could clone the target geography and run two worlds side by side – one with the lottery and one without – to have a true counterfactual for causal inference. Obviously this isn’t realistic! However, the Philly Vax Sweepstakes was designed to somewhat get around this challenge by introducing a randomized controlled experiment within its design. One zip code (of the 20 zip codes with the lowest vaccination rates as of late May) was randomly selected in each of the three drawings to have significantly higher odds (on average 73x) of winning a prize. This meant we could at least get to a pretty precise evaluation of the causal impact of higher odds on vaccination rates. This was, unfortunately, null – we don’t detect significant benefits – but our 95% confidence interval provides a clear upper bound of 9%.

Why do you think the first drawing led to an uptick in vaccinations but the overall effect (when taking into account all the drawings) was insignificant?

Unfortunately, we don’t have strong empirical evidence to explain the differential uptick in vaccinations across the three drawings. As we note in the paper, the first drawing received the most media attention both locally and nationally, so that may have boosted awareness and vaccination uptake. It also came earlier, and we know that as time goes on, holdouts on vaccination become harder and harder to convince. But other factors, observable or unobservable, may also have led to the variation in results.

Knowing the results of the intervention now, is there anything you wish you could go back and adjust about the experiment or the roll-out to see if it would lead to a different outcome?

Like many COVID-19-related research efforts, our research team and collaborators at the City necessarily moved fast to launch this lottery at the start of the summer, as the Delta variant increasingly spread and risked significant harm for unvaccinated individuals. If we could have gone back and paused time for a bit, we may have found more effective ways to increase awareness, vaccine accessibility, and other on-the-ground initiatives necessary to maximize the impact of the lottery and other vaccination initiatives.

What advice would you have for other cities or states thinking about potentially using lottery systems like this for vaccination?

First, I would caveat that what does and doesn’t work in Philadelphia and at this point in the pandemic may not generalize to other geographies and later time periods. To the extent we can extrapolate, however, our team would encourage policymakers to consider the full suite of tools available to help make vaccination easy and attractive. Vaccine lotteries are really only focused on increasing the financial benefit of getting a vaccine, and the expected cash value to individuals is low. Yet there are many, diverse reasons for which an individual may not yet be vaccinated. Other policy tools may better help address those reasons or may do so in a way that is more cost effective. Stronger measures such as mandates may be necessary at this point.

Any other takeaways to share?

First, I think the design features are really important to emphasize (as outlined above in #1) as policymakers consider rolling out vaccine lotteries or, frankly, any other COVID-19-relevant policy. To the extent that they can introduce an experimental element in a way that is fair, with benefits targeted at underserved communities (as ours was designed to do), this will better serve them and our broader global community as we try to learn what does and doesn’t work to fight the problems of this pandemic.

Second, it’s also worth noting that the Philly Vax Sweepstakes further differentiates from prior vaccine lotteries in that it is a “regret lottery.”  To our knowledge, all other vaccine lotteries draw from a pool of vaccinated individuals. The Philadelphia vaccine lottery drew from the pool of Philadelphia residents, vaccinated and unvaccinated. This meant that, hypothetically, you could find out that your name was drawn for a prize – up to $50,000 – but that you couldn’t claim it, because you didn’t have at least your first dose of the COVID-19 vaccine. Theoretically, the anticipated regret of missing out on the prize should provide additional motivation to get vaccinated, above and beyond the lottery incentive itself.

The authors on this SSRN pre-print were: Linnea Gandhi, Katherine L. Milkman, Sean Ellis, Heather Graci, Dena Gromet, Rayyan Mobarak, Alison Buttenheim, Angela Duckworth, Devin G. Pope, Ala Stanford, Richard H. Thaler, and Kevin Volpp.