Skip to Main Content

BIS Seminar: At warp speed: statistics and COVID-19 vaccine development

October 28, 2020
BIS Seminar: At warp speed: statistics and COVID-19 vaccine development
  • 00:00- [Fan] David Benkeser who is an assistant professor
  • 00:03at the department of biostatistics and bioinformatics
  • 00:07at Emory University.
  • 00:09Dr. Benkeser got his PhD in biostatistics
  • 00:11from University of Washington
  • 00:13and had his post-doctoral fellowship
  • 00:15from University of California at Berkeley.
  • 00:18Dr. Benkeser is an expert in methods for machine learning
  • 00:22and non-parametric statistical inference.
  • 00:24He has made important contributions
  • 00:26to integrate machine learning methods
  • 00:28to draw causal inferences with observational data.
  • 00:31He also has interesting work on preventative vaccines
  • 00:34and HIV prevention, which he's going to share with us today.
  • 00:37Welcome David, the floor is yours.
  • 00:44- [David] Thanks, yeah, it's a great pleasure
  • 00:46to be here today.
  • 00:48Well, here today, but with you guys today giving this talk.
  • 00:51So I did see that I think Tony Fauci
  • 00:55spoke at Yale yesterday, so it was very nice of you Fan
  • 00:58to book Tony Fauci as my opening act
  • 01:00and I'll try not to disappoint him with my followup.
  • 01:04So the talk I'm giving today is a very high-level talk.
  • 01:07So the title is statistics and COVID-19 vaccine development,
  • 01:10but it's really a talk mostly
  • 01:12about COVID-19 vaccine development.
  • 01:15There's not math until maybe slide like 29 out of 30.
  • 01:19So really these are sort of
  • 01:20just the high-level issues that have come up
  • 01:23as I've worked with companies and government organizations
  • 01:28on COVID-19 vaccine development.
  • 01:30So I think there's a lot of really interesting stuff
  • 01:32here and really, really glad to share it with you today.
  • 01:35So if you want to kind of slide along with
  • 01:39the slides they're available on GitHub
  • 01:41so there's a link at the bottom there,
  • 01:43and you can click on that
  • 01:44and that'll pull up the HTML slide back,
  • 01:46and I have sort of references hyperlinked in there.
  • 01:48So that's an easy way to access
  • 01:50the references there as well.
  • 01:52Okay so I'm going to start just kind of talking
  • 01:55about the biology a little bit of SARS-CoV-2,
  • 01:58and segue into sort of how we can think about
  • 02:01developing vaccines that will prevent
  • 02:03an infection and COVID-19 disease.
  • 02:06And so this is a nice little graphic that I ripped off
  • 02:09from The Washington Post, who's very much better
  • 02:12at making these cutesy little graphics
  • 02:13than I am using PowerPoint or something.
  • 02:16So let's kind of walk through this.
  • 02:17And the goal here is to try to understand,
  • 02:20you know, how SARS-CoV-2 is infecting your cells,
  • 02:22how it's replicating,
  • 02:23and then to understand what the mechanisms
  • 02:25that immunological mechanisms of the vaccine are
  • 02:28that can potentially block that infection
  • 02:30and prevent clinical disease.
  • 02:31So we'll just go quickly through this
  • 02:33and this is sort of the story for most viruses, right?
  • 02:36Is that viruses are really just genetic material
  • 02:39in this case RNA that's wrapped up in the glycoprotein.
  • 02:42So it's genetic material wrapped up in a protein.
  • 02:45And so for SARS-CoV-2 you may have heard of a couple
  • 02:49of these proteins in particular, the spike protein will play
  • 02:51a large role when we talk about a vaccine development
  • 02:54and why is this spike protein so important?
  • 02:56Well, that's the guy that sort of latches
  • 02:58onto your cell and it does that through this ACE2 pathway
  • 03:01and it grabs onto your cell and insert itself inside
  • 03:05you cell and once it's inside
  • 03:07it releases its genetic material, right?
  • 03:09It releases its RNA and kind of tricks
  • 03:11your cell into replicating the virus, right?
  • 03:13So that your cell is producing new copies of this virus,
  • 03:17they're pieced together out of proteins that are released
  • 03:19into your bloodstream to go infect more cells
  • 03:22and more people.
  • 03:23Okay so that's sort of the infection process
  • 03:25and where along the lines do you know
  • 03:27vaccines sort of halt this?
  • 03:29So I'll walk through a few different
  • 03:31of the major vaccine constructs that are being used
  • 03:33for SARS-CoV-2 vaccines,
  • 03:35and the details aren't super important here,
  • 03:37but I do think it's sort of helpful
  • 03:38to have a high level overview in comparison, right?
  • 03:40Because there's so many vaccine products being developed,
  • 03:43at least having some point of biological comparison
  • 03:45of how they're working is useful.
  • 03:47So to walk through these slides,
  • 03:48all of these slides are basically going to be the same
  • 03:50on the right hand side of the slide
  • 03:52and how they're gonna differ is what goes
  • 03:54into the vaccine on the left-hand side.
  • 03:56So let's actually start on the right-hand side, right?
  • 03:58And talk a little bit about immunology, right?
  • 04:00And how your body tries to fight off infection.
  • 04:03And we have a couple of different mechanisms
  • 04:05of your immune system to do that.
  • 04:06So there's a kind of T-cell responses, cytotoxic T-cells.
  • 04:10So those are T-cells that recognize cells in your body
  • 04:12that have been infected with a pathogen
  • 04:14and destroy those cells, right?
  • 04:15Because the cells are producing copies of the virus,
  • 04:17releasing in the bloodstream.
  • 04:19So if we're able to destroy infected cells,
  • 04:21we can potentially stop infection, prevent disease,
  • 04:24and then another key response
  • 04:26that your immune system has is through antibodies.
  • 04:28And that's sort of what's on the bottom here
  • 04:30and is that B cells are able to produce antibodies.
  • 04:34And what those antibodies do is they basically grab
  • 04:36onto these surface proteins, right?
  • 04:38So remember we talked about the spike protein,
  • 04:40and what antibodies do is basically just bind onto that
  • 04:43and sit there and so neutralizing antibodies.
  • 04:46So there's two classes of antibodies that are kind
  • 04:47of relevant for vaccines.
  • 04:48So neutralizing antibodies really, you're just doing that.
  • 04:50They're gonna sit on all of those spike proteins
  • 04:53and because they're sitting there now the virus can't grab
  • 04:55onto your cells to infect them.
  • 04:57There's also binding antibodies, which are somewhat
  • 05:01considered to be less important in this context,
  • 05:03but what those guys do is bind onto those surface proteins,
  • 05:05they don't neutralize the virus itself,
  • 05:07but they send out chemical signals to other cells
  • 05:09in your body that say, hey, here's a virus.
  • 05:11Please come eat it for me.
  • 05:13So those are the sort of antibody classes response
  • 05:15that you can have.
  • 05:16So there's these two sort of immune mechanisms
  • 05:18that we have to neutralize infections by viruses.
  • 05:22How do they learn to neutralize them?
  • 05:25Well, there's this sort of middleman.
  • 05:26So we're moving just to this middle panel here
  • 05:28with these APC cells,
  • 05:30so these antigen presenting cells, right?
  • 05:32Those are the guys that what they're doing
  • 05:33is basically digesting little bits
  • 05:36of the virus in this case of the surface protein, right?
  • 05:40And they're teaching or training your immune system
  • 05:43to recognize that pathogen, right?
  • 05:44So they're the ones that go and talk to the T cells,
  • 05:47talk to the B cells and say,
  • 05:48here's that how this virus looks,
  • 05:51please go produce some antibodies or please recognize cells
  • 05:53that have been infected with this
  • 05:55and neutralize them for me.
  • 05:56So really again, the whole right side of this plot
  • 05:59is about your immune system.
  • 06:00This is the way your immune system fights off infection.
  • 06:03And what's different between this slide
  • 06:05and the next few slides is basically how we present
  • 06:08pieces of the pathogen pieces
  • 06:09of the virus to these APCs, right?
  • 06:12So how do we get these APCs, the material that they need
  • 06:14for you to mount an immune response against SARS-CoV-2?
  • 06:19And so the first class of vaccines
  • 06:21I'll describe are nucleic acid vaccines.
  • 06:23And so I'm talking about first
  • 06:25because they're sort of the first wave of vaccines
  • 06:27that are in phase three trials in the US.
  • 06:29So Moderna and Pfizer, who are probably the most advanced
  • 06:33candidates for US licensure are both mRNA vaccines.
  • 06:37And so how are those vaccines made?
  • 06:39Well, we take a little bit of messenger RNA,
  • 06:42a little bit of viral genetic material,
  • 06:44and wrap that in a lipid shell, right?
  • 06:45That's the construct of the vaccine.
  • 06:47And when you're injected that lipid shell
  • 06:49latches onto your cell, right?
  • 06:51Delivers that mRNA into your cell,
  • 06:54just like a natural infection, right?
  • 06:55Remember the SARS-CoV-2 grabbed onto your cell
  • 06:57and inserted itself and then made copies of itself.
  • 07:00So what is the mRNA doing once it's in your cell,
  • 07:03it's actually just making copies
  • 07:05of the spike protein itself, right?
  • 07:07So you're manufacturing this protein within your own cells
  • 07:11that are then released for these APCs to detect.
  • 07:14So this is how we're getting these APCs,
  • 07:17spike protein with an mRNA vaccine.
  • 07:19We're basically using your cells as a warehouse
  • 07:22to produce the antigen of the vaccine
  • 07:24and so this is a really cool idea and a new idea, right?
  • 07:28So, am mRNA or DNA vaccine has never been licensed before
  • 07:31and that's not to say that we tried many times and failed.
  • 07:34It's just to say that this is a very new technology,
  • 07:36and it's sort of interesting that it's kind of come
  • 07:40to the forefront in this context.
  • 07:41So why do we like mRNA vaccines?
  • 07:44Well, they're very fast to manufacturer.
  • 07:46We'll talk about some of the other vaccine constructs
  • 07:48where we're making this spike protein in a lab,
  • 07:50and that is a long and arduous.
  • 07:52It needs to be very careful process
  • 07:54and when we're thinking about scaling up
  • 07:55vaccine manufacturing, mRNA vaccines are very appealing
  • 07:59in that sense, you can manufacture them
  • 08:02very quickly at scale.
  • 08:03They don't require a cold chain
  • 08:05and so that's another great advantage these vaccines enjoy
  • 08:10in terms of thinking about vaccine deployment,
  • 08:12particularly in developing world settings.
  • 08:15But again, this is a brand new technology.
  • 08:17We don't have any safety data
  • 08:18from past vaccines with this construct.
  • 08:20We don't have any efficacy data.
  • 08:22So, it's sort of an open question in the field
  • 08:25as to how well these things are gonna work.
  • 08:28So moving to sort of more classical, constructive vaccines
  • 08:30and viral vector vaccines.
  • 08:33So again, the right side of this picture
  • 08:34is exactly the same.
  • 08:35The story is how do we get an APC the right antigen?
  • 08:38How do we show an APC a little bit of the spike protein?
  • 08:41So a viral vector vaccine, right?
  • 08:45Is going to take a different virus and splice
  • 08:49a little bit of SARS-CoV-2 into that virus, okay.
  • 08:52So for example, AstraZeneca, that's the Oxford that you may
  • 08:56have heard of, they take a chimpanzee adenovirus,
  • 08:58that's like, it's a virus that causes the common cold
  • 09:02in chimpanzees and they splice in a little bit
  • 09:04of SARS-CoV-2 into that and so that sort of host virus,
  • 09:09that adenovirus holds genetic material
  • 09:12infects your cells and your cells then produce the antigen.
  • 09:16They produce the spike protein of SARS-CoV-2.
  • 09:20So AstraZeneca and Janssen are using this construct again,
  • 09:22both with adenoviruses, a very common virus vector.
  • 09:26And again, we like these types of vaccines
  • 09:29because they're quick to manufacturer,
  • 09:32but a challenge of them is that your body
  • 09:34can sort of develop separate immune responses
  • 09:37against the vector itself, right?
  • 09:39So you can develop a separate immune response
  • 09:42against say an adenovirus right?
  • 09:45Such that your body neutralizes those adenoviruses
  • 09:47before they're able to infect your cells
  • 09:50and produce the SARS-CoV-2 antigen.
  • 09:52So we do see tendency a kind of faster waning
  • 09:55vaccine effects with this class of vaccines.
  • 09:58So moving on to subunit vaccine.
  • 10:00So this is NovaVax and Sanofi's vaccine
  • 10:03will be subunit vaccines
  • 10:05and this is where I kind of mentioned before
  • 10:06actually what happens here is these spike proteins
  • 10:10or whatever the antigen is,
  • 10:12is created and purified in a lab.
  • 10:14So they actually use insect cells
  • 10:17that they infect with SARS-CoV-2,
  • 10:20those insect cells then produce the antigen that's purified
  • 10:23and that's what goes into the vaccine
  • 10:26are those protein subunits, right?
  • 10:27So there we're just directly giving you the spike protein
  • 10:30that we've grown outside of the host
  • 10:33and that's how we're getting these APCs, those antigens.
  • 10:37And so this is a commonly used vaccine construct.
  • 10:40So the hep B vaccine is highly effective.
  • 10:42HPV vaccine is highly effective.
  • 10:44That's the construct of these, but the downside of course
  • 10:46to it, so it's a well-trodden way of developing vaccines.
  • 10:51But the downside is that they're slower to manufacturer.
  • 10:53There's this whole process where we have to cultivate
  • 10:55and grow these viruses in a lab, we have to purify them,
  • 11:00and moreover they often also require an adjuvant.
  • 11:04So that's really just sort of adding something a little bit
  • 11:07extra that stimulates a better immune response in your body.
  • 11:11So basically at the site of injection,
  • 11:13it's something that increases
  • 11:14your inflammatory response actually
  • 11:16to kind of stimulate your immune system
  • 11:18into recognizing those antigens
  • 11:20and developing an immune response against them.
  • 11:22So there's subunit vaccines.
  • 11:24So the fourth class here is a weakened/inactivated vaccine.
  • 11:27And so this is, I think, what most people like what
  • 11:30my grandparents probably think all vaccines are,
  • 11:33is basically we take a pathogen
  • 11:35and we weaken it in some way, or we kill it, right?
  • 11:39And then that's the construct of the vaccine
  • 11:40and that's what's injected into you.
  • 11:42And we go through this similar process there
  • 11:45that literally mimics natural infection, right?
  • 11:47Where your cells are infected by this weakened form
  • 11:51of the virus, the virus replicates,
  • 11:53and that's how we get antigens to the APCs.
  • 11:56So this is the construct used in of course
  • 11:58some classic vaccines like MMR, polio vaccine,
  • 12:01but again, it's slower manufacturing, right?
  • 12:03Because we have to cultivate the virus
  • 12:04in the lab and then it also requires adjuvants.
  • 12:07So I don't think there's currently any plans
  • 12:10to have US phase three trials
  • 12:12of weaken inactivated vaccines, but there are in China.
  • 12:15So Sinopharm and Sinovac vaccines
  • 12:17were using this construct.
  • 12:21So that's just a bit of a background in immunology
  • 12:24and how all this works and how we think about preventing
  • 12:26infection with SARS-CoV-2 and hopefully preventing
  • 12:29clinical disease COVID-19 disease.
  • 12:32So now we're gonna segue to talk a little bit
  • 12:34about the vaccine development process, right?
  • 12:36'Cause this has all happened extremely fast.
  • 12:38So let's talk about sort of the process whereby
  • 12:40vaccine products are typically brought to market, right.
  • 12:43And what looks a little bit different
  • 12:45about the COVID-19 vaccine development process?
  • 12:49So this is a figure from a nice New England journal paper
  • 12:52that's referenced at the bottom
  • 12:54that's just talking about sort of what's different
  • 12:56this go around in terms of how are we accelerating
  • 12:58the vaccine development process.
  • 13:00And so I think as biostatisticians,
  • 13:01anyone who works on clinical trials is fairly familiar
  • 13:04with the traditional paradigm
  • 13:06for bringing products to market, right.
  • 13:08It involves sort of a lot of R&D
  • 13:11in the lab, preclinical work
  • 13:13and then you start doing human trials in phase one,
  • 13:16these are small dose finding safety trials,
  • 13:19checking whether these vaccines
  • 13:21generate any immune response.
  • 13:23And then what we'll often do is in vaccine trials
  • 13:25is run a small randomized trial.
  • 13:27That's a phase two trial, right?
  • 13:29We're we'll have a placebo control,
  • 13:31maybe pick out a particularly high risk population
  • 13:34and start to see if we're getting
  • 13:35any efficacy signal, right?
  • 13:37And this is a very deliberate process, right?
  • 13:39Phase one typically advances very slowly.
  • 13:41We have lots of safety concerns.
  • 13:42Phase two, we think very hard about whether the efficacy
  • 13:45signal was really worth it to advance a candidate to
  • 13:48phase three and it's a very deliberate process, right?
  • 13:51To get to this phase three licensure trial, right?
  • 13:53So the phase three trial is the big one involving
  • 13:56the most participants.
  • 13:57It's a randomized controlled trial, right?
  • 14:00Enrolling many, many subjects that's well powered
  • 14:02to detect ethicacy signals and based on the results
  • 14:04of that phase three trial and safety data
  • 14:07that's been accumulated throughout
  • 14:08this whole process, right.
  • 14:10We're able to provide licensure ideally for a product.
  • 14:14And so that's sort of the clinical development process,
  • 14:17but also in the context of COVID vaccines
  • 14:19it's important to think about
  • 14:20the manufacturing process, right.
  • 14:21And how that looks a little bit different.
  • 14:23So typically right, companies are very sort of hesitant
  • 14:27to scale up manufacturing before they know that they have
  • 14:31a product that will be licensed, right.
  • 14:32Which makes sense, you know, they're sort of risk averse.
  • 14:34We don't want to start manufacturing a product
  • 14:36that may ultimately be shot down by the FDA.
  • 14:39So really large scale manufacturing
  • 14:40is not happening until after product licensure.
  • 14:44So what's happening with COVID vaccine
  • 14:45is basically this whole long deliberate timeline
  • 14:49is being compressed into a shorter time period.
  • 14:52And so how do we do that?
  • 14:53Well, basically what happens is we've collapsed
  • 14:56the phase one and phase two trials, right?
  • 14:58So we're doing small safety studies.
  • 15:00We're checking whether these vaccines
  • 15:01are generating immune responses,
  • 15:03but we're really not doing that smaller efficacy study
  • 15:06that is typical of vaccine development.
  • 15:10And so we're collapsing the phase one and two process,
  • 15:13the phase three process is where we're at, right.
  • 15:15We're doing these large scale trials, right?
  • 15:16Because we need robust efficacy data
  • 15:19and we need robust safety data to gain licensure,
  • 15:22but a big thing that has changed, so the clinical process
  • 15:24yeah a little bit compressed, but mostly the same,
  • 15:27the big thing that's changed
  • 15:28is the manufacturing process, right.
  • 15:30Is we wanna make sure that once a vaccine is licensed
  • 15:33and is proven to be safe and effective that we're able
  • 15:37to start distributing that vaccine immediately.
  • 15:39So that means that manufacturing needs to start ramping
  • 15:41up right before we ever have a signal of efficacy
  • 15:45and that's a huge risk for companies to take.
  • 15:47So, I'll talk in a couple of slides about sort of how
  • 15:51the government has come in to try to remove
  • 15:54some of that risk from these companies
  • 15:56and then the next slide I think is just showing sort of
  • 15:59that it's really impressive that we're even talking
  • 16:01about potentially having a COVID vaccine available this year
  • 16:05or early next year, just given the timelines
  • 16:08that are required to bring effective vaccines to market.
  • 16:11And so here's just a few, you know,
  • 16:13polio, measles, chickenpox, mumps,
  • 16:14all multiple years of development for these vaccines,
  • 16:18you could add malaria on this list.
  • 16:19It took about 30 years
  • 16:20to get a partially effective malaria vaccine to market.
  • 16:24So this is typically a very long process, right?
  • 16:26And for COVID, we're looking at hopefully doing this
  • 16:29in just under a year or two.
  • 16:31So how is the US government playing a role in this?
  • 16:35Well, it's through this program that you may have heard of
  • 16:37called Operation Warp Speed,
  • 16:39which is this huge convoluted mess of an amalgamation
  • 16:44of programs across the government
  • 16:45from DOD to many branches of NIH, BARDA, NIAID,
  • 16:50so it's sort of all over the place.
  • 16:51And this is really just the same figure
  • 16:54that I showed you from the New England journal paper.
  • 16:57Just maybe a slightly more confusing
  • 17:01if you ask me, I don't think Edward Tufte,
  • 17:03he would be a big fan of graphic
  • 17:05but the point here I want to mention
  • 17:07is how is the government responding
  • 17:10to COVID vaccine development?
  • 17:11How are they contributing to that process?
  • 17:13Well, there's really two ways that they've offered
  • 17:15to accelerate the process.
  • 17:17The first is through funding
  • 17:19of phase three clinical trials, right?
  • 17:21So a number of companies, six of the major companies,
  • 17:24basically every company that's running a phase three trial
  • 17:26in the US besides Pfizer that you've heard about
  • 17:30is contracting with BARDA.
  • 17:32That's an arm of the NIH,
  • 17:34they're contracting with the government
  • 17:36to have the government fund their phase three trials.
  • 17:39So it's a joint agreement between the government
  • 17:40and these companies where the government,
  • 17:42you the taxpayer, right, are paying for these
  • 17:45phase three trials that will eventually lead to licensure.
  • 17:49So that's the first way that the government
  • 17:50is sort of throwing money at this problem.
  • 17:52It's through design and paying for these phase three trials.
  • 17:56The second way is that they're paying
  • 17:58for manufacturing, right?
  • 17:59They're removing that risk for these companies
  • 18:01by basically committing to buy a certain number of doses
  • 18:04before we ever have any efficacy data.
  • 18:06So we're in the hole basically to all of these companies
  • 18:08for a fixed number of doses right.
  • 18:11But that motivates the companies then to scale up
  • 18:13their manufacturing ahead of the time
  • 18:15that efficacy data are available.
  • 18:18And that type of agreement has been entered
  • 18:20into with Pfizer as well.
  • 18:21So all of these companies that OWS Operation Warp Speed
  • 18:24is running the phase three trials for
  • 18:27also have this manufacturing agreement.
  • 18:29Pfizer has that manufacturing agreement as well.
  • 18:33So what role have I played in any of this big messy thing?
  • 18:38So I work with a great group of scientists
  • 18:41in the COVID-19 Prevention Network.
  • 18:43So this was a clinical trials network established
  • 18:45by National Institute of Allergies and Infectious Disease
  • 18:48and NIAID so that's an arm of NIH,
  • 18:51and it's basically anyone who works in clinical trials
  • 18:54is fairly familiar
  • 18:55with these clinical trials networks, right?
  • 18:57It's an amalgamation of researchers and study sites,
  • 19:01laboratories, people who focus on recruitment and retention
  • 19:04of trial participants, statisticians.
  • 19:07So it's researchers who are really experts
  • 19:09in running clinical trials,
  • 19:10designing clinical trials
  • 19:12and ensuring their robust conduct.
  • 19:15So the CoVPN was formed by basically leveraging
  • 19:19four existing clinical trials networks.
  • 19:21One of which I was a part of,
  • 19:22which is the HIV vaccine trials network.
  • 19:24And so from our group, we've really brought a great group
  • 19:27of statisticians, many of whom are at the Fred Hutch
  • 19:30in Seattle as well as great groups of laboratories at U Dub.
  • 19:34And so what are the roles
  • 19:36that we're playing in these trials?
  • 19:38So in our statistical group,
  • 19:40there's a couple of statisticians who are designated
  • 19:44as like CoVPN representatives
  • 19:46for each of these phase three trials.
  • 19:48So I sit on calls with these trials and advise
  • 19:53on their design and analysis approaches
  • 19:55for their efficacy monitoring, for their safety monitoring.
  • 19:58We help them address DSMB and FDA comments
  • 20:01and sort of that's all happening in conjunction
  • 20:04with both government statisticians, right.
  • 20:06Representatives of BARDA and NIAID
  • 20:10as well as company statisticians.
  • 20:12And so we get on these calls and, you know,
  • 20:14nerd out over clinical trials,
  • 20:16statistical decision-making, and it's a good old time.
  • 20:21Another aspect that we really contribute a lot on,
  • 20:24or that CoVPN has sort of been tasked with taking
  • 20:26the lead on is the development of immune correlates.
  • 20:29And so that's the part of my talk
  • 20:31where I'll get a little bit into statistics
  • 20:32and talking about what immune correlates are,
  • 20:34some of the types of analytic approaches
  • 20:36we use to study those and the idea of immune correlates
  • 20:38just to give you a teaser
  • 20:40so you don't, you know, sign off Zoom early.
  • 20:42So immune correlates are really the idea there is
  • 20:45we're looking for immune responses that are predictive
  • 20:48of the vaccines working, right.
  • 20:52So what we'd really like to be able to do is understand,
  • 20:54okay, if we're able to generate this level
  • 20:56of neutralizing antibody,
  • 20:58then that will lead to this level of protective effect
  • 21:00of the vaccine, right?
  • 21:02So that's the whole goal there is identifying
  • 21:04what are these immune responses that are
  • 21:05responsible for providing protection?
  • 21:08Okay so I'm gonna walk through just a few of the design
  • 21:11and analysis questions.
  • 21:12And so these are things that have come up
  • 21:14as we've worked with these company statisticians,
  • 21:16as we thought about sort of the whole OWS vaccine program,
  • 21:20what are some of the issues that statisticians
  • 21:22are kicking around and people who have worked
  • 21:24on clinical trials, right,
  • 21:25a lot of these issues aren't gonna be new
  • 21:27and one thing that I think is sort of interesting about this
  • 21:30whole pandemic and operating as a public health professional
  • 21:34in this and a clinical trial statistician in particular,
  • 21:37is that a lot of things that we take for granted
  • 21:39as scientists are either very confusing
  • 21:42or sort of counterintuitive for a lot of the lay public.
  • 21:45And so it's been sort of interesting to have that laid bare.
  • 21:48In some of these issues, some of these things
  • 21:50that we think are no-brainers like doing interim analysis
  • 21:53for example are kind of highly controversial
  • 21:56and have ended up being, you know,
  • 21:57sort of areas of huge disputes.
  • 21:59And so I just want to run through some of these issues
  • 22:01that I think are quite fascinating, a lot of which,
  • 22:04you know, really don't have a correct answer
  • 22:06and they're really just sort of food for thought
  • 22:07the types of things that we're thinking about
  • 22:09when we're designing these trials.
  • 22:11So I'll start by just giving a sort of more specific idea
  • 22:16of what these trials look like and how they're conducted
  • 22:18and I've picked AstraZeneca because that's the one
  • 22:20I've worked on for the longest and most closely,
  • 22:23but all of the trials sort of follow this similar design.
  • 22:26And so the first thing I'll note
  • 22:27is that you can read these trial protocols.
  • 22:29So one of the interesting things that's happened
  • 22:31in this COVID-19 development processes
  • 22:33is there was a huge public push led by like Eric Topol
  • 22:36and others to have the protocols of these trials
  • 22:39made public, which when it happened was I guess
  • 22:43when that push started happening, you know,
  • 22:45I emailed all my colleagues and said,
  • 22:47really do we not usually make protocols public?
  • 22:50And that was just sort of interesting disconnect
  • 22:51for me as an academic who's used to sort of everything
  • 22:54being open science and that's a no brainer right.
  • 22:57Working in this setting, right,
  • 22:58where these protocols are really seen as trade secrets
  • 23:00for pharmaceutical companies.
  • 23:02So it's really unusual that actually these protocols
  • 23:05for clinical trials have been made public.
  • 23:06So it's sort of neat, but one of the things that happened
  • 23:09is all of these protocols went public and reporters
  • 23:12got their hands on them and said, wow,
  • 23:13these are really dense documents, right?
  • 23:15If you've ever looked at the clinical trial protocol,
  • 23:18it's like a hundred pages of very specific definitions
  • 23:21and safety monitoring and what symptoms lists
  • 23:24you're gonna use and what surveys
  • 23:26you're gonna give to people.
  • 23:27So they're very sort of detailed documents
  • 23:28that are kind of hard for the public to parse.
  • 23:32So it's been sort of a be careful what you wish for thing
  • 23:34in terms of releasing these protocols, but that's an aside.
  • 23:38So let's talk about actually what these trials look like.
  • 23:40So here's a schematic, and again,
  • 23:42this is AstraZeneca in particular,
  • 23:44but this is basically the design of most of these trials
  • 23:47will look something like this.
  • 23:48So who is the population?
  • 23:50Most of these trials are gonna be primarily in adults.
  • 23:53I think Pfizer has now started
  • 23:55to talk about including children.
  • 23:57I'm not exactly sure where that's happening,
  • 23:59but adults for the most part,
  • 24:01these are mostly healthy individuals
  • 24:05that don't have, you know, chronic diseases
  • 24:07that are at risk or high risk of death.
  • 24:10And we're really looking at targeting individuals
  • 24:12who are at an increased risk for SARS-CoV-2 acquisition
  • 24:15and severe COVID disease
  • 24:17and so the idea there is number one
  • 24:19these are the people that are bearing
  • 24:20the brunt of the pandemic, right?
  • 24:23So we want to be able to get a product to those people
  • 24:26as fast as possible.
  • 24:27But number two also, right, that means that we'll accrure
  • 24:29from a sort of cold hearted and statistician point of view
  • 24:32that means we'll accrue end points faster.
  • 24:34We'll observe more cases of COVID-19 disease
  • 24:37and potentially get an efficacy signal a little bit faster.
  • 24:40So there's a lot of interest in sort of recruiting
  • 24:43and retaining individuals at high risk for COVID-19.
  • 24:45So you can go onto the COVID-19 prevention trials network
  • 24:48and fill out a survey, right.
  • 24:49Then we'll basically under the hood
  • 24:51assess your risk for COVID-19
  • 24:53and if you're found to be at high risk,
  • 24:54we'll aggressively email you and try to get you
  • 24:55enrolled in one of these trials.
  • 24:57If you're at low risk,
  • 24:57we'll say, thanks for taking the survey,
  • 24:59we'll be in touch and likely
  • 25:01you won't hear from us anytime soon.
  • 25:03Okay so that's the trial population.
  • 25:05So how does the actual trial conduct look?
  • 25:07So there's kind of a mixture here.
  • 25:10AstraZeneca is using a two to one randomization scheme.
  • 25:13So you have two chances of getting the active vaccine
  • 25:16versus one chance of getting a placebo.
  • 25:18And in this case, it's a true placebo, just a saline dose
  • 25:22and then most of the vaccines, most all with Janssen
  • 25:26being the accepted are two dose vaccines.
  • 25:28So you receive the first dose at day one
  • 25:30and the second dose about a month later.
  • 25:32And in the interim, we take a couple of measurements.
  • 25:34We have a phone call to assess reactogenicity right.
  • 25:38Does your arm hurt, or have you experienced any adverse side
  • 25:41effects of the first dose of vaccine?
  • 25:44And then there's also an immune response measurement
  • 25:46that happens after a couple of days.
  • 25:48So we get an early signal
  • 25:49of how immunogenetic these vaccines are.
  • 25:51And so then individuals come in for their second dose
  • 25:53of vaccine and it's a similar story, right?
  • 25:55Did you have any reactions?
  • 25:57We measure your immune response and after that,
  • 25:59that's sort of when the clock starts for active follow-ups.
  • 26:02So this day 57, that's two weeks roughly after,
  • 26:07am I doing that math right?
  • 26:08Well, it looks like roughly two weeks after the second dose
  • 26:12of the vaccine is typically when this clock
  • 26:14is gonna start and we're gonna start counting COVID events.
  • 26:17And then it's sort of just the standard sort of game we play
  • 26:21in clinical trials.
  • 26:22We wait for events to accrue.
  • 26:23We have certain monitoring plan
  • 26:25for when we're gonna check for efficacy
  • 26:27and we'll talk about some of that.
  • 26:28So, I just want to note that there's sort of two ways
  • 26:30that we're ascertaining events
  • 26:32that are happening here, right?
  • 26:33The first is passive monitoring.
  • 26:35What that means is we basically wait for individuals
  • 26:37to present with symptoms of COVID 19, right?
  • 26:39So you get a cough, you lose taste, right?
  • 26:41You call the study site, right?
  • 26:45So I am having these symptoms.
  • 26:46They say, come on in.
  • 26:47You get a PCR test to see whether you're infected.
  • 26:50And in that case, you would count
  • 26:51as a COVID-19 endpoint, right?
  • 26:53If you check off some check boxes for symptoms
  • 26:56with COVID-19 disease, you have a PCR positive test.
  • 26:59You'd go down as a COVID 19 endpoint.
  • 27:01There's also these sort of active follow-up visits.
  • 27:04So these like day 90, day, 180 and day 360,
  • 27:08and at those visits we'll do a serology check.
  • 27:11And what that means is we basically take a blood draw
  • 27:13and we measure whether you have antibodies
  • 27:16against SARS-CoV-2, right, antibodies that are distinct
  • 27:19from the antibodies that are generated
  • 27:20in response to the vaccine.
  • 27:21So we're basically able to tell whether you were infected
  • 27:24in this sort of interim period,
  • 27:26when you show up for these visits.
  • 27:29So that's active follow up
  • 27:30and so there you're gonna be able
  • 27:31to pick up sort of asymptomatic cases, right?
  • 27:33'Cause if you never have symptoms, you'll never come in
  • 27:36and be captured by passive followup.
  • 27:38So we have to wait for these set clinic visits
  • 27:40to do the serology testing,
  • 27:42to ascertain it asymptomatic cases.
  • 27:44And so this is gonna actually play a role
  • 27:46in a little bit, when I started talking about, you know,
  • 27:48what are the end points that we're thinking about measuring?
  • 27:50Like, what do we want to know how well
  • 27:52the vaccine works at preventing?
  • 27:53Is it asymptomatic infection?
  • 27:55Is it disease?
  • 27:56Is it severe disease and so forth?
  • 27:57So we'll talk through some of those issues,
  • 27:59but just want to note already that the design has started
  • 28:02to inform some of the challenges that we might see
  • 28:04when we want to talk about how well the vaccine works
  • 28:07against certain forms of infection and disease.
  • 28:10And so I think if you read the newspaper and you'll see
  • 28:13the term vaccine efficacy tossed around a lot.
  • 28:15So the first thing I want to talk about is right,
  • 28:17what is the primary hypothesis
  • 28:19that these trials are trying to test?
  • 28:21And what is the parameter?
  • 28:23What is the estimate, right, that they're going after
  • 28:25in these trials and for whatever reason
  • 28:27nobody consulted me when they decided that VE
  • 28:29would be measured in this way.
  • 28:32But for whatever reason, we studied this that we quantify
  • 28:35the efficacy of a vaccine in a sort of weird way.
  • 28:37So a vaccine efficacy, we describe as the percent reduction
  • 28:40in relative risk comparing vaccine to placebo.
  • 28:43So it's this one minus a risk ratio.
  • 28:46There's a one minus a risk ratio where you take the risk
  • 28:49in the vaccine and the numerator and the risk
  • 28:51in the placebo and the denominator.
  • 28:53So, I mean,
  • 28:54we can just play a quick little intuitive game, right?
  • 28:56How do we get a VE close
  • 28:57to one that would be a perfect vaccine?
  • 29:00Well, we would make the risk
  • 29:01in the vaccine close to zero, right?
  • 29:03So that sorta makes sense.
  • 29:04If you have a perfectly effective vaccine,
  • 29:06there'll be no risk of infection and or disease
  • 29:08amongst the vaccinated.
  • 29:09So you would get VE close to one.
  • 29:11But on the other hand, how do we make VE zero?
  • 29:13Well, we would take the risk in the vaccine
  • 29:16and set it equal to the risk in the placebo, right.
  • 29:18In which case basically saying the vaccine's not doing
  • 29:21anything and then on the other hand,
  • 29:23a VE is negative, right?
  • 29:24That's indicating that there's actually higher risk
  • 29:26in the vaccine.
  • 29:28So just to give you sort of a few reference points, right?
  • 29:31So that VE of one is perfect, VE of zero is nothing
  • 29:34and what we're really hoping for with these COVID trials
  • 29:38is a VE of at least 50%.
  • 29:40And that's sort of the cutoff that FDA guidance
  • 29:42has stipulated is that you need to show a point estimate
  • 29:46of VE for your primary end point.
  • 29:48And again, we'll talk about what these primary end points
  • 29:50are but we need a VE against a primary end point
  • 29:53of at least 50%
  • 29:54and we need to definitively rule out the possibility
  • 29:59that the vaccine efficacy is less than 30%.
  • 30:02So basically we have to reject the null hypothesis
  • 30:05that VE is less than 30% along with having a point
  • 30:08estimate of VE being greater than 50%, right.
  • 30:11And we need to do that while controlling type one error
  • 30:13at two and a half percent.
  • 30:16Okay and so here, just one final note,
  • 30:19since this is a statistics talk,
  • 30:20I'll talk a little bit more
  • 30:21about what I mean by risk, right?
  • 30:23So risk here can be quantified in a number of ways
  • 30:26and it often is.
  • 30:27So we can quantify this using hazards, for example,
  • 30:29like you can imagine fitting a Cox model, right.
  • 30:31A proportional hazards model, right.
  • 30:33That only adjusts for vaccine, right.
  • 30:35And presenting like one minus a hazard ratio
  • 30:37from a Cox model, that's something that's commonly done.
  • 30:39You can also think about cumulative incidents, right?
  • 30:41So like mapping,
  • 30:43maybe one minus a survival probability as a way
  • 30:46of quantifying risk, incidents rate ratios.
  • 30:49So they're all sort of used for different vaccines.
  • 30:52And usually we like to sort of argue
  • 30:55about which one of these is better
  • 30:56and I've thought a lot about that in my career.
  • 30:58And in this setting, it turns out because COVID
  • 31:00is such a rare event that all of these ways of quantifying
  • 31:03rates are basically the same and you end up
  • 31:05with almost identical operating characteristics of a trial.
  • 31:08So it's really not worth sort of losing sleep over
  • 31:10whether we're talking about VE in terms of hazard
  • 31:12or incidents or incidents rate and so forth.
  • 31:16So how are folks going about estimating this VE?
  • 31:19Here's just a quick table of the four most advanced
  • 31:22phase three trials,
  • 31:23the ones that have released their protocols at least.
  • 31:25So we see for Moderna, AstraZeneca, and Janssen,
  • 31:28they're using pretty kind of the standard approaches.
  • 31:31Moderna a Cox model as I describe,
  • 31:33AstraZeneca a Poisson regression model,
  • 31:35it's like, okay, that's basically a Cox model,
  • 31:38and then Janssen is using a sort of exact binomial test
  • 31:41with this sequential probability ratio rest.
  • 31:44Pfizer is a little bit of the oddball.
  • 31:46So they have stipulated a bayesian approach
  • 31:49wherein they're basically specifying a prior
  • 31:53for vaccine efficacy and are using sort of
  • 31:55a beta-binomial bayesian approach to evaluate
  • 31:58the posterior probability of the vaccine efficacy
  • 32:00is greater than 30% and so at the end of the day,
  • 32:04there's four different statistical methods here.
  • 32:06Again, if you do a simulation study with parameters
  • 32:09that are approximately similar to what we expect to see
  • 32:11in these COVID trials,
  • 32:12you're really not gonna see much difference in terms
  • 32:14of operating characteristics across these.
  • 32:16So it's interesting to notice that assertions
  • 32:18that there's these different approaches,
  • 32:19but at the end of the day,
  • 32:20we're basically talking about how many vaccinated people
  • 32:22get infected, how many placebo people got infected,
  • 32:25and almost all of these methods are gonna yield
  • 32:27very similar inference.
  • 32:29When it comes down to brass tacks,
  • 32:31how many numbers fall into those categories?
  • 32:33So that's a little bit about sort of
  • 32:36how we quantify VE in these settings
  • 32:38but one of the big things I haven't described yet
  • 32:40is VE against what, right?
  • 32:42What is the end point that we're measuring here?
  • 32:43And so here's a figure from a paper we just had come out
  • 32:47in Annals of Internal Medicine, the link's here.
  • 32:50So this is where we were spending a lot of time,
  • 32:52you know, earlier this summer, thinking about,
  • 32:54you know, what's the right end point,
  • 32:56what's the right end point for a primary analysis
  • 32:58of the clinical trial.
  • 32:59And it's complicated for something like SARS-CoV-2, right?
  • 33:03Because we know we can start up here
  • 33:04with the SARS-CoV-2 infection, right?
  • 33:07That's sort of the base, you can become infected
  • 33:09and then a number of things can happen, right?
  • 33:11You can go on to be infected but develop no symptoms.
  • 33:14So we would call that an asymptomatic infection,
  • 33:17or you can develop symptoms, right.
  • 33:18In which case we don't call you
  • 33:20a SAR-CoV-2 infection anymore,
  • 33:21we call you a COVID-19 disease endpoint.
  • 33:25You have a clinical manifestation of your infection.
  • 33:28But even beyond that, right,
  • 33:30amongst people who exhibit symptoms
  • 33:31some of them, maybe many of them are quite mild, right.
  • 33:34So we have this kind of category of non-severe COVID,
  • 33:38whereas others we know that are extremely adversely
  • 33:41impacted by infection and end up with severe COVID disease.
  • 33:45So you have all of these choices of sort of
  • 33:49which end points you might want to talk about
  • 33:51and so I'll kind of walk through some what I see
  • 33:53as the positives and negatives of this and then I'll also
  • 33:56talk about this burden of disease
  • 33:58very briefly end point that we've put together
  • 34:01and so that's kind of a composite end point
  • 34:03that we've suggested that could kind of bring all
  • 34:05of these different end points together.
  • 34:07Okay so starting with SARS-CoV-2 infection, right?
  • 34:09Why might we like any sort of any infection, right.
  • 34:12Asymptomatic, symptomatic don't care,
  • 34:14let's count any infection as an event
  • 34:17and measure VE against preventing infection.
  • 34:20Okay and so that's definitely relevant, right.
  • 34:22It's relevant the context of a pandemic.
  • 34:24We're preventing infections,
  • 34:25we're preventing spread of the disease,
  • 34:27we're bringing our knot down, we're impacting the pandemic.
  • 34:30And moreover, we're going to see many more infections
  • 34:33than we will cases of symptomatic disease.
  • 34:36We know that many people who were infected
  • 34:37never go on to develop symptoms
  • 34:39so thinking about having an answer faster, right.
  • 34:42SARS-CoV-2 infection is a nice endpoint,
  • 34:44but then the question is,
  • 34:46is it a clinically relevant endpoint?
  • 34:47So it's really not describing an impact on patients at all.
  • 34:52So we could kind of question its relevance
  • 34:56from that perspective.
  • 34:57The other thing, right, is that we remember going back
  • 34:59to the study design, we're only able to ascertain
  • 35:01asymptomatic infections sort of very coarsely in time
  • 35:05and moreover you have this phenomenon that happens
  • 35:09is that when you're testing many, many individuals, right.
  • 35:12It's sort of the classic biostat
  • 35:14one-on-one problem that we give people, right.
  • 35:16You're testing many individuals, but the prevalence is low.
  • 35:19So even if you have high sensitivity and high specificity,
  • 35:22you could end up with low positive predictive value.
  • 35:25And the effect of that when you come to the time to analyze
  • 35:28the data is that you'll be biasing VE towards the knoll.
  • 35:31So it's actually, while it seems like maybe a nice end point
  • 35:35from the perspective of observing many infections,
  • 35:37it's a very challenging endpoint to analyze quantitatively.
  • 35:41So moving down we could talk about COVID.
  • 35:43So again, COVID is just infection,
  • 35:46PCR confirmed infection with clinical symptoms.
  • 35:49So that's of course more clinically relevant, right.
  • 35:51Because we're starting to talk about
  • 35:53an impact, excuse me, the endpoint that impacts patients.
  • 35:57All right so that's more clinically relevant and moreover
  • 36:00we'll expect to have a reasonable number of cases, right.
  • 36:03By including more mild cases, for example,
  • 36:06in this endpoint definition.
  • 36:09But then on the other side of that coin
  • 36:10is it really that clinically relevant
  • 36:12if we're just talking about mild symptoms?
  • 36:15We're talking about a disease where you get it
  • 36:16and you end up with a little cough for a couple of weeks
  • 36:18and that's it.
  • 36:19So then maybe you suggest using severe COVID right.
  • 36:22That's the most clinically relevant one.
  • 36:24We want to be protecting the most vulnerable individuals
  • 36:26so we should be quantifying how well our vaccines
  • 36:28work towards preventing those most severe end points.
  • 36:33And so most clinically relevant,
  • 36:35and also there's sort of a long history
  • 36:37of vaccine development where really we see the best VE
  • 36:40against severe cases of disease.
  • 36:43So that's really where we expect the vaccines
  • 36:45to have the most impact is maybe we are not preventing
  • 36:47you from being infected but we're lessening the symptoms
  • 36:51once you become infected.
  • 36:52So we're not totally blocking transmission
  • 36:55but we're making a clinical impact on disease
  • 36:57and that's sort of been seen
  • 36:58for a number of vaccines in the past.
  • 37:00The downside of this end point of course
  • 37:02is that there's very few cases expected to be observed.
  • 37:04So amongst all infections,
  • 37:05only a fraction have any symptoms.
  • 37:07Amongst those with any symptoms,
  • 37:09only a fraction develops severe symptoms.
  • 37:10So we're really whittling away the number of end points.
  • 37:13So we need to do larger trials
  • 37:15or have longer follow-up to evaluate this endpoint.
  • 37:19And so in that paper, I'm sort of pressed for time
  • 37:21so I won't spend too much time talking about this,
  • 37:24we also proposed this burden of disease measure
  • 37:26where you're sort of scoring these these outcomes, right?
  • 37:29So maybe you would get a score of zero
  • 37:31if you're an asymptomatic infection
  • 37:33'cause it's really no burden on you as a patient, right?
  • 37:36You don't have any symptoms.
  • 37:37And then we're sort of assigning arbitrarily
  • 37:39a score of one for non severe COVID so that's like
  • 37:42mild cases of COVID and a score of two
  • 37:45for severe cases of COVID and this end point actually
  • 37:48has some nice operating characteristics we think,
  • 37:51but of course it's subject to controversy, anytime you start
  • 37:53talking about an ordinal scoring system, right,
  • 37:57you start to raise questions about how you're assigning
  • 37:59the burden of disease score, right?
  • 38:01Why should severe cases be a two
  • 38:03versus a three versus a five and so forth?
  • 38:06So you can kind of get bogged down
  • 38:07in some of the specifics of that.
  • 38:10So what has FDA said about this?
  • 38:12So FDA guidance documents states that either
  • 38:15the COVID end point or SARS-CoV-2 infection
  • 38:18is an acceptable primary endpoint
  • 38:19and then somewhat ironically OWS has been telling companies
  • 38:22that infection alone is not acceptable
  • 38:24as a primary end point.
  • 38:25So we had one company that was interested in including
  • 38:28that as co-primary and for whatever reason we told them
  • 38:31please don't do that, and then beyond that so COVID
  • 38:36has sort of won out as the end point of choice.
  • 38:39But beyond that FDA guidance states that companies should
  • 38:42consider powering efficacy trials
  • 38:44for the severe COVID endpoint as a co-primary or at least
  • 38:48as a key secondary endpoint in the trial.
  • 38:51And so so far only Janssen has taken them up on that offer
  • 38:54of making severe COVID primary.
  • 38:56And that's why, if you look at the number of individuals
  • 38:58that are planning to enroll in their trial,
  • 38:59it's twice as many as any of the other OWS trials.
  • 39:03So like AstraZeneca is planning for 30,000,
  • 39:04Janssen is planning for 60,000 in their trial.
  • 39:08And that's the power, to see more cases of severe disease
  • 39:10to be sufficiently powered to detect VE against that.
  • 39:15So this is a controversial slide.
  • 39:17Or this is virtual topic I found,
  • 39:20again, something that clinical trials statisticians
  • 39:22sort of take for granted is doing interim analyses, right?
  • 39:26If the treatment is working and we have enough evidence
  • 39:28to claim that a treatment is working,
  • 39:29we'd like to stop that trial early
  • 39:31to get that treatment to patients, right.
  • 39:33One would think that that's true here
  • 39:35and so many of these trials
  • 39:36were designed with aggressive sort of interim looks, right?
  • 39:40Because we're in the middle of the pandemic
  • 39:41and we'd like to get a vaccine to individuals
  • 39:44as quickly as possible.
  • 39:45So I have a table, we won't go through it all here,
  • 39:47just sort of the planned interim analysis
  • 39:50for these different trials.
  • 39:52I would say Pfizer seems to be the most aggressive so far.
  • 39:56They have five interim looks or four interim looks
  • 40:00and a final look at their data, right?
  • 40:02So that's fairly aggressive.
  • 40:03OWS again, the trials that we're running,
  • 40:07we're really encouraging companies to be a bit
  • 40:09more conservative in the approach to this
  • 40:11and only maybe two or three
  • 40:13and so you see what's been adopted
  • 40:14by Moderna and AstraZeneca
  • 40:17and so this was really a big point of contention
  • 40:19I think when these protocols were made public is this idea
  • 40:22that like, can you really know that a vaccine works
  • 40:25based on 32 data points, right?
  • 40:27We're talking about a vaccine that's going to be given
  • 40:30to billions of people around
  • 40:32the world based on these results
  • 40:33and you're gonna do that based
  • 40:35on the results in 32 individuals?
  • 40:37And like, so I can stare at the math and say that like, yes,
  • 40:40that appropriately controls type one error and so forth,
  • 40:42but it still makes me just feel a little bit uncomfortable.
  • 40:45There's a bit of dissonance between sort of my life
  • 40:48as a statistician and just me being a human
  • 40:50and saying 32 data points is probably not enough
  • 40:52to decide to vaccinate billions of people.
  • 40:54And so a lot of people I think sort of shared
  • 40:57that viewpoint and in response FDA has now been sort of
  • 41:01backpedaling in a way and asking companies to provide more
  • 41:06data in order to grant an emergency authorization
  • 41:10for their vaccine.
  • 41:11So this EUA mechanism that FDA has of approving vaccines.
  • 41:14And so in addition to an efficacy signal,
  • 41:17now companies also are gonna be required, I think,
  • 41:20and this is sort of still a moving target so this is maybe
  • 41:23like data news at this point but I think prior to offering
  • 41:26an EUA, FDA has now said that companies need to have 50%
  • 41:29of participants complete at least two months of follow-up
  • 41:33for safety signals and that you need to have at least
  • 41:36six COVID cases in the oldest age group.
  • 41:39Of course, that's an age group of particular interest
  • 41:41in terms of severe cases and at least five cases
  • 41:44of severe COVID in the placebo group.
  • 41:45So they want to be able to see some data,
  • 41:48even if you're not specifying severe COVID
  • 41:50as a primary end point,
  • 41:51they want to be able to see some data,
  • 41:53some signal of efficacy against that
  • 41:55in order to grant licensure.
  • 41:57So I'll sort of, I won't go through this slide.
  • 42:01It's just to say that like,
  • 42:02sort of when Pfizer released their protocol,
  • 42:04everyone was like, ooh a bayesian analysis
  • 42:06and got very sort of skeptical, right?
  • 42:09Because the Pfizer CEO has been out there
  • 42:11sort of chest thumping and saying they're gonna have
  • 42:13a vaccine before the election and so forth
  • 42:15and then they came out with this bayesian design
  • 42:17that was a little atypical and so everybody was asking
  • 42:19the question, well, are they trying to hide something?
  • 42:21So I sort of did a quick analysis
  • 42:23and found that really it doesn't look that different
  • 42:25than a classic kind of post hoc monitored design.
  • 42:28And if you want to read more about that,
  • 42:30I have some slides up on my GitHub about it.
  • 42:33So let's see, I'm running low on time
  • 42:36so I'm gonna skip over sort of the question
  • 42:39of what happens if efficacy is declared early.
  • 42:41So I have some reasons that we should be excited, right?
  • 42:43If one of these trials stops earlier, I can get a vaccine.
  • 42:46There's good data that the vaccine works
  • 42:48and that's nice.
  • 42:50I'd like to go back to something resembling normal
  • 42:52as I'm sure you all would,
  • 42:54but of course there's reasons to be concerned, right?
  • 42:56If efficacy is declared early in particular,
  • 42:58if that means that blinded follow-up
  • 43:01in a study stops, right?
  • 43:02Because that means we have no way
  • 43:03to assess how durable the vaccine is.
  • 43:05We won't be able to assess VE
  • 43:07and key subgroups that we care about.
  • 43:09We might not be able to assess VE
  • 43:11formally against severe end points.
  • 43:13So there's real sort of concerns
  • 43:15about stopping these trials too early,
  • 43:17and what the implications of that
  • 43:18are both for evaluating the vaccine in question,
  • 43:21but as well as how it impacts
  • 43:23the other clinical trials that are ongoing.
  • 43:26And of course in the current political climate,
  • 43:29everybody's very concerned about the role
  • 43:31political pressure might play in all of this.
  • 43:33So yeah, so it's kind of a double-edged sword in some sense
  • 43:37in terms of what happens if efficacy is declared early,
  • 43:42but I want to save just a few minutes
  • 43:43to talk about vaccine correlates 'cause I promised
  • 43:45that I would show you some math and prove to you
  • 43:47that I'm a real statistician.
  • 43:48So let's do a little bit of that.
  • 43:51So again, we're kind of shifting gears here.
  • 43:53So that's the end of sort of talking about the primary
  • 43:55analysis of these trials,
  • 43:56what's gonna lead to their licensure.
  • 43:58And the correlates of protection
  • 44:00is sort of a key secondary analysis
  • 44:02and so why is it so important
  • 44:04that we're able to establish correlates of protection?
  • 44:07Well, because it's gonna speed up
  • 44:09the whole vaccine development process.
  • 44:12So again, a correlative protection is really just,
  • 44:14it's an immune response and really an assay
  • 44:18to measure that immune response that's been validated
  • 44:20to reliably predict vaccine efficacy.
  • 44:23So why is that so important?
  • 44:25Well, basically what we're hoping to achieve
  • 44:28is the establishment of a surrogate
  • 44:29endpoint for COVID disease right?
  • 44:32So I've sort of mentioned the numbers that we're talking
  • 44:34about in these phase three trials,
  • 44:36enrolling 30,000 participants, 60,000 participants
  • 44:40and ending up with one or two years of followup, right.
  • 44:42Just to be able to answer the primary question, right.
  • 44:44Does the vaccine prevent infection and/or disease?
  • 44:48So that's a huge, expensive clinical trial.
  • 44:50It takes a long time to get an answer
  • 44:52and so it would be very nice if all we had to do right
  • 44:56was give people the doses of vaccine that they need,
  • 44:59wait two weeks and measure their immune response
  • 45:02and understand does that vaccine work or not.
  • 45:05That would be a much faster vaccine development process
  • 45:07than where we're currently at
  • 45:09in having to run these enormous phase three trials.
  • 45:11So it's valuable for establishing a surrogate endpoint.
  • 45:14It's also valuable for accelerating approval
  • 45:17of vaccines that have been licensed in certain populations,
  • 45:22but not others.
  • 45:23For example, I mentioned that these phase three trials
  • 45:25are mostly being conducted in adults.
  • 45:27Well, what if we want to also obtain licensure for use
  • 45:30of this vaccine in children?
  • 45:32Well, if we had an established immune correlate
  • 45:34we wouldn't have to do
  • 45:35a large randomized trial in children.
  • 45:37We could do it just a small immunogenicity study
  • 45:39and use the correlates results to bridge the VE
  • 45:42that we observed from the phase three trial.
  • 45:45That's the immune response that we've observed
  • 45:47in these children or pregnant women for example are
  • 45:49another key population they're being
  • 45:51excluded from these phase three trials
  • 45:53but we'd like to understand if these vaccines
  • 45:55are safe and effective in those women as well.
  • 45:59So really this is one of the key goals
  • 46:01of this whole OWS program and the key role
  • 46:05that we're playing in the CoVPN is developing
  • 46:08the sampling plan and the statistical analysis plan
  • 46:12for the immune correlate studies
  • 46:14and so it's just a little bit of the statistical issues
  • 46:17that we're dealing with in these trials, right,
  • 46:20is that sort of running assays
  • 46:22so running these immuno assays on 30,000, 60,000 individuals
  • 46:26takes a long time, it's expensive, and as it turns out,
  • 46:30it's really overkill in terms of statistical power.
  • 46:34So we can actually be a little bit more
  • 46:35clever about how we design these correlate studies in order
  • 46:40to get answers faster and more cheaply.
  • 46:42So the way we do this is we use a case cohort design.
  • 46:45So we're not gonna measure immune responses
  • 46:47in all trial participants,
  • 46:48we're gonna measure them in a sub cohort
  • 46:50and that sub cohort will consist
  • 46:51of a stratified random sub cohort.
  • 46:54So we're gonna be sampling individuals randomly
  • 46:56based on their baseline infection status.
  • 46:58Were you infected with SARS-CoV-2 in the past?
  • 47:01Based on your race, ethnicity, and based on age.
  • 47:07And so based on that, we'll take a random draw
  • 47:09of the trial population, about 1600 individuals,
  • 47:13excuse me and everyone so I should mention right
  • 47:19in the trial design everybody is having their blood drawn.
  • 47:22And right now we're talking about whose blood
  • 47:24are we gonna use to measure these immune responses?
  • 47:27So we're gonna measure it in a random sample
  • 47:29and then we're gonna wait until the trial is over
  • 47:31or until one of these interim analysis concludes efficacy
  • 47:35and we're gonna measure immune responses
  • 47:38in all of the end points, right?
  • 47:39Remember that like power in these analyses is drive
  • 47:42by the individuals in which we observe endpoints.
  • 47:46So we're gonna make sure we get immune responses
  • 47:47in all the end point data, as in addition
  • 47:49to this random sub cohort and it turns out that that's about
  • 47:53as statistically efficient as running the immune assays
  • 47:57on all 30,000 individuals in the trial.
  • 47:59So this is this kind of classic case cohort design
  • 48:01that Ross Prentice has been writing about for years
  • 48:04that Norman Breslow did some sort of pioneering work
  • 48:06on in the 2000s and I'll just talk a little bit about sort
  • 48:11of how this complicates our life as statisticians
  • 48:13and then maybe we'll leave a few minutes for questions.
  • 48:16So here's the math, we made it.
  • 48:18Well, the moment you've all been waiting for it
  • 48:20to see some math.
  • 48:21So just introducing, you know,
  • 48:23why is this sampling design challenging
  • 48:26from a perspective of generating estimators, right?
  • 48:29Well, we can sort of immediately see that this isn't
  • 48:31a totally random sample of the trial population, right?
  • 48:35In particular we've over-sampled the individuals who end up
  • 48:38getting diseased and it's fairly obvious
  • 48:42that those individuals have potential to be very different
  • 48:45than a randomly selected individual in the population.
  • 48:47So we have a bias sub sample.
  • 48:49So we need some statistical methodology to try to back out,
  • 48:52you know, whatever this parameter is.
  • 48:54We want to be estimating it in the whole trial population,
  • 48:56not just in this biased sub samples.
  • 48:59So how do we do that?
  • 49:00So just a quick notation here,
  • 49:02let's call W baseline covariates,
  • 49:04A is a binary vaccine assignment,
  • 49:07Y is your binary COVID endpoint for example
  • 49:11and then we'll introduce this sort of indicators.
  • 49:13Delta is one, if you're selected into this immune response
  • 49:18sub cohort, either because you were a case,
  • 49:20you were an end point or because you were randomly selected
  • 49:23into the cohort.
  • 49:25And then we'll call S your immune response.
  • 49:28And then we'll just say, we'll represent this as Delta S,
  • 49:31which just means we'll arbitrarily set everybody
  • 49:33who's not in our sub cohorts immune response to be zero,
  • 49:36that's arbitrary doesn't really matter.
  • 49:38So let's talk about how estimation would happen.
  • 49:41So let's pick a very simple parameter, right?
  • 49:43Let's just say that we want to know what's the overall
  • 49:45immune response in the whole population,
  • 49:47not a particularly interesting parameter
  • 49:50for actually measuring correlates,
  • 49:51but just to motivate the types of statistical approaches
  • 49:54that we use in these settings.
  • 49:56So how can we control for the bias of the sampling design?
  • 49:59Well, one of the most straightforward ways
  • 50:01is to use the tried and true
  • 50:02Horvitz-Thompson or IPTW estimator, right.
  • 50:05Where we're just taking basically a sample mean
  • 50:08but all our observations are sort of inverse weighted
  • 50:11by their probability of being sampled into this sub cohort.
  • 50:16And so that's, IPTW estimator if you're in causal inference,
  • 50:18you're very familiar with this.
  • 50:19If you're in survey sampling,
  • 50:21very familiar with this.
  • 50:22Very classical way of adjusting for this selection bias.
  • 50:27It turns out that there's ways
  • 50:28that we can be more efficient in doing this.
  • 50:30We can use augmented estimators, AIPTW estimators.
  • 50:33And the key idea there is that we take the IPTW estimator
  • 50:37and we add a little bit of something to it
  • 50:39and the key thing is that that little bit
  • 50:41of something involves a regression of S the immune response
  • 50:46onto the covariates that were used to sample individuals
  • 50:50into the sub cohort.
  • 50:52And so what's the intuition as
  • 50:54to why this is more efficient?
  • 50:56Well, you can imagine what if we had a perfect predictor
  • 50:59of S measured at baseline, right?
  • 51:01Then this regression here is essentially imputing
  • 51:05the correct value of S
  • 51:06for every single person in the population.
  • 51:09So it's kind of like we're getting more data
  • 51:11in some sense, and the nice thing about
  • 51:15these approaches, these AIPTW approaches
  • 51:17is that they're double robust and so again,
  • 51:18if you work in causal inference a very familiar idea,
  • 51:21and it turns out because we know
  • 51:23the sampling probability by design,
  • 51:25this regression doesn't have to be consistently estimated
  • 51:28in order to obtain a consistent estimate
  • 51:30of the parameter measures.
  • 51:31So it's this really nice sort of double robustness property
  • 51:33that says, yeah, you might be turned off
  • 51:35from this augmented estimator
  • 51:36because you have to do a little bit of extra work,
  • 51:38you have to fit a regression model say,
  • 51:40and maybe you're worried about missspecifying
  • 51:42that regression well it turns out that because the sampling
  • 51:44probabilities are known by design,
  • 51:46you don't have to concern yourself with that.
  • 51:47So it turns out you can use any old regression estimator
  • 51:50here and still end up with a consistent estimate
  • 51:53of the parameter of interest.
  • 51:54And so we're applying this
  • 51:55to much more interesting parameters.
  • 51:57So we had a paper come out recently
  • 51:59in biometrics that's linked here
  • 52:01where we're starting to study a sort of causal inference
  • 52:03flavored parameters in this context,
  • 52:06things that we can really use to pin down,
  • 52:08you know, mechanisms of these vaccines working.
  • 52:10So, in this case, we're studying sort of the effect
  • 52:13of a stochastic intervention, we call it.
  • 52:16So it's sort of saying what would happen
  • 52:18if we took everybody's immune response,
  • 52:20this particular immune response that we observed,
  • 52:22and we shifted it up just a little bit
  • 52:25or we shifted it down just a little bit.
  • 52:26How would that impact the risk of disease amongst
  • 52:30the vaccinated individuals?
  • 52:30So that's what this big, gnarly parameter is right here.
  • 52:34And so you ended up looking at a plot
  • 52:35that's kind of like this.
  • 52:36So this is from an HIV vaccine trial.
  • 52:39So at zero we're saying that's just the observed risk
  • 52:42of the trial and as we move left we're saying,
  • 52:44what would the risk be if we decreased your immune response?
  • 52:47And so we can see in this example,
  • 52:49we found that the risk would be increasing, right.
  • 52:52And then if we're moving to the right
  • 52:53is what would happen if we increase your immune response.
  • 52:57And so we're kind of getting at something
  • 52:59that's like a controlled effects mediation type parameter
  • 53:03with this approach and so we're working out some
  • 53:06of the details of the correlates plan currently
  • 53:10and so when that's done
  • 53:11we'll have it available for public comment.
  • 53:13And again, we're academics, right?
  • 53:14So we'll do it all open science.
  • 53:16And then I'll just say like two words of conclusion
  • 53:18and I'll shut up and leave some time for questions.
  • 53:21So there's been a big concern
  • 53:23in the current political climate that we're gonna sneak
  • 53:26something through, that something's gonna be approved
  • 53:28without sort of the standard amount of evidence
  • 53:32that would be required, right.
  • 53:33That there's political interference at the FDA
  • 53:36and from where I sit, you know,
  • 53:39I can say that the science behind the vaccine
  • 53:41development program for COVID is extremely rigorous.
  • 53:43These are exactly the type of people who you would want
  • 53:46in charge of this decision making process
  • 53:48and the type of people that will raise red flags
  • 53:51as soon as sort of the process goes off the rails.
  • 53:54So right now I feel good about where things stand.
  • 53:57Of course, I watch presidential debates and hear, you know,
  • 54:00garbage science coming out and I get a little bit concerned,
  • 54:04but from where I sit right now,
  • 54:05everything's looking pretty good.
  • 54:07So overall, I'd say that the increased transparency
  • 54:09by releasing these protocols
  • 54:11has been good for scientists and consumers.
  • 54:13We want to bring vaccines to market,
  • 54:15but we also want people to trust those vaccine
  • 54:17so increasing transparency in whatever way we can is great.
  • 54:20And then finally, the final point is that a lot of these
  • 54:23issues that I've talked about,
  • 54:24how do we do interim monitoring, right?
  • 54:26What's the right end point to be studying?
  • 54:28What's the right S demand, right?
  • 54:30These are really hard decisions
  • 54:32and there are no right answers.
  • 54:34And so one of the things that's been a little bit
  • 54:37disconcerting or disheartening to me
  • 54:40is the extent to which in the pandemic era,
  • 54:43academic debates have been made very much public
  • 54:46and I'm not against academic debates.
  • 54:49It's just that most individuals aren't used to seeing them.
  • 54:52And so what I'm worried is happening is that people
  • 54:55see high profile academics debating these challenging
  • 55:00problems where there's no real right answer.
  • 55:02And they're saying, well, these guys don't know
  • 55:03what they're talking about.
  • 55:05So I think as academics and public health professionals
  • 55:08in this pandemic, one thing that we can do
  • 55:10is just to be very careful in how we're presenting,
  • 55:12you know, the science that we're doing
  • 55:15and acknowledge when there's not a right answer,
  • 55:17that you're presenting your opinion.
  • 55:19And that there is some validity, right?
  • 55:21That this is very gray, unfortunately,
  • 55:23that there's nothing black and white here.
  • 55:25So maybe that's a controversial statement to end on,
  • 55:28but I'll end there and then thanks again to Fan
  • 55:30for giving me the opportunity to talk
  • 55:32and I'm happy to take questions as there's time.
  • 55:35I don't have anything scheduled after this,
  • 55:36so I can stay a few minutes over as would be helpful.
  • 55:39So thanks again.
  • 55:41- [Fan] Thank you David for this very nice talk.
  • 55:44I think we do have three to four minutes for questions
  • 55:47from the audience, if there's any.
  • 55:54- [Woman] Hi David, I have a question
  • 55:55'cause right now for COVID situation and because of the time
  • 56:00and the faster progress of the disease
  • 56:04and it's a hard to keep the standard method,
  • 56:08but do you have other proofed vaccine for other disease
  • 56:14and have a quick trial have a similar way as COVID
  • 56:19and apply the method you're using right now
  • 56:24and we have standard results already
  • 56:27and then compare to see how good the current method is.
  • 56:32So that's my question.
  • 56:35- [David] Yeah it's an interesting question.
  • 56:37So let me try to restate, so you're saying,
  • 56:39are there any lessons from vaccine development
  • 56:42that we can try to draw from here
  • 56:44to evaluate our methodology, whether it work?
  • 56:48- [Woman] Yes, from other vaccines.
  • 56:52- [David] So I guess what I would say is that at this stage,
  • 56:55in phase three vaccines, these phase three trials
  • 56:58look completely normal.
  • 57:00So I would say the process of getting to the phase three
  • 57:03looked very different and much more accelerated
  • 57:05in terms of kind of squashing together
  • 57:07phase one and phase two in terms of the manufacturing,
  • 57:11but in terms of what's happening in a phase three trial,
  • 57:13this is probably the phase three trial
  • 57:15that would be done outside of the setting of a pandemic.
  • 57:18Maybe the interim analysis would be a little bit
  • 57:20less aggressive for some of these companies, but really,
  • 57:23I think the approaches that the companies are taking
  • 57:27would be fairly standard even in any other vaccine context.
  • 57:35- -Woman] Yeah. I mean, even though
  • 57:37for the established vaccine,
  • 57:41there could be some field trial
  • 57:43and that they also went through a phase three,
  • 57:47but you can do the similar thing to enhance,
  • 57:51to see whether it is possible to pass the current protocol
  • 57:57and become some sort of false positive.
  • 58:02- [David] Yeah and, you know, I think speaking,
  • 58:08I mean, speaking of failed vaccines,
  • 58:09as someone who works in HIV vaccines,
  • 58:11we're very familiar with failure and learning from that.
  • 58:15So again, I think the people who are running these trials
  • 58:17are sort of the right people in terms of looking out
  • 58:20for these false positive signals and so forth.
  • 58:24- [Woman] Thank you.
  • 58:27- [Fan] So I think we are just about the time
  • 58:30and I'm sure that David is happy
  • 58:32to take your questions afterwards by email.
  • 58:35So I'll thank David more time.
  • 58:37Again, thank you for sharing with us
  • 58:39and we'll see everyone again next week.
  • 58:43- [David] Thanks everybody.