Multi-Model Outbreak Decision Support (MMODS)

The Problem

The profusion of models for COVID-19, with differing structures, varied epidemiological scenarios, parameters and presentation, and sometimes conflicting projections, is a challenge for decision-makers. In a recent paper, we proposed a method for harnessing the power of multiple models by drawing from tools in decision analysis, expert judgment, and model aggregation (Shea et al. 2020). This project is meant to implement that proposal in the context of COVID-19. We aim to generate unbiased and well-calibrated aggregate projections under different interventions, that encapsulate scientific and logistical uncertainty, to better inform management decisions. In this framework, insights can be shared across groups to inform the same decision, while retaining the perspective of individual groups as part of the full expression of uncertainty.


The overall goals of this project are to implement these procedures for a series of COVID-19 decisions; engage a diverse set of modeling groups with expertise in structured, collaborative, ensemble projections; and develop efficient logistical processes for managing our broad communal effort. This is a complementary effort to the COVID-19 forecasting hub developed by Nick Reich and colleagues, with an explicit focus on interventions and decisions.


Specifically, we will run multiple projection exercises to address key decisions facing managers of COVID-19, including when and how to relax key social distancing interventions (exercise I). In later exercises, we will use model assemblages to assess more nuanced partial reopening strategies, intervention decisions at state and country levels, where best to trial vaccines and drugs, how to prioritize testing and how to optimize the roll-out of medical interventions. We will request, from each participating group, one or more models that encapsulate their group’s best understanding of the current pandemic (that is, we will treat each model as an hypothesis about the current outbreak). Results will be kept confidential within the group until presentation to decision-makers or in publication(s). When presented outside the group, only model participation will be disclosed (individual model results will be anonymized).



We will use principles of decision analysis to help structure model projections and analysis, and adopt well-established methods from the expert judgment literature so that the results from multiple modeling groups can all contribute to insights about the same decision context and contribute to a synthetic and long term resolution to the current pandemic.


For each exercise, we will take the following steps:

      1. Setting. We will present a decision setting, specifying the background epidemiology (location, outbreak trajectory), the targets of the decision maker (e.g., minimizing deaths, epidemic duration, etc.), and the intervention(s) to examine. Relevant epidemiologic and demographic data will be shared.

      2. Individual Projections 1. We will ask each modeling group to independently estimate the desired outcomes under the alternative interventions, with particular attention to expressing uncertainty. We will ask for probability distributions for each outcome and intervention scenario.

      3. Group Discussion. We will compile the results from the multiple modeling groups and display them (anonymously) in a format that permits ready comparison. We will convene a group discussion with all the modeling groups to explore the commonalities and differences, to share insights, and to discuss sources of uncertainty.

      4. Individual Projections 2. We will then ask each modeling group to independently project the same targets under the alternative interventions again, taking into account the insights from the group discussion to the extent they find them compelling. We will again ask for probability distributions.

      5. Aggregation and Analysis. We will then aggregate the second round of results into a set of ensemble projections that captures the uncertainty within and across modeling groups. We will also conduct a value-of-information analysis to identify sources of uncertainty that most affect the choice of an intervention. The summary of this work should be an analysis that conveys to the decision maker the expected performance of each of the interventions, using the ensemble projections, with an understanding of the role of uncertainty.


Exercise I of the elicitation: post-lockdown strategies

Please note, Exercise 1 is currently underway and closed to new participants. If you would like to be contacted with information regarding future elicitations, please email the project coordinators at


Setting and initial conditions:

We ask that you consider the setting of a US county of 100,000 people, with an age structure typical of the age structure across the US, that pre-emptively initiated, and adhered to, stringent social distancing guidelines (i.e., full lockdown with workplace and school closures) until May 15th, 2020. As of 15th May 2020, the town has recorded 180 confirmed cumulative cases and 6 total deaths (time series for both provided). Please assume current (i.e., partial) travel restrictions remain in place throughout the exercise, so that no international importation is allowed and domestic importations are limited.


The decision maker is the county executive, who has authority to specify guidance for opening workplaces. The focus is on decisions regarding social distancing and re-opening over the next few months, prior to the onset of the influenza season.


Projection outcomes/objectives:

The county executive has indicated they are interested in weighing the trade-offs among a number of outcomes, including the impact of the disease on public health, hospital resources, and the local economy. To reflect these objectives, we ask participating modeling groups to address 5 outcomes (metrics):

      1. cumulative number of all infected individuals through November 15
      2. cumulative number of COVID-related deaths from May 15 to November 15
      3. peak hospitalizations from May 15 to November 15
      4. probability of a new local outbreak (more than 10 new reported cases/day) from May 15 to November 15
      5. total number of days workplaces closed from May 15 to November 15


In this first exercise, we will only consider relaxation related to workplaces. We request that you provide model projections for the following 4 intervention scenarios:


      1. continue with current non-essential workplace and school closures from May 15 to November 15
        (baseline full control scenario)
      2. relax current social distancing 2 weeks after peak:
        ▪ open non-essential workplaces only (schools remain closed through November 15)
      3. relax social distancing when the number of new daily cases is at 5% of peak
        ▪ open non-essential workplaces only (schools remain closed through November 15)
      4. immediately relax all current restrictions on non-essential workplaces
        (schools remain closed through November 15)


For now, please assume no local testing/contact tracing and isolation of infected individuals; we will return to evaluate this in a future elicitation. You are however free to define and present results for any other relaxation process you feel is relevant or interesting.


Models should provide a full probability distribution of outcomes for each intervention, such that tail probabilities for the 2nd and 98th quantiles are relatively stable. Specifically, we want the probability distribution for each outcome for each intervention, by specifying the cumulative distribution function (i.e., with 100 quantiles).


Background information on your model: Please provide a short write up of your model, including assumptions made about key epidemiological parameters, with parametric uncertainty (e.g., transmission, recovery, R0, serial interval). Please document all sources of variation in your model using the checklist provided. We are looking for full expression of uncertainty in these projections. For example, uncertainty may be structural (e.g., should asymptomatic carriers be modeled explicitly?), or parametric with respect to the biology (e.g., what is the expected time between sequential cases in a chain of transmission?), the setting (what is the assumed rate of domestic importations?) or the interventions (e.g., what is the expected impact of social distancing?) or there may be other sources of stochasticity. Other key uncertainties you might scan across might include: controllability of social distancing, probability of novel incursions that might lead to a second wave of local infections, etc. Details of any model calibration or inference framework used should be provided (checklist will be provided). If we do not specify something, please use your best judgment and include that in your modeling of uncertainty (and please let us know in your short model description and in the checklist). Do not hesitate to send questions to, and please provide any other information you feel is pertinent so we can update our checklist for future exercises.


MMODS Core Team:

Katriona Shea1, Michael C. Runge2, Shou-Li Li3, William J.M. Probert4, Emily A. Howerton1, Rebecca K. Borchering1, Tiffany L. Bogich1, Wilbert van Panhuis5, Cécile Viboud6



1Department of Biology and Center for Infectious Disease Dynamics, The Pennsylvania State University, University Park, PA, USA.

2U.S. Geological Survey, Patuxent Wildlife Research Center, Laurel, MD, USA.

3State Key Laboratory of Grassland Agro-ecosystems, Center for Grassland Microbiome, and College of Pastoral, Agriculture Science and Technology, Lanzhou University, Lanzhou, People’s Republic of China.

4Big Data Institute, Li Ka Shing Centre for Health Information and Discovery, University of Oxford, Oxford, UK.

5Department of Epidemiology, University of Pittsburgh Graduate School of Public Health, Pittsburgh, PA, USA.

6Fogarty International Center, NIH, Bethesda, MD, USA.