Close

Multi-Model Outbreak Decision Support (MMODS)

Frequently Asked Questions

Collaboration logistics

 

Q: Who should I contact for more information?
A: Please see the website at https://midasnetwork.us/mmods/ or contact the MMODS team on mmods@midasnetwork.us or Katriona Shea on k-shea@psu.edu.

 

Q: Authorship?
A: We initially anticipated e.g. a PI and a postdoc or graduate student being recognized as co-authors on any papers arising from this elicitation exercise. Please let us know if your model has more than a few authors, and we can discuss specific situations. If the journal in which we try to publish limits the number of authors, or if we have a large number of co-authors, we will have a group name (the MMODS group) for publications with all authors indexed in e.g. pubmed and on the MIDAS MMODS web page. We can also identify additional collaborators in the acknowledgment section. Please note, submission of complete Round 2 results is a condition of authorship. This includes (1) an updated submission checklist, and (2) as complete a set of forecasts as possible, including full distributions, and not point estimates (with the exception of probability of an outbreak where either a point estimate or distribution will be accepted).

 

Q: Round 2 submission deadline?
A: The submission portal will reopen around July 1, including an updated submission checklist. We ask that all teams submit complete Round 2 results no later than July 12.

 

Model setting details

 

Q: Could we please have definitions for those mobility numbers?
A: These are Google mobility indicators and the units are %. The definition is the daily percent change in workplaces visited (alternatively change in time spent at home) relative to pre-pandemic baseline, where the baseline is the median value for the corresponding day of the week during Jan 3–Feb 6, 2020. https://www.google.com/covid19/mobility/data_documentation.html?hl=en

 

Defining the peak

 

Q: In trying to determine the peak for scenarios 2 and 3, should we:

  1. Use “real” data on reported cases prior to May 15th, then switch to projected data starting on May 15th OR
  2. Define the peak entirely on simulated data (some groups are simulating the epidemic starting at earlier dates than May 15)

A: Our intent was that the interventions would be triggered based on patterns in the observed data (smoothed by a 7-day moving average), so we kindly suggest that you implement option 1. There may be a jump between simulated and observed data on May 15th, but this jump will be smoothed out by considering a 7-day moving average. However, the final decision ultimately lies with your team.

 

Q: How much of the provided data should be used for calculating the peak?

A: Please compute the moving average starting on Jan 22. We have checked that the definition of ‘peak’ provided is not met before May 14th.

Submission Information

 

Q: Our model finds that the distribution for the probability of an outbreak is a point mass at 1. Thus, our cumulative distribution function is a step function at 1. Should we still provide just a median estimate of 1?

A: If your model is able to assess the distribution (and not just provide a point estimate) for the probability of an outbreak, please indicate so by reporting 1 for each of the requested quantiles (i.e., 1st through 100th).

 

Q: For the submission spreadsheet, results are requested by quantile, separately for each intervention. But, we’re finding that the results across interventions are correlated, such that, for example, intervention 1 is equal to or better than intervention 4 for all matched replicate scenarios (that is, replicates that share the same underlying parameter set). Should we not be reporting distributions for each parameter set, so that the correlation structure across interventions is maintained?

A: It was an active choice to only ask for marginal distributions of the output variables for reach intervention (i.e., the 4×5 sets of 100 quantiles). We are aware that we are losing the correlation structure, but we wanted to include a wide range of models (some of which would not be able to report the full results you suggest). We’ll consider the importance of the correlation structure in future elicitations.

 

Future elicitations

 

Q: We’re curious to know if you can provide insight on the types of exercises that will follow this initial round of projections
A: We have a LOT of possibilities in mind, and are actively working to develop a few of them, but we hope that this will be driven by decision makers, so that the modeling effort the group of participants expends will actually be useful, and used. The second elicitation is likely to focus on locations for vaccine trials, and there are questions about test allocations and state level reopening decisions also on the table. You are also very welcome to suggest ideas – please email us at mmods@midasnetwork.us with your suggestions!

Revised and clarified descriptions of interventions and outcomes (from June 24, 2020 Group Discussion notes):

 

As a result of the discussion, we ask that groups please update their models and results, including the following revised definitions and clarifications:

    • Measurement of all objectives should start from 0 on May 15, including the number of days closed (i.e., 184 days closed for the closed intervention), cumulative infections, and cumulative deaths.
    • Clarifying interventions (see * for reasoning):
      • The open intervention should be interpreted as the reopening of non- essential workplaces on May 15. Our goal is to compare the outcomes of interventions and be able to make general recommendations for counties of similar size (of which there are approximately 100). As such, we request that model groups use their best judgment in implementing assumptions related to reopening workplaces (e.g., compliance levels), keeping in mind that this is an important source of uncertainty that we hope to encapsulate through our multi-model approach. Please make sure to document your related assumptions in the updated checklist for Round 2 results.
      • We have changed the 1-percent intervention to a 5-percent intervention, that is, to relax social distancing when the number of new daily reported cases is at 5% of the peak (see new definition for peak below), and open non-essential workplaces only. We believe 5% to be better suited to counties of this size and will enable more groups to report results for this intervention as distinct from the closed scenario.
      • For the 2-week and 5-percent interventions, we ask that all groups use the same metric and method for calculating the peak (described in detail below). We acknowledge that this is only one of several metrics and methods that could be used to determine the peak. We chose one that could be implemented by a decision-maker (e.g., as opposed to an omniscient approach) and ask that all groups follow this in order to minimize uncertainty arising from different interpretations of the intervention (as the peak is part of the intervention definition). Please do let us know as soon as possible if you think this will be a problem for your model.
        • Metric (for both 2-week and 5-percent interventions): 7-day trailing moving average of the number of new daily reported cases (as opposed to infections). This is the information that a decision-maker might have access to; the moving average is meant to smooth out noise due to reporting and low population size. Let Cases_ma(t) be the 7-day moving average of daily reported cases at time t. Find the maximum of Cases_ma(*) up to time t.
        • Method:
          • 2-week intervention. Define the trigger to open as the first day for which:
            • Cases_ma(t) has been lower than the maximum for at least 14 days, AND
              • Cases_ma(t) has shown a day-to-day decline for at least 10 of the last 14 days (where a day- to-day decline occurs when Cases_ma(t+1) < Cases_ma(t)), OR
              • Cases_ma(t) is 0 (i.e., there have been 7 days without any new cases).
          • 5-percent intervention. Define the trigger to open as the first day for which Cases_ma(t) drops below 5% of the maximum.
          • Note that the peak that triggers a two-week intervention may not be the same peak that triggers a 5-percent intervention (e.g., if there is a second peak that is larger than the first one).

Please note that for the purposes of this exercise, the reopening of non-essential workplaces (in both the 2-week and 5-percent interventions) cannot occur before May 15.

  •  
  • Clarifying objectives:
    • Cumulative infections should be interpreted as all infections (with SARS- CoV-2) in the population (regardless of reporting).
    • Deaths should be interpreted as all deaths due to COVID-19 disease- induced mortality, regardless of reporting. Note this is based on infection status, not on testing status. It can include comorbidities but not indirect deaths, as we are only focusing on people who have been infected with SARS-CoV-2 and died from their infection.
    • Peak hospitalization should be interpreted as the largest daily number of occupied hospital beds.
    • For calculating the probability of an outbreak, an outbreak is defined as 10 new reported cases per day (not actual infections per day).
  • We recognize the challenges in calculating uncertainty surrounding the probability of an outbreak. The groups who were able to provide this have a two- level treatment of uncertainty: an outer loop that samples across parametric uncertainty; and an inner loop that samples across various stochastic processes. We understand that not all models are structured this way, and we don’t expect groups to be able to add this functionality on short notice. For this objective only, the reporting of point estimates will be allowed and this estimate will be considered the median (please enter at the 50th quantile in your submission data). However, please do report the full cumulative probability distribution (100 quantiles) if you are able to do so.


*Note: We know that we did not provide all the details you might want (e.g., hospital capacity); the reason for this is that we’d like the forecasts to be relevant for decision makers in similar counties, not just for one county executive. We caution, then, against tying the results to an individual county and instead ask for an integration over uncertainty (around compliance, geography, etc.) to ensure results are sufficiently generic for counties with a population size of approximately 100,000. We ask that groups (1) include this uncertainty in model projections, and (2) describe the assumptions and uncertainty in the submission checklist.