The 2019 coronavirus disease (COVID-19) pandemic saw the emergence of numerous websites compiling regional or global COVID-19 data. At the beginning of the pandemic, we observed a lack of centralized worldwide archive of official COVID-19 data from public health agencies, ministries of health, and other governmental websites. Our goal was to fill this gap.
We created the MIDAS Curated Archive of Global Public Health Data by automatically downloading, scraping, or screenshotting data of different formats from official sources for multiple countries and territories, eight regions, and at the global level. The archive contains case, death, hospitalization, demographics, and vaccination information at country and, when possible, at lower administrative levels, collected daily or weekly for each disease of interest. We have since expanded the archive to include other diseases, including COVID-19, Mpox, Flu, and RSV.
The data are verified against the source by a combination of manual and automated processes, deduplicated, and are made publicly available with curated metadata. The archived public health data are organized by disease, geographic location, and source in “collections.” Each collection is associated with archived data that can be viewed and downloaded, with metadata information available in the MIDAS Catalog.
The archived data are organized by collection. A collection can be defined as scraped, downloaded, and/or screenshotted data associated with a disease, a geographical region, and an official source. For more information, please consult the FAQ section What is a collection? below.
The MIDAS Curated Archive of Global Public Health Data includes data extracted from official websites. As various sites use different formats and structures for data storage, collections contain a number of different formats. Most data are available in a “computable” format (HTML, JSON, CSV, etc.), but some data are only available in a “non-computable” format; for example, PNG or JPEG format.
The contents of each collection are described in the in collection metadata available in the MIDAS Catalog.
All data from the MIDAS Curated Archive of Global Public Health Data are issued from specific sources and might be available only under specific licenses and/or copyrights. When available, license and rights information were collected from each source and are included in the associated metadata.
If you believe that any material contained on the MIDAS Catalog and/or on the MIDAS Curated Archive of Global Public Health Data infringes your copyright or other intellectual property rights, or is otherwise incomplete or inaccurate, please notify the MIDAS Coordination Center via email at questions@midasnetwork.us.
The MIDAS Curated Archive of Global Public Health Data process follows 5 steps:
Each step is associated with a specific question in this FAQ.
The MIDAS Curated Archive of Global Public Health Data is organized by collection. A collection can be defined as scraped and/or downloaded and/or screenshotted data associated with a specific disease, a specific geographical region, and a specific source.
A specific geographical region can be a country or region of the world, and, for some countries, the first-order administrative division (e.g., state or province).
The most common types of specific sources include:
As data storage and description practices have evolved since we began archiving data in the early days of the COVID-19 pandemic, the types of data and the formats included in various collections may differ over time. In some cases, data within a collection may be in different formats, even for a single time point.
Additionally, as the collections were initially defined at the time of creation, and as we needed to adapt to the changes of the reporting of information from each source, the definition of collections and evolution of information in a collection might be slightly different between two geographical units or sources.
For example, a collection for one specific location may contain scraping of a first source in the format of an HTML page for the beginning of the pandemic, then later contain JSON files downloaded from a dashboard (second source) as the dashboard might be embedded in the same web page that contains the information in HTML format previously; for example, Coronavirus Disease 2019 (COVID-19) web page from the Illinois Department of Public Health. For some other location, these two sources may be separated into two collections; for example, the COVID-19 Data for Pennsylvania web page from the Pennsylvania Department of Health and the Pennsylvania COVID-19 Dashboard from the Pennsylvania Department of Health. We first extract the HTML information from the COVID-19 Data for Pennsylvania web page and later on, the Pennsylvania COVID-19 dashboard was embedded in the web page. We separated these two sources into two collections to simplify the curation process as the web page continued to be scrapped after addition of the dashboard and the dashboard extraction contains multiple JSON files and screenshots.
The identification and selection of data sources follow three steps:
We use three processes to extract the data:
We regularly use these processes to extract updated data from each site. Depending on the frequency of update of the source, we scrape/download/screenshot data between 6 times per day and 1 time per week. For example, a weekly updated dataset will be extracted at least 3 times per week. At the beginning of the pandemic, all of the processes were run 6 times per day (every 4 hours) on all the sources to be assured to capture all the different sources’ updates. However, as multiple sources changed their frequency of update, we adapted our frequency of extraction.
As the data and the format of the data have evolved through time, it is possible to have different processes running in the same collection at the same and/or at different times. For example, we can have data that was scraped from a web page in an HTML format for a certain amount of time, and later, or at the same time, in a downloaded CSV file.
We use two processes to review extracted data:
Even “official” web sites undergo changes that lead data to be deleted, moved, or otherwise become unavailable. When we are unable to access one of the links that we are using to retrieve data, we will look to see if the data is available on a new page on the same site. Subsequent steps depend upon what we find:
As the collections were defined at the same time as we were creating them and as we needed to adapt to the change of the reporting of disease-specific information from each source, the definition of collections and evolution of information in a collection might be slightly different between two geographical units or sources.
As the data can be extracted up to 6 times per day, most collections initially contain significant redundancy. To remove the duplicates, we calculate checksums on all saved files and publish only the files identified as unique.
However, this process does not always identify all “unique” files. For example, some HTML files contain the same COVID-19 data, but as the text around the data might have changed, the files will not be noted as duplicates and both files will be published in the MIDAS Curated Archive of Public Health Data. Or, as another example, there are some screenshots where the pictures look the same, but the dimensions are slightly different.
In addition to true duplicate files identified by the checksums, we also limit the number of files containing individual case report (linelist) data since these files can become very large. These limits are applied when the files:
Files that meet these criteria are limited to the first unique file archived plus the last unique file each month.
The first publication of the MIDAS Curated Archive of Global Public Health Data contained COVID-19 data for the United States of America. Since then, other diseases and regions have been added. These collections will be updated monthly with data until the previous month. For example, if an update is planned in June, it will contain data until May.
New collections will be added to the Archive as they become available.
The archived data are accessible via the MIDAS Catalog. Each collection corresponds to one entry in the data catalog. Each of these entries contains a link to a file browser interface containing summary information about the collection (disease and pathogen information, geographical information, source, associated license and/or rights information, date, etc.) and links to individual files.
All individual files in a collection are presented in a table that can be searched, filtered and/or sorted. Files can also be downloaded individually from this page. A zipped set of all files of a specific file type or all files from the collection can be made available upon request; information on how to submit the request is provided on each collection page in the file browser.
Data is extracted from the sites regularly, depending on the frequency of updates on the site. Update intervals range from 6 times per day to once weekly.
The MIDAS Curated Archive of Global Public Health Data contains data in multiple formats. As source data are provided in multiple formats and structures , a single collection may contain multiple formats.
Most of the files are:
Other less common formats include XML, TXT, and RDS (an R data file format readable using R – for more information, please consult the R project website), etc. Each collection will have the available file format information associated with the data listed in the metadata, available in the MIDAS Catalog.
A collection can also contain multiple file formats in the same time frame; for example, if a page contains data in CSV format and as a table in the HTML of the page, then the collection will contain both formats.
The file browser interface for each collection contains a table with information associated with each file: the filename, the format of the file, the size of the file, and the date the file was created, which corresponds to the date the data was extracted.
For some files, the information is also available on the name of the file in the format: “filename_YYYY-MM-DD_HHMMSS”. Not all the files contain this information, as it is not always possible to rename the file or append the date in the filename.
A collection may contain a gap in the data as the downloading and/or scraping and/or screenshot of the web page was stopped for a certain amount of time before being restarted. It might happen, for example, in the case where the data were moved to a new page and we did not find the new page immediately; or, if another page had more detailed data than the original, we may have stopped the first one to avoid redundancy. However, if the detailed page disappears, then the original source is used again.
It is also possible that the source data was not updated during a certain period of time and all the redundant archived files were removed during the deduplication step.
Yes, it is possible to view some of the data before downloading them. All browsers can be used to view HTML, PDF, JSON, etc. files. To view the data, click on the filename of the file and it will open in the browser if possible; otherwise, the file will be downloaded.
Files in some collections use encodings that may not automatically render properly in the browser, so may need to be downloaded to view.
Viewing a file in the web browser is not always the best way to “see” the data. For example, the data can be hidden in the code of an HTML, or the HTML might contain a redirect link, or a link to embedded up-to-date content when viewed in the browser. We strongly advise the users to download the files to verify the data.
For some files, the data might be mixed with other information and might be difficult to find. For some frequent cases, we can provide some additional information:
For some collections, a screenshot is available to provide more context about the data and might help with locating the data. Additional information might also be available in the metadata Description field for special cases. Please feel free to contact us for any additional information or help.
As the verification and deduplication steps are imperfect, some collections might contain files with an error and/or without data for a certain amount of time.
Individual files can be downloaded directly from the file browser interface.
A zipped set of all files from a collection, or all files of a specific file type from a collection, can be made available upon request. More information on how to request such a file set is available on each collection page in the file browser interface.
Some collections have been used to generate COVID-19 Project Tycho datasets. In August 2022, Project Tycho released COVID-19 datasets for 237 countries. These datasets conform to the updated Project Tycho 2.0 pre-compiled dataset format v1.1. For more information, please consult the Project Tycho website.
All data from the MIDAS Curated Archive of Global Public Health Data are retrieved from specific sources and are subject to restrictions determined by those providers. The license and rights information were collected from each source, if available, and is provided for each collection in the associated metadata.
If you believe that any material contained in the MIDAS Catalog and/or in the MIDAS Curated Archive of Global Public Health Data infringes your copyright or other intellectual property rights, or is otherwise incomplete or inaccurate, please notify the MIDAS Coordination Center via email at questions@midasnetwork.us.
Several pieces of temporal information are recorded for each collection:
To illustrate these definitions, we can imagine an example of a collection that can contain multiple files about COVID-19 data. The collection started on 2020-04-01 (April 1, 2020) and stopped on 2022-02-01 (February 1, 2022), because the source page was not updated since 2022-01-01. In this case:
For each entry, we collect the “geographical resolution” and the “geographical scope”.
To illustrate these definitions, if we download the state, county and ZIP level data from the Pennsylvania COVID-19 Dashboard from the Pennsylvania Department of Health, the collection geographical information will be:
Whereas, if we download the Provisional COVID-19 Deaths by Week, Sex, and Age by the US National Center for Health Statistics web page available on the data.cdc.gov website, the collection geographical information will be:
For some entries, the associated metadata indicates “special region” or “health region” as geographical scope:
Currently, only the last link used for extracting the data is available. We might publish the historical links later; if so, we will update the documentation accordingly.
If you are using the MIDAS Curated Archive of Global Public Health Data, please consider citing the interface by using:
For specific collection:
Source, Title, Retrieved from: MIDAS Curated Archive of Global Public Health Data from Official Websites, URL. Accessed date
Example: Kansas Department of Health and Environment, Kansas COVID-19 Vaccination Overview, Retrieved from: MIDAS Curated Archive of Global Public Health Data from Official Websites, https://catalog.midasnetwork.us/scrapecat/607. Accessed 7 Dec. 2022.
To cite the MIDAS Curated Archive of Global Public Health Data from Official Websites:
MIDAS Curated Archive of Global Public Health Data from Official Websites, https://midasnetwork.us/midas-archive-faq. Accessed date
If you have any questions about the MIDAS Curated Archive of Global Public Health Data or MIDAS Catalog, please feel free to contact us at: questions@midasnetwork.us.
If you have any specific question about the archived data contained in a collection, depending on if the questions are about: