Close

DRAFT: Reusability and FAIR Data

Table of Contents
DRAFT - V0.0 February 2021

Introduction

As part of the MIDAS Coordination Center’s efforts to bring increased reproducibility and FAIR (Findable, Accessible, Reusable, and Interoperable)1 data principles to infectious disease modeling research, we have developed this series of recommendations for sharing of relevant software and data artifacts.

Our hope is that these recommendations will provide greater transparency and clarity to disease modeling efforts, both in terms of community methods and supporting communication with the public and policy-makers. When possible, these efforts should align with accepted and emerging community best-practices, such as the EPIFORGE 2020 guidelines for reporting epidemic forecasting and prediction research.2

These proposed guidelines have been developed by the MIDAS Coordination Center. In the interest of improving the clarity and utility of these suggestions, we’re interested in any feedback or comments that you might have. Please use the comment form below to let us know what you think.

Sharing Software

  1. Operating Systems where the code has been used, or is believed to be capable of running correctly
  2. Programming languages used, with versions (where available).
  3. Any related software dependencies, including, but not limited to
    1. Database engines (Postgres, MySQL, Neo4J,etc.)
    2. Operating environments such as R Studio, Conda, Jupyter, Cytoscape
  4. Libraries or related packages
  5. All components should include version numbers and links to appropriate sites for download/installation
  1. For command-line tools, list the parameters to be specified, describing each as required or optional.
  2. Enumerate assumptions regarding the format and content of all command-line parameters and data file fields. Examples may include data types (integer, floating point, string); restrictions (free-text vs. selection from one of a set of enumerated categories; positive integers); and any dependencies (parameter B must be given if parameter A is given)
  3. For any tools requiring input of data from a file or in a data structure provide a description of the file format, either in text or (when possible) in a machine-readable schema in a computable form such as JSON schema. JSON-LD or XSD.
  4. Provide pointers to any data dependencies, such as public data files that must be retrieved to run the tool. Whenever possible, specific versions of those data dependencies should be listed. When version numbers are not available, the date of download should be provided.
  1. Instructions for compiling, building, testing, and execution.
  2. Sample parameters, input files, and resulting outputs for command-line programs.
  3. For software libraries, minimally-functional examples of how various routines might be used, including construction of inputs and interpretation of results. These examples would ideally illustrate the content/format of the results of the calls in question.
  1. Pointers to code repositories (GitHub, GitLab, etc.)
  2. Pointers to issue trackers, online discussion groups, or individual email addresses where questions might be asked or bugs might be reported.
  1. Command-line flags: “-v ” is often used as means of retrieval of the version number for an executable
  2. Appropriate package metadata and/or API calls for libraries
  3. “About” screens for GUI applications
  4. Appropriate package metadata for programming libraries
  1. Source control
  2. Automated builds
  3. Automated testing (unit tests, Selenium)
  4. A long-term software management plan
  5. Modern best coding practices.7
  1. Epidemic model
  2. Agent-based disease transmission model
  3. Compartmental disease transmission model
    1. Susceptible-Infected-Susceptible Model
    2. Susceptible-Infected-Resistant Model
    3. Susceptible-Expoosed-Infectious-Recovered Model
  4. Metapopulation disease transmission model

Sharing Data

1. Provide detailed metadata8. Specific data field and definitions below include items from the Data Catalog Vocabulary (DCAT)9, the W3 HCLS Dataset Description Profile10, along with others specific to infectious disease modeling:
  1. Title: the name given to the item (dct:title)
  2. Creator: the entity responsible for producing the resource (dct:creator)
  3. Publisher: the entity responsible for making the item available (dct:publisher)
  4. Description: a free-text account of the item (dct:description)
  5. Geographic region(s) covered: in as much detail as possible, using country codes and amin-2 codes for states provinces, etc. from geonames or ISO 3166-2 names.
  6. Topic: One or more terms or keywords describing the topic of the data. The MIDAS data catalog has a list of terms that have been used to describe datasets: one or more of these terms should be used as possible. The Apollo-SV ontology has additional related terms.
  7. Pathogen(s) if relevant. The name of the organism(s) associated with the outbreak. Whenever possible, pathogens should be described using terms from standard vocabularies such as NCBI Taxon, the NCI thesaurus, or SNOMED CT.
  8. Dates covered: by the dataset: the start and end dates delineating the time period covered by the dataset. If the dataset is being regularly updated, the end date might be omitted or indicated via a term such as “ongoing”.
  9. License: a legal document under which the resource is made available (dct:license). This will generally take the form of a URL pointing to a textual description of a license.
  10. Access requirements: Information about who has the right to access the data, and under what terms (dct:accessRights or dct:Rights). This will generally take the form of a URL pointing to a textual description
  11. Provenance: datasets derived from other, pre-existing datasets should provide links (DOIs or URLs) to the original dataset, along with a description of how the original dataset was modified to create the present dataset.
  12. Link to to paper(s) describing the data11
  13. Version numbers and dates and descriptions of updates, where applicable. Infrequently-changed datasets should include explicit version numbers, along with dates of updates and brief descriptions of the nature of the updates in each version. Regularly updated datasets might use the date of last update (see “Dates”, above) as the version number.
  14. Links to other relevant data and metadata. Possibilities include related datasets, or shared descriptions of common data collection/curation methods.
  15. A data dictionary, describing the meaning of each of the data attributes/columns and the range and source of possible values (integral, continuous, or categorical, based on specific terms or entries from a public ontology)
2. Use public vocabularies and ontologies for metadata and attribute values whenever possible.8 Several examples, including the MIDAS list of modeling topics, are given above. Other ontologies found in resources such as Bioportal or Ontobee can be used to identify potentially useful ontologies with relevant biomedical content.
3. Data and metadata should be presented in a clear and understandable format. Data fields and column names should be clear, detailed, and ideally defined by a data dictionary defining how each field should be interpreted. Whenever possible, use a standard format for metadata, such as the Google dataset search model.
4. Provide a DOI or other persistent identifiers for both the data and (separately) for the metadata describing that data.6,8
5. Whenever possible, store data and metadata in a durable repository. Possibilities include Zenodo, data dryad and related repositories. Ensure accessibility of metadata if data aren’t available8
6. Metadata and data should be indexed in a searchable resource8. Possibilities include Google Dataset Search or the repositories discussed above.
7. Describe terms of use, limitations, and any authorization/authentication requirements8.
8. Share any relevant analysis code11

References

  1. Wilkinson MD, Dumontier M, Aalbersberg IjJ, Appleton G, Axton M, Baak A, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016 Mar 15;3(1):160018.
  2. Pollett S, Johansson MA, Reich NG, Brett-Major D, Del Valle SY, et al. (2021) Recommended reporting items for epidemic forecasting and prediction research: The EPIFORGE 2020 guidelines. PLOS Medicine 18(10): e1003793. https://doi.org/10.1371/journal.pmed.1003793
  3. List M, Ebert P, Albrecht F. Ten Simple Rules for Developing Usable Software in Computational Biology. PLOS Comput Biol. 2017 Jan 5;13(1):e1005265.
  4. Lamprecht A-L, Garcia L, Kuzak M, Martinez C, Arcila R, Martin Del Pico E, et al. Towards FAIR principles for research software. Data Sci. 2020 Jan 1;3(1):37–59.
  5. Lee BD. Ten simple rules for documenting scientific software. PLOS Comput Biol. 2018 Dec 20;14(12):e1006561.
  6. Garijo D, Kinnings S, Xie L, Xie L, Zhang Y, Bourne PE, et al. Quantifying reproducibility in computational biology: the case of the tuberculosis drugome. PloS One. 2013;8(11):e80278.
  7. Romano JD, Moore JH. Ten simple rules for writing a paper about scientific software. PLOS Comput Biol. 2020 Nov 12;16(11):e1008390.
  8. Jacobsen A, de Miranda Azevedo R, Juty N, Batista D, Coles S, Cornet R, et al. FAIR Principles: Interpretations and Implementation Considerations. Data Intell. 2020 Jan 1;2(1–2):10–29.
  9. Albertoni R, Browning D, Cox S, Gonzalez-Beltran A, Perego A, Winstanley P. Data Catalog Vocabulary (DCAT) – Version 2 [Internet]. [cited 2021 Dec 1]. Available from: https://www.w3.org/TR/vocab-dcat-2/#Property:resource_creator
  10. Gray AJG, Baran J, Marshall MS, Dumontier M. Dataset Descriptions: HCLS Community Profile [Internet]. World-Wide Web Consortium; 2015 May. Available from: https://www.w3.org/TR/hcls-dataset/
  11. Goodman A, Pepe A, Blocker AW, Borgman CL, Cranmer K, Crosas M, et al. Ten Simple Rules for the Care and Feeding of Scientific Data. PLOS Comput Biol. 2014 Apr 24;10(4):e1003542.
 

Comments

This set of recommendations is a living document.  We will continue to improve it based on community feedback.  If you have any comments, please leave them below.

 

If you’d like to email us instead of posting a comment, please email: questions@midasnetwork.us

 

Note: To protect this page against spambots, all comments are moderated. Your comment will not be visible until it is approved by a moderator.