Systematic review of COVID-19 policy evaluation methods and design ================================================================== Protocol registration 11/23/2020 Authors ------- Noah A Haber, ScD, (corresponding author), Meta Research Innovation Center at Stanford University, Stanford University, 1265 Welch Rd, Palo Alto, CA 94305, noahhaber@stanford.edu, Emma Clarke-Deelder, MPhil, Department of Global Health & Population, Harvard T. H. Chan School of Public Health Joshua A Salomon, PhD, Department of Medicine, Center for Health Policy and Center for Primary Care and Outcomes Research, Stanford University School of Medicine Avi Feller, PhD, Goldman School of Public Policy, University of California, Berkeley Elizabeth A Stuart, PhD, Department of Mental Health, Johns Hopkins Bloomberg School of Public Health Emily R Smith, ScD Departments of Global Health and of Exercise and Nutrition Sciences, Milken Institute School of Public Health, The George Washington University Contributions ------------- NH was the primary protocol developer, administrator, and corresponding author. EC, JS, AF, and ES provided contributions to the design and language of the protocol. Rationale --------- Policies concerning the response to COVID-19 are among the most impactful decisions of our time. Future policy is informed by what we know and learned from past policy. It is therefore critical that studies that estimate the impact of COVID-19 policies do so with strong and reliable methods, following best methodological and reporting practices, and with appropriate cautions. Unfortunately, there are substantial concerns that much of the existing literature may be methodologically flawed, which could render their conclusions unreliable for informing policy. Rather than being a review of a specific topic, this systematic evidence review examines the methodological landscape of the COVID-19 policy evaluation literature. We will examine what types of policies are being studied, what methods are being used, and the degree to which these studies meet basic requirements for study design related to policy analysis in COVID-19. The review instrument proposed was designed as consumer-oriented review guidance on the design of COVID-19 health policy impact research, and is an operationalized version of Haber et al., 2020 written by this author team, available here: https://arxiv.org/abs/2009.01940. This guidance was written to assist reviewers in identifying design issues with COVID-19 policy impact evaluation papers. This review examines the extent of these issues. Objectives ---------- The primary objective is to examine the methodological landscape of COVID-19 policy impact evaluation for markers of methods strength and reporting. We will assess what types of policies are being examined, what outcomes, what methods are being used, and numerous markers of methodological strength for policy impact estimation methods. As a secondary objective, we will be qualitatively reviewing the performance of the review guidance as it is deployed in a systematic review setting for consistency, ease of use, and clarity. Eligibility criteria -------------------- - The primary topic of the article must be evaluating one or more individual COVID-19 policies on direct COVID-19 outcomes - The primary exposure(s) must be a policy, defined as a government-issued order at any government level to address a directly COVID-19-related outcome (e.g. mask requirements, travel restrictions, etc). - COVID-19 outcomes may include cases detected, mortality, number of tests taken, test positivity rates, Rt, etc. - This may NOT include indirect impacts of COVID-19 on things such as income, childcare, trust in science, etc. - The primary outcome being examined must be a COVID-19-specific outcome, as above. - The study must be an impact evaluation study using an design from primary data (i.e. not primarily a predictive or simulation model or meta-analysis) - The study must be peer reviewed, and published in a peer-reviewed journal indexed by PubMed - The study must have the title and abstract available via PubMed at the time of the study start date - The study must be written in English Information sources ------------------- Literature will be acquired from any article published in PubMed in 2020, from January 1 until the start date of this study (expected November 25th). Search strategy --------------- The search strategy will be based on a simple boolean-based search of the PubMed literature. The search terms will combine four boolean-based search terms: a) a set of terms for COVID-19 research, b) a set of terms pertaining to regional government units (e.g. country, state, county, and specific country, state, or province, etc.), c) a set of terms pertaining to policy or policies, and d) a set of terms pertaining to impact. The search terms will be as follows: ((((wuhan[All Fields] AND ("coronavirus"[MeSH Terms] OR "coronavirus"[All Fields])) AND 2019/12[PDAT] : 2030[PDAT]) OR 2019-nCoV[All Fields] OR 2019nCoV[All Fields] OR COVID-19[All Fields] OR SARS-CoV-2[All Fields]) AND ("impact*"[TIAB] OR "effect*"[TIAB]) AND ("policy"[TIAB] OR "policies"[TIAB] OR "order*"[TIAB] OR "mandate*"[TIAB]) AND ("countries"[TIAB] OR "country"[TIAB] OR "state"[TIAB] OR "provinc*"[TIAB] OR "county"[TIAB] OR "parish"[TIAB] OR "region*"[TIAB] OR "city"[TIAB] OR "cities"[TIAB] OR "continent*"[TIAB] OR "Asia*"[TIAB] OR "Europe*"[TIAB] OR "Africa*"[TIAB] OR "America*"[TIAB] OR "Australia"[TIAB] OR "Antarctica"[TIAB] OR "Afghanistan"[TIAB] OR "Aland Islands"[TIAB] OR "Åland Islands"[TIAB] OR "Albania"[TIAB] OR "Algeria"[TIAB] OR "American Samoa"[TIAB] OR "Andorra"[TIAB] OR "Angola"[TIAB] OR "Anguilla"[TIAB] OR "Antarctica"[TIAB] OR "Antigua"[TIAB] OR "Argentina"[TIAB] OR "Armenia"[TIAB] OR "Aruba"[TIAB] OR "Australia"[TIAB] OR "Austria"[TIAB] OR "Azerbaijan"[TIAB] OR "Bahamas"[TIAB] OR "Bahrain"[TIAB] OR "Bangladesh"[TIAB] OR "Barbados"[TIAB] OR "Barbuda"[TIAB] OR "Belarus"[TIAB] OR "Belgium"[TIAB] OR "Belize"[TIAB] OR "Benin"[TIAB] OR "Bermuda"[TIAB] OR "Bhutan"[TIAB] OR "Bolivia"[TIAB] OR "Bonaire"[TIAB] OR "Bosnia"[TIAB] OR "Botswana"[TIAB] OR "Bouvet Island"[TIAB] OR "Brazil"[TIAB] OR "British Indian Ocean Territory"[TIAB] OR "Brunei"[TIAB] OR "Bulgaria"[TIAB] OR "Burkina Faso"[TIAB] OR "Burundi"[TIAB] OR "Cabo Verde"[TIAB] OR "Cambodia"[TIAB] OR "Cameroon"[TIAB] OR "Canada"[TIAB] OR "Cayman Islands"[TIAB] OR "Central African Republic"[TIAB] OR "Chad"[TIAB] OR "Chile"[TIAB] OR "China"[TIAB] OR "Christmas Island"[TIAB] OR "Cocos Islands"[TIAB] OR "Colombia"[TIAB] OR "Comoros"[TIAB] OR "Congo"[TIAB] OR "Congo"[TIAB] OR "Cook Islands"[TIAB] OR "Costa Rica"[TIAB] OR "Côte d’Ivoire"[TIAB] OR "Croatia"[TIAB] OR "Cuba"[TIAB] OR "Curaçao"[TIAB] OR "Cyprus"[TIAB] OR "Czechia"[TIAB] OR "Denmark"[TIAB] OR "Djibouti"[TIAB] OR "Dominica"[TIAB] OR "Dominican Republic"[TIAB] OR "Ecuador"[TIAB] OR "Egypt"[TIAB] OR "El Salvador"[TIAB] OR "Equatorial Guinea"[TIAB] OR "Eritrea"[TIAB] OR "Estonia"[TIAB] OR "Eswatini"[TIAB] OR "Ethiopia"[TIAB] OR "Falkland Islands"[TIAB] OR "Faroe Islands"[TIAB] OR "Fiji"[TIAB] OR "Finland"[TIAB] OR "France"[TIAB] OR "French Guiana"[TIAB] OR "French Polynesia"[TIAB] OR "French Southern Territories"[TIAB] OR "Futuna"[TIAB] OR "Gabon"[TIAB] OR "Gambia"[TIAB] OR "Georgia"[TIAB] OR "Germany"[TIAB] OR "Ghana"[TIAB] OR "Gibraltar"[TIAB] OR "Greece"[TIAB] OR "Greenland"[TIAB] OR "Grenada"[TIAB] OR "Grenadines"[TIAB] OR "Guadeloupe"[TIAB] OR "Guam"[TIAB] OR "Guatemala"[TIAB] OR "Guernsey"[TIAB] OR "Guinea"[TIAB] OR "Guinea-Bissau"[TIAB] OR "Guyana"[TIAB] OR "Haiti"[TIAB] OR "Heard Island"[TIAB] OR "Herzegovina"[TIAB] OR "Holy See"[TIAB] OR "Honduras"[TIAB] OR "Hong Kong"[TIAB] OR "Hungary"[TIAB] OR "Iceland"[TIAB] OR "India"[TIAB] OR "Indonesia"[TIAB] OR "Iran"[TIAB] OR "Iraq"[TIAB] OR "Ireland"[TIAB] OR "Isle of Man"[TIAB] OR "Israel"[TIAB] OR "Italy"[TIAB] OR "Jamaica"[TIAB] OR "Jan Mayen Islands"[TIAB] OR "Japan"[TIAB] OR "Jersey"[TIAB] OR "Jordan"[TIAB] OR "Kazakhstan"[TIAB] OR "Keeling Islands"[TIAB] OR "Kenya"[TIAB] OR "Kiribati"[TIAB] OR "Korea"[TIAB] OR "Korea"[TIAB] OR "Kuwait"[TIAB] OR "Kyrgyzstan"[TIAB] OR "Lao People's Democratic Republic"[TIAB] OR "Laos"[TIAB] OR "Latvia"[TIAB] OR "Lebanon"[TIAB] OR "Lesotho"[TIAB] OR "Liberia"[TIAB] OR "Libya"[TIAB] OR "Liechtenstein"[TIAB] OR "Lithuania"[TIAB] OR "Luxembourg"[TIAB] OR "Macao"[TIAB] OR "Madagascar"[TIAB] OR "Malawi"[TIAB] OR "Malaysia"[TIAB] OR "Maldives"[TIAB] OR "Mali"[TIAB] OR "Malta"[TIAB] OR "Malvinas"[TIAB] OR "Marshall Islands"[TIAB] OR "Martinique"[TIAB] OR "Mauritania"[TIAB] OR "Mauritius"[TIAB] OR "Mayotte"[TIAB] OR "McDonald Islands"[TIAB] OR "Mexico"[TIAB] OR "Micronesia"[TIAB] OR "Moldova"[TIAB] OR "Monaco"[TIAB] OR "Mongolia"[TIAB] OR "Montenegro"[TIAB] OR "Montserrat"[TIAB] OR "Morocco"[TIAB] OR "Mozambique"[TIAB] OR "Myanmar"[TIAB] OR "Namibia"[TIAB] OR "Nauru"[TIAB] OR "Nepal"[TIAB] OR "Netherlands"[TIAB] OR "Nevis"[TIAB] OR "New Caledonia"[TIAB] OR "New Zealand"[TIAB] OR "Nicaragua"[TIAB] OR "Niger"[TIAB] OR "Nigeria"[TIAB] OR "Niue"[TIAB] OR "Norfolk Island"[TIAB] OR "North Macedonia"[TIAB] OR "Northern Mariana Islands"[TIAB] OR "Norway"[TIAB] OR "Oman"[TIAB] OR "Pakistan"[TIAB] OR "Palau"[TIAB] OR "Panama"[TIAB] OR "Papua New Guinea"[TIAB] OR "Paraguay"[TIAB] OR "Peru"[TIAB] OR "Philippines"[TIAB] OR "Pitcairn"[TIAB] OR "Poland"[TIAB] OR "Portugal"[TIAB] OR "Principe"[TIAB] OR "Puerto Rico"[TIAB] OR "Qatar"[TIAB] OR "Réunion"[TIAB] OR "Romania"[TIAB] OR "Russian Federation"[TIAB] OR "Rwanda"[TIAB] OR "Saba"[TIAB] OR "Saint Barthélemy"[TIAB] OR "Saint Helena"[TIAB] OR "Saint Kitts"[TIAB] OR "Saint Lucia"[TIAB] OR "Saint Martin"[TIAB] OR "Saint Pierre and Miquelon"[TIAB] OR "Saint Vincent"[TIAB] OR "Samoa"[TIAB] OR "San Marino"[TIAB] OR "Sao Tome"[TIAB] OR "Sark"[TIAB] OR "Saudi Arabia"[TIAB] OR "Senegal"[TIAB] OR "Serbia"[TIAB] OR "Seychelles"[TIAB] OR "Sierra Leone"[TIAB] OR "Singapore"[TIAB] OR "Sint Eustatius"[TIAB] OR "Sint Maarten"[TIAB] OR "Slovakia"[TIAB] OR "Slovenia"[TIAB] OR "Solomon Islands"[TIAB] OR "Somalia"[TIAB] OR "South Africa"[TIAB] OR "South Georgia"[TIAB] OR "South Sandwich Islands"[TIAB] OR "South Sudan"[TIAB] OR "Spain"[TIAB] OR "Sri Lanka"[TIAB] OR "State of Palestine"[TIAB] OR "Sudan"[TIAB] OR "Suriname"[TIAB] OR "Svalbard"[TIAB] OR "Sweden"[TIAB] OR "Switzerland"[TIAB] OR "Syria"[TIAB] OR "Syrian Arab Republic"[TIAB] OR "Tajikistan"[TIAB] OR "Thailand"[TIAB] OR "Timor-Leste"[TIAB] OR "Tobago"[TIAB] OR "Togo"[TIAB] OR "Tokelau"[TIAB] OR "Tonga"[TIAB] OR "Trinidad"[TIAB] OR "Tunisia"[TIAB] OR "Turkey"[TIAB] OR "Turkmenistan"[TIAB] OR "Turks and Caicos"[TIAB] OR "Tuvalu"[TIAB] OR "Uganda"[TIAB] OR "UK"[TIAB] OR "Ukraine"[TIAB] OR "United Arab Emirates"[TIAB] OR "United Kingdom"[TIAB] OR "United Republic of Tanzania"[TIAB] OR "United States Minor Outlying Islands"[TIAB] OR "United States of America"[TIAB] OR "Uruguay"[TIAB] OR "USA"[TIAB] OR "Uzbekistan"[TIAB] OR "Vanuatu"[TIAB] OR "Venezuela"[TIAB] OR "Viet Nam"[TIAB] OR "Vietnam"[TIAB] OR "Virgin Islands"[TIAB] OR "Virgin Islands"[TIAB] OR "Wallis"[TIAB] OR "Western Sahara"[TIAB] OR "Yemen"[TIAB] OR "Zambia"[TIAB] OR "Zimbabwe"[TIAB] OR "Alabama"[TIAB] OR "Alaska"[TIAB] OR "Arizona"[TIAB] OR "Arkansas"[TIAB] OR "California"[TIAB] OR "Colorado"[TIAB] OR "Connecticut"[TIAB] OR "Delaware"[TIAB] OR "Florida"[TIAB] OR "Georgia"[TIAB] OR "Hawaii"[TIAB] OR "Idaho"[TIAB] OR "Illinois"[TIAB] OR "Indiana"[TIAB] OR "Iowa"[TIAB] OR "Kansas"[TIAB] OR "Kentucky"[TIAB] OR "Louisiana"[TIAB] OR "Maine"[TIAB] OR "Maryland"[TIAB] OR "Massachusetts"[TIAB] OR "Michigan"[TIAB] OR "Minnesota"[TIAB] OR "Mississippi"[TIAB] OR "Missouri"[TIAB] OR "Montana"[TIAB] OR "Nebraska"[TIAB] OR "Nevada"[TIAB] OR "New Hampshire"[TIAB] OR "New Jersey"[TIAB] OR "New Mexico"[TIAB] OR "New York"[TIAB] OR "North Carolina"[TIAB] OR "North Dakota"[TIAB] OR "Ohio"[TIAB] OR "Oklahoma"[TIAB] OR "Oregon"[TIAB] OR "Pennsylvania"[TIAB] OR "Rhode Island"[TIAB] OR "South Carolina"[TIAB] OR "South Dakota"[TIAB] OR "Tennessee"[TIAB] OR "Texas"[TIAB] OR "Utah"[TIAB] OR "Vermont"[TIAB] OR "Virginia"[TIAB] OR "Washington"[TIAB] OR "West Virginia"[TIAB] OR "Wisconsin"[TIAB] OR "Wyoming"[TIAB] OR "Ontario"[TIAB] OR "Quebec"[TIAB] OR "Nova Scotia"[TIAB] OR "New Brunswick"[TIAB] OR "Manitoba"[TIAB] OR "British Columbia"[TIAB] OR "Prince Edward Island"[TIAB] OR "Saskatchewan"[TIAB] OR "Alberta"[TIAB] OR "Newfoundland"[TIAB] OR "Labrador"[TIAB]) AND ("Journal Article"[PT] AND 2020[DP]) Data management --------------- Data will be stored in spreadsheet form on a cloud service. Data are not sensitive and do not require substantial security. Selection process ----------------- The title and abstract for all papers matching the search terms will be downloaded and screened for meeting the review criteria by two reviewers, with a third for arbitration in the case of a conflict. Prior to the full screening, the reviewers will be given 100 articles randomly selected from the papers matching our search terms, reviewed independently by all three screeners, and shared with each other for discussion and training purposes. When this is complete, the reviewers will screen the remaining paper titles and abstracts. During the full review and data extraction phase, a reviewer may believe that an article that was included did not meet the inclusion criteria. In that case, the paper will be reviewed for inclusion criteria by the other independent and a third arbitrating reviewer. Data collection process ----------------------- Reviewers will receive a training, which primarily consists of walking through the review instrument. The review questionnaire was adapted from a review checklist, available here: https://arxiv.org/abs/2009.01940. Reviewers will be asked to refer to a lightly edited version of the Haber et al., 2020 review guidance that corresponds with this systematic review. The edited version will be identical to the original, but with all references to specific COVID-19 policy analysis papers removed. Each article selected in the selection process will be reviewed in full by two independent primary reviewers. After completion of the training, reviewers will be assigned to a randomly selected set of papers to review, ensuring that each paper received two reviewers. Reviewers will first be given the opportunity to recuse themselves in cases of personal, professional, or financial conflicts (or any other reason that they see fit). and second will be asked if the study meets the inclusion criteria. In the case of a recusal, the paper will be reassigned to another reviewer. If there are no conflicts of interest and the study is deemed includable, the reviewers will continue to answer the remainder of the questionnaire to the best of their abilities. Once both reviewers have completed their reviews, they will be given the review from the other primary reviewer and an opportunity to discuss and resolve any differences. If a resolution cannot be met, a third arbitrating reviewer will be chosen at random from the reviewer pool, and will select what they believe to be the most accurate answers to the questions in the review tool, using the two primary reviewers’ responses as guidance. The entire dataset from this study will be reported in full and made publicly available. The only item that will be redacted is which reviews were produced by which specific reviewers. Data items ---------- - Policy types examined, by category (e.g. Workplace closing, stay at home order, contact tracing; see survey instrument for full list, generated from the Oxford Coronavirus Government Response Tracker: https://www.bsg.ox.ac.uk/research/publications/variation-government-responses-covid-19) - Outcomes examined (e.g., COVID-19 replication rate, number of cases, number of deaths, etc) - Main claim made in the abstract (copied sentence) - The main methods used for estimation of policy effects - General categories include cross-sectional, experiments/trials, interrupted time-series, and difference in differences designs - What proportion of studies met key design attributes of that study design? e.g. if an interrupted time-series, is the functional form of the trends shown graphically? Do the authors justify the functional form of the outcome (e.g., linear)? Etc. Will be reported as proportion of studies in each method category, and within each category, those meeting all criteria, proportion of criteria met, which criteria were and were not met, etc. - Metadata on the publications (e.g. journals, dates of publication, etc) Outcomes and prioritization --------------------------- Because this is a review assessing literature methodological strength and quality, rather than combining literature to summarize actual estimates of particular policies, this review neither collects nor summarizes effect estimates. The outcomes we collect pertain to the methods alone, and not the results. Risk of bias in individual studies ---------------------------------- This study is primarily a review of methods, rather than a study that will generate a summary impact estimate. As such, identifying these issues is the main purpose of this study. Data synthesis -------------- All studies in our sample will be reviewed using the review instrument, attached to this document. If data are appropriate for quantitative synthesis, describe planned summary measures, methods of handling data and methods of combining data from studies, including any planned exploration of consistency (such as I2, Kendall’s τ) Data will consist almost entirely of descriptive statistics of the arbitrator’s final reviews, and 95% confidence intervals. Comparison of proportions by groups will be done using risk differences and risk ratios with 95% confidence intervals, as appropriate. If there is sufficient power to do so, group comparisons may include types of policies, methods used, journals, etc. Meta-bias(es) ------------- Because this is a review assessing literature methodological strength and quality, rather than combining literature to summarize actual estimates of particular policies, meta-biases are part of the process resulting in our target sample of interest. As a result, this is a descriptive result of issues with the published literature, which may be a product of selective reporting and publication biases. Confidence in cumulative evidence --------------------------------- The body of evidence will be assessed using the criteria in our review instrument.