At IPPSR, we are underway building a comprehensive database full of policy-relevant research. A large focus of the database is to find randomized controlled trials (RCTs) of public policy experimentation. These studies enable causal inference by randomly assigning a policy intervention to some people or areas and comparing the results to a control group. So far, we are proud to have summarized roughly 60 published studies utilizing RCTs to examine the effects of policy implementation. These RCTs range in their focus, and can be categorized into 7 categories. In this post we briefly summarize each category, and end with a comprehensive list of all RCTs, complete with a brief summary of each.
Health
The biggest category, by far, involves health-related policy. Several studies utilize data from the famous Oregon Medicaid Experiment to assess changes in utilization, outcomes, and preventative screenings in Medicaid patients. Important findings from the Oregon experiment have shown that gaining access to Medicaid has important medical benefits (decreased depression and increased self-reported health), decreases financial hardship, and increases utilization of primary care visits and specialist referrals in community health centers. Other studies not related to the Oregon experiment have looked at methods of reducing childhood obesity, eliminating risky behaviors, and testing the efficacy of nurses’ home visitations.
Education
In terms of education policy, RCTs have proven to be quite popular. A popular subject focuses on testing the effectiveness of various pre-school and student tutoring programs, including the prominent Head Start Impact study. Many of these results paint a similar story: programs have early benefits that dissipate quickly once students move into standard classrooms. These studies have shown the necessity of devising new programs better able to maintain long-term benefits. Other studies have looked at methods of increasing university attendance and retention rates, relative effectiveness in differing approaches to sexual-education, and the cost-effectiveness of school vouchers.
Poverty, Welfare, and Social Services
Another burgeoning use of RCTs has focused on measuring the effectiveness of various social programs. Randomized experiments have been used to assess the success of homelessness prevention programs, welfare time-limits and employment restrictions, and job-training programs. Results have shown a great success in New York’s Homebase Community Prevention Program in terms of reducing homelessness. Further, analyses of Florida’s Family Transition Program (which added time-limits to welfare while providing job-search aide) concluded that the program did not increase the probability of recipients finding employment, and further increased mortality rates amongst recipients.
Crime and Criminal Justice
RCTs have become a useful tool in analyzing police and criminal policy. Through RCTs, different police policies including crime hot-spot patrolling, body-worn video cameras, and procedurally-just traffic stops have been assessed in terms of success and efficiency. Further, criminal justice policies such as ignition breathalyzer lock systems have been tested for crime prevention. These studies have shown that body-worn cameras successfully decrease allegations and complaints against police officers, while increasing police patrols in crime hot-spots successfully act as a deterrent to crime.
Business and the Economy
RCTs have been successful in different aspects of the economic and business world. Multiple studies have focused on methods for improving tax compliance, others have looked at mechanisms to increase retirement planning, and have tested the effectiveness of OSHA inspectors. Some results have shown that close supervision of business tax-compliance has a negative effect on on-time tax payments, while negative media exposure and increased perceptions of tax-complexity also decrease compliance rates.
The Environment
Though still moderately scarce, the environment poses a prime sector for RCTs in the future. Currently, our database includes two studies on the environment. One attempts to find the most productive method for enticing consumers to switch to an electricity rate-plan that incentivizes off-peak usage, eventually finding that these differing methods provided statistically zero difference. The other studies the impact of Delhi, India’s experiment to curb automobile pollution by mandating each car in the territory only be allowed to operate every other day. The results showed dramatic decreases in pollution, decreases in traffic congestion, and a significant increase in carpooling and public transportation usage.
Voting Behavior and Government Operations
Recently RCTs have been increasingly popular in this area with the sudden increase in voter restriction (see our post). Many studies focus on the effects voter ID, early voting, and absentee ballot laws have on turnout. Beyond voting, RCTs have been utilized to test mechanisms for increasing citizen engagement, the role of gender in policy-making, and the effect public opinion has on legislators’ votes. An important result for policy makers has shown that sending letters to constituents helps move public opinion towards the legislator’s position, while simultaneously avoiding a decrease in approval rating by expressing unpopular policy stances.
All-in-all, RCTs have proved to be an immensely popular and successful tool in policy evaluation across all policy-areas. In the future, their use is likely to increase, and we look forward to the knowledge they provide.
Below we list all RCTs in our database by category:
Education
Impact Evaluation of the U.S. Department of Education’s Student Mentoring Program
Evaluates the effects of the federal Student Mentoring Program
Investigates the effect of school-based mentoring (SBM)
Mentoring in Schools: An Impact Study of Big Brothers Big Sisters School Based Mentoring
Tests the impact of the Big Brothers Big Sisters mentoring program on students
Tests the relative effectiveness of peer-led sexual education classes
The High Scope Perry Preschool Study Through Age 40
Examines the effect of preschool on children from impoverished families
Improving Numeracy and Literacy Evaluation Report and Executive Summary
Evaluates the Improving Numeracy and Literacy program regarding students’ mathematical and reading skills
This report summarizes an experimental intervention that aims to improve the writing and grammatical skills of young students
Analyzes the usage of text messages to increase college attendance
Studies the effects of offering free breakfast to schoolchildren in Wales
School Vouchers and Student Achievement: First year Evidence from the Louisiana Scholarship Program
Examines the academic difference between students who won a lottery for private school vouchers, and those who lost
Evaluates the effect of Tennessee’s voluntary Pre-Kindergarten program (hereafter “pre-k”)
Head Start Impact Study Final Report
The Head Start Impact Study was a government run randomized experiment to measure the effect of the Head Start program
Parent-Child Information Frictions and Human Capital Investment: Evidence from a Field Experiment
Examines the effect of increased flows of information to parents on student success
Teachers, Race, and Student Achievement in a Randomized Experiment
Uses data from the Tennessee project STAR and tests the effect of minority students having a teacher from the same minority group
Efficacy of a Theory-Based Abstinence-Only Intervention Over 24 Months
Tests the effectiveness of abstinence-only education among youth
Health
Analyzes the effect Child Development Accounts (long-term investment accounts) have on mother’s depression
School Gardens and Physical Activity: A Randomized Controlled Trial of Low income Elementary Schools
Assess the impact school gardens have on physical activity
The Impact of Screening and Advice on Inattentive, Hyperactive and Impulsive Children
Analyzes the effect of in-school ADHD screenings
Examines the effects of nurse’s home visitation on incidence of abuse and neglect
Examines mechanisms for improving young consumers’ abilities to understand and compare food nutrition labels
Part of the Oregon Medicaid experiment, this study analyzes the effect Medicaid expansion has on the prevalence of cancer screenings
Effectiveness of a Nurse-led Intensive Home-visitation Programme for First time Teenage Mothers
Evaluates the effect of a home-visitation nurse program for first time mothers
Encouraging Primary Care Physicians to Help Smokers Quit: A Randomized, Controlled Trial
Evaluates various methods to encourage Physicians to convince patients to quit smoking
Evaluates the National Exercise Referral Scheme (NERS), a program set up in Wales to increase physical activity by proving exercise instruction, group exercise classes, and motivation and goal-keeping consultations with professionals
The Oregon Experiment - Effects of Medicaid on Clinical Outcomes
In 2008 Oregon expanded its Medicaid through a lottery, drawing 30,000 names out of 90,000 applicants. This was used as a randomized experiment to analyze multiple effects of Medicaid access
Community Health Center Use After Oregon Randomized Medicaid Experiment
In another study based on the Oregon randomized Medicaid experiment, this experiment analyzes community health center utilization by Medicaid patients
Promotion of Healthy Eating Through Public Policy: A Controlled Experiment
Analyzes the effectiveness of different methods of decreasing consumption of unhealthy food and beverages
A Family Intervention to Delay Nursing Home Placement of Patients with Alzheimer Disease
Evaluates a program providing counseling to spouses of Alzheimer’s patients
Health Insurance and the Demand for Medical Care: Evidence from a Randomized Experiment
Explores the effect of cost-sharing on the demand of medical care
Does Free Care Improve Adults’ Health? Results from a Randomized Controlled Trial
Measures the effect the provision of free healthcare has on health outcomes
Is Cost a Barrier to Screening Mammography for Low-Income Women Receiving Medicare Benefits?
Tests the effectiveness of offering free mammography screening to low income patients (specifically, older women)
Reducing Children's Television Viewing to Prevent Obesity
Measures the effect a curriculum to decrease television viewing has on childhood obesity
Six-year Follow-up of the First Waterloo School Smoking Prevention Trial.
Measures the effect of an educational program designed to resist peer-pressure to smoke
Communities Mobilizing for Change on Alcohol: Lessons and Results from a 15-community Randomized Trial
Attempts to evaluate a policy intervention that was designed to alter “alcohol-related behaviors”
Poverty, Welfare, and Social Services
Banking the poor via savings accounts: Evidence from a field experiment
Estimates the demand and utility that providing banking institutions to poor households in Nepal brings to Nepal citizens and the country
Evaluation of the Homebase Community Prevention Program
Explores the effect of the Homebase Community Prevention Program in New York City
The Family Transition Program: Final Report on Florida’s Initial Time Limited Welfare Program
Assesses Florida’s Family Transition Program (FTP), the first attempt of a state to initiate time limits on welfare benefits
Welfare Programs That Target Workforce Participation May Negatively Affect Mortality
Analyzes results from the Florida Family Transition Program, an experiment looking at time limits on welfare benefits
Analyzes the effects of the “Moving to Opportunity” experiment, where families from highly impoverished housing projects were given vouchers to move into less impoverished neighborhoods
Evaluates employment training and services through the Job Training Partnership Act (JTPA)
Crime and Criminal Justice
General Deterrent Effects of Police Patrol in Crime Hot Spots: A Randomized, Controlled Trial
Investigates the effects of police-officer patrols in high crime “hot spots” in Minneapolis
Police, Camera, Evidence: Londons Cluster Randomised Controlled Trial of Body Worn Video
Tests the effects of police use of body-worn video cameras
Measures the effectiveness of a program that instituted breath-alcohol detectors connected to the ignitions of drinking-and-driving offenders for one year
Shaping Citizen Perceptions of Police Legitimacy: A Randomized Field Trial of Procedural Justice
Assesses the impact “procedurally just” traffic stops have on perceptions of the police
Business and the Economy
Effects of supervision on tax compliance: Evidence from a field Experiment in Austria
Examines the effect supervision has on the prevalence of tax compliance
Randomized Government Safety Inspections Reduce Worker Injuries with No Detectable Job Loss
Analyzes the effect of Occupational Health and Safety Administration (OSHA) inspectors
Examines the effect of tax related media on consumers trust in government and on their perceived power of the government to enforce tax policy
The Role of Information and Social Interactions in Retirement Plan Decisions: Evidence from a Randomized Experiment
Assesses the impact of additional information on university employee enrollment in retirement savings plans
The Environment
Can Behavioral Economics Be Used to Encourage Consumers to Switch to Green Energy Tariffs?
Attempts to find the most productive method to entice consumers to switch to a rate plan incentivizing off-peak usage.
Delhi Odd Even Plan as a Public Policy Experiment
Evaluates Delhi, India’s experiment to to curb automobile pollution by mandating that cars with an even number registration would be allowed to drive on even dates, while cars with odd registration could drive on odd dates
Voting Behavior and Government Operations
The Effects of Voter ID Notification on Voter Turnout: Results from a Large Scale Field Experiment
Tests the effect of voter identification laws on voter turnout
Women as Policy Makers: Evidence from a Randomized Policy Experiment in India
Analyzes the effect of women policymakers on policymaking
Invitation Phone Calls Increase Attendance at Civic Meetings: Evidence from a Field Experiment
Evaluates differing methods to increase attendance at town hall meetings
Can Learning Constituency Opinion Affect How Legislators Vote? Results from a Field Experiment
Analyzes what effect learning public opinion has on a legislator’s stance on an issue
Taking advantage of voting rules in San Diego, an experiment was devised to test door-to-door canvassing effects on traditional voting compared to precincts that solely use mail-in ballots
Analyzes the effect of state legislators expressing opinions to their constituents
Tests different mechanisms to increase public participation in municipal committees
