Survey Design & Sampling
IMPAQ conducted randomized controlled trial (RCT) impact evaluations to examine the effects of interventions aimed at combating child labor in Costa Rica, Ecuador, India, Malawi, and Rwanda. The project objectives were to:
- Provide experimental evidence to inform the design and effectiveness of a project or government policy that has direct or indirect effects on child labor or forced labor.
- Utilize partnerships between researchers, contractors, and/or non-governmental organizations (NGOs) and local organizations in the country where the impact evaluation took place.
- Test interventions or combinations of interventions.
- Ensure that study designs, surveys, and analysis conform to international and national definitions and standards on child labor and forced labor.
Specifically, in each country, the RCTs had the following scope and objectives:
IMPAQ collaborated with the government of Costa Rica to conduct an RCT of the Working Children and Adolescents (NNAT) program. The RCT's main goal was to build rigorous evidence to determine if the intervention contributed to the reduction in child labor and hazardous child labor among program beneficiaries. The NNAT program is a conditional cash transfer program designed to keep youth in school by replacing their wages from working with a monthly subsidy of approximately USD $200. Besides solving the immediate problem of ensuring sufficient family income, the program also targeted long-term family income by requiring families to keep their children in school to continue receiving the subsidy. Through the RCT, IMPAQ evaluated whether the project had achieved its outcomes of reducing child labor and increasing students' schooling and/or aspirations for further education.
Together with Young Potential Development (YPD) Ecuador and the Secretariat of Education in Quito (SEQ), IMPAQ's RCT of the YPD program aimed to measure the impact of YPD on the likelihood of student participation in hazardous child labor. The evaluation investigated whether a program fostering socioemotional skills had an effect on reducing hazardous child labor and other outcomes. IMPAQ examined whether YPD achieved expected changes in socioemotional skills, and whether these outcomes led to changes in other youth outcomes, such as labor market educational aspirations and academic performance.
IMPAQ worked with the Kailash Satyarthi Children's Foundation (KSCF) on this RCT impact evaluation of the Bal Mitra Gram (BMG or Child-Friendly Village) program. The evaluation aimed to measure the effectiveness of the BMG program in reducing the prevalence of child labor in India. It investigated the role of information- and rights-based campaigns in reducing child labor. As KSCF's signature intervention to combat child labor, the BMG program follows a community-based approach to tackling child labor and improving access to education for children across India. With this RCT, IMPAQ examined whether the BMG program had an impact on: 1) participation of BMG children in child employment, labor, and hazardous child labor; 2) enrollment and attendance of BMG children in school; 3) adult attitudes toward child labor, marriage, and education; and 4) adult awareness and knowledge regarding child labor.
The RCT in Malawi examined the linkages between village savings and lending associations (VSLAs) and child labor through an impact evaluation of the second phase of the Child Labour Elimination Actions for Real Change (CLEAR) program. CLEAR II, funded by the Eliminating Child Labour in Tobacco-Growing (ECLT) Foundation, aimed to reduce child labor prevalence by adopting the same multi-pronged approach from the first iteration: 1) identifying and removing children engaged in child labor; 2) raising community awareness and advocating for policy changes at national and local levels; and 3) implementing a livelihood intervention for relaxing households' liquidity constraints. However, the second phase focused specifically on supporting the formation and training of VSLAs to provide training in child labor prevention, financial literacy, and procedures for obtaining VSLA certification. The RCT aimed to contribute to the evidence base on the link between increased access to credit and savings and child labor.
In this RCT, IMPAQ evaluated the Model Farm Schools (MFS) program to reduce the worst forms of child labor in Rwanda. Beginning in 2016, Winrock integrated the Model Farm School (MFS) program into its Rwanda Education Alternatives for Children in Tea-growing Areas (REACH-T) program targeting vulnerable, out-of-school youth. MFS aimed to provide training for small groups of 16-17-year-old youth and to connect them to safe work opportunities by equipping them with both technical and life skills. The evaluation intended to understand the impact of MFS on 1) likelihood of participating in hazardous labor; 2) the type of work youth would like to obtain in the next few years; 3) the educational aspirations of these youth; and 4) youths’ level of confidence and locus of control.
In 2016, IMPAQ and the government of Costa Rica identified a target population of 554 cases of child labor to be randomly assigned to the NNAT program. We conducted a lottery to create a treatment and a control group of similar sizes. The treatment group consisted of 276 cases of child labor, scheduled to receive the NNAT subsidy in 2017. The control group consisted of 275 cases of child labor to receive delayed treatment. IMPAQ designed a project-specific survey that was administered to the target groups at baseline in 2016 and at endline in 2018. The survey instrument covered information on personal and socio-demographic characteristics, educational outcomes, work, workplace conditions, and household chores.
In addition to the participant survey, IMPAQ also gathered qualitative data through key informant interviews (KIIs) and focus group discussions (FGDs). We integrated the quantitative and qualitative data in order to facilitate understanding of the mechanisms through which the program leads beneficiaries to change their behavior and to quantify the outcomes produced.
IMPAQ randomly assigned 806 students, ages 15–17, into classrooms within seven municipal schools in Quito. Treatment students were taught by YPD-trained teachers, while control students were taught by regular teachers. At the end of the academic year, the evaluation team compared student outcomes in treatment classrooms to those in control classrooms to measure the impact of YPD.
An implementation study complemented the impact evaluation to assess whether the program was implemented as intended and provide contextual understanding of the changes resulting from the program. IMPAQ designed a project-specific survey questionnaire administered to the treatment and control group students at the beginning of the school year (2016) and at the end of the school year (2017). The survey included sections on students' socioemotional skills (self-efficacy, social skills, and perception of school climate), educational and career aspirations, and labor market activities and household chores.
In addition to the student survey, IMPAQ also gathered school administrative data (test scores, behavior grades, and attendance) from SEQ, YPD's program implementation records, as well as conducted key informant interviews (KIIs) with program implementers and focus group discussions (FGDs) with teachers and students after program completion.
IMPAQ randomly assigned 40 villages in the Giridih district of Jharkhand evenly into treatment and control groups. This area was selected because of the high prevalence of mica mining, a sector known for employing children. Villages in the treatment group began receiving the BMG intervention activities in March 2016. Prior to implementation, IMPAQ executed a baseline survey. In 2018, after two years of program implementation in treatment villages, IMPAQ conducted the endline survey. The data collected were obtained from a household survey, which was administered to treatment and control group households who had at least one child aged 5–17. Household heads responded to questions about children's economic activity, educational attainment, and adults' perceptions towards children's work, school, marriage, and child labor laws.
Using regression analysis, the impact of the program was measured by comparing the average outcomes of the treatment group with the average outcomes of the control group. A qualitative data analysis complemented the impact evaluation to provide understanding of the mechanisms of change resulting from the program.
Before the launch of program activities, IMPAQ randomly assigned 18 communities to treatment and control groups. Data for the impact evaluation was collected at baseline in 2016 and at endline in 2019. IMPAQ designed a project-specific survey administered to households and children. The survey gathered information related to household composition, education, employment, tobacco cultivation, informal group membership, and savings and investments. Respondents were household members who were at least 18 years of age and had at least one child between the ages 5–17. The children’s survey was administered to those aged 12–17.
Using regression analysis, the impact of the program was measured by comparing the average outcome of the treatment group with the average outcome of the control group. We measured differences between treatment and control groups on the following outcomes: child and hazardous child labor, school enrollment and attendance, household savings, and credit access and investments. A qualitative study was also implemented to facilitate understanding of changes experienced by households through their participation in VSLAs. IMPAQ conducted key informant interviews with program implementers and community leaders, and focus group discussions with VSLA members in treatment communities.
IMPAQ randomly assigned 962 youth, ages 15–17, from 15 sites in the Southern and Western regions of Rwanda to treatment and control groups. Youth in the treatment group were selected to participate in MFS trainings through a random public lottery.
The evaluation team estimated MFS effects by comparing the average outcomes of the treatment group with the average outcomes of the control group. The team also compared the outcomes of all treatment students who participated in the training with the control group.
IMPAQ designed a project-specific survey questionnaire administered to the treatment and control group youth between December 2015 and January 2016, as well as between March and April 2018. The survey included sections on students' household characteristics and demographics, labor activities and ambitions, and program participation.
A qualitative study complemented the impact evaluation to investigate the program implementation fidelity and provide in-depth understanding of the mechanisms of changes resulting from the program. IMPAQ conducted key informant interviews with local leaders and focus group discussions with MFS participants and youth in the control group after program completion.
Experience & Expertise
The IMPAQ International Development team has a record of integrity and collaboration while being able to mobilize quickly and adapt to varying contexts and obstacles. Despite the complexity in carrying out RCTs across five countries, IMPAQ was able to ensure that each of the RCTs maintained a level of rigor to deliver crucial research for understanding program outcomes and effectiveness. Our in-house quantitative and qualitative experts provided strong oversight to ensure that all deliverables maintained strict quality standards.
On this project, the IMPAQ team comprised senior researchers and economists with significant impact evaluation experience using experimental methods. We also had a team of experts who helped to build and implement surveys that allowed us to use similar definitions of child labor across varied contexts. The qualitative researchers across our RCTs helped facilitate a more in-depth understanding of the mechanisms of change and contextualize program findings. Lastly, each RCT engaged experienced local data collection firms with significant knowledge of the country landscape and cultural context. These firms understood how to mobilize teams quickly to collect relevant and high-quality data while accounting for potential difficulties in the field.