Working Papers

How can developing countries address heterogeneity in students’ preparation for school? A review of the challenge and potential solutions (with S. Djaker)

Over the past two decades, low- and middle-income countries (LMICs) have rapidly expanded access to schooling by building new schools and making education more affordable. These expansions have included segments of the population in these countries that had remained at the margins of the school system. Yet, they also presented schools with a new challenge: how to educate larger and more heterogeneous student groups. In light of this heterogeneity in students’ preparation for school, this paper has four goals: (a) reviewing the evidence on the degree to which students of different preparation levels coexist in the same classrooms; (b) proposing a conceptual framework to understand why current incentives and capacity do not lead principals and teachers to address heterogeneity; (c) discussing the extent to which existing experimental and quasi-experimental evidence has addressed these constraints; and (d) identifying unexplained pieces of this puzzle and recommend future directions.

Out of sight, out of mind? The gap between students’ test performance and teachers’ estimations in India and Bangladesh (with S. Djaker & S. Sabarwal)

This is one of the first studies of the mismatch between students’ test scores and teachers’ estimations of those scores in low- and middle-income countries. Prior studies in high-income countries have found strong correlations between these metrics. We leverage data on actual and estimated scores in math and language from India and Bangladesh and find that teachers misestimate their students’ scores and that their estimations reveal their misconceptions about students in most need of support and variability within their class. This pattern is partly explained by teachers’ propensity to overestimate the scores of low-achieving students and to overweight the importance of intelligence. Teachers seem unaware of their errors, expressing confidence in estimations and surprise about their students’ performance once revealed.

The untapped math skills of working children in India: Evidence, possible explanations, and implications (with A. V. Banerjee, S. Bhattacharjee & R. Chattopadhyay)

It has been widely documented that many children in India lack basic arithmetic skills, as measured by their capacity to solve subtraction and division problems. We surveyed children working in informal markets in Kolkata, West Bengal, and confirmed that most were unable to solve arithmetic problems as typically presented in school. However, we also found that they were able to perform similar operations when framed as market transactions. This discrepancy was not explained by children’s ability to memorize prices and quantities in market transactions, assistance from others at their shops, reliance on calculation aids, or reading and writing skills. In fact, many children could solve hypothetical transactions of goods that they did not sell. Our results suggest that these children have arithmetic skills that are untapped by the school system.

Journal Articles

The impact of combining performance-management tools and training with diagnostic feedback in public schools: Experimental evidence from Argentina (with R. de Hoyos, S. Djaker & P. A. Holland)

Economics of Education Review, 2024, 9, 102518

Providing principals with low-stakes information on their students' test scores has been shown to improve school management, instruction, and achievement in upper-middle income countries. We evaluate this approach by itself (“diagnostic feedback” or T1) and combined with tools and training (“performance management” or T2) through an experiment in 396 public primary schools in Salta, Argentina. After two years, T1 had null or adverse effects on on students' performance in school, but T2 reduced grade repetition (especially, among cohorts with more exposure), even a year after the interventions ended. We cannot rule out small-to-moderate effects on achievement. T2 also impacted teacher quality, student beliefs, bullying and discrimination, and extracurricular activities for high-exposure cohorts. Our results suggest that tools and training can effectively complement information in contexts of low principal capacity.

Augmenting state capacity for child development: Experimental evidence from India (with K. Muralidharan & C. R. Walters)

Journal of Political Economy, online first.

We use a large-scale randomized experiment to study the impact of augmenting staffing in the world’s largest public early childhood program: India’s Integrated Child Development Services (ICDS). Adding a half-time worker doubled net preschool instructional time and led to 0.29 and 0.46 standard-deviation (SD) increases in math and language test scores after 18 months for children who remained enrolled in the program, and corresponding increases of 0.13 and 0.10 SDs for the larger population of all children enrolled at baseline. Rates of child stunting and severe malnutrition were also lower in the treatment group for children who remained enrolled. A cost-benefit analysis suggests that the benefits of augmenting ICDS staffing are likely to significantly exceed its costs. Several features of our study setting and design suggest that these effects are likely to replicate even at larger scales of program implementation.

Measuring and predicting teachers’ commitment to implementing evidence-based programs (with T. Cramer, E. Cappella, & P. Morris)

Early Childhood Research Quarterly, 2023, 64(3), 405-415

Practical wisdom and empirical evidence indicate that teachers’ commitment to learn and enact a practice (i.e., their commitment to implement) is a critical but overlooked mechanism for supporting changes in teacher practice. Shortcomings in this area of scholarship stem from research gaps regarding: (1) how to operationalize teachers’ commitment to implementing new practices and (2) which factors predict whether teachers are committed to using new practices. To address these gaps, this study collected survey and open-ended response data from 107 early childhood education teachers implementing an evidence-based social-emotional learning program called Tools of the Mind (TotM). After comparing intention, planning, and motivation approaches for operationalizing teachers’ commitment to implement, a motivation approach emerged as the most consistent and valid method. Teachers’ belief that TotM would benefit children's development was identified as the strongest predictor of teachers’ commitment to implement based on structural equation modeling and open-ended survey data. Findings are discussed in the context of implementation science and teacher workforce development.

Can positive school climate ameliorate racial-ethnic disparities in pre-kindergarten quality? Evidence from a large urban school district (with J. Siegel & E. Cappella)

Early Childhood Research Quarterly, 2023, 64, 313-323

We investigated whether school climate, specifically effective school leadership, strong family ties, and trusting relationships, moderates the well-documented disparities in pre-kindergarten quality for Black and Latine students relative to their peers. In a sample of 615 public schools in a large, urban school district serving a minoritized student population (the average school in the sample served 66% Black and Latine students), we first explored patterns of disparities and found that Black and Latine children, on average, experience lower-quality pre-kindergarten in terms of classroom organization, emotional support, and instructional support. Using structural equation modeling, we found that a multi-dimensional index of school climate moderated this relationship with pre-kindergarten quality in terms of classroom organization and emotional support, but not instructional support. These findings suggest a potential role for school climate in mitigating existing inequities in the quality of pre-kindergarten.

Which students benefit from computer-based individualized instruction? Experimental evidence from public schools in India (with A. de Barros)

Journal of Research on Educational Effectiveness, 2023, online first

This is one of the first studies to evaluate the impact of computer-based individualized instruction in a developing country. We randomly assigned 1,528 students in grades 6-8 in 15 “model” public schools in Rajasthan, India who were using a computer-adaptive learning software to: a control group, in which they were only able to access the activities for their enrolled grade level; or a treatment group, in which they were able to access exercises appropriate for their performance level. After nine months, computer-based individualized instruction had a null average effect on math achievement. However, treatment students with low initial performance outperformed their control counterparts by 0.22 standard deviations. Our results suggest that computer-based individualized instruction is most beneficial for low-performing students.

The foundational math skills of Indian children (with A. de Barros)

Economics of Education Review, 2023, 92, pp. 102336.

We leverage data on learning for 101,084 public-school students in grades 4, 6, and 8 across 19 Indian states to diagnose their mathematic skills. These data allow us to diagnose their achievement on less frequently assessed skills. We use a novel approach to estimate the share of students who can meet fourth-grade standards. We find that the foundational skills of children are even lower than previously documented: 52% mastered frequently assessed skills, but only 27% mastered typically unassessed skills. These children also make less progress than believed. Gender gaps in these skills emerge between grades 4 and 6 and persist.

Which students benefit from independent practice? Experimental evidence from a math software in private schools in India (with A. de Barros & A. Venkatachalam)

Journal of Research on Educational Effectiveness, 2021, online first

This study is one of the first evaluations of independent (i.e., self-guided) practice in math in a developing country. We randomly assigned 4,461 students in grades 4-7 in high-performing private schools across seven Indian cities who were using a computer-assisted learning software to: (a) a control group, in which they moved from one unit to the next upon completion; or (b) a treatment group, in which they had to complete practice exercises before progressing to the next topic. After six months, the additional practice had a precisely estimated null effect on the math achievement of the average student. However, treatment students with low initial performance outperformed their control counterparts by 0.14 standard deviations (SDs). Our results suggest that independent practice may help private-school students in need of catching up. This is an important finding given the large and growing enrollment in this sector. Yet, it also suggests that, if affluent private schools provide more opportunities for practice, this may exacerbate inequities of the broader school system.

The role of teachers' commitment to implement in delivering evidence-based social-emotional learning programs (with T. Cramer, P. Morris, & E. Cappella)

Journal of School Psychology, 2021, Vol. 88, pp. 85-100

Although there are reasons to believe that teachers' commitment to learn and enact an evidence-based program (i.e., their commitment to implement) predicts their implementation fidelity, there is surprisingly little quantitative research testing this relationship. Using a national large-scale evaluation of three preschool social-emotional interventions, this study investigated how strongly teachers' commitment predicted implementation fidelity and whether commitment was a meaningful predictor of fidelity as compared to other individual factors (i.e., teacher stress at baseline) and contextual factors (i.e., collegial supports, classroom behavioral problems, and classroom quality at baseline). We surveyed 230 preschool teachers in their first year of implementing the interventions; data sources include surveys from teachers and 52 intervention coaches as well as classroom observational data. We found that teachers' baseline commitment consistently predicted implementation fidelity across time and that commitment predicted unique variation in fidelity after accounting for other individual and contextual factors. In addition, implementation fidelity had a moderate positive relationship with teachers' baseline classroom quality and a small negative association with baseline classroom behavior problems. Findings are discussed with respect to implementation science in education.

Teaching with the test: Experimental evidence on diagnostic feedback and capacity-building for public schools in Argentina (with R. de Hoyos and P. A. Holland)

The World Bank Economic Review, 2021, Vol. 35, No. 2, pp. 499-520

We examine the impact of two strategies to use large-scale assessment results to improve school management and classroom instruction in the Province of La Rioja, Argentina. We randomly assigned 104 public primary schools to: a diagnostic-feedback group, in which we administered standardized tests at baseline and two follow-ups and made results available to schools; a capacity-building group, in which we also conducted workshops and school visits; or a control group, in which we administered tests at the second follow-up. After two years, diagnostic-feedback schools outperformed control schools by .33σ in math and .36σ in reading. In fact, feedback schools still performed .26σ better in math and .22σ in reading in the national assessment a year after the end of the intervention. Additionally, principals at these schools were more likely to use assessment results for management decisions and students were more likely to report that their teachers used more instructional strategies and rated them more favorably. Combining feedback with capacity building does not seem to lead to additional improvements, but this might be due to schools assigned to receive both components starting from lower learning levels and participating in fewer workshops and visits than expected.

Can principal training improve school management? Short-term experimental evidence from Argentina (with S. H. Freel)

Papeles de la Economía Española, 2021, Vol. 166, 67-83

This paper presents one of the first evaluations of a principal-training program in a developing country. We randomly assigned 100 public primary schools in the Province of Salta, Argentina to a treatment group in which school principals attended a six-week, intensive, training workshop provided by an international foundation, or to a business-as-usual control group. Take up of the program was lower than expected: of the 52 schools assigned to the workshop, only 37 had a staff member sign up, 36 had someone attend, and the average participant attended 19 of the 26 sessions. Yet, satisfaction among participants was high: 99% of attendants reported the workshop was similar or better to other opportunities and 97% would recommend it to a peer. One month after the workshop, the program did not improve the availability or adequacy of school facilities, but it increased the links between schools and community organizations (e.g., teacher-training institutions and universities), as reported by principals. It had no impact on school climate, as reported by students, or on students’ performance in math and language in the national assessment. Less than three months after the workshop, the program had no effect on students’ performance in school. In fact, we can rule out small-to-moderate positive effects. The null results illustrate the difficulty of observing impacts of principal-training interventions in the short run, given that they often require non-trivial changes in management and instruction.

Growth mindset interventions at scale: Experimental evidence from Argentina

Educational Evaluation and Policy Analysis, 2020, Vol. 42, No. 3, 417-438

This is one of the first evaluations of a “growth-mindset” intervention at scale in a developing country. I randomly assigned 202 public secondary schools in Salta, Argentina, to a treatment group in which Grade 12 students were asked to read about the malleability of intelligence, write a letter to a classmate, and post their letters in their classroom, or to a control group. The intervention was implemented as intended. Yet, I find no evidence that it affected students’ propensity to find tasks less intimidating, school climate, school performance, achievement, or post-secondary plans. I rule out small effects and find little evidence of heterogeneity. This study suggests that the intervention may be more challenging to replicate and scale than anticipated.

Hard cash and soft skills: Experimental evidence on combining scholarships and mentoring in Argentina (with F. Barrera-Osorio, M. L. Biehl, & M. A. Cortelezzi)

Journal of Research on Educational Effectiveness, 2020, Vol. 13, No. 2, pp. 380-400

We evaluate a long-standing program run by one of the largest education foundations in Argentina that offers scholarships and non-academic mentoring to secondary school students. We randomly assigned 408 grade 6 students within 10 public schools in the Province of Buenos Aires to either receive the program throughout secondary school or not to receive it. After three years, the program improved students’ academic behaviors (e.g., studying before an exam or catching up on missed work), but we find little evidence that these changes translated into broader improvements in students’ academic mindsets (e.g., self-beliefs about performance and efficacy), perseverance (e.g., grit), or learning strategies (e.g., metacognition). The program also improved students’ performance in school during the first year (e.g., grades, attendance, and passing rates), but we do not find similar gains in subsequent years. This may be due to a large share of treatment students being expelled from the program for not meeting its requirements. The program did not improve student learning or personality traits (e.g., conscientiousness). Finally, we find some heterogeneous effects for female students and students from low-income families.

Disrupting education? Experimental evidence on technology-aided instruction in India (with K. Muralidharan & A. Singh)

American Economic Review, 2019, Vol. 109, No. 4, pp. 1426-1460

We present experimental evidence on the impact of a personalized technology-aided after-school instruction program on learning outcomes. Our setting is middle-school grades in urban India, where a lottery provided winning students with a voucher to cover program costs. We find that lottery winners scored 0.36σ higher in math and 0.22σ higher in Hindi relative to lottery losers after just 4.5-months of access to the program. IV estimates suggest that attending the program for 90 days would increase math and Hindi test scores by 0.59σ and 0.36σ respectively. We find similar absolute test score gains for all students, but the relative gain was much greater for academically-weaker students because their rate of learning in the control group was close to zero. We show that the program was able to effectively cater to the very wide variation in student learning levels within a single grade by precisely targeting instruction to the level of student preparation. The program was cost effective, both in terms of productivity per dollar and unit of time. Our results suggest that well-designed technology-aided instruction programs can sharply improve productivity in delivering education.

More than words: Expressed and revealed preferences of top college graduates entering teaching in Argentina (with M. Alfonso & A. Santiago)

Comparative Education Review, 2017, Vol. 61, No. 3, 581-606

School systems are trying to attract top college graduates into teaching, but we know little about what dissuades this group from entering the profession. We provided college graduates who applied to a selective alternative pathway into teaching in Argentina with information on what their working conditions and pay would be if they were admitted into the program. Then we observed whether they reported that they wanted to go into teaching and whether they did so. We found that individuals who received information about working conditions or pay were more likely to report that they no longer wanted to pursue their application to the alternative pathway but no more likely to drop out of the program’s selection process. This could be due to prominence effects. Students with higher GPAs were more likely to drop out if they received information on working conditions but not if they received information on pay.

The barking dog that bites: Test score volatility and school rankings in Punjab, Pakistan (with F. Barrera-Osorio)

International Journal of Educational Development, 2016, Vol. 49, pp. 31-54

This paper is the first to explore the implications oftest score volatility for school accountability policies in a lower-middle-income country. Using two new datasets from the province of Punjab, Pakistan, we find that rankings based on school-level averages of students’ test scores fluctuate considerably from one year to the next due to factors unrelated to school quality. First, most variation in test scores is within, not between schools, allowing noise to play a large part in determining school rankings. Second, variation in school test scores is partially attributable to fluctuations in the characteristics of student cohorts. Third, an even larger share is explained by one-time shocks. Fourth, these problems are more pronounced in small schools. As a result, rankings that rely on test score levels or changes often fail to identify top-performing or rapidly improving schools, since they are influenced by factors beyond schools’ control.

Improving educational outcomes in developing countries: Lessons from rigorous evaluations (with R. J. Murnane)

Review of Educational Research, 2016, Vol. 86, No. 3, pp. 719-755

In this article, we reviewed and interpreted the evidence from 223 rigorous impact evaluations of educational initiatives conducted in 56 low- and middle-income countries. We considered for inclusion in our review all studies in recent syntheses that have reached seemingly conflicting conclusions about which interventions improve educational outcomes. We grouped interventions based on their theory of action. We derived four lessons from the studies we review. First, reducing the costs of going to school and expanding schooling options increase attendance and attainment, but do not consistently increase student achievement. Second, providing information about school quality, developmentally appropriate parenting practices, and the economic returns to schooling affects the actions of parents and the achievement of children and adolescents. Third, more or better resources improve student achievement only if they result in changes in children’s daily experiences at school. Fourth, well-designed incentives increase teacher effort and student achievement from very low levels, but low-skilled teachers need specific guidance to reach minimally acceptable levels of instruction.

Why do some school-based management reforms survive while others are reversed? The cases of Honduras and Guatemala

International Journal of Educational Development, 2016, Vol. 47, pp. 33-46

In the 1990s, El Salvador, Nicaragua, Guatemala, and Honduras enacted school-based management (SBM) reforms that allowed communities to make key decisions about their schools that were previously reserved for state-appointed officials. Yet, these reforms have recently begun to slide back. What explains this trend toward reform reversal? This paper argues that two factors determine the likelihood of the reversal of an SBM program: the scope of the reform and the level of national investment in it. Using the cases of Honduras and Guatemala, I contend these two factors determine the extent to which an SBM reform is vulnerable to events that can bring about its termination, such as changes in government, union strength, or parental pressure.