Teacher turnover is a perennial concern, especially in low-performing, high-poverty schools. While districts and schools may try to anticipate and mitigate turnover by surveying teachers about their future plans, existing research on whether teacher-reported intent is predictive of actual turnover behavior is mixed. Using unique survey data from teachers in 35 low-performing, high-poverty districts in Michigan linked at the teacher level to statewide administrative data, we are able to measure turnover behavior one, two, and three years following reported intent. We find that intent is a significant predictor of turnover and becomes increasingly predictive over time. We also find organizational commitment and school organizational conditions are important factors in teachers’ intent and, to a lesser degree, actual turnover behavior.
Federal incentives and requirements under the Obama administration spurred states to undertake major efforts to reform teacher evaluation systems. We examine the effects of these reforms on student achievement and attainment at a national scale by exploiting their staggered implementation across states. We find precisely estimated null effects, on average, that rule out impacts as small as 0.017 standard deviations for achievement and 1.2 percentage points for high school graduation and college enrollment. We highlight five factors that likely limited the efficacy of evaluation reforms at scale: political opposition, decentralization, capacity constraints, limited generalizability, and the absence of compensating wages.
11. Harbatkin, E., Pham, L., Redding, C., & Moran, A.* (2024). What are the Side Effects of School Turnaround? A Systematic Review. Review of Research in Education, 48(1), 58-88. https://doi.org/10.3102/0091732X241248151. [ungated]
School turnaround has long been a tool for improving persistently low-performing schools. A large literature has identified wide variation in the intended effects of turnaround, and an overlapping literature examines unintended effects on educators, students, and the communities they serve. While two recent meta-analyses combine current quantitative evaluations to summarize of the average intended effects of turnaround and the mechanisms underlying those effects, no research thus far has synthesized the research on the unintended effects of turnaround. In this systematic review, we examine research from 2009 through 2022 to methodically identify and classify the unintended effects of turnaround in the United States. To do so, we develop a conceptual framework that classifies three different types of turnaround side effects—spillover effects, systemic side effects, and internal side effects—and differentiates these side effects from unintended negative intervention effects. We identify four broad categories of turnaround side effects within this framework that we characterize based on the population they impact: the broader community, school systems, educators, and students. We find that some of the most prevalent side effects in the turnaround literature are related to educator experiences, staffing, community reaction, education governance, and the proliferation of external educational actors. We conclude with a call for future research to explicitly examine these common side effects alongside the intended effects of a turnaround intervention.
10. Burrell, N.* & Harbatkin, E. (2024). Beyond the School Building: Examining the Contribution of Out-of-School Factors to Multidimensional School Grades.Education Policy Analysis Archives, 32. https://doi.org/10.14507/epaa.32.8497 (open access).
Many states report school performance grades as a way to inform the public about school quality. However, past research has shown that when these grades drew largely on proficiency-based measures, they served to capture variation in school and community demographics rather than school quality. We extend this literature by examining whether a multidimensional measure of school quality such as those required under the Every Student Succeeds Act is less confounded by out-of-school factors than the proficiency measures that characterized previous generations of accountability. Drawing on school accountability grades from Florida combined with school and community demographic data, we find that more than half the variation in multidimensional measures of school quality can be explained by observable school- and county-level factors outside the school’s locus of control. Together, our findings show that even school grades that draw on multiple measures misattribute the contribution of demographics and socioeconomics to school quality—but subcomponents based on learning gains perform better than those based on proficiency. We conclude with policy implications and recommend that states focus public reporting on school quality measures that driven less by out-of-school factors and more by the school’s true contribution to student outcomes.
9. Harbatkin, E., Strunk, K.O., & McIlwain, A.* (2023). School Turnaround in a Pandemic: An examination of the outsized implications of COVID-19 on low-performing turnaround schools, districts, and their communities. Economics of Education Review. 97, 102484. https://doi.org/10.1016/j.econedurev.2023.102484. [ungated]
Turnaround schools and districts that were charged with making rapid and dramatic improvements before the onset of the COVID-19 pandemic faced considerable challenges carrying out improvement efforts during pandemic schooling. Using survey and administrative data collected during the pandemic, we document some of the ways in which students and educators in Michigan’s turnaround schools and districts experienced the pandemic. We show that the communities in which turnaround schools are located were hardest hit by the pandemic and school and district operations were substantially disrupted. By extension, turnaround districts and especially the lowest performing schools in those districts that were targeted for school-level turnaround experienced high rates of student absenteeism, low student and parent engagement, and, ultimately, significantly smaller gains on math and reading benchmark assessments than non-turnaround districts. Our findings have implications for policy as states resume school and district turnaround efforts that were disrupted by the pandemic.
8. Burns, J.A., Harbatkin, E., Strunk, K.O., Torres, C., McIlwain, A.,* Waldron, S.F.* (2023). The Efficacy and Implementation of Michigan’s Partnership Model of School and District Turnaround: Mixed-Methods Evidence from the First Two Years of Reform Implementation. Educational Evaluation and Policy Analysis. https://doi.org/10.3102/01623737221141. [ungated]
The recent Every Student Succeeds Act (ESSA) requires states to identify and turn around their lowest performing schools, but breaks somewhat from prior policies by granting states significant autonomy over how they identify and turn around these schools. This mixed-methods study, which draws on administrative, qualitative, and survey data, examines the effectiveness of Michigan’s approach to school turnaround under ESSA. We find that students in turnaround schools experienced significant achievement gains in math and to a lesser extent in ELA, with effects concentrated among the lowest achieving students. Analyses of qualitative and survey data suggest that these outcomes were influenced by state-level supports, strategic planning, the threat of accountability for continued low performance, and improved leadership quality in turnaround schools.
7. Henry, G.T., McNeill, S., & Harbatkin, E. (2022) Accountability-Driven School Reform: Are There Unintended Effects on Younger Children in Untested Grades? Early Childhood Research Quarterly, 61, 190-208. https://doi.org/10.1016/j.ecresq.2022.07.005. [ungated]
Test-based accountability pressures have had mixed effects on the student outcomes that they are intended to improve. Accountability policies have also resulted in transfers of less effective teachers into untested early grades and more effective teachers in early grades into tested grades, which could yield unintended negative consequences. In this study, we use a sharp regression discontinuity design to examine the effects of an accountability-driven school reform on student outcomes and teacher mobility in 38 elementary schools assigned to reform in North Carolina. We find evidence of a small increase in chronic absenteeism and grade retention in grades K-2 in the first year of reforms. We also find suggestive evidence of negative effects on early literacy and reading comprehension, measured using formative reading assessments, in the first year that rebounded somewhat in the second year. Schools labeled low performing reassigned low effectiveness teachers from tested grades into untested early grades, though these assignment practices were no more prevalent in reform than control schools. Our results suggest that accountability-driven school reform can yield negative consequences for younger students that may undermine the success and sustainability of school turnaround efforts.
6. Wolf, B. & Harbatkin, E. (2023).Making Sense of Effect Sizes: Systematic Differences in Treatment Effect Sizes by Outcome Measure Type. Journal of Research in Education Effectiveness, 16(1). 134-161. https://doi.org/10.1080/19345747.2022.2071364. [ungated]
One challenge in understanding “what works” in education is that effect sizes may not be comparable across studies, raising questions for practitioners and policymakers using research to select interventions. One factor that consistently relates to the magnitude of effect sizes is the type of outcome measure. This article uses study data from the What Works Clearinghouse to determine average effect sizes by outcome measure type. Outcome measures were categorized by whether the group who developed the measure potentially had a stake in the intervention (non-independent) or not (independent). Using meta-analysis and controlling for study quality and intervention characteristics, we find larger average effect sizes for non-independent measures than for independent measures. Results suggest that larger effect sizes for non-independent measures are not due to differences in implementation fidelity, study quality, or intervention or sample characteristics. Instead, non-independent and independent measures appear to represent partially but minimally overlapping latent constructs. Findings call into question whether policymakers and practitioners should make decisions based on non-independent measures when they are ultimately responsible for improving outcomes on independent measures.
Recruiting and retaining effective teachers is critical to school turnaround. However, research on how improving educator quality in low-performing schools contributes to school improvement is largely situated in urban settings. This study examines staffing practices through a descriptive analysis of the first cohort of Comprehensive Support and Improvement (CSI) schools in North Carolina. In the 94 schools slated for turnaround, ineffective teachers in rural schools were less likely to turn over than in non-rural schools. Instead of filling vacancies with more effective teachers than they lost, rural schools tended to assign their more effective teachers to tested grades and subjects.
4. Goldhaber, D., Imberman, S., Strunk, K.O., Hopkins, B., Brown, N., Harbatkin, E., & Kilbride, T. (2022). To What Extent Does In-Person Schooling Contribute to the Spread of COVID-19? Journal of Policy Analysis and Management, 41(1), 318-349. https://doi.org/10.1002/pam.22354. [ungated]
In this paper, we use data from two states—Michigan and Washington—on COVID case rates at the county level linked to information on the type of instructional modality offered by local public school districts during the 2020-21 school year to assess the relationship between modality and COVID outcomes. We focus primarily on COVID case rates, but also provide estimates for hospitalizations (in Washington only) and deaths. District and month fixed effects models that exploit within district (over time) variation in instructional modality and account for time-invariant district factors show that, in both states, in-person/hybrid relative to remote instruction can lead to increases in COVID spread in communities with moderate to high levels of pre-existing COVID cases. However, the causal effect is arguably small in magnitude and not consistently statistically significant across the pre-existing case distribution. Event study estimates show sharp increases in Michigan case rates after a district opens for in-person or hybrid schooling, but these fade out completely over two months.
A growing body of research has found that student-teacher race match is associated with higher test scores, teacher expectations, and teacher perceptions of students. This paper contributes to the student-teacher race match literature by investigating the effect of race match on course grades. To the extent that race match is associated with higher course grades for minority students, a more diverse teacher workforce is one mechanism that may help to narrow the achievement gap. Using a series of fixed effects models exploiting within-student variation across year and subject matter, I find that having a race-matched teacher is associated with a small but significant increase in course grade, on average. The positive effect of race match is driven largely by the experience of Black students.
2. Henry, G.T. & Harbatkin, E. (2020). The Next Generation of State Reforms to Improve their Lowest Performing Schools: An Evaluation of North Carolina’s School Transformation Intervention. Journal of Research on Educational Effectiveness, 13(4), 702-730. https://doi.org/10.1080/19345747.2020.1814464. [ungated]
In contrast to prior federally mandated school reforms, the Every Student Succeeds Act (ESSA) allows states more discretion in reforming their lowest performing schools, removes requirements to disrupt the status quo, and does not allocate substantial additional funds. Using a regression discontinuity design, we evaluate a state turnaround initiative aligned with ESSA requirements. We find the effect on student test score growth was not significant in year one and -0.13 in year two. Also in year two, we find that teachers in turnaround schools were 22.5 percentage points more likely to turn over. While the increased teacher turnover in NCT schools in 2017 opens the possibility that reform schools were intentionally replacing less effective teachers with more effective ones, we do not find evidence that strategic staffing occurred. The negative effects on student achievement appear to be tied to implementation of one of the few required components for serving low-performing schools under ESSA—a comprehensive needs assessment on which school supports are planned. These findings may serve as a cautionary tale for states planning low-performing school reforms under ESSA.
1. Bleiberg, J., & Harbatkin, E. (2020). Teacher Evaluation Reform: A Convergence of Federal and Local Forces. Educational Policy. 34(6), 918-952. https://doi.org/10.1177/0895904818802105. [ungated] [equal authorship, author names in alphabetical order]
This article employs event history analysis to explore the factors that were associated with rapid state uptake of teacher evaluation reform. We investigate three hypotheses for this rapid adoption: (a) downward diffusion from the federal government through Race to the Top (RTTT), (b) upward diffusion from large school district policies, and (c) the influence of intermediary organizations. Although RTTT clearly played a role in state adoption, our analysis suggests that having a large district implement teacher evaluation reform is the most consistent predictor of state adoption. Intermediary organizations appeared to play a role in the process as well.
Manuscripts under review
Harbatkin, E. & Wolf, B. State Accountability Decisions Under The Every Student Succeeds Act and the Stability, Validity, and Equity of School Ratings. Working paper available at edworkingpapers.com/ai23-863. Revise and resubmit.
Harbatkin, E., Burns, J.A., & Cullum, S.* The Role of Culture and Climate in School Turnaround. Working paper available at edworkingpapers.com/ai23-817. Revise and resubmit.
Harbatkin, E., & Moran, A.J.* Does Teacher Turnover Disrupt Collaboration in Schools? Evidence from Turnaround Districts. Revise and resubmit.
Manuscripts in preparation
Harbatkin, E., Strunk, K.O., & Cullum, S.* Did the COVID-19 Pandemic Exacerbate Staffing Challenges in Turnaround Schools? Evidence from Michigan.
Harbatkin, E., Allen, D., & Henry, G.T. Examining the Barriers to School Improvement: A Mixed Methods Approach.
Cullum, S.* & Harbatkin, E. Choosing the Goalposts: District-Defined Accountability Under the Every Student Succeeds Act.
Harbatkin, E. & Henry, G.T. Why Do Schools Fail? An Examination of School-Level Factors Associated with Low Performance.
Harbatkin, E. An Early Warning System for Low-Performing Schools.
Henry, G.T. & Harbatkin, E. Turnover at the Top: Estimating the Effects of Principal Turnover on Student, Teacher, and School Outcomes. Working paper available at http://edworkingpapers.com/ai19-95
Book chapters
Watson, C.,* Singer, J., Bertrand, A.,* & Harbatkin, E. (accepted) “Teacher Well-Being in Michigan’s Turnaround Schools through the COVID-19 Pandemic.” Contemporary Perspectives on School Turnaround and Reform. Charlotte, NC: Information Age Publishing.
Harbatkin, E., Henry, G.T., & Pham, L. (in press). “Planning a Quantitative Analysis” in Bickman & Rog (eds.) The Evaluation Handbook: An Evaluator’s Companion. New York: Guilford Publications, Inc. (chapter submitted)
Pham, L., Henry, G.T., & Harbatkin, E. (in press). “Conducting a Quantitative Analysis” in Bickman & Rog (eds.) The Evaluation Handbook: An Evaluator’s Companion. New York: Guilford Publications, Inc. (chapter submitted)
Selected public writing and blog posts
Harbatkin, E. & Nguyen, T.D. (2023, October 19). The relationship between teacher intentions, turnover behavior, and school conditions. Brookings Institution Brown Center Chalkboard. Available here.
Harbatkin, E. & Henry, G.T. (2019, October 21). The cascading effects of principal turnover on students and schools. Brookings Institution Brown Center Chalkboard.
Singer, J., Woulfin, S.L., Strunk, K.O., Watson, C.*, Bertrand, A.*, Lizarraga, L.*, Moran, A.J.*, & Harbatkin, E. (2023). Human Capital Challenges in Round 4 Partnership Districts. East Lansing, MI: Education Policy Innovation Collaborative, Michigan State University. Available here.
Harbatkin, E., Strunk, K.O., Watson, C.,* Bertrand, A.,* Cullum, S.,* Singer, J., & Woulfin, S. (2023) Teacher Mobility in Turnaround Schools: A Summative Report From the First Two Cohorts of Partnership. East Lansing, MI: Education Policy Innovation Collaborative, Michigan State University. Available here.
Cullum, S.*, Harbatkin, E., & Strunk, K.O. (2023). Student Enrollment in the First Two Cohorts of Partnership Schools. East Lansing, MI: Education Policy Innovation Collaborative. Available here.
Cullum, S.* & Harbatkin, E. (2023) Student Achievement in the First Two Cohorts of Partnership Schools. East Lansing, MI: Education Policy Innovation Collaborative. Available here.
Strunk, K.O., Harbatkin, E., McIlwain, A.,* Cullum, S.,* Torres, C., & Watson, C.* (2022). Partnership Turnaround: Year Four Report. East Lansing, MI: Education Policy Innovation Collaborative, Michigan State University. Available here.
McIlwain, A.* & Harbatkin, E. (2021). Human Capital in Michigan’s Partnership Schools and Other Low-Performing Schools During the COVID-19 Pandemic. East Lansing, MI: Education Policy Innovation Collaborative. Available here.
Hatch, E.* & Harbatkin, E. (2021). COVID-19 and Michigan’s Lowest Performing Schools. East Lansing, MI: Education Policy Innovation Collaborative. Available here.
Strunk, K.O., Harbatkin, E., Torres, C., McIlwain, A. Cullum, S., & Griskell, C. (2021). Partnership Turnaround: Year Three Report. East Lansing, MI: Education Policy Innovation Collaborative, Michigan State University. Available here.
Strunk, K.O., Burns, J. Torres, C., McIlwain, A., Waldron, S., & Harbatkin, E. (2020). Partnership Turnaround: Year Two Report. East Lansing, MI: Education Policy Innovation Collaborative, Michigan State University. Available here.
Harbatkin, E. & Henry, G.T. (2019). Impact of the North Carolina Transformation (NCT) Initiative to Reform Low-Performing Schools. Nashville, TN: North Carolina Transformation Initiative. Available here.
Harbatkin, E. & Henry, G.T. (2019). Barriers to Improvement in North Carolina’s Lowest Performing Schools. Nashville, TN: North Carolina Transformation Initiative. Available here.
McNeill, S.M., Harbatkin, E., Jenkins, J.M., Penner, E.K., & Henry, G.T. (2019). Early Literacy in North Carolina and Its Lowest Performing Schools. Nashville, TN: North Carolina Transformation Initiative. Available here.
Henry, G.T., Pham, L., Guthrie, J.E., & Harbatkin, E. (2018). Guiding Principles for Improving the Lowest Performing Schools in Tennessee. Nashville, TN: Tennessee Education Research Alliance. Available here.
Park, C., Arshan, N., Harbatkin, E., & Murphy, K. (2014, March). Evaluation of the College-Ready Writers Program: Baseline Report. Menlo Park, CA: SRI International.
Wechsler, M., Greenwald, E., Bienkowski, M., Arshan, N., Bosetti, K., Shields, P., Harbatkin, E., Black, A., & Comstock, M. (2013, November). Innovative Technology Experiences for Students and Teachers (ITEST) Program Evaluation: Final Report. Prepared for the National Science Foundation. Menlo Park, CA: SRI International.
Keating, K., Young, V., Mitchell, N., Tse, V., Yee, K., Heying, E., Cassidy, L., & Harbatkin, E. (2013, June) Strategies for Preparing At-Risk Youth for Postsecondary Success: Case Study Report. Prepared for the U.S. Department of Education. Menlo Park, CA: SRI International.
Wechsler, M., Greenwald, E., Bienkowski, M., Arshan, N., Bosetti, K., Shields, P., Harbatkin, E., Black, A., Comstock, M. (2013, March). Innovative Technology Experiences for Students and Teachers (ITEST) Program Evaluation: Interim Report. Prepared for the National Science Foundation. Menlo Park, CA: SRI International.
Harbatkin, E., Patel, P., and Jakopic, J. (2011, November). Job Opportunities for Low-Income Individuals (JOLI) Program Assessment Report. Prepared for the Office of Community Services, Administration for Children and Families, U.S. Department of Health and Human Services. Fairfax, VA: ICF International.