In this post Michael Siegel and Mary Filardo write about a recent supply and analysis analysis released by D.C. Public Schools that focused on the lack of high-quality schools in high-poverty areas.
Siegel, who specializes in public and environmental finance, has 35 years of experience in preparing and analyzing studies related to economic and fiscal impact analysis, project feasibility, public sector cost, revenue forecasting and demand studies, utility rates and tariffs, and public education. He has, among other education projects, advised the D.C. Council on school modernization legislation, and is a member of the District’s Public Education Finance Reform Commission.
Filardo is a leading national authority on school facility planning, management and public private development. She is the founder and Executive Director of the Washington, DC-based 21st Century Schools Fund, which provides urban communities with leadership, innovative financing solutions, research, and public policy analysis of school facility issues. She, too, is a member of the District’s Public Education Finance Reform Commission.
By Michael Siegel and Mary Filardo
Background
Last month the District of Columbia Deputy Mayor for Education released “Quality Schools: Every Child, Every School, Every Neighborhood .” The study was paid for by the Walton Foundation and prepared by the Illinois Facilities Fund (IFF), a Chicago-based charter finance and real estate advisory organization. Preparation of the study was overseen by IFF’s director of research, Dr. Jovita Baber.
The study comes at a time when the District is within about five years of completing an ambitious modernization and right-sizing program for its public school facilities, and a long and substantial decline in enrollment in DCPS’s traditional public schools has slowed and may have begun to reverse.
Perhaps not un-relatedly, some of the District’s neighborhoods have also begun to reverse decades of declining population that saw the loss of much of its middle class.
Summary
IFF’s study purports to identify “service gaps” between the supply of and demand for “performing seats” in both DCPS and public charter schools. IFF describes its study:
At its core, this study is a supply and demand analysis. It subtracts the number of seats in performing schools from the number of students in the public system and provides that data by cluster for each of the 39 neighborhood clusters designated by the DC government for community planning purposes. To identify schools providing performing seats, the study averages the percent proficient in 2011 DC Comprehensive Assessment System (DC-CAS) and the predicted percent proficient in 2016, for each grade division (K-5, 6-8 and 9-12). To predict whether a school is likely to meet or exceed current state standards in the next five years (2016), the study uses an extrapolative regression of DC-CAS results over the past five years (2007-2011). Based on the mean of the 2011 standardized test scores and a predicted projection for 2016, each school is ranked. The top quartile of schools is considered performing and referred to as Tier 1 in a four tier system. Tier 1 schools have a high level of achievement on the 2011 DC-CAS results, a steep improvement slope over the past five years or both. (IFF p. 6)
The report then makes recommendations that it presumes to close these service gaps. IFF summaries its recommendations:
Increasing the number of performing seats is paramount. This study demonstrates that the actions with the greatest value for students will occur if DCPS and the Public Charter School Board (PCSB) work together to concentrate on the ten priority neighborhood clusters. In particular, IFF recommends:
1. Invest in facilities and programs to accelerate performance in Tier 2 schools.
2. Close or turnaround Tier 4 DCPS schools. Close Tier 4 charter schools.
3. Fill seats in Tier 1 schools. Sustain the performing capacity of Tier 1 schools.
4. Monitor Tier 3 schools. (IFF, p.6)
Unfortunately, the study’s methodology, analysis, and recommendations are so seriously flawed, they fail to provide a valid basis for any actionable policies, or for guiding public investment or for school facilities planning.
Were the study’s recommendations to be implemented, the immediate effect would be to halt and hamstring scheduled modernizations of schools throughout the District. They would cause unwarranted disruption and uncertainty for thousands of students and families, while reducing accountability and fostering public distrust. They would also close and transfer governance of dozens of DCPS schools to charter operators, and turn over control of these real estate assets to the Public Charter School Board for use as “incentives” and disposition as it sees fit.
IFF’s expected outcomes are based on conjecture
* The key premise of the study is that higher-performance follows charter operators to low-performing schools, and that undefined investment will do the same for tier 2 schools.
* When asked, IFF’s research director was unable to identify a low-performing school in any of the five large urban cities where it has made similar recommendations that were transformed to high-performing status upon transfer to a charter operator.
* Much of the “creaming” effect that can result from re-sorting existing student populations into new and/or reorganized schools may have already occurred in the District which has the second-highest percent of students attending charter schools nationally, and a number of successful specialty schools.
No evidence for IFF’s speculated outcomes
* Transfer of Anacostia, Dunbar, and Coolidge High Schools to a charter operator in 2008 did not elevate them to high-performing status. The charter manager for two of these schools was asked to leave after two years.
* A review and analysis of transformation case studies prepared for the U.S. Department of Education found fewer than three dozen successful examples nationwide, for which only “minimal” evidence was found that transformation strategies had any affect whatsoever on outcomes.
* A 2011 study of an urban school system found the post-closure and transfer performance of students from closed schools was adversely affected. Adverse affects were minimized when students were combined with the student body of an existing higher-performing school. This is not what IFF’s recommendations would accomplish in the District.
* Because few (if any) high-performing District schools have excess capacity, and most charter students are likely to attend schools near their place of residence, the existing student body of schools IFF targets for closure can be expected to transfer largely intact to new charter schools, many to be located in the identical buildings or in a nearby replacement.
* A 2009 “matched pair” study designed to eliminate differences in student demographics as a factor found charter schools to have under-performed their matched pair traditional public schools.
Conflicts and lack of qualifications
* IFF’s director of research, who oversaw its study of the District, lacks any prior publications, quantitative analysis, or expertise in education and pedagogic research.
* IFF is a self-described charter school “stakeholder” that provides real estate consulting services to charter operators. As a conduit lender to charters it also has a potential financial interest in the implementation of its recommendations.
* IFF’s recommends closing dozens of (mostly DCPS) schools it profiles as “low-performing” and the transfer of their publicly-owned real estate assets to the Public Charter School Board for use as “incentives” and disposition as it sees fit.
Invalid methodology and analysis
* Single-variate regression models are used by IFF to estimate trend lines based on five or fewer data points for each school. This approach is unlikely to have generated high R-square values or statistically significant results for many (and possibly most) schools.
* Data for some schools are corrupted by cheating that reportedly occurred from 2008 to 2011.
* IFF’s methodology does not employ a more appropriate longitudinal model. Rather, its static single-variate model (erroneously) presumes the attributes of the modeled cohort apply to the following cohort, and fails to account for other factors known to affect outcomes (e.g., special education, ELL, economically disadvantaged).
* IFF’s tier assignments are essentially arbitrary. For example, the upper two tier 4 slots are occupied by Roosevelt and Cardozo high schools. IFF relegates them for closure and transfer (and perhaps for demolition), yet their forecast proficiency exceeds about a dozen tier 3 schools.
It’s also of no small concern that IFF’s arbitrary replacement cost hurdle could consign to demolition some of the District’s most prominent and historic legacy properties. Had its recommendations been in effect at the time, Wilson, Cardozo, and Walls Senior High Schools together with Janney Elementary could have met with the wrecking ball.
Discussion of methodological and analytical issues
IFF’s analysis is flawed by a paucity of data points, corrupted data, and a methodology that relies on the single variable of time to forecast the performance of students in 2016 – many of whom have yet to enroll in school or take their first DC-CAS tests.
One can have little confidence in the predictive value of such a model. IFF explicitly acknowledges its model is inappropriate for the purposes for which it is utilized:
“This model cannot and does not purport to forecast the percent that will be proficient in a school in 2016,” the report says on Page 9.
Yet, this is precisely what IFF inexplicably proceeds to do.
Its forecasts of proficiency are the core of IFF’s study. They are used to profile schools into four tiers and to condemn those in the third to dis-investment (and likely future closure), and the fourth tier to summary closure and transfer to charter operators.
Demonstrating an astounding disregard for the limitations of its methods and data also enables IFF to foresee the achievement of each District school six decades hence:
“Assuming the current trajectory of improvement, it will take approximately 33 years (2045) to have 75 percent of the students testing at grade level in math and 63 years (2075) for 75 percent to be at grade level in reading,”
No credible researcher with a passing familiarity of quantitative analysis would make such a statement, or attempt to apply such a model for the purposes IFF’s study has done.
When questioned about the effect of reported widespread cheating on standardized tests, Dr. Baber responded that the study’s large data set would wash out any effects. To the contrary, her study tells us that each school’s forecast is based on as few as two, and no more than five, actual data points.
IFF’s methodology and analysis used to assign schools to a particular tier according to their forecast median performance on standardized tests is essentially arbitrary and invalid.
IFF’s recommendations are without basis
The premises for IFF’s recommendations are also without basis and appear to be the product of confused thinking:
“The economic diversity of students in Tier 1 schools [higher performing schools] and the geographic distribution of performing schools reaffirms that all students across the District can and do succeed when given the chance to attend a performing school,” it says on page 25, with materials in brackets added.
IFF’s “service gap” maps (maps 10, 12, and 14) shows a concentration of Tier 1 schools in upper Northwest, west of 16th Street. But even in these schools, “all students” do not succeed.
Consider Murch Elementary and Wilson High School, both in affluent upper northwest Washington D.C. A significant number of out-of-boundary students attend Murch, while Wilson draws students city-wide. Murch’s 2010 DC-CAS results show 50 percent of its 36 “economically disadvantaged” students in that year (representing 18 percent of all students tested in the school) to have not met the “adequate yearly progress requirement of No Child Left Behind, in reading 2010. At nearby Wilson Senior High School, 36 percent of such students did not do so.
Similar reasoning is behind IFF’s premise that closure and transfer of a school and its student body to a charter operator suffices to confer high-performing status upon their “seats.” This false premise occurs because IFF conflates a “seat” – the physical capacity for one student – with the student sitting in it. A student sitting in a seat in a school transferred to a charter that operates a high-performing school elsewhere is deemed to be sitting in a high-performing seat.
But “seats” and charter operators do not take DC-CAS tests – students do. IFF’s conflation of seats and students leads it to conclude that performing seats can be ordered up like flapjacks at the fireman’s supper:
“To accelerate performance in the District, add 27,070 performing seats in the Top Ten priority neighborhood clusters by 2016,” it says on Page 42.
IFF’s study indicates that a closed school’s student body would remain largely intact, often in the very same school building. Accordingly, there is no basis to expect closed Tier 4 (or later, Tier 3) schools to be performing seats upon re-opening when all that has occurred is a transfer of operational management (change in governance).
Charter management was imposed on three chronically low-performing DCPS public high schools in 2008 (Dunbar, Anacostia, and Coolidge). The performance of these three schools has not been elevated as IFF suggests. In fact, the charter manager of two of these schools was asked to leave after two years due to lack of improvement and widespread community dissatisfaction.
IFF recommends targeting investment to Tier 2 schools. Presumably, this would mean they would be modernized and/or obtain other operational enhancements. In either case, IFF confers Tier 1 status upon them. For reasons unexplained, IFF arbitrarily prefers this (assumed) outcome to investing in Tier 3 and 4 schools that, by the same logic, would elevate them to Tier 2 or 3.
IFF would withhold facility investment altogether from Tier 4 schools by imposing a burdensome, costly, time-consuming, and undefined “cost-benefit” requirement upon them that most would undoubtedly fail for equally spurious reasons.
“Transformation” is rare and little understood
The unstated premise behind IFF’s recommendation for Tier 4 schools is that their closure and transfer to charter operators will result in the a priori transformation of a lower-performing school to a higher-performing school. For the parents and students attending these schools, this would likely to prove to be a cruel illusion.
Consider that IFF has made similar recommendations in St. Louis, Chicago, Denver, Milwaukee, and Kansas City. Yet neither IFF’s study nor its director of research can cite a single school where this remarkable turnaround has occurred. When queried, Dr. Baber vaguely referred to the U.S. Department of Education website for case studies of successful transformations.
The Education Department’s transformation case studies were prepared by the Institute of Education Science. A literature search found documentation for only 35 schools in which post-transformation student performance improved. Notably, the institute could find only “low” or “minimal” evidence to link its recommended transformation strategies to improved student performance in these schools. The institute could not distinguish why efforts at these schools succeeded when so many others failed. Consequently, the institute warns:
“The recommendations in this guide are based on a collection of case studies of low-performing schools that improved student achievement in one to three years. The panel feels compelled to emphasize that the level of evidence is low because none of the studies examined for this practice guide is based on a research methodology that yields valid causal inference.”
In reality, there is no evidence that a change in governance of Tier 4 schools will assure their transformation to high-performing status. A well conceived and executed study (Miron, 2009) published by CREDO at Stanford University compared performance scores of students in public charter schools to a matched traditional public school.8
Paired schools were matched by race, gender, special education, and English language proficiency to a demographically comparable public school. The results for charters were unimpressive. Only 17 percent outperformed their matched public school, while 36 percent performed worse. The remainder of the pairs performed virtually the same.
A study of performance-based school closures (Engberg, 2011) also provides little support for IFF’s premise, particularly as the District has no effective surplus capacity in its higher-performing schools. This study found students from closed lower-performing schools experienced adverse affects when transferred, and that such affects could be minimized when students were transferred to higher performing schools.
IFF’s recommendation that charters be allowed to open in closed public school buildings before they are closed would likely to preclude any such beneficial effect as, upon re-opening as new charter schools, they would retain substantially the same student body as attended the closed school.
IFF shows DCPS to operate a number of higher-performing schools, and a number of lower-performing schools to be operated by charters. IFF exhibits no interest as to what factors have contributed to these outcomes.
Conclusion
In the final analysis, there is no valid evidence to justify the outcomes IFF’s rankings and recommendations. Their only predictable results would be the disruption of the lives of thousands of students and families; the imposition of an arbitrary process to select schools for dis-investment, investment, demolition, and closure; the transfer of control of school facilities to a publicly-unaccountable appointed Board; and, the attendant loss of public trust that would result.
Although IFF’s study must be rejected, we are in agreement that the District should focus on its chronically lower-performing schools. It should do so in a manner that engages our citizens, educators, stakeholders, and capable local researchers to formulate a strategy for success. Such efforts could be informed by a study prepared for the District by the 21st Century School Fund, the Brookings Institution, and the Urban Institute in 2008 that recommends actions to increase education quality and housing opportunities. As part of an integrated strategy, we would not preclude the participation of charter operators with successful turnaround experience for a particular school or cluster.
We would also urge the District to stay the course on school modernizations and capacity-based right-sizing, while re-doubling and expanding its restructuring efforts, and directing new resources towards its chronically under-performing schools.
This will require additional resources which, together with other members of the Public Education Finance Reform Commission, we have recommended by assigning a funding “weight” to students who are economically-disadvantaged and substantially behind grade level. The commission also recommends an adequacy study that we hope will also identify specific school-based resources needed to improve pupil performance in lower-performing schools.
For these schools, DCPS must also concentrate on recruiting dynamic and successful principals and educators to spearhead restructuring efforts, and also align its central office to support them.
For all such schools, it will also be necessary to develop use other metrics of success that would focus on stability, security, and fostering a culture of learning. Such metrics might include drop-out rates, pupil transiency and turnover, tardiness and absenteeism, incidents of violence, promotion and suspension rates, books checked out of the library, in-school tutoring, and participation at parent-teacher conferences. Improvement in these indicators can be expected to precede rising test scores.
None of the foregoing will instantaneously transform a lower-performing school to a higher-performing school. There are no magic wands or easy answers that would accomplish this, despite what IFF would have us believe. We are confident, however, that positive results can and will be obtained with focused, well-informed, and sustained efforts.
Let’s get to work.
I have been reading your posts regularly. I need to say that you are doing a fantastic job. Please keep up the great work.
ResponderExcluir