New York’s Outcome: Trust No Education Measurement!

New York has long stood at the forefront of the nation’s political and social worlds. George Washington, for instance, took the first Presidential oath of office at the corner of Wall and Broad Streets. A century later, the woman’s suffrage movement blossomed under New York’s Susan B. Anthony. And during the twentieth century, Mayor Fiorello Laguardia fought corruption by pioneering a professional civil service.

More recently, Wall Street billionaire Mike Bloomberg helped launch a national trend by spending his personal fortune to win political office and institute business friendly reforms. Despite serving as mayor during a period of immense economic strife, Bloomberg has been lauded strongly for his success. In particular, his focus on public education has attracted praise for resulting in the improvement of minority student performance.

To produce this outcome, New York’s Board of Education needed to accomplish two goals. First, it needed to target poorly performing minority students and address their deficiencies. Then, it needed to create a system of measurement to quantify levels of improvement (or lack thereof) and adjust strategies accordingly. The Bloomberg administration appeared to succeed in these endeavors in a dramatic fashion; that is why so many were shocked to learn recently that New York’s unprecedented improvements in minority education were attributable, in fact, to false interpretations.

Beating the Baseline

To understand how accurate measurements can lead to false interpretations regarding reading comprehension skills and math calculation abilities, it is helpful to review how politicians and educators customarily define performance standards. Generally speaking, in the field of public education, a specific level of performance is established for each year or grade from kindergarten through high school. Success is then defined as the percentage of students within each grade who achieve a target outcome on an examination.

Sounds complicated? It’s actually a relatively simple task, assuming that the education system establishes appropriate standards. For instance, if a school system creates an examination that measures a child’s ability to read at a third grade level, its Board of Education can simply require each third grader to complete the exam. If all students pass that exam, then 100% of the students can be classified as having beaten the baseline and having achieved the requisite level of competency to progress to the fourth grade. The school system can thus pride itself on having left no child behind, i.e. none whatsoever!

Of course, most school systems cannot possibly achieve perfect 100% scores; instead, they strive to achieve significant percentage improvements each year. For example, a financially strapped inner city school with a significant student drop-out rate may consider itself highly successful if it increases its percentage from 70% to 75% in a single year. On the other hand, an affluent suburban school may be content if it simply nudges its percentage from 95% to 96% during the same year.

A Widening Gap

So how could New York have so badly misinterpreted performance levels among its minority student population? How could its cadre of statistical experts have misunderstand simple changes in percentage scores?

Apparently, the snafu originated in the establishment of baseline performance levels. Let’s assume, for instance, that students are classified as having passed a baseline examination when they earn a “C” grade score. Let’s also assume that half of the minority student population earns a “C Minus” on the exam, while the other half earns a “C.” Furthermore, all of the mainstream student population earns a “B Plus.”

Given such a situation, what would happen if new teaching methods drive all of the mainstream students from a “B Plus” level to an “A Plus” level? And what if those same methods, far less dramatically, simply nudge the “C Minus” minority students up a notch to the “C” level? Every minority student would end up with a “C” and every mainstream student with an “A Plus;” the gap between minority and mainstream students would thus widen significantly on a year-to-year basis.

Defining Success

If success, however, is simply defined as a grade of “C” or higher, a statistician might assert that the percentage of successful minority students has leaped from 50% to 100%, while the percentage of successful mainstream students has remained at 100%. In essence, this is how the Bloomberg administration could claim an unprecedented narrowing of the gap between minority and mainstream student scores — even while the underlying evidence showed no such outcome.

In fact, what would be the outcome in our hypothetical situation if the school district decides to nudge its definition of success from a “C” to a “C Plus” grade? Every minority student would be scored as a failure while every mainstream student would be scored as a success, a truly dismal comparative statistic that could be masked by simply tweaking the baseline definition of success back down to a “C” grade!

Incidentally, public knowledge of New York City’s misinterpretation of minority performance only spread after its Board of Education decided to nudge up its standards of success in this manner. Educators across the nation may thus learn the following lesson from this story: decisions to raise baseline standards of performance may reveal that interpretations of success may have been faulty all along!