Mark Twain said there are three types of liars, “Liars, Damn Liars, and Statisticians.” Few examples illustrate the truth of that old maxim better that psychoeducational assessments and IEP eligibility determinations. Any parent that has been through an initial IEP, has experienced how statistics can be used to lie. At far too many IEP meetings and in far too many school district reports, the juggling of statistics becomes a weapon used to guard the gateway to special education eligibility. And, few statistical terms better exemplify the ability of statistics to mask lies than the term “average.”
Twain’s observation wasn’t the first and mine will likely not be the last. Books such as Measured Lies: The Bell Curve Examined and Inequality by Design: Cracking the Bell Curve Myth, have examined in detail the unfairness of statistics and the damage done by blind adherence to statistical “norms.” Clearly, I’m not going to go into this to that depth, but I want to warn parents, advocates, and attorneys of the many ways one simple statistical term is manipulated to deny students needed supports and services. That term Is “average.”
Most of us know the term average. Many of learned that there are three types of average: mean, median, and mode. Some of us still remember which is which. But none of that prepared of for the many faces of average demonstrated in reporting educational assessments. In that world, “average” might mean 50%, it might mean “meeting grade standards,” it might mean a range (between 35%-65%, 28%-72%, 20%-70%, or even 7%-93%), or it might be merely an arbitrary measured assigned by a test publisher. During a single meeting, or even a single report the term average may be used to mean any or even several of these dramatically different measurements.
All assessments used for psychoeducational assessments are “normed” against a representative sample population to establish a distribution of scores. Most test results consist of a collection of subtests and each subtest has its own statistical range, and various subtest can be combined to create different “Indexes” that better represent certain specific strengths, weaknesses, or disability profiles. Typically, each individual score can be reported in five or more ways: the raw score, scaled score, percentile rank, age equivalent, and grade equivalent. On top of those five different ways of reporting, many publishers add an additional arbitrary designation of their choosing (low, low average, average, high average, high or extremely below average, significantly below average, below average, average……). Each publisher can designate their own terminology and the percentile ranges that define those terms.
Statistical average usually means within 1 standard deviation from the mean, or 34% above or below the “real average.” That means that 68% of students are “average.” Obviously, the student scoring it the 16th percentile may bear little resemblance to the student scoring in the 84th percentile. But, according to most school psychologists, they are both “average.” And, your 16th percentile child just missed out on the special education lottery. Even more concerning are the games played with which representations of the score are used in the reports, and at the IEP. Typically, school psychologists will only report one or two measures in their report, and you will need to make a record request (they are required to provide the records in five days) for the test “protocols,” which are the instructions and scoresheets to get the full spectrum of the score values. Don’t expect them to report that your child tested 2-3 grade levels below in the grade equivalent measure, and expect to be told that those measures are inaccurate unless they happen to help justify a decision to deny services.
Score that are in areas of strength will be highlighted by reporting the scaled score or percentile ranking, areas of weakness will be reported as “average” or using one of the publisher’s arbitrary designations. Equally deceptive is the use of indexes to average out strong and weak scores. They will “average” out a 98th percentile in one subtest with a 2nd percentile in another to make 50%. Miraculously, another “average” result. The results of some tests can be completely invalidated when there are significant subtest variants. That approach ignores the value of the subtests variants in narrowing in on the actual area, or areas, of deficit and crafting accommodations most likely to work for that individual student. Because isn’t that ultimately the goal of the assessment process.
So the next time you are at an IEP and hear someone use the term “average” keep asking questions until you understand just what that particular “average” means. Maybe then you will know if you are dealing with a liar, a damn liar, or just a statistician.