Late last year, we offered this puzzle, to which no answer was provided at the time:
McKinsey recently came out with a study of how a shift to remote learning has affected math test scores of students in elementary school. The impact of school closures has been large and negative and has had the effect of exacerbating the gap between White students and students of color. Leaving aside the racial gap, what explains the increasing drop-off as students (both White and nonwhite) get older? I don’t have the answer, but, if you think you do, send us a note.
We got a lot of responses, and, more than anything, they illustrate so-called Type III Error: “answering the wrong question.” (Read more here about how Type III error this can cause AI projects to fail.)
The Responses
Recall the question:
“Leaving aside the racial gap, what explains the increasing drop-off as students get older?”
Most respondents answered a different question, usually one of the following:
- What explains the difference between white and nonwhite scores? or
- Why do students perform worse in math as they get older?
The focus on racial differences in some answers is interesting, given that this was explicitly not the question of interest. This is obviously a topic of great interest, and a lightning rod for attention, but it is not what the puzzle was.
People who answered question #2 seemed to be answering this formulation of the problem:
“As students get older, why do they fall increasingly short of some (unspecified) math performance standard?”
But that was not the question either. The plot that readers were asked to explain showed the drop-off for a given grade in this year’s scores, compared to the same grade’s average for the prior three years.
This is the question that relates directly to the effects of the pandemic. Only a third of the respondents answered it, but here are some points they made:
Parental Help Less Useful
With virtual learning, kids rely more on parents. As children progress through math, the simple concepts of first grade are readily amenable to parental help, but the greater complexity of fifth grade math eludes more parents. As one respondent put it:
“one contributing factor is the changes in mathematics pedagogy that leave parents unable to figure out what the heck their kids are doing in math! I know parents with advanced degrees that can’t help their fourth-grade kids with math homework.”
Other Responsibilities at Home
As kids get older, they may have other responsibilities in the home that take time away from studies. These responsibilities and distractions were not there in prior years when they were in a classroom.
Parents Assume Older Kids Don’t Need Help with Virtual Learning
Parental help with virtual learning is less forthcoming for older kids, particularly if parents must also cope with younger siblings, so parental time/attention is diminished
Diminished Motivation
As kids get older, the natural urge to learn diminishes. (ed. note: presumably this would be true both in class and at home; perhaps the respondent meant to suggest that the structure of being in school would counteract this better than the home environment.)
Conclusion
The pandemic will provide a wealth of data to examine in years to come, answering questions like the one posed here. What became immediately clear in our short Puzzle survey is the powerful pull of Type III error. There is much hand-wring over general innumeracy in society. Our readers fall in the “numerate” class, yet two-thirds of them fell down the Type III error rabbit-hole. On the other hand, we mustn’t be too hard on ourselves. Statistical plots presented in the news media are often confusing or inaccurate and lead us astray. It might be natural to look at a plot like this not as a scientific problem, but merely as a prompt to share already-formed thoughts on the subject.