There are two axes on which we can map and evaluate the excellence of a standardized test, or in other words, external measurement of learning outcomes. The first axis is does it measure what matters: does the work it requires students to do, the learning it requires students demonstrate, matter to successful citizenship and professional accomplishment? Does it assess not content measurement but application of knowledge to original, complex problems; does it assess students’s ability to do that most important of things, TRANSFER?
The second axis is what is done with the results of the test? Are the results used to punish schools, teachers, kids? Are the results published in ways which rank kids, or rank teachers– even in the newspaper? In a neutral position on this axis is the null effect: are the results provided to educators, without repercussions or rewards, but also without great guidance, insights, analysis, comparisons, examples? Or, at the high end of this axis, are the results used as and for compelling and creative information, illumination, and inspiration to improve learning for ALL students?
Map your favorite– or least favorite– tests on these axes– and then watch Andreas Schleicher in the TED talk above discuss the PISA test and map PISA.
(Full disclosure, I’m no longer, as of about a month ago, a neutral or objective commentator on this topic, having taken on an engagement for a PISA related project. But, I was thrilled to seek and accept the engagement because of my passion for the power of PISA.)
Tony Wagner was one of my influences here, particularly as he argued for the value of PISA, particularly what it measures, in a postscript to his film, the Finland Phenomenon.
“The PISA tests measure different things from TIMMS, which measures factual recall of memorized content knowledge. This is a test asking you to apply knowledge to a new question or problem you’ve never seen before.”
Wagner interviews Roger Bybee, Ph.D., who explains that
“TIMMS and NAEP are similar in that they both are curriculum based, seeing what the standard curriculum contains and designing an assessment to see how well students have attained what is in curriculum. PISA is an assessment that takes a literacy approach– not just the content one learns from the curriculum but how well you can apply that content in life situations in everyday use of the knowledge and the abilities one has.
About measuring what matters, Schleicher has a funny line:
So with PISA, we try to change this by measuring the knowledge and skills of people directly. And we took a very special angle to this. We were less interested in whether students can simply reproduce what they have learned in school, but we wanted to test whether they can extrapolate from what they know and apply their knowledge in novel situations.
Now, some people have criticized us for this. They say, you know, such a way of measuring outcomes is terribly unfair to people, because we test students with problems they haven’t seen before.
But if you take that logic, you know, you should consider life unfair, because the test of truth in life is not whether we can remember what we learned in school, but whether we are prepared for change, whether we are prepared for jobs that haven’t been created, to use technologies that haven’t been invented, to solve problems we just can’t anticipate today.
The bulk of the talk is dedicated to what I described above as the second axis: what do we do with these results? OECD and PISA have dedicated 15 years to making sense of international testing results and comparisons– and it is a treasure trove. (more…)