21k12

–

There are two axes on which we can map and evaluate the excellence of a standardized test, or in other words,  external measurement of learning outcomes.   The first axis is does it measure what matters: does the work it requires students to do, the learning it requires students demonstrate, matter to successful citizenship and professional accomplishment?   Does it assess not content measurement but application of knowledge to original, complex problems; does it assess students’s ability to do that most important of things, TRANSFER?

The second axis is what is done with the results of the test?  Are the results used to punish schools, teachers, kids?  Are the results published in ways which rank kids, or rank teachers– even in the newspaper?  In a neutral position on this axis is the null effect: are the results provided to educators, without repercussions or rewards, but also without great guidance, insights, analysis, comparisons, examples?    Or, at the high end of this axis,  are the results used as and for compelling and creative information, illumination, and inspiration to improve learning for ALL students?

Map your favorite– or least favorite– tests on these axes– and then watch Andreas Schleicher in the TED talk above discuss the PISA test and map PISA.

(Full disclosure, I’m no longer, as of about a month ago, a neutral or objective commentator on this topic, having taken on an engagement for a PISA related project.   But, I was thrilled to seek and accept the engagement because of my passion for the power of PISA.)

Tony Wagner  was one of my influences here, particularly as he argued for the value of PISA, particularly what it measures, in a postscript to his film, the Finland Phenomenon.

“The PISA tests measure different things from TIMMS, which measures factual recall of memorized content knowledge.   This is a test asking you to apply knowledge to a new question or problem you’ve never seen before.”

Wagner interviews Roger Bybee, Ph.D., who explains that

“TIMMS and NAEP are similar in that they both are curriculum based, seeing what the standard curriculum contains and designing an assessment to see how well students have attained what is in curriculum.  PISA is an assessment that takes a literacy approach– not just the content one learns from the curriculum but how well you can apply that content in life situations in everyday use of the knowledge and the abilities one has.

About measuring what matters, Schleicher has a funny line:

So with PISA, we try to change this by measuring the knowledge and skills of people directly. And we took a very special angle to this. We were less interested in whether students can simply reproduce what they have learned in school, but we wanted to test whether they can extrapolate from what they know and apply their knowledge in novel situations. 

Now, some people have criticized us for this. They say, you know, such a way of measuring outcomes is terribly unfair to people, because we test students with problems they haven’t seen before. 

But if you take that logic, you know, you should consider life unfair, because the test of truth in life is not whether we can remember what we learned in school, but whether we are prepared for change, whether we are prepared for jobs that haven’t been created, to use technologies that haven’t been invented, to solve problems we just can’t anticipate today.

The bulk of the talk is dedicated to what I described above as the second axis: what do we do with these results?  OECD and PISA have dedicated 15 years to making sense of international testing results and comparisons– and it is a treasure trove.

Now, knowing what successful systems are doing doesn’t yet tell us how to improve. That’s also clear, and that’s where some of the limits of international comparisons of PISA are.That’s where other forms of research need to kick in, and that’s also why PISA doesn’t venture into telling countries what they should be doing. But its strength lies in telling themwhat everybody else has been doing. 

And the example of PISA shows that data can be more powerful than administrative control of financial subsidy through which we usually run education systems.

PISA doesn’t punish schools or educational systems, and PISA doesn’t tell nations how they should educate; instead they offer information, powerful data, and leave it to educators to decide what to do next.

As Schleicher explains in the talk, there are three domains upon which they focus their reporting and research: average performance, equity of performance, and extremes of performance.   For each of these three things, a second dimension is added– change of performance over time.

So while the highlighted take away is overall national average performance, such that we know that Korea, Shanghai, Finland perform extremely well, Schleicher also really wants us to look at disparities: are nations closing the socio-economic gap?

 Educators like to talk about equity. With PISA, we wanted to measure how they actually deliver equity, in terms of ensuring that people from different social backgrounds have equal chances. And we see that in some countries, the impact of social background on learning outcomes is very, very strong. Opportunities are unequally distributed. A lot of potential of young children is wasted.

We see in other countries that it matters much less into which social context you’re born. We all want to be there, in the upper right quadrant, where performance is strong and learning opportunities are equally distributed. Nobody, and no country, can afford to be there, where performance is poor and there are large social disparities.

And then we can debate, you know, is it better to be there, where performance is strong at the price of large disparities? Or do we want to focus on equity and accept mediocrity?

But actually, if you look at how countries come out on this picture, you see there are a lot of countries that actually are combining excellence with equity. In fact, one of the most important lessons from this comparison is that you don’t have to compromise equity to achieve excellence.These countries have moved on from providing excellence for just some to providing excellence for all, a very important lesson.

Without this data, without these findings, we might find ourselves stuck thinking that things can’t change– they are culture bound, or economically determined, or just not possible.   But with the power of the PISA data, we know they can change, and we can learn about the practices employed where they do.

It is not just average and equity– it is low end and high end.  Some countries have very few students, one or two percentage points, not achieving the basic proficiencies necessary for baseline participation in complex society– and some have far, far too many not achieving this baseline.   Some countries have good average performance, but not as much truly high end master performances, and that may limit them too in generating the intellectual elites necessary for innovating and deep problem-solving capacity.     Korea is an example.

In the year 2000, Korea did already very well,but the Koreans were concerned that only a small shareof their students achieved the really high levels of excellence.They took up the challenge,and Korea was able to double the proportion of studentsachieving excellence in one decade in the field of reading.

For readers here from independent schools and from high achieving affluent suburban districts, the Korean example is especially compelling.  Within  your particular school or districts, you’re probably reasonably confident your average performance is strong– up there with Finland Korea, quite likely, and closing equity is not within your grasp in your particular school.  But, how large a proportion of your students are performing at the very top– and can you use this tool first to determine that, and second, to motivate, inform, and assess your progress toward that improving that proportion?