This sixty page guide is really several things in one.

  • It is in part a guide to this particular tool, the OECD Test for Schools (Based on PISA), a test which individual schools, public and private, can participate in.
  • It also provides some high level treatment of the test’s alignment of PISA testing with 21st century skills and  “Deeper Learning.”  See the Appendix.
  • Finally, in the first full section, “leading your OECD program” and in the Case Studies section, the assessment example is OECD testing, but the framework and the treatments can serve as a guidance generally for how schools can best manage a new assessment tool project, using that new test or tool to advance student learning outcomes.



There are two axes on which we can map and evaluate the excellence of a standardized test, or in other words,  external measurement of learning outcomes.   The first axis is does it measure what matters: does the work it requires students to do, the learning it requires students demonstrate, matter to successful citizenship and professional accomplishment?   Does it assess not content measurement but application of knowledge to original, complex problems; does it assess students’s ability to do that most important of things, TRANSFER?

The second axis is what is done with the results of the test?  Are the results used to punish schools, teachers, kids?  Are the results published in ways which rank kids, or rank teachers– even in the newspaper?  In a neutral position on this axis is the null effect: are the results provided to educators, without repercussions or rewards, but also without great guidance, insights, analysis, comparisons, examples?    Or, at the high end of this axis,  are the results used as and for compelling and creative information, illumination, and inspiration to improve learning for ALL students?

Map your favorite– or least favorite– tests on these axes– and then watch Andreas Schleicher in the TED talk above discuss the PISA test and map PISA.

(Full disclosure, I’m no longer, as of about a month ago, a neutral or objective commentator on this topic, having taken on an engagement for a PISA related project.   But, I was thrilled to seek and accept the engagement because of my passion for the power of PISA.)

Tony Wagner  was one of my influences here, particularly as he argued for the value of PISA, particularly what it measures, in a postscript to his film, the Finland Phenomenon.

“The PISA tests measure different things from TIMMS, which measures factual recall of memorized content knowledge.   This is a test asking you to apply knowledge to a new question or problem you’ve never seen before.”

Wagner interviews Roger Bybee, Ph.D., who explains that

“TIMMS and NAEP are similar in that they both are curriculum based, seeing what the standard curriculum contains and designing an assessment to see how well students have attained what is in curriculum.  PISA is an assessment that takes a literacy approach– not just the content one learns from the curriculum but how well you can apply that content in life situations in everyday use of the knowledge and the abilities one has.

About measuring what matters, Schleicher has a funny line:

So with PISA, we try to change this by measuring the knowledge and skills of people directly. And we took a very special angle to this. We were less interested in whether students can simply reproduce what they have learned in school, but we wanted to test whether they can extrapolate from what they know and apply their knowledge in novel situations. 

Now, some people have criticized us for this. They say, you know, such a way of measuring outcomes is terribly unfair to people, because we test students with problems they haven’t seen before

But if you take that logic, you know, you should consider life unfair, because the test of truth in life is not whether we can remember what we learned in school, but whether we are prepared for change, whether we are prepared for jobs that haven’t been created, to use technologies that haven’t been invented, to solve problems we just can’t anticipate today.

The bulk of the talk is dedicated to what I described above as the second axis: what do we do with these results?  OECD and PISA have dedicated 15 years to making sense of international testing results and comparisons– and it is a treasure trove. (more…)

oecd pisaThe country reports made the point that, in many Asian countries, classes are much larger than in the United States and teachers typically use whole-group instruction through the entire class period. They also pointed out that, in these countries, one sees little lecturing by the teacher.

Instead, the teacher gives real-world problems to the whole class and, having observed the students attempting to solve those problems, asks several to come to the blackboard to talk about their approaches to the problem, knowing that some of those students have made errors in the strategy they have selected for solving the problem.

As described in the country reports for Japan and Shanghai-China, the teacher uses these differences in strategy to develop a class discussion that focuses on the underlying concepts involved in problem-solving, and thereby promotes a deep understanding of the topic under discussion among both the quickest and the slowest students in the class. Nothing could so vividly demonstrate the point that instructional practice matters.

lessons-from-pisa-for-the-united-states_9789264096660-enAsian teachers often complain about class sizes getting too small to find a useful range of student solutions to a problem in order to conduct a good class, instead of complaining that the class is too large to teach effectively, as in the United States.

[Strong Perfomers and Successful Reformers in Education: Lessons from PISA for the United States, p 241]

PISA has been a fascination of mine for the past few years; like many other educations committed to educational innovation and learning from best practices, I was intrigued by the success of Finland in the 2009 PISA administration, and the value of PISA testing was affirmed for me by my friend and mentor Tony Wagner in his video about the Finland Phenomenon.  (I most recently wrote about PISA testing in April: PISA-OECD Test: Using Results to Improve Learning in Fairfax County)

OECD, which manages the PISA test, generates a terrific stream of quality monographs about what can be inferred from PISA testing; the quote at top is just one such example of many– and I intend to share more in the months to come.   I’m deeply committed, as regular readers may know, to the idea that problems should come first and that “inverting” the familiar/traditional dynamic of teaching content first, presenting problems second, is the truly compelling”FLIP” teaching.   This concept is well explained in Ted McCain’s Teaching for Tomorrow:

we need to invert the conventional classroom dynamic: instead of teaching information and content first, and then asking students to answer questions about it second, we should put the question/ problem first, and then facilitate students with information and guidance as they seek the answer and hold them accountable for the excellence of their solutions and of their presentation of their results”

As we can learn from PISA, this methodology is widely deployed in those (particular) Asian educational programs which demonstrate strong success at the PISA testing– and for the many of us who tend to favor smaller classes, isn’t it fascinating to see the logic employed by Asian teachers to prefer larger classes?

edleader21Learning from PISA will be one of my major projects for the next six months or so: I’m delighted to share here news that I recently signed on with my friends at EdLeader21 for a contract engagement with them to write this fall and winter, in conjunction with Edleader21’s Chief Learning Officer Valerie Greenhill, a guide with the working title  OECD TEST FOR SCHOOLS (BASED ON PISA): An IMPLEMENTATION TOOLKIT.   (EdLeader21 is headed up by Ken Kay, founder and former long-time President of the Partnership for 21st century Skills).

This project, which has funding support from the Hewlett Foundation Deeper Learning program, will include articulating the alignment of the PISA/OECD test for schools with 21st century skills and learning outcomes, sharing case studies of how districts and schools are using the OECD test for schools based on PISA results to improve learning for their students, and capturing a set of best practice strategies for implementing the testing program and applying its reports effectively.

As Valerie Greenhill has written about the EdLeader21 work with PISA/OECD test for schools,

EdLeader21 has played a unique role throughout the establishment of the OECD Test for Schools. As members of the advisory board, we have represented the voices of district and school leaders around issues of implementation. We have continually stressed the need for resources and energy to be applied to what we see as the most important part of this project: What happens AFTER school leaders receive their results.

The OECD Test for Schools result report is lengthy and quite complex. For the results to translate into meaningful changes that improve student outcomes, school and district leaders will need targeted advice for implementation. The Implementation Toolkit will contain such practical guidance for education leaders who have received their results

Wish me luck in this in this initiative, and please offer me your thoughts on valuable PISA testing resources and on best using PISA and other test results for improving student learning and 21st century college, career, and life preparation.