This sixty page guide is really several things in one.

  • It is in part a guide to this particular tool, the OECD Test for Schools (Based on PISA), a test which individual schools, public and private, can participate in.
  • It also provides some high level treatment of the test’s alignment of PISA testing with 21st century skills and  “Deeper Learning.”  See the Appendix.
  • Finally, in the first full section, “leading your OECD program” and in the Case Studies section, the assessment example is OECD testing, but the framework and the treatments can serve as a guidance generally for how schools can best manage a new assessment tool project, using that new test or tool to advance student learning outcomes.

Enjoy.

 

kegley100910stg1821Call me crazy: Common Core Assessments aren’t too long in testing duration and shouldn’t be shortened.

Forgive me for being contrary: I know I threw a few friends when I wrote last week we shouldn’t assess projects in PBL (though my full argument was far more nuanced than my headline/thesis), and now I know I take the risk of irking more friends by making the argument which follows.

Among the many caveats to my argument, I’ll prioritize these two:

First, I too am appalled by the misuse and abuse of current or future standardized testing, particularly in regards to punishing schools and teachers.  What Bill Ferriter wrote recently on this topic is nothing short of brilliant. “It’s time that you start asking your policymakers some difficult questions about their positions on value-added measures of teacher performance.    If Jackson is right, those policies — which have rapidly become the norm instead of the exception in most states in America — are wasting our time AND our money.

I want quality testing to be used for meaningful purposes: advancing student learning, not teacher-bashing.

Second, these important advances in testing are certainty not the end of the line; they don’t represent a complete arrival at a place of testing excellence.  They are instead a significant and meaningful advance from the status quo toward that place of excellence, an advance I think we should applaud.  For more on the continued advances needed, see this recent Edweek post and the report from the Gordon Commission on the Future of Assessment in Education upon which it is commenting.

But here goes: Common Core Assessments PARCC and SBAC (Smarter Balanced) tests shouldn’t be any shorter in their time duration than they are planned to be.

Why?

1. Because we shouldn’t be so quick to call this testing time “lost” to teaching and learning. In even only a moderately good testing experience, testing time is learning time– sometimes superior learning time.

2. Because these new tests assess in ways far more authentic and meaningful than any previous generation of standardized K-12 educational tests, and assess the deeper learning our students greatly need to learn to be successful (learning which far too few are indeed learning), assessment information we need to improve their “deeper learning.” 

But both of these things will be compromised or lost if the tests get any shorter.

The length of these tests is being hotly debated and combated.

Edweek published last week a short article about the duration of the tests, and it is worth reviewing.

New tests being designed for students in nearly half the states in the country will take eight to 10 hours, depending on grade level, and schools will have a testing window of up to 20 days to administer them, according to guidance released today.

The tweets which followed the Edweek piece were not at all positive: the following tweet is entirely representative of the attitude in the feed of tweets about the Edweek post, although it is not entirely representative of the tone of those tweets, because many were more vulgar.

Let me flesh out my argument:

1.   We shouldn’t be so quick to call this testing time “lost” to learning: in even a moderately good quality testing experience, it is quite the opposite.

I don’t believe that time spent taking a good test is “time away from learning.”  It doesn’t even have to be a great test– just a good test will do.  When I look back at my K-16 education, I am certain that on average, I learned more, was more engaged, more challenged, more interested, more analytical and creative, when I was taking a half-decent test than I was when I was sitting in class watching a teacher talk in the front of the room.

Quite often– though not always– my test-taking times as a student were among the very most intellectually exciting and growth-oriented events and experiences in my education. (more…)

[Note: I’ve added further clarification to my argument in a comment below the post]

We call it PBL– Project Based Learning– but inevitably as we immerse ourselves in the practice and the thinking about it, we recognize the limitations of this term.

As I increasingly write and share thoughts about PBL, I find, and I know others do too, especially helpful the clarification provided by explaining what it isn’t– as for, example, it’s not Project-Oriented Learning, not an enrichment activity or an independent assignment– and the illumination offered by the metaphors– PBL is not the dessert, it’s the entree.

One of my concerns is that the term PBL makes a mistake in reinforcing the understandably pervasive, but in my eyes problematic, perception that the project itself– the thing students make or present or publish or post– is what the PBL experience is all, or mostly/especially, about.    It isn’t.    But saying PROJECT based learning makes it sounds as if it is.

“Learning by doing,” (LbD), or “Learning by Project Preparation” (LbPP) would, I’m arguing here, better capture convey the heart of what I call high quality, 21st century PBL.  (as do inquiry based learning and challenge-based learning, both of which are in the conversation but, in my experience, far less prominent/established as PBL).

Rethinking PBL this way helps us rethink how best we assess student learning in PBL, something for which that many teachers in surveys report feeling especially under-prepared for and lacking in proficiency.

The heart of my argument here is that despite the understandably common norm,  we shouldn’t assess the project.  Don’t.  

Don’t choose or create and apply a rubric for a “quality project,” and don’t tell student that their grade is based on the quality of their project.

I’m not saying we shouldn’t grade student learning– although I realize there are many who wish we could dramatically reduce or eliminate grading.   No– I’m operating inside the current conventional world of schooling and grades, the same one in which most of us live.

Assess not the project, but instead, assess the 3-6 things you’ve decided you want students to want students to know, or know better, and be able to do, or do better, at the end of the unit.

In assessing these concrete, specific, public things, you will most certainly seek and accept as evidence of their learning their final projects/products, but we’ve now re-conceptualized that assessment in meaningful ways.   And, essential to remember, is that we will seek and accept other things in addition to the project as evidence of their learning.

Time out here for a minute.  Let’s review what other key PBL resources say about assessment.

First up, the excellent free guide from High Tech High entitled Work that Matters.    Here from that book is the guru Ron Berger on the topic.

Assessment isn’t just about the final product.   Teachers often mistakenly presume that a project’s final product is the only thing they should assess, which leads them to assume that they should be able to tell whether the kids learned what they needed to learn by looking at the final product.

Actually, assessing what kids know is ongoing throughout a project. The product is the motivation for learning the material, but it won’t demonstrate that they learned it all. For example, in the physics standards project (see page 51) each kid only demonstrated one physics concept, so how do you know that they learned the rest of the material?

The answer to this question is that the project isn’t the assessment. You can assess what they’ve learned before the book project comes out, and afterwards. In Physics Standards they gave all the students a physics test with all the concepts in it. You need to do assessment throughout the project so that when they’re doing great artistic stuff, you know that they know what they need to know. You can’t leave it all to the end.

Ron Berger, Chief Programme Officer,
Expeditionary Learning

Note the emphasis he places here on not leaving assessment all to the end, and indeed, the project might not be able to capture and convey all the things you want students to learn.   You have to assess more than just the project, and you can give tests along the way, even.  Tests! 

work-that-mattersOn the following page of the High Tech High booklet, this is reinforced: “You may want to have a separate assessment of knowledge, such as an exam.”

And more:

Has the student learned the curriculum content required for this project?

The process followed here is much the same as for the assessment of skills,with one important difference: your project plan should include the essential curriculum content for the project. As a result, much of this will be considered ‘non-negotiable’, though it is still important to have students co-construct the process.

So, for example, you might ask students to determine how they will present the content knowledge they’ve acquired through the project (they might do this through an essay, quiz, presentation, film, etc). (more…)

It was a dynamite four days in Philly last week at the NAIS annual conference: although I was unsure how it would feel to be attending in a different capacity, not as a Head but in my new role of writer/consultant/presenter, it ended up very fun and engaging.   As always, the best parts are outside the formal conference in the camaraderie and fellowship found there with so many pursuing with parallel passion the meaningful and rewarding work of remaking learning for our fast-changing times.

The slides above come from a most fascinating session sharing what I’d argue is genuinely breakthrough work from the folks at the Index group on what they call their new Mission Skills Assessment, MSA, for Middle School Students.

(It was a big team presenting, including Lisa Pullman from Index, Tim Bazemore from New Canaan Country School (CT), Jennifer Phillips from Far Hills Country Day (NJ), and Rich Roberts from ETS; see the last slide for all their names and contact info)

As they explained, and as I often try my best to pursue here at 21k12, we have long as educators believed and proclaimed that character development, defined broadly, is of importance equal to that of intellectual and academic development, and yet truly, outside of the not-always-deeply successful advisory programming and a few assemblies here and there, how far do we usually go with this character education?

And, when students know that grades are the coin of the realm and that nearly all of the grades they earn and the feedback they get is on the academic-intellectual side, how well are we signaling to them the importance we place or guiding them with the feedback which is so important on the non-cognitive side of the equation?

Here with the MSA, the group has identified, after review of both the research of what makes for success out there, and of what our schools state in our missions we do in here, six key traits, and I love this list:

Teamwork, Creativity, Ethics, Resilience, Curiosity, Time Management. 

As the slides demonstrate, this has been an investigation carried out in the most serious of ways, spread out over five years and drawing upon the expert resources of and collaboration with ETS.  Their ETS partner, Rich Roberts, explained that as surprising as it might seem, ETS has been working on Noncog for over a decade, and indeed, the pursuit of noncog assessment which can match the quality of cognitive assessment goes back more than 60 years.

Roberts argued that the consensus view after decades of study is that noncog is not, no it is NOT, twice as important as cognitive skills and attributes for success in life– but it is EQUAL.

But assessing it has never been easy– this is the rub.  But, the research here conducted finds strong validity and reliability for a tripartite approach, as described in the image below, of student self-report, teacher evaluation, and a third tool for “triangulation.” NAIS and the Mission Skills Assessment from the Index Group   21k12

These third tools are discussed in slides 36-38, and include Situational Judgement Tests (SJTs), which were similarly touted at the Boalt Hall Law School study I described here, biographical studies, and Creativity Performance Tests.

For those that are skeptical that even with this triangulation we get to an effective measurement, check out the discussion of reliability and validity on slides 48-55, where reliability is found to be just a tad less than on the SAT and validity in prediction better than standardized test scores and GPA for student quality ratings and student well being and just a little less well than standardized test scores for GPA.

As for the inevitable question– whether and when this tool will become more broadly available, beyond the membership of the Index group, it appears as I view it that these questions have yet to be answered.   As soon as they are, I’ll do my best to report that news here.

But, there is no reason for schools outside of Index to not use these ideas and resources to advance their own work in assessing student development of these essential qualities.

Last week I presented (for a second time) a webinar for Simple K12 on the topic, Performance Task Assessment is 21st century Assessment.

Those slides are embedded above, and the webinar is available here (free for members, for a fee if you’re not).

In that presentation I discuss various strategies for designing and developing your own performance tasks for assessment, and suggest that one avenue is to borrow an existing one and adapt for your purposes.     In the PBL world where I also spend a lot of time, we refer people often to PBL libraries (BIE has a list of them here), and so it is important we match them with performance task libraries.

Performance Task assessment is becoming increasingly important, as I’ve posted here several times before, because of its role in Common Core assessments, (more…)

Delighted today to have the opportunity to share these slides and thoughts with folks here at OESIS today.  I continue to think that using technologies, current and emerging, to reinvent testing and assessment is among the primary projects for 21st century K-12 learning in the current decade, and I’m going to continue to do my best to support this reinvention.

As I explained at some length in the opening of my session, and I realize I may stand a bit alone here, I still love tests– of all kinds, including the “test” that is asking students to demonstrate their learning in challenging ways– and a huge part of my personal mission is to make testing more engaging and meaningful for students: let’s improve the way we use assessment as, for, and of learning! 

In “conversation” yesterday at educon about the CWRA and the data it generates, Pam Moran and I spoke about how important it is we all develop our skills in interpreting data– and even more than that, and I have to say, this has previously been a bridge just beyond my full focus– using data to conduct our own research as teachers and administrators.

Pam emphasized this is a critical project: we need to support teachers, offering education, resources, and time, to undertake their own action research projects and generate their own findings.

I began, belatedly, this conversation as a Head of School last year, and it is a big project, to be sure, to help busy, often over-loaded, teachers to get to this place.

And so it is with great respect and admiration I share with readers here this very interesting report, generated by teachers and administrators, on their research and findings.    The folks at St. Andrew’s Episcopal School (MD), particularly Glenn Whitman, were kind enough to send me their recent publication, Think Deeply and Differently:  The Transformational Classroom:How Research in Educational Neuroscience Enhances Teaching and Learning at St. Andrew’s.

Too often, I fear, and certainly I’ve been very much guilty of this, professional learning in schools lacks focus, coherence, continuity and a sense of accomplishment and completion, and part of the reason it lacks focus is because it doesn’t have a finish line and any type of finished product around which its efforts can be centered.

But, it would see here in this example from St. Andrew’s, they’ve tackled exactly this problem by organizing themselves, their professional learning, and their action research toward the end of a publication that speaks from and to their organizational mission and their pedagogical initiatives and next steps.   That is great.

To quote the preface,

St. Andrew’s Episcopal School and its Center for Transformative Teaching and Learning is a sterling example of how educators are informing the teaching and learning process through research-based practices.

These practices encompass every aspect of the educational experience including how we approach the learning environment, how we plan instruction to promote mastery of skills and concepts, how we assure that students are engaged in higher-order thinking and creative problem-solving, and how we use the arts and technol-ogy to maximize each child’s learning potential.

St. Andrew’s is on the forefront of not only practicing but also advancing this knowledge by engaging in research and discovery that has the potential to inform their own teaching practice as well as the entire field of education.  (more…)

Pam Moran and I shared this and facilitated this conversation today at educon: our thanks to the attendees for the rich and meaningful conversation.

This is the third of four posts about the USC Rossier Attributes that Matter Conference.

Morning Sessions: Non-cognitive Variables in Action and Attributes of Good Students and Good Professionals

Where Bill Sedlacek (see previous post) laid out the intellectual concept of noncognitive assessment with a bit of history and a lot of theory, sharing his decades of research and his passionate advocacy, the following two sessions took us from theory to practice, as five university administrators and researcher told us about the fascinating work they’d done in this field.

Attributes

Two bold and innovative directors of admissions at the university level, (Oregon State and DePaul), came to report that, despite their best efforts, their experiments with noncog assessment have had only very limited success in predicting student performance on campus.

As Eric Hoover reports in the Chronicle of Higher Ed about Noah Buckley’s leadership at OSU.

In 2004 the university added to its application the Insight Résumé, six short-answer questions.  One prompt asks applicants to describe how they overcame a challenge; another, to explain how they’ve developed knowledge in a given field.

The answers, scored on a 1-to-3 scale, inform admissions decisions in borderline cases, of applicants with less than a 3.0 GPA. “This gives us a way to say, ‘Hey, this is a diamond in the rough,'” Mr. Buckley says. For students with GPAs of 3.75 or higher, the scores help determine scholarship eligibility.

The Insight Résumé is a work in progress, Mr. Buckley says.

Reading 17,000 sets of essays requires a lot of time and training. Meanwhile, he believes the addition has helped Oregon State attract more-diverse applicants, but it’s hard to know for sure. A recent analysis found that although the scores positively correlated with retention and graduation rates, they did not offer “substantive improvements in predictions” of students’ success relative to other factors, especially high-school GPAs.

Details about the Insight Resume can be found in the slides above; it includes

Six short-answer questions asked as part of admissions application:
•Leadership / group contributions
•Knowledge in a field / creativity
•Dealing with adversity
•Community service
•Handling systemic changes / discrimination
•Goals / task commitment

Similarly, at DePaul as at OSU, very meaningful evidentiary results still stand further in the future.   (more…)

Presentation7

[graphic from Digital Learning Now]

This post continue a small project here at 21k12 of viewing the coming Common Core Standards through a backwards prism: the testing assessments that will evaluate student and school success at learning and teaching Common Core standards.  These new assessments sit at a junction of topics I’m interested in and working on regularly: integrating technology, next generation and digitally enhanced assessment, computer adaptive assessment, and  performance task assessment.

These new Common Core (CCSS) assessments are the product in part of Secretary Arne Duncan’s call for a new generation of Assessments, Assessment 2.0 he calls it, about which I have written before.   To advance this vision of moving “beyond the bubble,” the US DOE is spending, via Race to the Top funding, more than $300 M in developing new kinds of tests and testing systems, split between two major programs, PARCC and Smarter Balanced.

As the Ed Leadership article by Nancy Doorey reports,

The assessment consortia are drawing on new advances in technology, cognitive science, and measurement as they develop this improved generation of assessments.

They hope these new systems will address concerns about existing state assessments—that many assessments measure skills too narrowly; return results that are “too little, too late” to be useful; and do not adequately assess whether students can apply their skills to solve complex problems, an ability students need to succeed in college, the workplace, and as citizens.

Both tests are administered digitally and online, and will require in most states and districts a massive technological infrastructure improvement to be implemented.   Administering them digitally and online offers many advantages, including the ability to offer adaptive testing (which is currently intended for SB only, not PARCC), and faster results returned to teachers for instructional purposes.

Eight questions worth asking about the the new assessments:

1.  Will they come on-time or be delayed, and will the technology be ready for them?    Although the test design is funded (enormously), the technological infrastructure upgrades are not externally funded, and it remains a very open question whether and from where this funding will come.   If districts are unable to meet the requirements, will the 2014-15 launch date for these digital and online tests be postponed?

Engaging Ed fears they will.

Digital Learning Now, in a recent report embedded below, pleads with the consortia: Don’t Delay.

Don’t phase in. With two years left to prepare, the combination of a long test window and supporting outdated operating systems allows almost all schools to support online testing now. Going further to support paper-and-pencil testing in and past 2015 is unnecessary, expensive, and reduces comparability.

It is also is unwise for districts to seek to compromise by the use of less than 2:1 ratios of computers to students.    Imagine the schools which are trying to use current computer labs to assess their students– it will take 12 disruptive weeks to roll all students through the labs, and the labs themselves won’t be available for any other learning during that time. (more…)

Above are the slides for my presentation today to the Association of Colorado Independent Schools Heads and senior administrators, a three hour workshop.   Sadly, I (again!) made the mistake of trying to stuff in too much information, and several of our intended activities and videos had to be cut.

Below are first, some of the key links to think to which I referred, and below that, some of the videos I showed or intended to show as part of the presentation.

A very valuable reference is the excellent presentation, Measuring What We Value by Lyons and Niblock.   (Note: I borrowed/adapted a small number of these slides for integration into my presentation: My thanks to Lyons and Niblock. )

My thanks to Lee Quinby, the very fine ACIS executive director, and all those who took the time for the session this morning.

Links:

Think TankNot everything that can be counted counts, and not everything that counts can be counted. — Albert Einstein

So much of what really matters in education just can’t be measured. — Independent school educators everywhere

Count me in. The quotes above are words I’ve uttered not dozens or scores but hundreds of times during my 15 years of independent school administration—and I very much believe I am in good company. Indeed, how can I argue with Albert Einstein?

But perhaps I am wrong. I’ve been enjoying reading this month a book which shakes my conviction that there is much of value that cannot be measured—and which gives very good guidance in how we can improve the way we capture in data just about anything we desire to know more about. The book is entitled How to Measure Anything by Douglas Hubbard—and although in my experience it is not a much discussed book in educational circles, I think it should be.

Grant Wiggins, author of the essential education book, Understanding by Design, is a fan of this book, directed me to Hubbard’s work in a blog post entitled “Oh You Can’t Measure That.”

Recently, I read a great book that might be of interest to anyone who wants to get beyond a knee-jerk reaction about what can and can’t be measured. The book makes the point from the git-go, in its title: How to Measure Anything: Finding the Value of Intangibles in Business, by Douglas Hubbard. Don’t let the ‘in Business” part throw you off. Almost everything in the book speaks to educational outcomes.

Hubbard writes with an axe to grind, and what becomes clear in the reading is that education is far from the only field or profession where managers express, frequently, their view that something, or most things, can’t be measured. This is Hubbard’s bête noir, one he is determined to confront with this book.

Often an important decision requires better knowledge of the alleged intangible, but when an executive believes something to be immeasurable, attempts to measure it will not even be considered.

As a result, decisions are less informed than they could be. The chance of error increases. Resources are misallocated, good ideas are rejected, and bad ideas are accepted.

Hubbard embeds as foundations to his argument three genuinely inspiring and impressive stories of measurement—times when individuals generated creative, ingenious methods for measuring something thought to be immeasurable—most famously and wondrously, Eratosthenes’ uncannily accurate measurement of the circumference of the earth two hundred years before the Common Era, using nothing more than shadows of the sun.

Click here to read the full post.

brookhart book“We still really don’t know how to assess problem-solving,” I heard a university professor of engineering say last week, and it resonated because it is so clear to me that while we all want to do more to educate our students in the work of solving complex problems and creative thinking, and we know the importance of assessment in driving this instruction, we nevertheless stumble in our clarity about what and how we ought to assess these things.

Most often the books I write about here are what might be viewed as the superstructure books– the writing about the future of learning and the most innovative practices for reinventing what and how we should be teaching.

Examples of this would be my reviews of Net Smart by Rheingold, Networked by Wellman and Rainey, Future Perfect by Johnson, and Zhao’s World Class Leaners. 

But sometimes it is useful to return to the foundations, and firm up our terms and concepts at more basic, but critical, levels— indeed, if we don’t do so, the superstructures will be that much more unwieldy.

This 2010 title, from ASCD, is exactly that, and I hope readers will forgive the “primer” nature of this post.   It would seem to me that schools which simply do the work to try to unify and make more consistent our language and practice around higher order thinking skills assessment will be well poised to then experiment, iterate, and innovate in this essential realm.

Brookhart begins by defining the core elements of what we mean by higher order thinking:

  • Transfer: relating learning to other elements beyond those they were taught to associate with it.
  • Critical thinking (judgment): reasonable, reflective thinking focused on what to believe or do, applying wise judgment and producing an informed critique.
  • Problem solving, including creative thinking: the non-automatic strategizing required for solving an open-ended problem, involving identifying problems, creating something new as a solution.

establishing three core components of what exactly effective assessment entails:

  1. Specify clearly and exactly what it is you want to assess.
  2. Design tasks or test items that require students to demonstrate this knowledge or skill.
  3. Decide what you will take as evidence of the degree to which students have shown this knowledge or skill.

and elaborating with three more principles of higher order thinking assessment:

  • Presenting something for students to think about, usually in the form of text, visuals, scenarios, resource material, problems.
  • Using novel material–material new to students, not covered in class and not subject to recall.
  • Distinguishing between level of difficult, easy versus hard, and level of thinking, lower order thinking/recall versus higher order thinking) and control for each separately. (more…)