“We still really don’t know how to assess problem-solving,” I heard a university professor of engineering say last week, and it resonated because it is so clear to me that while we all want to do more to educate our students in the work of solving complex problems and creative thinking, and we know the importance of assessment in driving this instruction, we nevertheless stumble in our clarity about what and how we ought to assess these things.
Most often the books I write about here are what might be viewed as the superstructure books– the writing about the future of learning and the most innovative practices for reinventing what and how we should be teaching.
But sometimes it is useful to return to the foundations, and firm up our terms and concepts at more basic, but critical, levels— indeed, if we don’t do so, the superstructures will be that much more unwieldy.
This 2010 title, from ASCD, is exactly that, and I hope readers will forgive the “primer” nature of this post. It would seem to me that schools which simply do the work to try to unify and make more consistent our language and practice around higher order thinking skills assessment will be well poised to then experiment, iterate, and innovate in this essential realm.
Brookhart begins by defining the core elements of what we mean by higher order thinking:
- Transfer: relating learning to other elements beyond those they were taught to associate with it.
- Critical thinking (judgment): reasonable, reflective thinking focused on what to believe or do, applying wise judgment and producing an informed critique.
- Problem solving, including creative thinking: the non-automatic strategizing required for solving an open-ended problem, involving identifying problems, creating something new as a solution.
establishing three core components of what exactly effective assessment entails:
- Specify clearly and exactly what it is you want to assess.
- Design tasks or test items that require students to demonstrate this knowledge or skill.
- Decide what you will take as evidence of the degree to which students have shown this knowledge or skill.
and elaborating with three more principles of higher order thinking assessment:
- Presenting something for students to think about, usually in the form of text, visuals, scenarios, resource material, problems.
- Using novel material–material new to students, not covered in class and not subject to recall.
- Distinguishing between level of difficult, easy versus hard, and level of thinking, lower order thinking/recall versus higher order thinking) and control for each separately.
The case for teaching and assessing higher order thinking skills might be self-evident for most of us, but there is still value in what Brookhart shares about the research evidence for doing so, citing multiple studies and one meta-analysis showing the strongly statistically significant impact upon cognition, student achievement in particular subject area, and most of all, in student attitudes and motivation.
Students do not become engaged with their studies in the abstract, nor do they become motivated in the abstract. Rather, they become engaged in thinking about particular things and motivated to learn particular things. Higher order thinking increases students’ sense of control over ideas. Thinking is much more fun than memorizing.
Happily, this is true of low achievers as well as for high achievers, and at all grade levels.
Throughout the book there is good common sense, such as her advice, often recognized but less often practiced,that we all
be careful how we read student papers. Analyze the thinking and the writing separately. Then you will be able to give more targeted and helpful feedback.
She also provides useful resources, such as carefully selected exemplars of excellent rubrics, such as this one from NWREL for mathematical problem solving, broken into conceptual understanding, strategies and reasoning, computation and execution, communication, and insights.
Brookhart, having built her platform of the core elements of assessment, then applies them to a small array of higher order thinking areas, including evaluation, analysis, logic and reasoning, judgment, problem solving and creativity/creative thinking.
Critical thinking, we all understand, is an essential tool for digital literacy, and research, as I’ve written before, makes clear that teaching and assessing critical thinking explicitly and emphatically makes a big difference for student mastery here.
As for all areas, she breaks it down to give teachers and students more clarity about what and how to assess. For Critical thinking, which she also calls crap detection, she says students have to be provided a subject of study, and then asked to
- evaluate the credibility of the source
- identify implicit assumptions
- identify rhetorical and persuasive strategies
For problem-solving, she borrows from a classic, the Bransford Stein The Ideal Problem Solver, a book I’m looking forward to.
- Identify the problem
- define and represent the problem
- Explore possible strategies
- act on the strategies
- look back and evaluate the effects of your activities.
Brookhart adds a few more elements:
- identify irrelevancies
- identify obstacles or additional information for solving a problem
- reason with data
- use analogies
- solve a problem backwards
If we are going to talk, as I have done so often, about elevating the importance of teaching and assessing problem-solving, we should, or maybe it is only me– I should– be more clear and more specific in my communications about what this contains– and Brookhart offers good foundational guidance here.
As she dives into creativity, the author encounters a definitional challenge which she adeptly navigates her way through. Is creativity exclusively generative– the bringing forward of new ideas– or does in include the selective: determining which of the many ideas are most applicable, which could be viewed as exercising critical judgment separate from creativity.
Taking the second path, she offers this:
Creativity means putting together things in new ways either conceptually or artistically observing things other might miss, constructing something novel, using unusual or unconventional imagery that nevertheless works to make an interesting point, and the like.
Creativity and creative thinking is not certainly not limited to artistic creativity, though it includes it, but it excludes a nicely colored report cover or anything aesthetically pleasing or cute.
Continuing with the all important definition, she cites Ken Robinson (“a process of having original ideas of value”), and then draws upon the Partnership for 21st century skills and others to provide a list of what “creative students do”:
- recognize the importance of a deep knowledge base and continually work to learn new things
- are open to new ideas and actively seek them out.
- find source material for ideas in a wide variety of media, people and events.
- look for ways to organize and reorganize ideas into different categories and combinations, and then evaluate whether the results are interesting, new, or helpful.
- Use trial and error when they are not sure how to proceed, viewing failure as an opportunity to learn.
So this all sounds good, but what can and what can’t you evaluate and grade in creativity?
This is where she distinguishes more deeply creativity from creative thinking.
Creativity is understood in this distinction to be limited to the epiphany, the generative moment, which she argues, and my own jury is out regarding whether I concur, cannot be graded.
Of course it is fine to give narrative feedback, but “simple generation of something new, in my opinion, should not be graded or scored in a rubric which ends up in a final grade. The act of grading itself is a kind of critique or evaluation.”
Ken Robinson is brought in for moral support:
Whether there should be an individual grade for creativity, that’s a larger question. Certainly giving people credit for originality, encouraging it, and giving kids some way of reflecting on whether these new ideas are more effective than existing ideas is a powerful part of pedagogy. But you can’t reduce everything to a number in the end, and I don’t think we should.
It is not just about a number:
Another problem is that, in order to grade, you need criteria and a scale. By definition, if the student has a truly new idea or product, you can’t have already listed all the elements of it you would observe and the criteria by which you would evaluate them. So you wan’t have a solid basis on which to grade.
A compelling argument– with which I have many sympathies and I know at other times I would have signed on to whole-heartedly. But recall that former Tuft Provost Robert Sternberg believes he can assess, with a number, the creativity of applicants to college.
As I wrote in a previous post: At Tufts, teams of assessors developed comprehensive rubrics, worked on sample essays to develop their scoring skills, compared results, and argued until arriving at near-consensus on their approach.
Essays were evaluated on how
- “original—novel and different from others—
- and compelling—well-crafted and adding value— they are, as well as
- how appropriately they accomplish the task at hand.”
Now the latter of these three, Brookhart is fine scoring, and the second she might believe is fine for scoring but usually belongs in a separate category of communication. But for the first of the Sternberg set, she would seem to deny scoring.
If creativity, defined narrowly, is out for scoring, creative thinking, which entails both the generation of ideas plus the evaluation of those ideas and the application of them. is still entirely available.
The very best way to stimulate creativity is to inspire it by making assignments that are, in their own right, creative. To assess creative thinking, an assessment should do the following:
- Require student production of some new ideas or a new product, or require students to reorganize ideas in some new way.
- Allow for student choice on matters related to the learning target, not on on tangential aspects such as format.
- If graded, evaluate student work against the criteria students were trying to reach, as well as conventional criteria for real work in the discipline.
As useful as the book is on the whole, this section leaves a little bit to be desired– I think there is a bit of a hole here on the discrete break-down of creative thinking and tools for evaluating it, and I think she may be mistaken in not at least offering a tentative approach to assessing and scoring creativity alone.
Grant Wiggins, as I cited before in my Sternberg piece, believes we can and should measure creativity. In fairness, in his post to my eyes he speaks of creativity and creative thinking as if they were almost interchangeable, eliding the difference, so perhaps he is not altogether different from Brookhart’s position.
Now, of course, the naysayers are quick to say that you cannot measure creative thinking. This is silly: here is a rubric for doing so: Creative. We can and do measure anything: critical and creative thinking, wine quality, doctors, meals, athletic potential, etc.
All in all, Brookhart’s is a fine and substantial contribution to the construction of a foundation for what we mean by assessment, how we can do it better, and how we should best practice assessing higher order thinking skills. The question of creativity, though, admittedly complicated and controversial, is less well illuminated than is problem solving and critical thinking.