OU blog

Personal Blogs

Design Museum

MY NOTES : Inside the Black Box

Visible to anyone in the world

Inside the Black Box by Paul Black and Dylan Wiliam (1998)

My intention with this notes is to qualifiy what I am picking out from what I learn as I go along. ASo watch out, I'll be back several times to adjust what I've put here with examples and reflection. 

Raising stands of learning that are achieved through school education. (p.1)

System engineer = black box and inputs to deliver outputs.

What is happening 

Some things counter-productive

Something policy-makers can do?

Teaching and learning have to be interactive (p.2)

Feedback to modify the activities

Effect size 0.4 = one grade at GCSE

Improved formative assessment helps the (so-called) low attempt more than the rest.

Frequent assessment feedback helps both groups enhance their learning (Fuchs et al, 1997) Effects of Task Focused Goals on Low Achieving Students. (p.5)

Actively involved (p.5)

Requires guidance on how work can be improved.

Advice not grades are needed (p.6)

Personal improvement is more important than setting one against another with grades. 

Vs. students getting into the habit of ‘just getting by’. (p.8)

NOTE :> The ultimate use of assessment information which is elicited in order to improve learning is the pupil.

Vs. - ‘the overall result’ (with grades/gold stars) ‘is to enhance the frequency and the extent of under-achievement’.

  • A culture of success

  • A belief that all can achieve

NOTE:> Feedback to any pupil should be about the particular qualities of his or her work, with advice on what he or she can do to improve, and should avoid with other pupils.

Formative assessment is enhanced through self- and peer- assessment.

Pupils are honest in assessing themselves and others.

Their own assessments become an object of discussion with their teachers and with one another. (p.10)

The desired goal

Present position

A way to close the gap (Sadler, 1989) Formative Assessment and the design of instructional systems (Instructional science) 18 pp 119-144

NOTE : > For formative assessment to be productive, pupils should be trained in self-assessment so that they can understand the main purpose of their learning and thereby grasp what they need to do to achieve.

Careful scrutiny of the main components of a teaching plan. As the argument develops it becomes clear that instruction and formative assessment are indivisible.

NOTE : > Opportunities for pupils to express their understanding should be designed into any piece of teaching for this will initiate the interaction whereby formative assessment aids learning. (p.11)

The dialogue between pupils and a teacher should be thoughtful, reflective, focused to evoke and explore understanding, and conducted so that all pupils have an opportunity to think and to express their ideas.

Untapped potential not fixed IQ (p.14)

To take new practices on board what teachers ‘need is a variety of living examples of implementation, by teachers with whom they can identify and from whom they can both desire conviction and confidence that they can do better, and see concrete examples of what doing better means in practice. (p.16)

Drawing > Median > via swimming and making a jigsaw > full time in the West for creativity.

Facts or thinking

Permalink Add your comment
Share post
Design Museum

Embedded Formative Assessment by Dylan Wiliam

Visible to anyone in the world


Pghe2QUydgC8RiqGNR_40Nuvy1cCflch60xtgto46zvdi6KG-bqJKalQTelVd_9MbAgDQUVzW-TseQHVOxV3OqpMezznrISOkgqoBatyTSepKqWFd9ojF49SZyvgJ3qlGjYQ7U6q

William. D (2017) Embedded Formative Assessment: (Strategies for Classroom Assessment That Drives Student Engagement and Learning) (The New Art and Science of Teaching) : Second Edition. Bloomington, USA. Solution Tree Press. 

Dylan Wiliam is to formative assessment what Chris Witty is to Covid

I’m getting used to Dylan Wiliam - I think of him as the David Walliams of Education, the Will-I-Am of Formative Assessment.

I’ve been pointed at his webinars, sat through a keynote or two, read the book and the read pamphlet ‘Inside the Black Box: raising standards from classroom assessment’ which he produced with Paul Black in 1998 and established his reputation. 

In my first pass of note taking I had over 60 quotes and ideas here, not all from Dylan William. With some editing I’ve got this down to 19 ‘Top Tips’ which are all set out ready to quote. 

I’ll go back to the book if needs be for clarification. In this case it is an eBook which makes it very easy as the digital version will correlate to actual print page numbers.

This is how I learn. 

It’s a slow and repetitive process: skim read a book to get the lay of the land, read it with an open notebook, then transfer the notes to a document like this. Then edit the notes and where necessary go back to the original book. Most important of all - give it a go. Over the last five weeks I’ve had a chance to apply some of the thinking to both online remote teaching and in the classroom face-to-face.  

When a book strikes me as really important I am likely to have it in print and eBook - they read differently and you take different things from each. 

These notes are for me to browse, refer back to and use when I next have a formal rationale or observation to write, as well as for every day reflection on the learning and e-learning experience I am currently going through. 

Myth Busting

Wiliam puts the evidence of learning above myth and has a number of bugbears. 

Good teaching is difficult

It is relatively easy to think up cool stuff for students to do in classrooms, but the problem with such an activity-based approach is that too often it is not clear what the students are going to learn. (Wiliam 2017 p.94) 

Wiliam disregards learning styles, ‘which have no discernible impact on student achievement at all’. (Wiliam 2017 p.11 also Adey, Fairbrother, William, Johnson, & Jones, (1999). Rather - ‘as long as teachers vary their teaching style, then it is likely that students will get some experience of being pushed beyond it’. (Wiliam 2017 p.48) 

Nor is Wiliam a  fan of ‘performance of a learning task’ as a predictor of long-term retention of learning (Wiliam 2017 p.47 from Bjork, R.A. (1994) Any mention of ‘neuroscience’ as the panacea annoys him. (Wiliam 2017 p.50) 

There is no shortcut

Wiliam’s firm belief is that formative assessment improves performance. (Wiliam 2017 p.11) His view - ‘the use of assessment for summative purposes - grading, sorting, and ranking students - gets in the way’ of learning. (Wiliam 2017 p.56) Education is overly prescriptive with rubrics. (Wiliam p.93 in Alfied Kohn (2006)

No one can do the learning for the student who does not engage.

Our classrooms seem to be based on the principle that if teachers try really hard, they can do the learning for the learners. (Wiliam 2017 p.225) 

According to Wiliam there is little evidence that the following ‘tricks’ have any impact of student achievement:

  • Summarisation

  • Highlighting

  • Keyword mnemonic

  • Image use for text learning

  • Rereading (Wiliam 2017 p.225) 

Wiliam sets out five key stages of formative assessment : 

  1. Clarifying, sharing and understanding learning intentions and success criteria.

  2. Eliciting evidence of learning

  3. Providing feedback that moves learning forward

  4. Activating learners as instructional resources for one another

  5. Activating learners as owner of their own learning

The Assessment Reform Group puts it this way. Assessment to improve learning requires five elements to be in place (cited Broadfoot et al., 1999) (Wiliam 2017 p.61)

  1. Providing effective feedback to students

  2. Actively involving students in their own learning

  3. Adjusting teaching to take into account the assessment results

  4. Recognising the profound influence assessment has on student’s motivation and self-esteem, both of which are crucial influences on learning.

  5. Needing students to be able to assess themselves and understand how to improve.

Challenge them!

I like Wiliam’s thinking that ‘good instruction creates desirable difficulties’ - that we learn and retain knowledge through struggling with it. (Wiliam, 2017 p.11) I know from personal experience that the greater the struggle I am willing to endure, the deeper and the longer lasting my learning. It is when I can face the struggle … that I struggle.  Good instruction creates what he describes as ‘desirable difficulties’ (Wiliam, 2017 p.47) quoting Bjork (1994, p.193).

Pedagogy Over Curriculum

‘A bad curriculum well taught is usually a better experience for students than a good curriculum badly taught’. (Wiliam, 2017 p.11) I rather think we can apply this to many situations, for example, that pedagogy comes first - not EdTech. 

Formative Assessment above all else 

The Wiliam mantra is that, ‘attention to minute-by-minute and day-to-day formative assessment is likely to have the biggest impact on student outcomes’. (Wiliam 2017 p.42) He defines formative assessment as ‘the process used by teachers and students to recognise and respond to student learning in order to enhance that learning, during the learning’. (Wiliam 2017 p.59). Formative assessment, according to Bloom (1969) is a kind of evaluation - ‘a brief test used by teachers and students as aids in the learning process. (Wiliam, 2017 p.53 in Bloom (1969) It is ‘the process used by teachers and students to recognise and respond to student learning in order to enhance that learning, during the learning. (Cowie & Bell, 1999 p.32) It is ‘assessment carried out during the instructional process for the purpose of improving teaching or learning’. (Wiliam 2017 p.59 in Shepard et al., 2005 p.275) 

I like this : ‘Frequent, interactive assessment of students’ progress and understanding to identify learning needs and adjust appropriately.’ (Wiliam 2017 p.59 from Looney 2005)

And if you need more ways to think of it, try this: ‘assessment, for learning tells ‘us’ ‘what progress each student is making toward meeting each standard while the learning is happening - when there’s still time to be helpful (Wiliam 2017 p.62 from Looney 2005 (pp1-2) Stiggins (2005)

All kinds of formative assessment are not equal

(Wiliam 2017 p.62) 

‘The evidence is clear that the shorter the assessment - interpretation - action cycle becomes the greater the impact on student achievement’. (Wiliam, 2016). He continues, ‘short-cycle formative assessment has to be the priority for schools and teachers, because the impact on students is greater. (Wiliam p.75) 

‘ … regular use of minute-by-minute and day-to-day classroom formative assessment can substantially improve student achievement’. (Wiliam 2017 p.81)

Design backwards from the learning outcome

(Wiliam 2017 p.81 from Grant Wiggins and Jay McTighe (2000)

Three issues in the development of learning intentions and success criteria that may be useful to think about. (Wiliam p.95)

  1. Task-specific versus generic scoring rubrics.

  2. Product-focused versus process-focused criteria

  3. Official versus student-friendly language

NOTE :> By being specific about what we want, we focus the students learning too much. ‘He who never made a mistake, never made a discover’ (Wiliam 2017 p.95 quoting Samuel Smiles (1862 p.275)

There are a number of techniques which appealed when I read about them and I have used them. These are techniques to help students understand and achieve learning intentions. (Wiliam 2017 p.104 from Clarke (2001)

This is simple, these phrases work 

WALT ‘We are learning to … ‘

WILF ‘What I’m looking for … ‘

TIB ‘This is because … ‘ 

Student Engagement Techniques

The teacher asks a question, selects a student to answer the question and then responds to the student’s answer, which is generally some kind of evaluation of what the student said. (Wiliam 2017 p.124) Teacher-Led classroom discussion.

Try some of these:

  • Wait time

  • Evaluative listening

  • Interpretive listening

  • Questions shells

  • Hot-seat questioning

  • All student response systems

  • ABCD Cards

  • Mini Whiteboards

  • Exit passes

  • Discussion vs. diagnostic questions

  • Alternative questions

Pose a question, pick a student at random

Pose-pause-pounce-bounce

  • Pose

  • 5 seconds - pause

  • Pounce-random choice

  • Bounce-what do you think (Wiliam 2017 p.129) 

App or Lollipop sticks (Wiliam 2017 p.129) 

These techniques are used to ensure that all students realise that all are expected to tke part and that it is ok to make mistakes. If someone does not provide a response then come back to them and tell them  “OK, I’ll come back to you”.  (Wiliam 2017 p.132) 

Then seek Response 2 and 3 and then return to the person who gave no response and ask them which reply they liked the best and ask them why.

‘Engagement and responsiveness - are at the heart of effective formative assessment’. (Wiliam 2017 p.142) 

Managing Challenging Behaviour

To change the behaviour criticise the behaviour not the student. (Wiliam 2017 p.171) 

It is quality rather than the quantity of praise that is most important - teacher praise is far more effective if it is infrequent, credible, contingent, specific, and genuine. (Brophy, 1981 in Wiliam 2017 p.171)

The use of feedback improves performance when it is focussed on what needs to be done to improve, and particularly when it gives specific details about how to improve. (Wiliam 2017 p.180)

Motivation

“It's up to me, and I can do something about it”.  (Wiliam 2017 p.183)

When students have to struggled in the learning task, the quality of their performance on this task reduces, but the amount of learning that takes placed increases (Wiliam 2017 p.190)

Feedback functions formatively only if the learner uses the information feedback to him or her to improve performance. If educators intend the information fed back to the learner to be helpful but the learner cannot use it to improve his or her performance, it is not formative.

Motivation is not a cause but a consequence of achievement (In Wiliam 20176 p.234) from Garon-Carrier et al., 2016)

Like sports coaching, teaching takes time to master

It takes years for even the most capable of coach to break down a long learning journey from where the student is right now - to where he or she needs to be. (Wiliam 2017 p.193)

Feedback should cause thinking

Feedback for Future Action. (Wiliam 2017 p.194) 

To be effective, feedback needs to direct attention to what's next, rather than focusing on how well or poorly the student did on the work, and this rarely happens in the typical classroom.

The response from the student to feedback should be ‘cognitive rather than emotional’ (Wiliam 2017 p.205) In other words, feedback should cause thinking by creating desirable difficulties.

Peer tutoring can be more effective than one-on-one tutorial instruction from a teacher. This is because of the ‘change in power relationships’. (Wiliam 2017 p.209)

And regarding students online not using their webcams he believed you can see how a student is really taking it by seeing their faces (Wiliam 2017 p.209)

Student Reporter

Put students in a group towards the end of the class so that they can discuss then report back on what has been taught.

Two Techniques that work 

(Wiliam 2017 p230)

  1. Practice Testing 

  2. Distributed Practice

These received high ratings because they were effective with learners of different ages and abilities and were shown to boost students’ performance across many kinds of tasks, and there was plenty of evidence that they worked in educational contexts.

If students complete a practice test and get immediate feedback on their answers, students will get the benefit of the hypercorrection effect for those questions where they were correct. (Wiliam 2017 p.231)

Setting Goals

Students are more motivated to reach goals that are specific, are within reach, and offer some degree of challenge. (Wiliam 2017 p.236 in Bandura, 1986)

When the goals seem out of reach students may give up on increasing competence and instead avoid harm, by focusing on lower-level goals they know they can reach or avoiding failing altogether by disengaging from the task. (Wiliam 2017 p.236)

TIPS

  1. Think ‘how am I going to teach this and what are the pupils going to learn?’ (Wiliam 2017 p.79) 

  1. Having an in-depth understanding of the curriculum may be of more benefit to student progress than advanced study of a subject on the part of the teacher.

  1. Students don't learn what we teach. (Wiliam 2017 p.77) 

  1. Teaching the goal. Driving as teaching. (Wiliam 2017 p.78)

  1. ‘When the pressure is on, most of us behave as if lecturing works but deep down inside we know it’s ineffective’. (Wiliam 2017 p.80)

  1. The teacher’s job is not to transmit knowledge, nor to facilitate learning. It is to engineer effective learning environments for students. (Wiliam 2017 p.80)

  1. Learners of all ages need to understand what it is that they need to learn and be able to monitor their progress toward their goal. (Wiliam 2017 p.95) 

  1. Don’t simply plan the instructional activity, but also plan how you are going to find out where the students are in their learning. You need to be clear about what we want students to learn (Wiliam 2017 p.115)

  1. Ask questions either to cause thinkin and to provide information for the teacher about what to do next. (Wiliam 2017 p.126)

  1. Beware - those avoiding engagement are forgoing the opportunities to increase their ability.

  1. The most important factor influencing learning is what the learner already knows - the teachers job is to ascertain this and to teach accordingly. (Wiliam 2017 p.166) 

  1. Much of the feedback that students get has little or no effect on their learning, and some kinds of feedback are actually counterproductive. (Wiliam 2017 p.167) 

  1. Do not mix grades and comments, just stick to comments. (Wiliam 2017 p.167) 

  1. Oral feedback is best. (Wiliam 2017 p.174)

  1. Too often feedback is counterproductive. (Wiliam 2017 p.178)

  1. Don’t provide students with feedback unless you allow time, in class, to work on using the feedback to improve their work. (Wiliam 2017 p.195) 

  1. Feedback should be more work for them than you! (Wiliam 2017 p.195) 

  1. A simple approach to feedback. Pick out two things to praise, then express what they need to do a constructive wish. He calls the technique ‘two stars and a wish’. (Wiliam 2017 p.214)

  1. Teachers have a crucial role to play in designing the situations in which learning takes place, but only learners create learning. Wiliam 2017 p.246)

REFERENCES

Adey, Fairbrother, William, Johnson, & Jones, (1999) p.36 A review of learning styles and learning strategies. London. King’s College London Centre for Advanced Thinking. 

Ausubel (1968) Educational psychology. A Cognitive view. New York. Holt, Rinehart & Winston

Bjork, R.A. (1994) Memory and metamemory consideration in the training of human beings. In J.Metcalfe & A.P. Shimamura (eds) Metacognition : Knowing about knowing (pp.188-205) Cambridge, MA MIT Press. 

Bloom, B.S. (1969) Some theoretical issues relating to educational evaluation, In H.G. Richey & R.W. Tyler (Eds). Educational evaluation : New roles, new means, part 2 (Vol.68, pp. 26-50_ Chicago : University of Chicago Press.

Brophy, H (1981) Teacher praise: A functional analysis. Review of Educational Research, 51 (i), 5-32

Clarke, S (2001) Unlocking Formative Assessment. London, Hodder & Stoughton

Cowie, B & Bell, B (1999) A Model for formative assessment in science education. Assessment in Education : Principles, Policy and Practice. 6(1), 101-116

Looney, J (ed) (2005) Formative Assessment: improving learning in secondary classroom. Paris. Organisation for Economic Co-Operation and Development.

Shepard et al., 2005 p.275 in L.Darling-Hammon & J.Bransford (Eds) preparing teaching for a changing world : what teachers should learn and be able to do (pp.215-326)

Soderstrom, N.C., & Bjork, R.A. (2015) Learning versus performance : An integrative review. Perspectives on Psychological Science 10 (2), 176-199. 

Stiggins R.J. (2005) Assessment for learning defined. 

Wiggins, G., & McTighe, J. (2000) Understanding by design. New York. Prentice Hall.

William, D. (2016) Leadership for teacher learning . Creating a culture where all teachers improv so that all students succeed. West Palm Beach , FL, Learning Sciences International.

William, D (2017) Embedded Formative Assessment: (Strategies for Classroom Assessment That Drives Student Engagement and Learning) (The New Art and Science of Teaching) : Second Edition 

Permalink Add your comment
Share post
Design Museum

Dylan Wiliam on Improving Teacher Effectiveness

Visible to anyone in the world
Edited by Jonathan Vernon, Wednesday, 16 Dec 2020, 12:57

A Chirstmas Brandy Fruit Cake

This SSAT presentation (the first of two) is as dense as the most succulent brandy Christmas cake. I cannot get through it in one sitting. I am now on the fifth helping and still have a long way to do. I have my notes, I have created a transcript, I have looked up names and added references, I have added some of his slides and made some of my own.

I am getting there.

My summary conclusion is simple - to improve effective of student learning teachers should

  • Prioritise what works in your context and make time by dropping other things.
  • Only do what is known to be effective from the research (though question the research rigourously)
  • Make time to seek ways to improve through working with fellow teachers instead of reinventing the wheel

And what Dylan Wiliam failed to mention, let alone develop:

  • Use big data. Surveilliance and monitoring of student (and teacher) performance is made readibly possible through use of digital platforms. 
  • Use A.I. Openly use smarter smart speakers and personalised A.I. that can monitor and therefore personalise and differentiate a student's learning. 



Permalink Add your comment
Share post
Design Museum

Dylan Wiliam is keeping me up at night!

Visible to anyone in the world

How we learn chart showing a cycle process

I'm on day 3 of a 5 day challenge to pick my way multiple times through the recent talk Dylan Wiliam gave to SSAT on Formative Assessment. I have establised that it is possible to unscamble egss. By the end of this process there will be six of them neatly placed in the door of my fridge.

It is taking time. He name drops academics with every thought of fact and I have wanted to chase down every one of these too. He fires off ideas like a catherine-wheel that once ignited has spun off its perch and is now dancing around on the floor. Oh, to get him in a room on my own for a bit of tutorial rough and tumble; I'd like to keep the sparks flying. Maybe come with some sparklers of my own.

I'll get to the end of this exercise later today. I'd be done if I hadn't decided to get to the end of Kate Clanchy first. This is a new development for me: don't have six books on the go - just a couple. Always get to the end of one of them before picking up the third. 

That said, I've got the Dylan Wiliam library for Christmas. Nothing like hearing the same ideas delivered in multiple ways and at different times, with different metaphors, connected stories and references. 

Permalink Add your comment
Share post

This blog might contain posts that are only visible to logged-in users, or where only logged-in users can comment. If you have an account on the system, please log in for full access.

Total visits to this blog: 11297259