OU blog

Personal Blogs

Design Museum

Embedded Formative Assessment by Dylan Wiliam

Visible to anyone in the world


Pghe2QUydgC8RiqGNR_40Nuvy1cCflch60xtgto46zvdi6KG-bqJKalQTelVd_9MbAgDQUVzW-TseQHVOxV3OqpMezznrISOkgqoBatyTSepKqWFd9ojF49SZyvgJ3qlGjYQ7U6q

William. D (2017) Embedded Formative Assessment: (Strategies for Classroom Assessment That Drives Student Engagement and Learning) (The New Art and Science of Teaching) : Second Edition. Bloomington, USA. Solution Tree Press. 

Dylan Wiliam is to formative assessment what Chris Witty is to Covid

I’m getting used to Dylan Wiliam - I think of him as the David Walliams of Education, the Will-I-Am of Formative Assessment.

I’ve been pointed at his webinars, sat through a keynote or two, read the book and the read pamphlet ‘Inside the Black Box: raising standards from classroom assessment’ which he produced with Paul Black in 1998 and established his reputation. 

In my first pass of note taking I had over 60 quotes and ideas here, not all from Dylan William. With some editing I’ve got this down to 19 ‘Top Tips’ which are all set out ready to quote. 

I’ll go back to the book if needs be for clarification. In this case it is an eBook which makes it very easy as the digital version will correlate to actual print page numbers.

This is how I learn. 

It’s a slow and repetitive process: skim read a book to get the lay of the land, read it with an open notebook, then transfer the notes to a document like this. Then edit the notes and where necessary go back to the original book. Most important of all - give it a go. Over the last five weeks I’ve had a chance to apply some of the thinking to both online remote teaching and in the classroom face-to-face.  

When a book strikes me as really important I am likely to have it in print and eBook - they read differently and you take different things from each. 

These notes are for me to browse, refer back to and use when I next have a formal rationale or observation to write, as well as for every day reflection on the learning and e-learning experience I am currently going through. 

Myth Busting

Wiliam puts the evidence of learning above myth and has a number of bugbears. 

Good teaching is difficult

It is relatively easy to think up cool stuff for students to do in classrooms, but the problem with such an activity-based approach is that too often it is not clear what the students are going to learn. (Wiliam 2017 p.94) 

Wiliam disregards learning styles, ‘which have no discernible impact on student achievement at all’. (Wiliam 2017 p.11 also Adey, Fairbrother, William, Johnson, & Jones, (1999). Rather - ‘as long as teachers vary their teaching style, then it is likely that students will get some experience of being pushed beyond it’. (Wiliam 2017 p.48) 

Nor is Wiliam a  fan of ‘performance of a learning task’ as a predictor of long-term retention of learning (Wiliam 2017 p.47 from Bjork, R.A. (1994) Any mention of ‘neuroscience’ as the panacea annoys him. (Wiliam 2017 p.50) 

There is no shortcut

Wiliam’s firm belief is that formative assessment improves performance. (Wiliam 2017 p.11) His view - ‘the use of assessment for summative purposes - grading, sorting, and ranking students - gets in the way’ of learning. (Wiliam 2017 p.56) Education is overly prescriptive with rubrics. (Wiliam p.93 in Alfied Kohn (2006)

No one can do the learning for the student who does not engage.

Our classrooms seem to be based on the principle that if teachers try really hard, they can do the learning for the learners. (Wiliam 2017 p.225) 

According to Wiliam there is little evidence that the following ‘tricks’ have any impact of student achievement:

  • Summarisation

  • Highlighting

  • Keyword mnemonic

  • Image use for text learning

  • Rereading (Wiliam 2017 p.225) 

Wiliam sets out five key stages of formative assessment : 

  1. Clarifying, sharing and understanding learning intentions and success criteria.

  2. Eliciting evidence of learning

  3. Providing feedback that moves learning forward

  4. Activating learners as instructional resources for one another

  5. Activating learners as owner of their own learning

The Assessment Reform Group puts it this way. Assessment to improve learning requires five elements to be in place (cited Broadfoot et al., 1999) (Wiliam 2017 p.61)

  1. Providing effective feedback to students

  2. Actively involving students in their own learning

  3. Adjusting teaching to take into account the assessment results

  4. Recognising the profound influence assessment has on student’s motivation and self-esteem, both of which are crucial influences on learning.

  5. Needing students to be able to assess themselves and understand how to improve.

Challenge them!

I like Wiliam’s thinking that ‘good instruction creates desirable difficulties’ - that we learn and retain knowledge through struggling with it. (Wiliam, 2017 p.11) I know from personal experience that the greater the struggle I am willing to endure, the deeper and the longer lasting my learning. It is when I can face the struggle … that I struggle.  Good instruction creates what he describes as ‘desirable difficulties’ (Wiliam, 2017 p.47) quoting Bjork (1994, p.193).

Pedagogy Over Curriculum

‘A bad curriculum well taught is usually a better experience for students than a good curriculum badly taught’. (Wiliam, 2017 p.11) I rather think we can apply this to many situations, for example, that pedagogy comes first - not EdTech. 

Formative Assessment above all else 

The Wiliam mantra is that, ‘attention to minute-by-minute and day-to-day formative assessment is likely to have the biggest impact on student outcomes’. (Wiliam 2017 p.42) He defines formative assessment as ‘the process used by teachers and students to recognise and respond to student learning in order to enhance that learning, during the learning’. (Wiliam 2017 p.59). Formative assessment, according to Bloom (1969) is a kind of evaluation - ‘a brief test used by teachers and students as aids in the learning process. (Wiliam, 2017 p.53 in Bloom (1969) It is ‘the process used by teachers and students to recognise and respond to student learning in order to enhance that learning, during the learning. (Cowie & Bell, 1999 p.32) It is ‘assessment carried out during the instructional process for the purpose of improving teaching or learning’. (Wiliam 2017 p.59 in Shepard et al., 2005 p.275) 

I like this : ‘Frequent, interactive assessment of students’ progress and understanding to identify learning needs and adjust appropriately.’ (Wiliam 2017 p.59 from Looney 2005)

And if you need more ways to think of it, try this: ‘assessment, for learning tells ‘us’ ‘what progress each student is making toward meeting each standard while the learning is happening - when there’s still time to be helpful (Wiliam 2017 p.62 from Looney 2005 (pp1-2) Stiggins (2005)

All kinds of formative assessment are not equal

(Wiliam 2017 p.62) 

‘The evidence is clear that the shorter the assessment - interpretation - action cycle becomes the greater the impact on student achievement’. (Wiliam, 2016). He continues, ‘short-cycle formative assessment has to be the priority for schools and teachers, because the impact on students is greater. (Wiliam p.75) 

‘ … regular use of minute-by-minute and day-to-day classroom formative assessment can substantially improve student achievement’. (Wiliam 2017 p.81)

Design backwards from the learning outcome

(Wiliam 2017 p.81 from Grant Wiggins and Jay McTighe (2000)

Three issues in the development of learning intentions and success criteria that may be useful to think about. (Wiliam p.95)

  1. Task-specific versus generic scoring rubrics.

  2. Product-focused versus process-focused criteria

  3. Official versus student-friendly language

NOTE :> By being specific about what we want, we focus the students learning too much. ‘He who never made a mistake, never made a discover’ (Wiliam 2017 p.95 quoting Samuel Smiles (1862 p.275)

There are a number of techniques which appealed when I read about them and I have used them. These are techniques to help students understand and achieve learning intentions. (Wiliam 2017 p.104 from Clarke (2001)

This is simple, these phrases work 

WALT ‘We are learning to … ‘

WILF ‘What I’m looking for … ‘

TIB ‘This is because … ‘ 

Student Engagement Techniques

The teacher asks a question, selects a student to answer the question and then responds to the student’s answer, which is generally some kind of evaluation of what the student said. (Wiliam 2017 p.124) Teacher-Led classroom discussion.

Try some of these:

  • Wait time

  • Evaluative listening

  • Interpretive listening

  • Questions shells

  • Hot-seat questioning

  • All student response systems

  • ABCD Cards

  • Mini Whiteboards

  • Exit passes

  • Discussion vs. diagnostic questions

  • Alternative questions

Pose a question, pick a student at random

Pose-pause-pounce-bounce

  • Pose

  • 5 seconds - pause

  • Pounce-random choice

  • Bounce-what do you think (Wiliam 2017 p.129) 

App or Lollipop sticks (Wiliam 2017 p.129) 

These techniques are used to ensure that all students realise that all are expected to tke part and that it is ok to make mistakes. If someone does not provide a response then come back to them and tell them  “OK, I’ll come back to you”.  (Wiliam 2017 p.132) 

Then seek Response 2 and 3 and then return to the person who gave no response and ask them which reply they liked the best and ask them why.

‘Engagement and responsiveness - are at the heart of effective formative assessment’. (Wiliam 2017 p.142) 

Managing Challenging Behaviour

To change the behaviour criticise the behaviour not the student. (Wiliam 2017 p.171) 

It is quality rather than the quantity of praise that is most important - teacher praise is far more effective if it is infrequent, credible, contingent, specific, and genuine. (Brophy, 1981 in Wiliam 2017 p.171)

The use of feedback improves performance when it is focussed on what needs to be done to improve, and particularly when it gives specific details about how to improve. (Wiliam 2017 p.180)

Motivation

“It's up to me, and I can do something about it”.  (Wiliam 2017 p.183)

When students have to struggled in the learning task, the quality of their performance on this task reduces, but the amount of learning that takes placed increases (Wiliam 2017 p.190)

Feedback functions formatively only if the learner uses the information feedback to him or her to improve performance. If educators intend the information fed back to the learner to be helpful but the learner cannot use it to improve his or her performance, it is not formative.

Motivation is not a cause but a consequence of achievement (In Wiliam 20176 p.234) from Garon-Carrier et al., 2016)

Like sports coaching, teaching takes time to master

It takes years for even the most capable of coach to break down a long learning journey from where the student is right now - to where he or she needs to be. (Wiliam 2017 p.193)

Feedback should cause thinking

Feedback for Future Action. (Wiliam 2017 p.194) 

To be effective, feedback needs to direct attention to what's next, rather than focusing on how well or poorly the student did on the work, and this rarely happens in the typical classroom.

The response from the student to feedback should be ‘cognitive rather than emotional’ (Wiliam 2017 p.205) In other words, feedback should cause thinking by creating desirable difficulties.

Peer tutoring can be more effective than one-on-one tutorial instruction from a teacher. This is because of the ‘change in power relationships’. (Wiliam 2017 p.209)

And regarding students online not using their webcams he believed you can see how a student is really taking it by seeing their faces (Wiliam 2017 p.209)

Student Reporter

Put students in a group towards the end of the class so that they can discuss then report back on what has been taught.

Two Techniques that work 

(Wiliam 2017 p230)

  1. Practice Testing 

  2. Distributed Practice

These received high ratings because they were effective with learners of different ages and abilities and were shown to boost students’ performance across many kinds of tasks, and there was plenty of evidence that they worked in educational contexts.

If students complete a practice test and get immediate feedback on their answers, students will get the benefit of the hypercorrection effect for those questions where they were correct. (Wiliam 2017 p.231)

Setting Goals

Students are more motivated to reach goals that are specific, are within reach, and offer some degree of challenge. (Wiliam 2017 p.236 in Bandura, 1986)

When the goals seem out of reach students may give up on increasing competence and instead avoid harm, by focusing on lower-level goals they know they can reach or avoiding failing altogether by disengaging from the task. (Wiliam 2017 p.236)

TIPS

  1. Think ‘how am I going to teach this and what are the pupils going to learn?’ (Wiliam 2017 p.79) 

  1. Having an in-depth understanding of the curriculum may be of more benefit to student progress than advanced study of a subject on the part of the teacher.

  1. Students don't learn what we teach. (Wiliam 2017 p.77) 

  1. Teaching the goal. Driving as teaching. (Wiliam 2017 p.78)

  1. ‘When the pressure is on, most of us behave as if lecturing works but deep down inside we know it’s ineffective’. (Wiliam 2017 p.80)

  1. The teacher’s job is not to transmit knowledge, nor to facilitate learning. It is to engineer effective learning environments for students. (Wiliam 2017 p.80)

  1. Learners of all ages need to understand what it is that they need to learn and be able to monitor their progress toward their goal. (Wiliam 2017 p.95) 

  1. Don’t simply plan the instructional activity, but also plan how you are going to find out where the students are in their learning. You need to be clear about what we want students to learn (Wiliam 2017 p.115)

  1. Ask questions either to cause thinkin and to provide information for the teacher about what to do next. (Wiliam 2017 p.126)

  1. Beware - those avoiding engagement are forgoing the opportunities to increase their ability.

  1. The most important factor influencing learning is what the learner already knows - the teachers job is to ascertain this and to teach accordingly. (Wiliam 2017 p.166) 

  1. Much of the feedback that students get has little or no effect on their learning, and some kinds of feedback are actually counterproductive. (Wiliam 2017 p.167) 

  1. Do not mix grades and comments, just stick to comments. (Wiliam 2017 p.167) 

  1. Oral feedback is best. (Wiliam 2017 p.174)

  1. Too often feedback is counterproductive. (Wiliam 2017 p.178)

  1. Don’t provide students with feedback unless you allow time, in class, to work on using the feedback to improve their work. (Wiliam 2017 p.195) 

  1. Feedback should be more work for them than you! (Wiliam 2017 p.195) 

  1. A simple approach to feedback. Pick out two things to praise, then express what they need to do a constructive wish. He calls the technique ‘two stars and a wish’. (Wiliam 2017 p.214)

  1. Teachers have a crucial role to play in designing the situations in which learning takes place, but only learners create learning. Wiliam 2017 p.246)

REFERENCES

Adey, Fairbrother, William, Johnson, & Jones, (1999) p.36 A review of learning styles and learning strategies. London. King’s College London Centre for Advanced Thinking. 

Ausubel (1968) Educational psychology. A Cognitive view. New York. Holt, Rinehart & Winston

Bjork, R.A. (1994) Memory and metamemory consideration in the training of human beings. In J.Metcalfe & A.P. Shimamura (eds) Metacognition : Knowing about knowing (pp.188-205) Cambridge, MA MIT Press. 

Bloom, B.S. (1969) Some theoretical issues relating to educational evaluation, In H.G. Richey & R.W. Tyler (Eds). Educational evaluation : New roles, new means, part 2 (Vol.68, pp. 26-50_ Chicago : University of Chicago Press.

Brophy, H (1981) Teacher praise: A functional analysis. Review of Educational Research, 51 (i), 5-32

Clarke, S (2001) Unlocking Formative Assessment. London, Hodder & Stoughton

Cowie, B & Bell, B (1999) A Model for formative assessment in science education. Assessment in Education : Principles, Policy and Practice. 6(1), 101-116

Looney, J (ed) (2005) Formative Assessment: improving learning in secondary classroom. Paris. Organisation for Economic Co-Operation and Development.

Shepard et al., 2005 p.275 in L.Darling-Hammon & J.Bransford (Eds) preparing teaching for a changing world : what teachers should learn and be able to do (pp.215-326)

Soderstrom, N.C., & Bjork, R.A. (2015) Learning versus performance : An integrative review. Perspectives on Psychological Science 10 (2), 176-199. 

Stiggins R.J. (2005) Assessment for learning defined. 

Wiggins, G., & McTighe, J. (2000) Understanding by design. New York. Prentice Hall.

William, D. (2016) Leadership for teacher learning . Creating a culture where all teachers improv so that all students succeed. West Palm Beach , FL, Learning Sciences International.

William, D (2017) Embedded Formative Assessment: (Strategies for Classroom Assessment That Drives Student Engagement and Learning) (The New Art and Science of Teaching) : Second Edition 

Permalink Add your comment
Share post
Design Museum

Daphne Koller: The University has flipped and The OU should have been there first

Visible to anyone in the world
Edited by Jonathan Vernon, Wednesday, 22 Oct 2014, 14:27

Fig. 1. Daphne Koller TED lecture on YouTube

Daphne Koller is a professor of Computer Science at Stanford University and a Third generation PhD. In this insightful talk we learn how e-learning is changing learning opportunities globally. Scale is at the heart of it.

A Machine Learning Class at Stanford with an undergraduate enrolment of some 400 when put online is followed by 100,000. And the lessons from scale led to the creation of Coursera where anyone can take the world’s top classes for free - delivered by the best instructors from the best universities.

  • Personalised curriculum
  • A coherent concept in 8 – 10 minutes
  • Students can traverse the content in different ways background, skills or interest.
  • Support or enrichment.

Practising with the material is important.

Video is interrupted to pose questions. Students are expected to engage.

  • Multiple choice
  • Short answer questions
  • Grade math and models

To be told when you are right or wrong is essential to student learning.

How do you grade 100,000 students?

Peer grading is a surprisingly successful strategy (Sadler & Good, 2006) .

  • Teacher and student grades extraordinarily similar, even self-grades.
  • And the student learns from the experience.

And learning is socialised

  • Around each of our courses a community of students has formed.
  • Some meet online, others locally.
  • Students respond to each other’s queries.

‘The median question to response time was 22 minutes because somewhere around the globe there was someone awake’. (Koller, 2012)

From 0:14:11

‘There are some tremendous opportunities to be had from this kind of framework’.

‘First it has the potential of giving us a completely unprecedented look into understanding human learning because the data that we can collect here is unique. You can collect every click, every homework submission and every form post from tens of thousands of students so you can turn the study if human learning from the hypothesis driven mode to the data driven on transformation that for example has revolutionized biology.

Fig. 2. Correcting misconceptions and poor learning paths

0:14:40

You can use the data to understand fundamental questions like what good learning strategies are versus ones that are not and in the context of particular courses you can ask questions like what are some of the misconceptions that are more common and how can we help fix that. 2000 students give the same wrong answer ... produce a targeted error message to give personalized feedback.

Fig. 3. Benjamin Bloom (1984) , 2 Sigma problem.

Lecture, mastery based approach, taught one on one with a tutor. individual gives you 2 sigma improvement 50/50 Individual 98% above average But cannot afford to provide every student with an individual tutor. Mastery will grade multiple times and show you the same video over and over without getting bored.

How can we push towards the 2 Sigma curve.

‘The mind is not a vessel that needs filling, but wood that needs igniting. From Ian Kidd's translation of Essays’. Plutarch

0:18:50 More time required igniting their creativity, their imagination and their problem solving skills by talking with them. We do that by active learning in the classroom.

Performance improves by every metric:

  • attendance
  • engagement
  • standardized tests

It would do three things:

  • Establish education as an absolute fundamental human right.
  • Enable lifelong learning
  • A wave of innovation

FURTHER READING

Guskey, TR 2007, 'Closing Achievement Gaps: Revisiting Benjamin S. Bloom's "Learning for Mastery"', Journal Of Advanced Academics, 19, 1, pp. 8-31, Academic Search Complete, EBSCOhost, viewed 17 February 2013.

REFERENCES

Bloom, BS 1984, 'The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring',Educational Researcher, 6, p. 4, JSTOR Arts & Sciences IV, EBSCOhost, viewed 17 February 2013.

Koller, D (2012) Ted Lecture Daphne Koller: What we're learning from online education (accessed 17 Feb 2013 http://www.youtube.com/watch?feature=player_embedded&v=U6FvJ6jMGHU )

Sadler, P, & Good, E 2006, 'The Impact of Self- and Peer-Grading on Student Learning', Educational Assessment, 11, 1, pp. 1-31, ERIC, EBSCOhost, viewed 17 February 2013.

Permalink 3 comments (latest comment by Sharif Al-Rousi, Thursday, 21 Feb 2013, 00:05)
Share post
Design Museum

Ditch the E and the M - it's learning

Visible to anyone in the world
Edited by Jonathan Vernon, Friday, 28 Sep 2012, 14:02

It's just learning ... the greatest thinkers on educational are largelly pre-digital: Dewey, Vygotsky, Piaget, Kolb, Bloom, Briggs, Engeström, Gagnéet al.

E and M learning are advances on the Guttenberg Press or Power Point, but they are technologies all the same.

Permalink Add your comment
Share post
Design Museum

Web-Based Training (Part One)

Visible to anyone in the world
Edited by Jonathan Vernon, Friday, 2 Nov 2012, 07:19

DSC04712.JPG

Web-Based Training (WBT) (2000)

Margaret Driscoll

I bought this book in 2001. Nearly a decade on I am delighted how apt it remains, even if the term may now have been superseded by e-learning - while cyberlearning had currency for a few years too and before this we had 'interactive learning'.

Even a decade on I recommend the book.

Training and learning are in different camps, one supposing a component of applied engagement (health and safety, fixing photocopiers, burying uranium trioxide, driving a delivery van, making cars, selling phones, employee induction) while the other is essentially cognitive (though with its physicality in this kind of prestidigitation).

Yet we made ‘training programmes’ on things like cognitive behavioural therapy. Corporate and government clients had the money to do these things.

Driscoll’s definition of WBT is somewhat longer than Weller’s definition (2007) of e-learning (electronically enhanced learning with a large component of engagement on the Internet). A bit ‘wishy-washy’ my exacting Geography A’ Level teacher would have said.

Hardly a clear definition if it has a let out clause. But when was anything clear about what e-learning is or is not, or should be? The term remains a pig in a poke; its most redeeming factor being that it is a word, not a sentence, and fits that cluster of words that includes e-mail.

Web-based training Driscoll (2000) says should be:

  • Interactive
  • Non-linear
  • Easy to use graphic interface
  • Structured lessons
  • Effective use of multimedia
  • Attention to educational details
  • Attention to technical details
  • Learner control

I like that. I can apply it in 2011. I did.

I was reading ‘Web-Based Training’ (and using the accompanying CD-rom) in 2001 and then active in the development of learning, or knowledge distribution and communication websites for the NHS, FT Knowledge ... and best of all, Ragdoll, the home of Pob, Rosie and Jim, Teletubbies and the addictive pleasures of ‘The Night Garden.’

We may call ourselves students, mature students even, or simply post-graduates, but would we call ourselves ‘Adult Learners.’

It’s never a way I would have defined my clients, or rather their audiences/colleagues, when developing learning materials for them in the 1980s and 1990s. Too often they were defined as ‘stakeholders,’ just as well I saw them as people and wrote scripts per-the-script, as if for only one person.

That worked, producing for an umbrella term does not.

Adult Learning doesn’t conjure up innovative e-learning, perhaps because of the connotations Adult Learning has inelation to the catalogue of F.E. courses then comes through the door every July or August.

This definition of an ‘adult learner’ would apply to everyone doing an OU course surely?

The special characteristics of adult learners Driscoll (2000:14)

  • Have real-life experience
  • Prefer problem-centred learning
  • Are continuous learners
  • Have varied learning styles
  • Have responsibilities beyond the training situation
  • Expect learning to be meaningful
  • Prefer to manage their own learning

With some of these definitions baring more weight than others, don’t you think?

REFERENCE

And additional references used by Driscoll but not cited above:

References (Adult Learning)

Knowles (1994) Andragogy in action. Applying modern principles of adult learning.

Brookfield (1991) Understanding and facilitating adult learning

Cross (1992) Adults as learns: increasing participation and facilitating learning.

Freire (1970) A cultural action for freedom

Merriam and Caffarella (1991) Learning in adulthood

Kidd (1973) How adults learn

 

Permalink
Share post

This blog might contain posts that are only visible to logged-in users, or where only logged-in users can comment. If you have an account on the system, please log in for full access.

Total visits to this blog: 5975191