OU blog

Personal Blogs

Design Museum

Learning how to assess - not an AL

Visible to anyone in the world
Edited by Jonathan Vernon, Wednesday, 17 Dec 2014, 19:35
From E-Learning V

 

I'm doing my learning with The OU, but on another platform. I'm emerging from an extraordinary eight weeks on the 'Start Writing Fiction' MOOC on FutureLearn. This draws upon the 'Start Writing Fiction' OpenLearn content and on the various BA and MA Creative Writing courses from The OU.

What's worth reflecting upon as a person who has taken this journey is the nature of the learning process in a highly charged, collaborative and 'massive' environment, the community self-help spirit that is engendered, partly by its being free and open, but also because scale means that that percentage of people who are contributory and giving is large enough to make an impact. And finally, as a learning experience - have I learnt something? Does it 'change behaviours' ? which is the ultimate test of learning - you come out changed.

The course is about creative writing, with the emphasis in a few hours a week on one thing only: characterisation. 

I've taken the view that even if I finished the course a week ago, that it is still term time so I have a duty to stick around. As well as carefully going back through all seven weeks so far I have now done 16 reviews. The reward is always two way.

In this instance I see in every writer ways I'd like to get it right, as well as ways that I too need to stop getting it wrong. An hour spent on a review is typical, though not always possible if either the piece is brilliant and a greater joy each time I read it, or because the piece struggles to reveal that the author has taken part or taken in much at all from the last seven weeks. The reviewer cannot relive the course pointing these things out that have been missed. Perhaps this is the difference between me and a professional tutor or lecture marking an assignment and given constructive feedback. 

Even amongst thousands you get to recognise a hundred or so people who are active in the discussion groups, desperate to put right what they are getting wrong and willing to try anything - I'm in that category and can truthfully say that my writing has been changed for the better, forever. I hope with these reviews that I am able to put back in a fraction of what I have been able to take out. From the learning perspective my behaviour has been changed. I know that writing is 20% of the task, so now I bash on with it as fast as I can ... as the real job, the 80% is the edit. Maybe with experience the balance will shift a bit. More 30% to 70% as I get things right first time. 

There's a discussion about how reviews are shared out. The system has to be automated. I believe the participation at the start of the MOOC was 20,000 and has gone up to 23,000 even 25,000. I know that stats so an exponential decline (is that the right term) sees 50% never even start and another 50% drop out after weeks one or two. Under 10% complete, possibly under 6%.

Reviews of work cannot and are not carried out by ALs. The cost would be astronomic and it would take years. Instead we rely on peer review. Over the eight weeks, beautifully choreographed (learning design) there was been a review of a 250 word piece, then a 500 word piece ... and now the equivalent of our EMA and a 1000 word piece. Without exception people are finally understanding that 1000 words means exactly that. People had the ignorance, arrogance or temerity to post 2,500 word pieces in the firs assignment. Some ignore the course, but wanted people to review their brilliance sad This is what occurs on an open platform. 

Regarding these pieces occasional requests have been made to have all of these on public view so that we could pick and choose the pieces to review. It would be quite wrong though to reveal what can be a sensitive and personal exchange between author and reviewer in a very vulnerable moment. It would, as I've seen in open 'classes' turn into a bit of a bun fight where, in the worst instances, like in the playground, you get people applauding one author and ridiculing another ... or simply join in on the back of what others have said. i.e. the learning experience is thwarted, even abused. It matters that the reviewer knows that their own words matter, without being influenced by what others have written. That said, some reviewers take a cavalier approach saying they don't like a thing, and then saying no more. If that is a student's only review received you can well understand their frustration. Even with the numbers involved somehow these pieces need to be returned and churned through the system, ideally until three to six reviews are received each. More work needs to be done to help students do reviews too so that they feel confident about doing so.

What we all benefit from this process is both learning to review, and learning to receive feedback.

The recommendation I make to everyone is to keep reviewing until you become good at it. If and when you can master reviewing, then you will be in a far better position to fairly review and edit your own work - a lesson that has finally sunk in. Writing is easy, the fun part. Jazz writing I call it. Top of the head stuff. The editing is the pain that crafts a piece so that others can enjoy it too. This pain is reduced the better you get at it.

What's revealing here is, as I've seen in the reviews, is that there are pieces that suggest that the author hasn't learnt anything at all from the course. I cannot make that assumption, so I review on the basis that it is genuine. From a formal assessment point of view, as I've learnt as a student with The OU for four years, is that a tutor looks for repeated indications that the student is using what they should have learnt from the course - if that is not present then alarm bells should ring. How can I give them points?

The problem of course in a MOOC (Massive Open Online Course) is that those who don't do the this task properly are especially taking away from genuine participants who want several considered reviews in return. Only if we get perhaps three or more reviews each will the 1 in 3 'non-review' be cancelled out.

Of 12 pieces I have come to expect 2 excellent and 2 ... how can I put it politely other than to say 'dire'. I'm not to know why this is, I can only judge what I have to review and have in mind the brief and the content of the course over the last seven weeks. Do some people paste in something they wrote months ago that has no bearing at all on the course? It seems that way. Might someone post in a piece that has been published, that is on brief? That is possible too. Of the remaining eight pieces these tend to be where errors and corrections based on the lessons of the course are most easy to make. It takes time. Time and focus. I admire those tutors, here and elsewhere who so clearly have gone to such lengths. Decades after the event I see lengthy comments on pieces for A levels from teachers who were clearly putting in a huge amount of work ... with no word limit on essays too.

In contrast, though not from The OU, I have had reviews of work that were laughable - one may have muddled me up with another student, while the other might have been written in the pub over a pint. One I made a polite complaint, gained 10 points and a distinction. The other I am about to challenge as this 'pub' idea might be close to the truth. And they are paid to assess a piece. In this instance the criticism over my missing a key point is unfounded as I make the required point a) in the introduction b) in the conclusion and c) developed the idea in the main body of the essay. Their comment, 'looks rushed' - which to my sensibilities is an indication of exactly how the tutor behaved - they are the one who were in a rush. 

Students need to be put on a confidence building exercise as they start university so that they feel, as fee payers, able to 'complain' without being stonewalled. This is another theme, but fee-paying students should and will change the attitude of institutions to their fee paying clients, rather than students on a grant-based 'freebie'.

Permalink Add your comment
Share post
Design Museum

Essay writing style: clay or concrete aggregate?

Visible to anyone in the world
Edited by Jonathan Vernon, Wednesday, 1 Oct 2014, 15:10
From River Ouse Low Tide

My tried and tested methodology, beyond the doomed 'winging it' is 'concrete aggregate'. Other weeks or months I accumulate a lot of stuff, much of it in a blog like this; not quite a relational database but the 'stuff' is here, tagged and of reasonable relevance. In a now defunct OU ePortfolio called 'MyStuff' or 'MyOU' - I forget, you could then shuffle and rank your gobbets of nonsense and so, discounting the volume of stuff, potentially, have a treatment that could then be turned into an essay.

Such stuff, if it contains, 10,000 words, often with chunks of verbatim passages, can be a hell of a task to hack into shape. You build in bold forms out of concrete and can only get it to look like a garden, or park sculpture, with a pneumatic drill and chisel. Sometimes it works. You get there. It is dry and workable. You'll more than pass. It depends on the subject, the module and the specific expectations of the assignment. Where you need to tick many boxes this approach may work well.

Clay is the better way forward in most situations. Here you build up your arguments in logical steps then refine them at the end. This, particularly in the social sciences, is where the tutor wants to see how you argue you case, drawing together arguments and facts, mostly those you've been exposed to in the module, though allowing for some reading beyond the module. You have to express your opinion, rather than listing the views of others. Get it right and this is the only way to reach the upper grades? Get it wrong, which is the risk, and you may end up with a hollow or limp structure with grades to match. 

 

Permalink Add your comment
Share post
Design Museum

Not recommended

Visible to anyone in the world
Edited by Jonathan Vernon, Friday, 3 Oct 2014, 09:09
From E-Learning V

Fig.1. Over doing the reading

I'm trying to put to bed what might be my 27th assignment: the last three have been non-OU but the same rules apply: whether tutor marked assignment, end of module assignment or an essay.

Have I ever been cut out for this?

Clearly, getting from TMAs in the 50s and a couple of EMAs in the 40s to TMAs in the 80s (and beyond) and EMAs in the 60s and 70s (though never beyond) indicates that I've learnt how to provide what is required ... and by default, that I have also learnt something (though my brain will complicate, and bury everything that goes in so fast that it's like putting rotten tomatoes onto the compost heap).

This is what I prefer by far: 'writing from the hip' I call it, or 'jazz writing' where a stream of consciousness, or drivel, fills the page.

I am taking a moments break from the nightshift.

This nightshift, awake at 2.00 am and writing by 2.30am has, over several years, become my default position whenever I need a three hour run at something; even the dog is asleep. I have to struggle to hear much more than a buzz in my head and either the tapping on this keyboard or scratching away of pen on paper.

Can I bring to some kind of conclusion this 'learning journal' in relation to writing 'the perfect essay'? 

Despite my best wishes I am NOT a strategic worker or thinker: my curiosity is too much of a pull. I do exactly what I was warned against a year ago - 'vanishing down rabbit holes'. I am the White Rabbit and Alice combined; an intriguing reference enthrals me so off I go. If I can I will source the paper, even get the long out of print book - I may even read the thing, take notes and then pop my head of this hole and wonder what the feck I'm playing at.

An essay needs a copse, not a forest. Imagine what it is like trying to turn an forest into an essay: too much wood (far too much paper). Not simply tough to digest, but any intrinsic pleasure from the act of writing at last is diminished by my knowledge of how much I will have to leave out. 

In the dead of night.

Giving up alcohol and coffee has not helped.

In every respect the alcohol was by far the easiest thing to cut back and cut out - just the conclusion to a ten year minor skirmish that ultimately was or is a medical irritant (allergic to it). Last week I managed 48 hours, or as it 36 hours without coffee. Hardly an achievement given that I was comatose, walking wounded or asleep for the duration. A mug first thing since has found me taking an afternoon siesta and still sleeping for seven + hours. I am sitting with the requisite jug of coffee now. 

I'll get to the end of this and do it justice

'On reflection' the couple of EMAs in the 40s I received was because having done the work, and got reasonable TMAs I blew it with this last struggle and deadlines ending up submitting the latests draft as the seconds disappeared. This time I have had months, really, months and even now I have another three days. I just want to do what I know has to be done: get a good draft finished a few days before, then do the re-read and edit. Nothing less will do and only then can I feel I've done all that I feel I am capable of. The truth is this does not, nor never has come naturally to me. I prefer being up on my feet doing and taking part with a team of people.

Take a nap, then, trusting to my wits and the fresh sea air, I'll be bobbing around offshore by mid-morning taking part in some global sailing charity event. 

Permalink Add your comment
Share post
Design Museum

This will not do!

Visible to anyone in the world
Edited by Jonathan Vernon, Tuesday, 17 Jun 2014, 09:49

Fig. 1 My A' level Geography exercise book

Somehow a handful of exercise books and files from my A'level years have survived three+ decades. On very rare occasions on stumbling across these in a box, in a garage or shed, or attic I've glanced at a few carefully drawn diagrams or maps, smiled proudly at the grades achieved in the final pages, but never stopped to take a careful look at the truth.

The truth when I read it made me faint. Was I that bad? Was the teacher so blunt? How come I didn't go ape-shit and promptly quit the course? 

If I wish to I could seek enlightenment in the diary I kept at the time; I have little doubt that on a teen quest for 'love' there were other priorities in my life. 

Reading this I fear my brain has buried my feelings; it's as if I am reading it for the first time. Every page, 'only twelve', are lightly scored through. In all credit to a brilliant teacher - I believe, ultimately that all but one or two in the class got an A grade 18 or so months later, he offers a bullet-pointed suggestion of an essay plan. The 'half-term' reply to this gets a C-- and a middling response that clearly indicates that only an 'A' will do. All of this experience is new to me in my schooling to date. Threats usually failed, seven years previously having done badly in a French Test I was threatened with a caning if I didn't get x% minimum in a retake ... after half-term. My arse was saved; I suspect my response could have been as violent as the teacher's had he come near me ... 

When it comes to the 'carrot and stick', my experience is that I need both applied and given firmly. 

Four essays and as many weeks later this teacher's response is somewhat different: 

 Fig. 2. Same Geography exercise book, four to five weeks later.

It is odd, though invaluable to be reflecting on this some 35 years after the event. It relates to learning; I'm not teacher-trained. It relates to e-learning too, at all the levels where it is offered. How does or can the technology be used so that a teacher or tutor can provide blunt, constructive feedback in a way that achieves its aims for the individual student? How did this teacher get extraordinary results out of a class, many of whom I knew well and would have rated themselves at the D/E level of likely attainment?

Fundamentally, I feel, is that he, and most others I had during these years were a) vocational educators b) brilliant at their subject c) practitioners.

This teacher was cold, but hugely enthusiastic and knew the subject down to the tiniest detail. He knew his art and gave classroom presentations that came from his soul, not from an Edexcel textbook on his desk. A year later he frankly said that he would teach us what we'd get in our first year at university to maintain our interest ... to keep us simmering as exams approached. And as the 'Physical Geographer' he would often start a lesson with photographs of a climbing trip in Scotland or flying a glider over Northumberland.

 Fig.3. What kind if daisy is your essay?

At the end of this particular exercise book the teachers fills half a side of A4, in red biro, doing his best, now for the umpteenth time, to get me to understand what an essay at this level requires; it is here that he draws takes up drawing a flower, a daisy, suggesting that the perfect essay has five or six petals and a simple step, not the twelve petals, or lopsided, or trunk of an essay that I could produce. 

The purpose of the essay, especially assignments that are not graded, are multiple but never, from I can see, used in the e-learning produce by the Open University. Where is the chance to find your feet, to have a go and fail, to learn through trial and error how to get it write? What role, in any case, would a tutor play to improve an essay style, or to simply help a student get their approach right? And what is the value in assessing a student in a modular form when they cannot expect to be anywhere near to mastering their subject until they've been studying for a year?

How can this vital, human component, make its mark in e-learning? 

Artificial Intelligence must surely offer a smart answer; how else can the many millions who are denied an education at this level have a chance at the experience?

Tellingly, I see that there are comments made in these text books that I struggle with still. I write too much, but in the essay where the purpose is to gather in and focus multiple ideas from several sources in order, at a later date, to refine and prioritise what YOU personally think matters, then more is far, far preferable and necessary then the essay with the limited word count. What happens otherwise is that too early in the learning process the student is expected to reduce down a substance, their research and thoughts, which in the early stages are bound to be on the thin side. From this 'thin; input a thinner essay is meant to be the basis on which the student takes their learning forward. 

There has to be a better way.

Or alternatives.

Permalink Add your comment
Share post
Design Museum

When the penny drops or the fog clears

Visible to anyone in the world
Edited by Jonathan Vernon, Friday, 30 May 2014, 06:57

 To come close to mastering a subject at undergraduate and graduate level takes a couple of years - it should be at the end of this two year period that formal assessments take place. I question the logic of formal assessments before them, other than to trap the unwary into taking modules that may or may not lead to something else, that will probably, for the first couple at least, deliver less than pleasing grades. If I took a couple of modules after the MAODE it would be to wipe-out the first two feeling dissatisfied with low 50s where a couple of mid 70s is where I peaked. 

At some point as you study the 'fog will clear', like learning a language, the 'language' of your subject will come into focus. There might be a 'Eureka Moment', when the 'Penny Drops', some text, something someone says that changes everything and you cross a line forever.

For me this requires considerable reading and discussion around the subject - I have to make my own discoveries, to hear it from half a dozen voices first, ideally from different angles, until one voice, or collectively, they come into harmony.

Permalink Add your comment
Share post
Design Museum

H818 - 69 - again

Visible to anyone in the world
Edited by Jonathan Vernon, Friday, 30 May 2014, 07:03

That's two 69s in the same week - from the OU this is so far from a distinction of 84+ not to warrant consideration, whereas from the University of Birmingham 69 is one mark short of their hurdle of 70 to call it a distinction (my previous effort for them got me to 67).

Was I ever cut out for writing about about education? Is it too 'wishy-washy' for me?

By comparison I find far higher scores achievable in subjects such as History, Geography and even Art. I like the essay/assignment where you pick ONE question from twelve or more and get on with it, whereas repeatedly with the OU I find I am not picking one question, but having to answer 60 questions each with a sentence or word, and then somehow hold this together in a compelling narrative. At least that is how the MAODE has panned out, and H818 in particular. 

Art is a performance, once you can demonstrate the required skill levels the 'grade' is subjective. 

The exception to this was the research into e-learning module where a 93 in an TMA had my tutor saying 'you are made for this kind of thing'. I should have been a barrister, that's what. 

 

 

 

Permalink Add your comment
Share post
Design Museum

The right way and wrong way to assess

Visible to anyone in the world
Edited by Jonathan Vernon, Sunday, 4 May 2014, 09:46

Looking back over four years can be revealing. In 2010 I was struggling with my third Tutor Marked Assignment (TMA03)  in the Master of Arts Open and Distance Education (MAODE). I queried the assignment process. In particular I felt as if it was akin to making a tapestry. Three years on I feel my last couple of TMAs and EMAs weren't simply making a tapestry - but doing so wearing roller-skates on a ship in a heavy sea.

Nothing would stitch together. I had far too much wool. 

The criteria, meant to be helpful, detailing paragraph, by paragraph if not sentence by sentence what the examiner is looking for ties my head in knots. 

I contrast this, favourably, with the MA I am doing with the University of Birmingham. Clunky but effective. I have a reading list. We attend lectures and take part in seminars and I write a 4000 word essay drawn from a list of 12 to 16 titles.

This allows me to be fluid, rather than the ground beneath my feet.

Throughout the MAODE I think the only module the regularly had this 'essay writing' approach was H809: Research practices in educational technology. 

I can be accused of over thinking and over preparing a TMA or EMA - yet this, too often, is how the things have been designed. Less would be more. Simpler would not be easier, writing is hard enough without having to second guess what a third party will be thinking as they read while running down a check list to give you a tick and therefore a mark and ultimately a grade.

Reflecting on four years I can see marks in TMAs, and EMAs especially, improve. I think TMAs in 60s, and 70s and the odd EMA in the 40s, then 50s give rise to TMAs of 80s, even in the 90s, though my best EMA was a 76. Of course, in their wisdom, my student grades for each module simply reads 'PASS'. I feel this rather diminishes the effort and evidence. There is certainly a different between a candidate scoring in the 40s and 50s between one scoring in the 60s and 70s and 80s. 

I met an MBA student who had achieved a distinction in every module. I was in awe. Not your usual OU student (are any from the Business School). She had a first in her first degree from Oxford: Classics. Some people have a mind for these things. Perhaps it is my head that sloshes around like the proverbial storm, rather than the system I have been part of?

Permalink Add your comment
Share post

This blog might contain posts that are only visible to logged-in users, or where only logged-in users can comment. If you have an account on the system, please log in for full access.

Total visits to this blog: 5335602