OU blog

Personal Blogs

E801: Action 3.14: Implications of using tests

Visible to anyone in the world

Murphy, S. 'Literacy assessment and the politics of identities' [Course website]

Notes on implications and equity issues in individual assessment.

General problems

  • Low scoring students results based on smaller sample of responses than higher scoring students
  • No credit for partial knowledge
  • Summaries masks differences between test scores i.e. student scores uniformly low or one test is very low and rest above average
  • Knowledge of format of tests affects performance
  • Limited generalisability from test to context e.g. comprehension test can be answered without reading the passage
  • Limited relevance to non-school environment
  • One narrow path to success - perceived as unfair by those with strengths in other areas.

Systematic bias

Testing takes place in context. It has behavioural rules on inter-personal communication and participation

  • Interaction between examiner/examinee
  • Anxiety
  • Over-testing and speed of testing for African-American children
  • Low achievers more anxious
  • Negative about process that sorts them for race and class
  • Freq. of specific word use differs across languages so students may not recognise words even if translated to their own language
  • In translation the resultant tasks may not be equivalent

Teachers administering tests

  • Successful result of the system
  • Forced to participate in system even if do not agree
  • Psychologists do not know child; teacher/parent does so ideas of teacher/parent often confirmed even though results are ambiguous
  • Teachers focus on those pupils likely to show gains in test results
  • Teachers reportedly alter results, teach to the test, use test items in class. Resistance but pressure to get pupils to achieve.

Psychologists administering tests

  • Seem to be autonomous in selection and administration of tests
  • Milofsky found psychologists in suburbs worked in environment and identified barriers; those in urban environment were too busy and identified individual differences
  • Objectivity of test legitimises the work;
  • Working in marginal position so power important to personal identity
  • Legal requirements require audit trail

 

Permalink Add your comment
Share post

E801: Action 3.13: Assessment

Visible to anyone in the world

Reid (2009) Chapter 5

1. What specific tests are mentioned and what do they assess?

Cognitive Measures: WISC -IV (Wechsler intelligence scale) assumes IQ tests are valid/reliable; IQ and reading share causal dependency; no information on how to intervene

Processing Skills: PAL-11 (Process Assessment of the Learner) explains why; suggests how to intervene;

WIAT-11 (Wechsler Individual Achievement Test) correlates with WISC; extent of difficulties but no guidance on which areas are involved

CTOPP (Comprehensive test of phonological processing); more specific; precise diagnostic information and evaluation of progress.

Woodcock Reading Mastery Tests- formal; structured

GORT-4 (Gray oral reading tests) - both top down and bottom up processes; uses Miscue analysis but diff. marking than Goodman

LPAD - Freuerstein's learning potential Assessment Device - dynamic test/assisted assessment - intervention AND assessment

What is meant by 'standardised and psychometric criteria'?

Standardised utilises a basis for comparison; a reference point against which other things can be evaluated (norm-referenced). E.g. IQ tests, reading age. Need care to avoid bias in test construction.

Psychometrics is the field of study concerned with the theory and technique of educational and psychological measurement, which includes the measurement of knowledge, abilities, attitudes, and personality traits. The field is primarily concerned with the construction and validation of measurement instruments, such as questionnaires, tests, and personality assessments. (Wikipedia). Tries to establish a norm.

WISC - psychometric, standardised test.

What do you understand by the term 'screening'?

Screening is a strategy used in a population to detect a disease/difficulty in an individual without signs of the disease/difficulty.

At what age, which skills, how should results be used?
All children?

 

What are the key aspects of curriculum assessment?

In context; performance in natural environment; using meaningful activities

  • Cloze procedure
  • Silent reading
  • Reading aloud
  • Free writing
  • spelling

What is miscue analysis?

Miscue analysis is a tool for looking closely at the types of reading strategies a reader uses. The kinds of miscues (incorrect guesses) a reader makes when reading from a text will give the listener clues about how familiar or unfamiliar the reader finds the content matter, and how easy or difficult they find the text to read. Reading tests do not give this sort of information because reading is so much more than just looking closely at each letter and every word. Based on Goodman (1976)

What is meant by the 'components approach' to assessment?

  • Distinguish dyslexia from slow learner
  • Distinguish dyslexia from comprehension deficits
  • Adapted for teacher and psychologist
  • Complete diagnosis: qualitative and quantitative

The advantages of observational assessment?

  • Gives info which can lead to personalised development programme
  • Less stress for learner
  • Flexible
  • Adaptive
  • Contextual

Give some examples of 'assessing in context'

Specific difficulties relating to subject e.g. relative importance of information and ordering information in history.

History example was assessment designed to bolster self-esteem

2. Look at the assessment materials in Reid (2009) Appendix 1 and the assessment materials on the course website.

Which tests are suitable for large numbers of students and which are individualised?

Individual

Group

TOPA-2+ (5-8yrs)

TOPA-2+ (5-8yrs)

CTOPP (5-25)

 

Launch into reading success (young)

 

GORT-4 (6-19)

 

TOWRE (6-25)

 

DIBELS

 

Bangor

 

DST (3 levels - up to adult)

 

CoPS

CoPS

SNAP (facilitates communication)

 

WIAT-11

 

WRAT 4

 

 

In which ways would you differentiate between these assessments and decide which to use?

Although it seems one of the least important factors, I would have to mention that the availability of the tests would have to play a part as I would like to use a selection of tests and they are expensive so it would have to depend on which tests the university would be willing to pay for or already possessed.

Age is an important factor - I work with a population of adults over a wide range from 18years in a University context.

I would like to use a screening approach initially with in depth discussion on family history, past and present educational experiences and other life experiences.

I do not feel that an IQ test is appropriate with these adult learners who have achieved university entry. However it may be necessary to include if required by the Student Loans Company to justify their payment of Disabled Students Allowance.

I think that I would like to use the Process Assessment of the Learner Diagnostic Assessment for Reading and Writing (PAL-11) and then progress to dynamic assessment models but this would all depend on the demands of the Student Loans Company. I intend to do some more research in this area in order to find out what exactly is required.

 

Permalink Add your comment
Share post

E801: Action 2.6: Inclusion and Globalisation

Visible to anyone in the world

E801: Action 2.6: Inclusion and Globalisation

Johnson, D. & Kress, G. (2009) 'Globalisation, literacy and society: redesigning pedagogy and assessment' in in Soler, J., Fletcher-Campbell, F. & Reid, G. Understanding Difficulties in Literacy Development: Issues and Concepts, London, Sage.

The inclusion of digital literacies as part of the portfolio of literacies required by modern learners has been studied in many contexts - that of the different forms of digital literacy, the skills required for accessing information in a digital format, the skills required for extraction and evaluation of information etc.

A lot of institutions initially believed that a techno-based curriculum would cure all problems and level the playing field for people with disabilities. A lot of money was spent on various forms of technology and much of it was wasted.

Currently, in online and distance learning, the readjustment is towards  learner-centred activities with universal design for online learning materials BUT with an emphasis on user-controlled flexibility. Collaborative learning is on the increase with the use of peer networks via Twitter, forums and blogging.

Policies and institutions may take a while to catch up but individual teachers are leading the way with some fabulous activities for their students.

Assessment is also improving dramatically with online portfolios and peer feedback.

The following is from a previous blog post:

Universal Design for Learning (UDL) Guidelines - Version 1.0 (CAST, 2007)

UDL has three primary principles that provide the structure for these Guidelines:

Principle I: Provide Multiple Means of Representation (the "what" of learning). Students differ in the ways they perceive and comprehend the information presented to them.

I am currently working with a student who has a severe visual impairment. She lost most of her sight at the age of 16 years by which time she had already discovered that her preferred learning style was visual. She still has enough sight to revise by drawing out large diagrams but it is not easy for her.

Principle II: Provide Multiple Means of Expression (the "how" of learning). Students differ in the ways they are able to navigate a learning environment and express what they know.

I have experienced the following adjustments in the universities where I work: allowing speech impaired people to plan and design PowerPoint presentations using the inbuilt speech features; allowing a student with ME to verbally present the information rather than spend all evening writing a report on a field course;

Principle III: Provide Multiple Means of Engagement (the "why" of learning). Students differ markedly in the ways they can be engaged or motivated to learn.

One third year module at Keele University is Inspirational Landscapes in Geography. Assessment is 20% test and 80% project. Previous student projects have included:

    • Impact of the Malvern Hills on Elgar's music
    • Video diary of a walk in Wordsworth's footsteps
    • Photomontage of the experience of Dovedale
    • Influences of Indian landscape on fashion design
    • Johnny Depp: face, costume and landscape
    • Landscapes of Lord of The Rings
    • Thomas Hardy and the "Wessex" landscape
    • Landscape design for computer games

The module sounds fascinating and I know several students who really enjoyed it. http://www.esci.keele.ac.uk/people/pgk/geg-30014/handbook.html

 

 

 

Permalink Add your comment
Share post

H810: Week 13 : Activity 25.3

Visible to anyone in the world

 

Week 13: Activity 25.3: Online Assessment

These are similar to some of the guidelines that you looked at in Block 2, but they also include much more discussion of teaching and learning.

  • How far do the points made in these guidelines match the points discussed in the previous activity?
    They mention 'high-stakes' testing where accessibility provisions may have an effect on validity - equates to 'professional body' point in previous exercise.
  • Which staff role do you think these guidelines would be most useful for in your context?
    Really not keen on these guidelines as they are long winded, full of waffle and do not keep to the point. Much preferred the last exercise. Sections 9.3 and 9.4 do not seem to be complete.
  • Which guidelines that you looked at in Block 2 would be helpful for a web developer in addition to these?
    The JISC e-assessment training program:
    http://www.techdis.ac.uk/resources/sites/staffpacks/Staff%20Packs/E-Assessment/Presentation%20-%20EAssessment.xml

Really do not agree that 'low stakes' assessment are 'not a serious issue'. Formative assessments set up expectations of a student's performance in a lecturer's mind and effect later marking. They also have a big effect on a student's learning and motivation.

 

 

Permalink Add your comment
Share post

H810: Week 13 : Activity 25.2

Visible to anyone in the world

 

Week 13: Activity 25.2

Make notes on the three important points which are noted in the Introduction:

  • whether particular assessments or examinations are core to the course

I have been working with a student with no usable sight studying for a computing degree. One of the modules covers coding for graphics. The university are happy to make reasonable adjustments but these generally involve working with a support worker which reduces his independence. He is very frustrated and suggested that the department let him study an alternative module but they will not do this as it is a core module and they have no time to write a module from scratch for him.

  • what adjustments are permissible within particular assessments or examinations without compromise to academic, or other prescribed, standards, such as competences required by professional bodies

I work with a blind student studying law and they have several exams coming up in May where they answer three questions in three hours. Last year the department organised her 100% extra time and rest breaks and she took the exams but was completely exhausted at the end of an 8 hour stretch and she had to postpone one exam and sit it in August as a first attempt. This week the department approached her and suggested that she answered one question in the exam and the other two as an assignment. In this way she will not be too exhausted to do her best work.

Another university has found problems on physiotherapy and medical courses as the professional body will not allow notetakers for dyslexic students. This one is a current dispute between the university and the NHS!

  • whether the successful achievement of the highest grades and awards, based on performance in examinations and other assessments, is equally attainable by disabled students.

Another university has a student who uses notetakers on field trips for geology. After discussion, it has been decided, that the notetaker should take notes in the assessed notebook but to label each section with the name of the person speaking i.e. the student who is dictating notes or one of the lecturers. In this way the student is only marked on their work rather than that of the notetaker's ability to record the lecturer's words. This would seem to be a reasonable approach but actually it works to the student's detriment as the other students mingle their thoughts with that of the lecturer's comments and get marked on their good ideas which may actually have come from the lecturer.

Would you have emphasised the same three points?

I think that these points are important but I also think that the perception of the other students needs to be considered. In the case of the law student, her friends and colleagues think she is being assessed just as stringently as they are being assessed. In the case of the notetaker on field trips, one student I worked with went to great lengths to explain to the other course members the way her assessment worked and the fact that her DSA paid her notetaker on several occassions. She informed me that there had been very rude comments from other students about the fact that she was marked on a professional notetaker's work and that the others had to pay for it too!

Are there any positive or negative aspects of online assessment for disabled students?

Online assessment can work well but too many institutions consider online assessment to mean multiple choice tests with a time limit so that students do not have an opportunity to research the questions. This can disadvantage many disabled students. For example, in the last few weeks one student, who has Asperger syndrome and is severely dyslexic, has had mid-term tests in his first year at university. The two modules he has found most difficult had straight tests and he achieved first class honours in both. The two modules he has no problems with, used multiple choice tests and he has failed both. Researchers put the problem down to problems with working memory and poor eye coordination.

 

Creating Accessible Examinations and Assessments for Disabled Students

Evaluating Practice

  • Staff are consciously aware of, and in agreement about, what aspects of student attainment or performance they are trying to assess.
    This seems to vary between the three universities where I am employed and also between departments. I know that the OU have course team meetings to try to ensure that this happens but I have personally been penalised for following guidelines issued by one tutor which my tutor did not agree with.
  • Students are aware of the aspects of attainment or performance which are the subject of assessment.
    Not always
  • The nature of marking criteria are kept under regular review: such matters as the importance of spelling, grammar, the ability to calculate, and the ability to remember dates and constants are collectively evaluated by the staff including part-time staff and teaching assistants.
    There is a lot of disagreement on this one. Some lecturers place great emphasis on grammar and spelling in examination conditions and penalise heavily. Lack of knowledge, and in some cases, disbelief concerning dyslexia leads to a response that is less than helpful. This summer I was assisting a student on a field course that involves spot tests in the field. I know all the students well and several were upset as they could not immediately recall the information. These were students with dyslexia and I spoke to the lecturer who was very concerned that he had disadvantaged those students. He had no idea that short term memory problems were an issue.
  • Policies concerning electronic aids to spelling, grammar and calculation in examinations are kept under regular review.
    A real annoyance of mine! One university insists that they will not allow electronic spelling aids in examinations despite the student's access report stating that they need one. They supply dictionaries for students with dyslexia, which the majority cannot use!
  • Where a student is unable by reason of an impairment to show evidence of relevant attainment or performance in the standard way, alternative arrangements are put in place if it is possible to do so.
    Yes, at all three universities
  • The flexibility referred to above is available in terms of deadlines and timetabling of assessments.

Yes, but this varies between universities. Two universities require an extenuating circumstances form to be completed; the third states that extenuating circumstances are for unexpected occurrences and extensions required due to disability are just to be granted with consultation between student, tutor and lecturer.

  • Alternative assessment arrangements as referred to above are well controlled to ensure consistency and fairness, vis-à- vis both the students taking them and other students.
    No, at one university everyone who is assessed with dyslexia gets 25% extra time; everyone who needs rest breaks is allowed 15 minutes despite disability or length of exam.
  • Assessment feedback to students is accessible to all our students, both in terms of content and format.
    No, some departments put handwritten reports on a printed version of an assessment. This results in a student with reading difficulties or visual impairment having to ask a support worker or friend to read this private material to them.
  • Those responsible for our examinations and assessment appeals are well versed in the ways in which procedures may need to be adjusted in acknowledgement of the needs of some disabled students.
    Definitely not!

 

 

 

Permalink Add your comment
Share post

This blog might contain posts that are only visible to logged-in users, or where only logged-in users can comment. If you have an account on the system, please log in for full access.

Total visits to this blog: 461694