Activity 12.1: Read and critique the paper (3 hours)
Imagine you have been asked to peer review Davies and Graff's paper as an academic referee. The request has come from an international journal of elearning and the editor wants your opinion of the paper.
a) Introduction
Harvard convention is cf not e.g.
4 authors should be et al. not listed in full
English is very informal and sounds contrived in places e.g. 'there is therefore' and 'what needs to be investigated' p.658
'One reason for the importance....is...' p.658 Is this really this definite?
Research question - why does this need to be investigated?
b) Research design
Is this university/college/distance learning?
All on same degree which is consistent but were they studying same six modules?
Blackboard is a trade name and should be acknowledged, referred to with a capital letter and referenced.
Blackboard statistics-what are these? It is not clear what is recorded - number of times logged on or time spent online. Students could log on and straight off again or leave the program running in the background. Later on (p.659) the authors report that it is number of times logged on that is recorded.
Is this a blended course? Could these students be meeting f2f as well? Is this accounted for? Some students may be arranging their own support sessions in the library or coffee shop - is this checked, accounted for?
c) Analysis
Does access to group or communication area = participation? Were they just reading the boards?
Kruskall-Wallis needs the following data types:
- Data points independent from each other - yes
- Distributions do not have to be normal and variances do not have to be equal - yes
- Ideally more than five data points per sample - just 4
- Individuals selected at random from the population - whole population used
- All individuals must have equal chance of being selected - whole population
- Sample sizes should be as equal as possible - 10 to 31 so large differences
Not sure about this. Would it have been better to analyse without grouping? This would have given 98, 101, 85, 99, 70 and 80 scores per sample which is certainly greater than five data points and more equal than the group size.
The correct format is to report the direction of the data e.g. the right-tail probability (0.0052) is lower than 0.05 the H0 must be rejected.
The null hypothesis is not stated - a positive hypothesis is proposed stating that more time spent in communication achieves better grades. (one way analysis) but authors report it is number of times logged on that is recorded. Assuming the null hypothesis to be: there is no difference between more time spent in online communication and better grades.
In all cases (for the total analysis) the null hypothesis should be rejected indicating that there may be a relationship between more time spent in online communication and better grades.
The authors suggest (p.659) that greater activity as measured by Blackboard usage is likely to lead to better performance in terms of module grade. This cannot be presumed from the initial data analysis without further analysis with a non-parametric multiple comparison test.
P.660 - Using the proportion of time spent in interaction compared to task areas as a percentage of total usage. Time was not actually used -it was 'hits'. However, this figure is probably not reliable as greater weight will be given to lower total participation rates. For example, if a poor student only logged on three times and twice accessed the interactive areas, then they would record 67%; another student accessing the site 100 times and equally accessing the task and interaction sites would only record 50%. Hence a reader could not rely on the statistics generated from these figures.
d) Discussion
I am not convinced that 'the students achieving high or medium passing grades engaged more actively with the course' (p.661) as this is not confirmed by the data analysed which measures the number of times logged onto Blackboard.
The 'proportion' analysis is flawed and cannot be used to support conclusions and this 'observed difference' was not statistically significant anyway.
Page 662 Connolly et al. is missing full stop after Latin phrase on both occasions.
e) Conclusion
No evidence presented for second paragraph.
Activity 12.3: Further research (20 minutes)
The authors talk about testing whether 'the frequency of interactions are more important in providing support, whereas the quality and dynamics are the more important factors in learning and performance'. Note down the kinds of problem you think such testing might involve.
- Monitoring frequency of interactions - especially difficult on a blended course but also online as I personally interact with other course members by Skype, email, Twitter and Facebook as well as the forums and blogs.
- What constitutes 'support'? How is it defined? Academic or emotional? The distinction between the two can be unclear.
- Performance can be affected by many factors other than interaction. It could be difficult to measure the contribution of interaction amongst other factors such as IQ, past experiences, emotional and physical difficulties, current lifestyle stresses etc.