I read Jonathan Rees’ post on the flaws of peer grading in MOOCs with interests.
Jonathan says:
Because of the size of the course I think I can safely assume that many of my fellow MOOC students inevitably had no history background at all, yet the peer grading structure forced them to evaluate whether other students were actually doing history right.
The implicit assumption of any peer grading arrangement is that students with minimal direction can do what humanities professors get paid to do and I think that’s the fatal flaw of these arrangements. This assumption not only undermines the authority of professors everywhere; it suggests that the only important part of college instruction is the content that professors transmit to their students.
Read more: http://www.insidehighered.com/views/2013/03/05/essays-flaws-peer-grading-moocs#ixzz2MiKxNP7b
Inside Higher Ed
What are the assumptions behind the background of MOOC students? Do we know enough if one’s fellow MOOC students had any (history) background at all?
What are the assumptions behind peer grading? I could see the values, merits and limitations of peer grading in certain fields, such as evaluations of group projects, individual assignments, but in the case of MOOCs, would there be huge variations in the grading, when subject to the assessment of different peers or professors? The use of 0, 1, 2, 3 etc. as a grading scale is appropriate when those performance criteria is clearly understood, with a concise marking guide. However, given the “unknown” abilities of the peers in assessment of the work, such as this “professor”, how would one be able to judge professionally, except from the report from this student professor?
Professional judgment in assessment requires one to comprehend the significance and application of validity, reliability, authenticity and sufficiency in evaluation and assessment of a piece of work (essay, report, project or artifact). That’s why we need quality assessment (control over variation in “standards”, which could be measured based on concrete and reliable performance standards). Could super professors do this for hundreds, or tens of thousands of students in MOOCs? That is mission impossible. May be one could develop a course in “training” the students in how to assess in a professional manner first (based on what professors would normally do). Even then, one would realize that there are always variation in the assessment tools, methodology when students are requested to do the assessment, as an “experiment” in MOOCs.
There are many assumptions made here, and I just can’t help but to quote the
Assumptions Theory that I suggested. The xMOOCs are based on assumptions that people could learn from the best professors in the world, with these peer assessment and grading rendered possible due to the advances in technology. We have also assumed that participants (students in particular) have got the skills to peer assess, and provide valued comments to other students. I just like to continue with stating the assumptions, but would think it better for you to share your assumptions on this interesting topic.
Learner Weblog 
Pingback: What are the assumptions behind MOOC, in particular peer assessment and grading? | CUED | Scoop.it
A good account John! Assessment really is a big problem for humanities MOOCs. Maybe some sort of hierarchical training scheme – something like: super professor -> professor -> TA -> ‘super student’ -> participant. ‘Super student’, for example, might be a participant of proven abilities from a previous MOOC. This has an exponential quality – if each one trains 10 trainees 10,000 participants are exposed to a super student on a 10:1 trainee:trainer basis! Obviously there are organisational and funding problems – the hybrid MOOC of the future with layers of ‘premium’ access to ‘informed’ assessors – 10 dollars per super student, 1,000 dollars per professor? 🙂 Gordon
Gordon, that is a wonderful model that I like very much. I suppose “we” should get some funding for promoting and experimenting on that! How about individual mentoring and peer-mentoring?
I think a varied assessment strategy may work. I agree with all that has been said, but I’m wondering about self-assessment – how does the learner define if they’ve been successful? This may be more for a cMOOC than an xMOOC environment, but I’m thinking a combination of peer and self, facilitated by an intro workshop or collaborative live session on “how to assess” (i.e. your “train the trainer” model)…a suggested pre-requisite to any xMOOC? Just a thought…Thank you for this post!
Wonderful. That’s sounds a good approach. One method that hasn’t been used much in the xMOOC is the Learning Contract or Agreement. This is based on the definition of the purpose of the project, with scope, learning and assessment strategies proposed and action plans developed. This could then be negotiated, agreed and acted upon by the participants, followed by monitoring and review of the learning in action. In principle, this is based on a mentoring approach, where the mentee works with the mentor (professor, or a knowledgeable other) to work out learning projects or problems, in order to achieve the set goals by the learner. The learner could then define success based on his/her own learning goals and plans. This is a common practice in a contemporary mentoring program. I have used in for the last decade, and it seems to work pretty well. In the case of digital networks, and the x or c MOOCs, such model of mentoring could work if there are policy, procedures well established for the course and training provided to participants so they feel comfortable in choosing their mentors, or co-mentors, mentees. This might also rely more on a behavioral/cognitivist/constructivist approach at the start, though I suppose a connectivist approach could be adopted if both the mentor and mentees appreciate the importance of adopting networked learning with distributed learning platforms in the development of capabilities and skills of digital literacies. I wonder if xMOOC providers would find this approach useful, as this is very much similar to the face-to-face model of apprenticeship in the case of PhD program, which might be so high value added that professors wouldn’t have time for each of the participants of MOOCs. Besides, this is not a few weeks’ program, but a program which may last for a few months – in a typical mentoring program. This also depends on the needs and expectations of the participants (as mentees or mentors). Some people don’t feel comfortable to be the mentees (especially if they are advanced learners, graduates (with Masters or PhDs), or Professors), and many would prefer to be the mentors. Would this also change the role of the professor from content transmitter to mentor coordinator or facilitator? If the MOOC is content oriented, as in the case of most xMOOCs, then this approach might have limited success. In the case of cMOOCs where content might be negotiated, or even be decided by the participants, then this approach could yield wonderful results, provided that both the professor and participant understand and apply an adaptive learning methodology in the MOOCs.
Thanks again for your visit and comments. Much appreciated.
Can automated scoring of essays be done (by computer softwares)? See this paper
Pingback: How MOOCs change the world? | Learner Weblog
Pingback: What are the assumptions behind MOOC, in partic...
Pingback: What are the assumptions behind MOOC, in partic...
I have been saying this for a while… since my first Coursera MOOC. I now refuse to do the peer assessments. I have taught online in a Masters of Education program and used peer assessments successfully. I also have a Masters in Education in eLearning, so I understand why people are trying this. But it does not work in this type of classes.
When I taught my students knew each other online, they were all going for a degree, and they had similar background both in education and profession. Most of my students were also K-12 teachers and had used rubrics in their own classes. Plus. I could intervene if there was a problem. And there were a couple of personality problems.
But for Coursera this is not true. The students are from 15-85, from various backgrounds, different languages and various levels of education. Many are unfamiliar with this kind of assessment. Plus we are assuming that these students care when grading others. That is not true either. And there is no one to complain to if a students disagrees with the assessment or can it be changed.
I love taking these classes and love the fact that we are now talking about education and assessment. But I will never again participate in a peer grading, which if I want a certificate will not help me.
MOOCs usually use multiple peer assessments and take the median to exclude eccentric marking, which should be good enough given a clear enough rubric. But engaging markers who are one step ahead of the learners, eg those who previously successfully completed the course with good grades, to mark and offer feedback for a small fee could offer better quality and be part of the monetization model.