Jenny Mackness provides an excellent review on what she has learnt from the course OldGlobeMOOC “Update-on-oldglobemooc-and-peer-assessment.”
I would like to share some of her points and responses to my questions here.
She says:
The OldGlobeMOOC is a great experience in terms of the diversity of participants. Unfortunately the younger participants, in their teens, who signed up, seem to have fallen out of the discussion forums. This does not mean that they are no longer participating through observation and reading – it’s difficult to know. But I have wondered how an 11 year old might review the assignment of an academic Professor, or how an academic Professor might respond to a learner with special needs, or a very young participant, or someone whose first language is not English, and so on. The assignment submission is anonymous. Do these differences have implications for the equity of the peer review process?
I ask:
Thanks Jenny for your insightful responses. It is interesting to have peer assessment on such topics. I am wondering if the peer assessment by peers are assessing based on whether the stories are resonating to their personal perceptions. Assessment may also relate to the degree of relatedness to ageing. After reading through those peer reviews, I have a few questions in mind. 1. Since the experience relates to the writer’s story and opinions, would the assessment relate more on how the story and opinions on ageing (emotional response) or the story structure and written expression (clarity and coherence)? 2. How to ensure a fair assessment under such blind assessment arrangement? As you mentioned, it may likely that a novice (11 old teenager) assessing an academic professor, or a veteran or geek. 3. How would one appeal to any “inappropriate assessment or review”? Are there any appeal mechanism or procedures in place? Should there be one, as you mentioned that you have been penalized though you have submitted all five reviews on time. I ask these for the sake of discourse, not as a challenge to the authority, or a “complaint”. I think it is important to ensure any assessment be viewed as open, transparent, and are based on the honest feedback, rather than hasty response. Though there might be good reasons why some people just give others a “2″, I just wonder if this would be “acceptable” in a peer assessment system. What do you think would be a better way to ensure a fairer assessment with xMOOC? John
Jenny elaborates:
Hi John – thanks for all these questions. I don’t think OldGlobe is typical of xMOOCs. In fact Sarah Kagan has called it a cMOOC and it does have a lot of the characteristics of a cMOOC – it is being run mostly on the Coursera platform, but there is also a Facebook group and Twitter, but I’m not aware of other bloggers. It is open in the sense that access is open, the assessment is open and there is very little course content, e.g. there are weekly videos, but no readings, and discussion can follow any path the participants would like. The course has great diversity of participation and discussion, and participants do have a lot of autonomy if they are not bothered about assessment. So a lot more like a cMOOC than an xMOOC. And judging from comments in the forums, the assessment in OldGlobe has been different.
The nature of the assessment questions does lead to storytelling and for me this is a strength because it means that anyone from any background can engage with the assessment – but as I have described it does lead to some difficulties with the peer review system. I think it’s possible for reviewers to take a number of approaches to the peer assessment – but this is how peer reviewers are asked to respond:
Please type your 100-250 word peer assessment below.
What do you think about this participant’s portfolio item choice to answer this question of the week?
How does this participant’s perspective differ from your point of view?
How is your point of view similar?So in OldGlobe the criteria for peer assessment are quite loose and non-academic. So I don’t think it’s possible to think in terms of fairness, as we normally understand it in academic assessment, but rather in terms of empathy, kindness and respect. Reviewers are told:
An assignment only receives a zero if it is incomplete or did not follow the guidelines set for this week. Don’t be afraid to be generous!
And yes it is possible to appeal, by putting a question in the Help Forum – but participants have to have a genuine cause for appeal. It’s no good saying ‘I didn’t have time to do the peer reviews this week’ – hence the peer review I got which said
peer 2 → I’m headed for an airplane so don’t have time to review, and I won’t be back until after evaluation time ends so I’m just giving everyone a 2.
Based on what Jenny says above, the course does have a lot of characteristics of a cMOOC. Also the nature of assessment questions does lead to story telling. Indeed, one of the most attractive nature of online conversation is story telling and sharing, where bloggers share their anecdotes or learning scenarios, and reflect on those experiences that they have learnt.
The assessment criteria of this MOOC thus focuses on the reflection of similarities and differences in views and perceptions, and possibility the resonance or dissonance that one experiences in life upon reading the writings, especially when the author and reader relates to their perception or feelings of old age. This could also be a powerful lesson for any one to imagine what it means to undergo the various stages of old age, physically, mentally and spiritually.
This sounds quite an interesting way to assessment for the participants too, as it could lead both the learner and the reviewer (who could also be a blogger or another learner) to assess based on his/her affections (feelings, emotions) towards what the learner think about old age, and thus provide a point of view which is distinct from the learner.
Would this assessment be designed to measure certain attitudes, and to a certain degree the emotional awareness, control and responses, and empathy- which are related to emotional and social intelligence? See my posts here, here, and here. This seems not to be explicit in the assessment, though I haven’t enrolled into the course, and thus not be aware of how the assessment is related to emotional intelligence.
Another dimension that I reckon such assessment would lead to is clarity and brevity in expressing one’s thoughts in writing (as there is a 250 words limit). It seems that this is based on writing in English, and that grammar, use and choice of words, and structure of the writing would need to be taken into consideration in the assessment. Would this be more about assessment on written English, based on the context and content? How would the length of writing (i.e. 100 -250 words) affect the overall “quality” of the writing? Are there any penalty if the writing exceeds the limit? May be not, with such form of assessment.
Finally I could sense that such assessment may help the author and the reviewer to understand and apply the following basic principles in blogging and online conversation, based on writings:
1. Be concise and clear in writing
2. Be empathetic – understand others’ points of view, and empathize
3. Be kind, generous and sincere in comments and feedback – don’t patronize
4. Be open to others’ views and comments, even if we don’t agree with our readers’ points of views
5. Be patient and listen to the reader’s comments and feedback
6. Be thankful to our commentators and readers
7. Be ourselves – in stating our views
8. Be supportive to each others’ learning, through such sharing of views and experiences
9. Be tolerant even if others are harsh on us, though at times, we need to be assertive
10. Be responsive to others, and be responsible for our own work
Does it also mean that sometimes it is the MacGuffin (as suggested by Stephen Downes) that sets off a good online conversation? It is the emotional attenuation that would help people relate to our story, so they could also share their stories with us.
What else have I missed?
Again, I think Jenny has been an master (and exemplary) blogger who role models how to write comprehensive blog posts and how to respond to comments in blogging and assessment.
You would find lots of master pieces in her blog posts. I would encourage you to read her blog posts, where you would surely find lots of words of wisdom.
Thanks Jenny for her wonderful insights.
Learner Weblog 
Jenny: I am wondering if assessment through a mentoring approach might help the students to learn and assess more consistently and “professionally” – see the last few comments in my previous post https://suifaijohnmak.wordpress.com/2013/03/06/what-are-the-assumptions-behind-mooc-in-particular-peer-assessment-and-grading/
See below:
One method that hasn’t been used much in the xMOOC is the Learning Contract or Agreement. This is based on the definition of the purpose of the project, with scope, learning and assessment strategies proposed and action plans developed. This could then be negotiated, agreed and acted upon by the participants, followed by monitoring and review of the learning in action. In principle, this is based on a mentoring approach, where the mentee works with the mentor (professor, or a knowledgeable other) to work out learning projects or problems, in order to achieve the set goals by the learner. The learner could then define success based on his/her own learning goals and plans. This is a common practice in a contemporary mentoring program. I have used in for the last decade, and it seems to work pretty well. In the case of digital networks, and the x or c MOOCs, such model of mentoring could work if there are policy, procedures well established for the course and training provided to participants so they feel comfortable in choosing their mentors, or co-mentors, mentees. This might also rely more on a behavioral/cognitivist/constructivist approach at the start, though I suppose a connectivist approach could be adopted if both the mentor and mentees appreciate the importance of adopting networked learning with distributed learning platforms in the development of capabilities and skills of digital literacies. I wonder if xMOOC providers would find this approach useful, as this is very much similar to the face-to-face model of apprenticeship in the case of PhD program, which might be so high value added that professors wouldn’t have time for each of the participants of MOOCs. Besides, this is not a few weeks’ program, but a program which may last for a few months – in a typical mentoring program. This also depends on the needs and expectations of the participants (as mentees or mentors). Some people don’t feel comfortable to be the mentees (especially if they are advanced learners, graduates (with Masters or PhDs), or Professors), and many would prefer to be the mentors. Would this also change the role of the professor from content transmitter to mentor coordinator or facilitator? If the MOOC is content oriented, as in the case of most xMOOCs, then this approach might have limited success. In the case of cMOOCs where content might be negotiated, or even be decided by the participants, then this approach could yield wonderful results, provided that both the professor and participant understand and apply an adaptive learning methodology in the MOOCs.
Thanks again for your visit and comments. Much appreciated.
Pingback: What is most important when conversing online? ...