Application of Game Theory in the design, delivery and assessment in MOOCs

This Lifelong education on Steroids provides an excellent and insightful overview about MOOCs.  What I would add is that MOOCs could be one the game changers in Higher Education, not just online education.  Why?  Higher Education has been a game in business, where each of the game players are playing a fair, though competitive game in a global arena for decades.  

The strategic alliance and partnership is one of the macro approaches in game playing where institutions are working with various other education providers or services in order to enhance the overall education and learning experiences of the learners, or consumers and customers.

How would Game Theory help in the design, delivery and assessment in MOOCs?

There are two main approaches that we could consider – a macro and a micro approach.

Macro approach:

First, to design MOOCs based on Game Theory, on a macro scale. What this involves is to compare and contrast the various design of x and c MOOCs, based on a set of principles where networked learning and mastery learning is leveraged, especially when an institutional education model is based.  This could be demonstrated and applied by taking into consideration the payoff and expected return with each probability (i.e. un-bundling of each of the present services of typical MOOCs services as described here) and re-bundling them with values and benefits for each cohort of learners and educators.

Second, to deliver MOOCs based on Game Theory principles which include those elaborated in this Understanding the MOOC Trend.

Third, to assess MOOCs based on a combination of automation and human intervention, where learning analytics and big data are used to provide feedback to both educators and learners on a continuous basis.  This paper on assessment on MOOCs provides an insightful approach to incorporate

Micro approach

This involves strategically designing MOOCs based more on the games with various multimedia and interactive game story, where assessment and learning are built in to engage both professors and learners to co-explore and learn through the education process.  Games could also be used for assessing learners in a personal and adaptive way, though this would involve a total different design from the instructivist approach.  This includes peer-teaching and learning as proposed by Eric Mazur and other educators.  Indeed peer teaching and learning is one of the pedagogy adopted in a connectivist approaches towards learning.

It should be noted that the majority of peer-tutoring programs for students are intended to complement, not substitute for, regular classroom instruction. Tutoring should never be a substitute for professional teaching. An ideal learning atmosphere is as a rich blend of peer and adult instructional strategies.

Cited From: http://www.opencolleges.edu.au/informed/features/peer-teaching/#ixzz2cTBZTEiL

In summary, game theory could be used in the design, delivery and assessment in MOOCs, with an overall improvement in the learning and education experiences of learners.

On Study Skills and Examination

Another useful video by Dr Stephen Chew on study skills and examination.

Examinations could be highly effectively in mass assessing students of Higher Education in an objective way.  Most of us (as educators) have gone through the examination processes in Universities and Colleges.  Indeed examinations could be critical in determining the grades of students in a College or University degree, and such practice might not change in the foreseeable future.

Examinations are still useful for undergraduate and graduate studies up to PhDs, or even professional association admission or accreditation.  So it is important to learn those examination skills, in order to achieve good results and meet the goals.

Examinations are, however summative assessment tool and there are little that the learners could do to change the results of the examination, unless there are feedback to the learners on where they are fallen short of, in terms of their “mistakes” or “wrong answers” so they could correct.  

Whilst examinations are still important tool in assessing students in Higher Education, there is now a trend towards using various combination of formative and summative assessments – authentic or real life assessment tasks, problem based assignments, workplace projects, and workplace based assessment as a more holistic educational tool in the assessment process, apart from the formal examinations.

In my post here, I share the following:

If assessment is so important in formal education, why do people still prefer to adopt the instrumental teaching based principally on mass lecture, tests and examination rather than assessment as an effective pedagogy?  Take a test or examination, and if you could pass it, you are qualified for a pass of the unit.  Isn’t it simple?

Some of us might have watched this video.

So, a lot of students would ask a basic question: Are the lecture materials delivered by the teacher during the lesson to be tested in the tests and  examinations? If not, could we focus just on what is to be tested or examined, and leave the rest to be “learnt” outside the classroom?  This is exactly the type of questions most students are asking in each semester, in a traditional lecture type of education and learning. Is that what the educators are most concerned too?  Teaching the content of examination or test to the students, so students could achieve high marks in the assessments. So, why not teaching to the test?

A test and or examination is a typical assessment tool used in education for decades. That’s where students could demonstrate their competency, and that is how assessment is conducted in most schools.  And if students are learning in online distance education, then they would be expected to submit the standard assignments (say completing a 2,000 words essay or answering a series of questions as required in the problem or project set), attend the examination, and if they pass in both assessment, congratulations!

Doing assessment requires more than the mere completion of the written assignments.  An excellent example of assignments as shown here requires the preparation and collection of evidences, and through an exploration and research process in the assessment, the learners would be able to demonstrate the competency required.  Also learners could identify their own learning needs and gaps in the learning process, when working through the assignments.  With the feedback from peers and or facilitators, the learner could also identify what would need to do to improve his or her learning.  These will all involve sensemaking (giving meaning to experience) and metacognition (cognition about cognition or knowing about knowing).

What is most important when conversing online?

Jenny Mackness provides an excellent review on what she has learnt from the course OldGlobeMOOC “Update-on-oldglobemooc-and-peer-assessment.”

I would like to share some of her points and responses to my questions here.

She says:

The OldGlobeMOOC is a great experience in terms of the diversity of participants. Unfortunately the younger participants, in their teens, who signed up, seem to have fallen out of the discussion forums. This does not mean that they are no longer participating through observation and reading – it’s difficult to know. But I have wondered how an 11 year old might review the assignment of an academic Professor, or how an academic Professor might respond to a learner with special needs, or a very young participant, or someone whose first language is not English, and so on. The assignment submission is anonymous. Do these differences have implications for the equity of the peer review process?

I ask:

Thanks Jenny for your insightful responses. It is interesting to have peer assessment on such topics. I am wondering if the peer assessment by peers are assessing based on whether the stories are resonating to their personal perceptions. Assessment may also relate to the degree of relatedness to ageing. After reading through those peer reviews, I have a few questions in mind. 1. Since the experience relates to the writer’s story and opinions, would the assessment relate more on how the story and opinions on ageing (emotional response) or the story structure and written expression (clarity and coherence)? 2. How to ensure a fair assessment under such blind assessment arrangement? As you mentioned, it may likely that a novice (11 old teenager) assessing an academic professor, or a veteran or geek. 3. How would one appeal to any “inappropriate assessment or review”? Are there any appeal mechanism or procedures in place? Should there be one, as you mentioned that you have been penalized though you have submitted all five reviews on time. I ask these for the sake of discourse, not as a challenge to the authority, or a “complaint”. I think it is important to ensure any assessment be viewed as open, transparent, and are based on the honest feedback, rather than hasty response. Though there might be good reasons why some people just give others a “2″, I just wonder if this would be “acceptable” in a peer assessment system. What do you think would be a better way to ensure a fairer assessment with xMOOC? John

Jenny elaborates:

Hi John – thanks for all these questions. I don’t think OldGlobe is typical of xMOOCs. In fact Sarah Kagan has called it a cMOOC and it does have a lot of the characteristics of a cMOOC – it is being run mostly on the Coursera platform, but there is also a Facebook group and Twitter, but I’m not aware of other bloggers. It is open in the sense that access is open, the assessment is open and there is very little course content, e.g. there are weekly videos, but no readings, and discussion can follow any path the participants would like. The course has great diversity of participation and discussion, and participants do have a lot of autonomy if they are not bothered about assessment. So a lot more like a cMOOC than an xMOOC. And judging from comments in the forums, the assessment in OldGlobe has been different.

The nature of the assessment questions does lead to storytelling and for me this is a strength because it means that anyone from any background can engage with the assessment – but as I have described it does lead to some difficulties with the peer review system. I think it’s possible for reviewers to take a number of approaches to the peer assessment – but this is how peer reviewers are asked to respond:

Please type your 100-250 word peer assessment below.

What do you think about this participant’s portfolio item choice to answer this question of the week?
How does this participant’s perspective differ from your point of view?
How is your point of view similar?

So in OldGlobe the criteria for peer assessment are quite loose and non-academic. So I don’t think it’s possible to think in terms of fairness, as we normally understand it in academic assessment, but rather in terms of empathy, kindness and respect. Reviewers are told:

An assignment only receives a zero if it is incomplete or did not follow the guidelines set for this week. Don’t be afraid to be generous!

And yes it is possible to appeal, by putting a question in the Help Forum – but participants have to have a genuine cause for appeal. It’s no good saying ‘I didn’t have time to do the peer reviews this week’ – hence the peer review I got which said

peer 2 → I’m headed for an airplane so don’t have time to review, and I won’t be back until after evaluation time ends so I’m just giving everyone a 2.
:-)

Based on what Jenny says above, the course does have a lot of characteristics of a cMOOC.  Also the nature of assessment questions does lead to story telling.  Indeed, one of the most attractive nature of online conversation is story telling and sharing, where bloggers share their anecdotes or learning scenarios, and reflect on those experiences that they have learnt.

The assessment criteria of this MOOC thus focuses on the reflection of similarities and differences in views and perceptions, and possibility the resonance or dissonance that one experiences in life upon reading the writings, especially when the author and reader relates to their perception or feelings of old age.  This could also be a powerful lesson for any one to imagine what it means to undergo the various stages of old age, physically, mentally and spiritually.

This sounds quite an interesting way to assessment for the participants too, as it could lead both the learner and the reviewer (who could also be a blogger or another learner) to assess based on his/her affections (feelings, emotions) towards what the learner think about old age, and thus provide a point of view which is distinct from the learner.

Would this assessment be designed to measure certain attitudes, and to a certain degree the emotional awareness, control and responses, and empathy- which are related to emotional and social intelligence?  See my posts here, here, and here.  This seems not to be explicit in the assessment, though I haven’t enrolled into the course, and thus not be aware of how the assessment is related to emotional intelligence.

Another dimension that I reckon such assessment would lead to is clarity and brevity in expressing one’s thoughts in writing (as there is a 250 words limit).  It seems that this is based on writing in English, and that grammar, use and choice of words, and structure of the writing would need to be taken into consideration in the assessment.  Would this be more about assessment on written English, based on the context and content?  How would the length of writing (i.e. 100 -250 words) affect the overall “quality” of the writing?  Are there any penalty if the writing exceeds the limit?  May be not, with such form of assessment.

Finally I could sense that such assessment may help the author and the reviewer to understand and apply the following basic principles in blogging and online conversation, based on writings:

1. Be concise and clear in writing

2. Be empathetic – understand others’ points of view, and empathize

3. Be kind, generous and sincere in comments and feedback – don’t patronize

4. Be open to others’ views and comments, even if we don’t agree with our readers’ points of views

5.  Be patient and listen to the reader’s comments and feedback

6. Be thankful to our commentators and readers

7. Be ourselves – in stating our views

8. Be supportive to each others’ learning, through such sharing of views and experiences

9. Be tolerant even if others are harsh on us, though at times, we need to be assertive

10. Be responsive to others, and be responsible for our own work

Does it also mean that sometimes it is the MacGuffin (as suggested by Stephen Downes) that sets off a good online conversation?  It is the emotional attenuation that would help people relate to our story, so they could also share their stories with us.  

What else have I missed?

Again, I think Jenny has been an master (and exemplary) blogger who role models how to write comprehensive blog posts and how to respond to comments in blogging and assessment.

You would find lots of master pieces in her blog posts.  I would encourage you to read her blog posts, where you would surely find lots of words of wisdom.

Thanks Jenny for her wonderful insights.

Is MOOC about Community Building?

I just come across this post on MOOC. So Coursera community is starting to introduce student profiles to enable students to build connections with each others.  That’s a great move towards community building.

There are suggestions to develop groups in these courses, and again that would surely help in supporting connectivity in MOOC.

The challenge in MOOCs is: how do we know who someone really is, in their identity online?  Also building groups with similar language and cultural background may help in clustering learners who speak and write the same languages, but then this would weaken the chance of those people meeting others coming from different backgrounds too.

If someone is coming from a “weak” educational background, then would that hinder them from connecting with those more experienced learners with a strong educational background?

As Roy points out:

On the question of elitist or egalitarian MOOCs … I think there is room for both, but the only really important point for me is it should be upfront: the polite equivalent of something like …

  1. ‘this MOOC is a conversation in which experts are free to participate in expert discourse, without regard for novices’
  2. ‘this MOOC is a conversation in which experts are free to participate in expert discourse, but they should be aware of the need to relate to others who may be novices in particular fields’ (which includes just about everyone)
  3. ‘this MOOC is a conversation in which experts are not free to participate in expert discourse if that leads to the exclusion and marginalisation of novices’.

What is Coursera built upon?  Is it an elitist or egalitarian platform?  That would determine how the conversation would flow, and who would be engaged in those conversations.  This is also a question for c MOOC too, as experts’ interests could be very different from those of the novices.

So, it is rather hard to resolve the issue – on whether experts and novices could be encouraged and supported in the discourse or conversation, by merely having a central forum in the course, where tens of thousands of students resided.

Has this been a challenge in c MOOCs – where MOODLE forum was used?   A definite YES.

In the post:

The connectivist model is more visionary in that it understands that one of the most dynamic assets of an unbound open learning system is the people, but in execution there is a lot of same chaos in identifying, absorbing, and building upon meaningful contributions. At the end of the day it is still too many voices overflowing seemingly never-ending streams.

There were suggestions on how these be rectified by:

1. Managing information: purposing a meaningful discussion

– Setting up the proper framework and controls

– Facilitating digestion of information

– Encouraging engagement

2. Facilitating action: tangible engagement beyond words

– Forging meaningful connections

– Translating into real-life interaction

Sounds great.  I suppose most of the actions mentioned are extremely useful and worthwhile to pursue, if most if not all students are coming from similar background experience, and thus with similar expectations.  However, in the case of MOOC, I think it would be quite a challenge to ensure the meaningful discussion and the provision of facilitation, unless these are based on self-organising action and volunteer master facilitators who could help and support the facilitation.

I also see those nuances in learning differently.  First connectivist model requires an appreciation of chaotic learning, due to the interaction between different agents and information sources, and are therefore highly valuable for people who would like to master sensemaking and wayfinding, as a goal in the learning.  Would that be applicable in structured courses such as x MOOCs?  May be, may be not.  As shared, the prescriptive learning that is based on known and declarative knowledge would not be learnt most effectively with approaches other than mastery learning, as there are definite answers that the learners are expected to respond to in assignments and examinations.

Second, even in the case of peer learning and assessment, the emergent learning that emerged out of the interaction in assessment would be “structured” around a few learners only.  Each of the peers need to have a certain mastery of the knowledge and skills before they could make a “valid” and reliable assessment of their fellow peers.  This is similar to the peer-review in articles for publication in journals or conference, only that the review here is related to certain course work, rather a formal paper for publication.

To achieve a mastery of skill in vetting and grading peers’ assignment required mastery of assessment too.  This might require certain “validation” of the skills before one could take up such assessment role.

Under a formal structured course of instruction, the instructivist approach seems to be the norm rather than exception, in endorsing and assessing the learners to peer assess.

If one wants to know whether the responses in assessment are meeting the standards set up by the education authorities or professors or not, then such assessment has to go through another assessment process – like auditing or peer-review by experts.  How would this work in x MOOCs?

How about the cheating and plagiarism issue that are challenging the Coursera?  Are these also part of the learners’ roles to check and report while assessing their peers’ work?

The peer assessment, however, may best be approached with a connectivist approach, provided any suggestions to improvement and development are co-created in an educator-learner environment.  This is both a challenge to the professor, and to that of fellow learners, who might have believed that chaos in learning is undesirable, thus causing frustrations among learners, who would then drop out of the course, without any raising of the concerns.

In summary, building community to approach learning in MOOC would surely be the way to go, when all those learning are happening in open space, rather than the limited closed space in forum or videos instruction.  There are however challenges which still need to be resolved, when language barriers, chaotic learning, peer assessment and cheating and plagiarism issues are emerging out of the MOOCs.

What are the solutions?

Postscript: Cheating has become a huge concern in Higher Education.  See this post.  Here is the post relating to the language and peer grading issues.

Photo: image from Google

What may be the biggest problem in online course assessment? Cheating and Plagiarism!

What are the issues relating to cheating in online courses?

Cheating goes high tech:

“This is the gamification of education, and students are winning,” the professor told me.

The Shadow Scholars could just be the tip of the icebergs – in cheating.  Copying and plagiarism in online courses are also a concern for educators and education authority.

How to solve these cheating and plagiarism problems?

1. Use of technology and tools –  like face recognition, or other electronic identification, as outlined in this post interview to identify the persons, and tools such as turnitin or playchecker to check on plagiarism.

I have used playchecker in checking my own writings based on plagiarism, and here is the result.

2. Course terms and conditions – checking and monitoring of students’ submission of work based on agreement to terms and conditions, and an honor code.  Coursera and Udacity provide comprehensive terms and conditions of the registration and use of course materials, and submission of work.  This may not exclude students from cheating or plagiarism but could deter any students from cheating with intention in online courses.

3. Human intervention – where professors and instructors would interview the students via virtual conferencing or by referring to third party agents (employers, college or university authority, or authorized local representative) to check on the identity of students.

4. Test and examination centres – where students are required to sit for the tests and examination under surveillance, and monitoring if there are any cheating or plagiarism.  This could be a costly exercise, though worthwhile to prevent and reduce cheating.

Prevention is better than cure.  I think the excellent terms set out in the courses such as Coursera and Udacity would help students in appreciating the importance of integrity and honesty.

Here are the Best Practice Strategies to Promote Academic Integrity in Online Education.

Comments?

Postscript: This post on Dozens of Plagiarism Incidents highlights the seriousness of plagiarism.

Another related post.

A post on Plagiarism retrieved on 26 August 2012.

Nice post about cheating.

A post on cheating accessed on 8 Sept 2012.

#Change11 On MOOC – my reflection part 2

Interesting article on MOOC here.  Here is my response (also on FB): I was surprised that there wasn’t any mention of the pre-Stanford MOOCs, like CCK, CritLit, PLENK2010, led and developed by George Siemens, Stephen Downes, and Dave Cormier; David Wiley and Alec Couros Courses on Open Education and Tech etc. What might be the reasons for that? In the wikipedia, MOOCs have been well documented.

MOOCs are not new phenomenon, and are only new if we view Stanford, Udacity, MITx and edX as the main MOOCs recently introduced. On the assessment part, using auto-assessments have been in place for decades. Any one taking online knowledge test in driving would have experienced that first hand. Using auto-assessment to test knowledge (where knowledge is known, with correct answers) could be done based on softwares and technology. Peer assessment based on voting of best answers are sound if there are more than one “right” answers, but would be challenging if the assessment criteria are “too open”, leading to different interpretation. There are also possibility of over or under appraising one’s capability due to the inherent “issues” and “weaknesses” of the assessment system or tools. For instance, there are no ways of contextualization of the assessment tools to suit individual situations, leading to the mere checking of “factual” or “procedural” knowledge, and little on emergent or advanced knowledge which involved application at work. The use of eportfolios could be a good option, but this requires human intervention (experts, knowledgeable others, or crowd-assessment). There are also challenges on accreditation, where assessment system based on standardization would unlikely be appropriate in auditing such a complex system or network. How about the authenticity of the assessment – who are the participants really taking the test, quizzes, and assignments? Are they really who they are? Would someone be testing the system using non-human (like machine, AI)? How to ensure that there aren’t any plagiarism, cheating and “copying” of answers, assignments or portfolio evidences from others? How to quality assure the whole system of online education?

#Change11 Assessment, Active Learning and Project-based Learning

Is assessment part of learning and instruction?

If we assume that assessment forms an important part of learning, then assessment itself could also be considered a good pedagogy, in formal education and learning environment.  This is evidenced in most formal online courses, such as Artificial Intelligence (AI) and Machine Learning ML offered by Stanford University.  Here assessment determines the quality of the courses offered.  This stems from the notion that assessment should align with the learning outcomes, which the course is also based upon.  Accreditation of courses and award of qualification are normally based on the achievement of learning outcomes by the students or learners in formal education.

If assessment is so important in formal education, why do people still prefer to adopt the instrumental teaching based principally on mass lecture, tests and examination rather than assessment as an effective pedagogy?  Take a test or examination, and if you could pass it, you are qualified for a pass of the unit.  Isn’t it simple?

Some of us might have watched this video.

So, a lot of students would ask a basic question: Are the lecture materials delivered by the teacher during the lesson to be tested in the tests and  examinations? If not, could we focus just on what is to be tested or examined, and leave the rest to be “learnt” outside the classroom?  This is exactly the type of questions most students are asking in each semester, in a traditional lecture type of education and learning. Is that what the educators are most concerned too?  Teaching the content of examination or test to the students, so students could achieve high marks in the assessments. So, why not teaching to the test?

A test and or examination is a typical assessment tool used in education for decades. That’s where students could demonstrate their competency, and that is how assessment is conducted in most schools.  And if students are learning in online distance education, then they would be expected to submit the standard assignments (say completing a 2,000 words essay or answering a series of questions as required in the problem or project set), attend the examination, and if they pass in both assessment, congratulations!

Doing assessment requires more than the mere completion of the written assignments.  An excellent example of assignments as shown here requires the preparation and collection of evidences, and through an exploration and research process in the assessment, the learners would be able to demonstrate the competency required.  Also learners could identify their own learning needs and gaps in the learning process, when working through the assignments.  With the feedback from peers and or facilitators, the learner could also identify what would need to do to improve his or her learning.  These will all involve sensemaking (giving meaning to experience) and metacognition (cognition about cognition or knowing about knowing).

Is this active learning familiar to you?

I think such active learning have been adopted by experienced educators (especially in vocational education and training) for years.  May be this form of Active Learning is still new to those who are new to the teaching profession or that in the online learning.

We have learnt that lectures are still the predominant methods of transferring knowledge in Higher Education though there have been some changes as  reported here, with the use of small study groups of students discussing the questions posted by the professor.  Such study mode in classes was used as a driver to learning, with peer instruction as a means of active learning.

In this active learning in the classroom, everyone is engaged.

For those educators who have undergone formal education and training, active learning has always been part of the training.  I would assume that it is already part of any lessons for most experienced educators.

How about the reality?  Is it the case in Community College and Higher Education Classroom learning?

What are the challenges, when active learning is “transferred” and migrated to an online or virtual learning environment? What sort of pedagogical approaches would be more relevant and effective in such online courses?  I reckon that these have been explored in our past MOOCs here and here.

There are some assessment challenges that are still inherent in MOOCs, as past researches indicated that not all participants would undertake the assessment as designed for the course.  The learning also takes on different forms, with some active learning, whilst others would prefer to lurk in the course.

Learners need to have more control over their assessments in an online course, as remarked by Jenny.  But where does this leave ‘the expert’ and will students have the skills to take control of their assessment?

Here Jim Groom has adopted digital media projects as the main assignment for the MOOC.  I reckon these projects are highly relevant for students learning in formal online courses.   How would these project-based learning be used in MOOCs? Would these projects be adopted by life-long learners in MOOCs?

References

Kop, R., Fournier, H., Mak, S.F. J. (2011). A Pedagogy of Abundance or a Pedagogy to Support Human Beings? Participant Support on Massive Open Online CoursesThe International Review of Research in Open and Distance Learning. Vol 12, No. 7 (2011).

Kop, R., Carroll, F. (2011). Cloud Computing and Creativity: Learning on a Massive Open Online Course.  European Journal of Open, Distance and E-Learning. (2011).

Note:

I reckon one of the slides in the video relating to hearing, seeing and doing is not correct.  In my previous post, I commented on the use of visuals in lesson presentation.  I also cited George’s comments: ” Will Thalheimer debunks/questions the validity of this claim. This automatically calls into question related statements in the article (not cited properly) about the prominence of visuals in learning and retention.”

Also refer to this post where the author says:

“Not only is it easier to communicate something using a picture, but it’s also much easier for people to remember things that have been communicated to them visually. Psychologist Jerome Bruner of New York University has studied the art of communication, and his studies have shown that:

  • People remember 10% of what they hear;
  • 20% of what they read; and
  • 80% of what they see and do.

Most people are visual learners; a recent study by the U.S. Federal Government suggested that up to 83% of human learning occurs visually. The study also indicated that information which is communicated visually is retained up to six times greater than information which is communicated by spoken word alone.”

There are still lots of assumptions about the percentage of people remembering what they hear, read, see and do in human learning, and further research is required to validate those claims.

Postscript: A relevant post here by Stephen – on experimentation with new forms of higher education.

A useful video on problem based learning.