Measurement of effectiveness of cMOOCs

Here is my response to Christina’s post on difficulties researching cmoocs.

How to measure the effectiveness of a cMOOC?

There are 4 semantic conditions of networks that Stephen Downes has proposed. As Stephen has commented, those properties – openness, diversity, autonomy and connectedness & interactivity is not perfect in cMOOCs. Besides Connectivism as applied in cMOOCs could likely best be based on an informal learning, rather than a traditional institutional model.

I have reiterated that the constraints typically imposed with an institutional model would be huge challenge for administrators and educators to adapt, as is witnessed even in xMOOCs, where a totally new approach (such as flipping the class or flipped learning) as perceived by professors would be at odds with the mass lecture approach typical in mass-education, with a broadcasting model. How to overcome those challenges, and ensure learning is more effective, when cMOOCs are embedded in an institutional model?

Here is my response  that I perceive as a way to measure the effectiveness of cMOOCs – in its

1. awareness of Networked Learning and Connectivism as an “informal learning paradigm”,

2. an adoption and leveraging of the 4 properties- openness, diversity, autonomy and connectedness & interactivity when networking,

3. an achievement of personal goals with immersion in the network and community (and community of practice) on personal basis,

4. adoption of Personal Learning Environment and Network PLE/PLN in pursuit of life-long learning, and

5. a shift of frame of reference and paradigm from knowledge transmission to knowledge sharing and creation model under a knowledge ecology.

John Mak

Personalization of education and learning

What should our future education be aiming for?  Massification of education or personalization of learning?

In this paper on Instructional Theory by Reigeluth C. (2012), he highlights the need of having more personalized approach towards learning, through a post-industralist instructional approach, where learner becomes the centre for learning.

In this Mastery Learning and this paper on Mastery Learning, there are benefits of adopting its philosophy in MOOCs.  That’s also the central pedagogy adopted by most xMOOCs providers.

As I have shared in my previous post, students may master what is expected to be learnt if all teachers are teaching solely to the test.  However, it seems that many people might have mis-understood the initial intention of Mastery Learning, where the intention is NOT to ask the teacher to teach only those concepts for the sake of assessment or testing, but to allow the learners to master their learning at their own pace, in a progressive manner with immediate feedback in order to reinforce their understanding of concepts, and to correct any mis-understood concepts where possible.  Besides, Mastery Learning could be effectively employed in a mentoring and apprenticeship program where the mentor could guide the mentee through the program.

The future of education though would lie with personalization rather than massification of education as Aoki concludes here

This massification of online education appears to go in an opposite direction to personalization that elearning and use of ICT in education should aim for the purpose of providing more effective individualized learning experiences to learners.

How to progress from massification to personalization of online education?  I have shared that here.

Giving  students the correct answers strict away may sound a good instructional approach towards teaching.  However, have the students learnt how to arrive to those calculations?  Have the students mastered the concepts CORRECTLY?  How do we know if the students could apply their skills and transfer them from one area to another, in solving problems?

Aoki elaborates further on how personalization of learning could be achieved:

With the vast amount of data gathered through learners, personalization will become possible eventually with proper learning analytics and data mining. Furthermore, quality of learning outcomes may be further assured with the evidence of learning.

Research into professors’ experience in teaching and learning in x and cMOOCs

Do we need to conduct such researches?

Throughout the past years, there had been researches done on the learning experiences of participants of cMOOCs, and more recently, with the participants of x and cMOOCs.

I don’t see many systematic and inter-disciplinary researches into professors’ or facilitators’ experience in teaching and learning in x or cMOOCs, conducted by independent researchers or participants of MOOCs though.  There might be self-reflective blog posts by the professors of x or cMOOCs, and surveys and research interviews on the professors, by the professors, but not much by the participants or independent researchers.

What MIGHT BE the reasons for such rare research studies on professors?

Most researches are based on participants’ experience where learning could be investigated, with MOOCs. However, when it comes to research into x or cMOOCs, most of the professors might have preconceived what might be best as a pedagogy for their MOOCs, or that such pedagogy (Mastery Learning, an Instructivist – with behavioral/cognitivist approach) has already been pre-determined in the MOOC platform or in their design.  It could also be challenging to request professors to self-reflect fully on their experiences in teaching and learning in x and c MOOCs, as they would need to disclose their impressions, feelings and emotions, and how and what they have taught, facilitated and learnt through the MOOCs.  Besides, such researches might only be possible if they are conducted with independent researchers who have a good mastery of MOOCs, with adequate design of the research, and that researches are done in an ethical manner.  Besides, would MOOC professors be interested in participating in such a research?

Why this sort of research is important?

We have often heard about learners’ experience relating to MOOCs, and the constructive criticism for further improvement and innovation, with the use of technology or networks in MOOCs.  To what extent has such learning and development took place, by the professors and designers of MOOCs?

I would like to consider adopting such approach into the research on x and cMOOCs.  Some of the questions are included here.

Questions for the Designers, Instructors, or Professors of MOOCs:

Decisions to take part in MOOCs

1. Why would you like to create a MOOC?

2. What would you like to achieve with a MOOC?

3. What prior teaching or learning experience do you have with online education or MOOCs?

Design and delivery of MOOC

1. What would you like to include and expect in the design of MOOC? What are the design criteria? Why are they important to you and the participants?

2. What would you like to include and expect in the delivery of your MOOC? What are the delivery factors that you have considered? Why are they important to you and the participants?

3. What are the essential elements of a MOOC that would enhance the learning of the participants? Why do you think they are essential?

4. How would you evaluate the learning of the participants in MOOC?

5. What would you suggest to improve in your MOOC?

Research into MOOC

1. What research areas would be helpful to you in MOOC?

2. How would you conduct such research into MOOC?

3. What have you learnt through the design, delivery and review of MOOC?

4. What would you have done instead if you were to re-create a MOOC or start another MOOC?

More questions for professors and you

What do you think about such researches?

What research topics of MOOCs interest you?

Motivation and Intention in participating and engaging with MOOCs

Is intention an appropriate measure of success of MOOCs?

I reckon each person’s intention in MOOCs is different, though the participation and engagement could likely fall into patterns similar to the four archetypes of MOOCs.

My proposition and assumptions relating to motivation and intention in participating and engaging with MOOCs include:

Psychological factors, Like/dislike of MOOCs (as public/commoditised/monetised goods), credentials achivement, & pedagogy used in MOOC as perceived by people could make a difference.

1. How would people’s perception impact on their intention to learn with MOOCs?

1.1 What factors would determine people’s intention to enroll into MOOCs?

– These students/participants intend to browse and audit the programs.  These participants could include: (a) professors, educators and experts in their field or other fields who would like to have a sense of feel on what MOOCs are, and how they are run; (b) researchers and Master or PhD students who would like to conduct researches on MOOCs, as part their faculties requirements or qualification requirements; (c) participants who are life-long learners, who might have got a degree in the field, or in other fields, but are interested in the field of study.  There might be some people who like the pedagogy, and others who dislike the pedagogy.

– These students/participants intend to engage and interact with part of the course content and or other participants with discussion boards.  These participants could include those of the above, but with an intent to complete a few to most of the activities, assessments or examinations,  but have no intention of getting credits or expecting credentials out of the MOOCs

– These students/participants intend to engage and interact fully with the course content and other participants with the LMS.  These participants are more inclined to like the pedagogy adopted, though again there may be a minority of participants who dislike the approach, but not willing to disclose their emotions or feelings in open public.  These sort of feelings towards courses are typical in learners attending most institution based courses.  Feelings of loneliness, lack of interaction with others and professors, and lack of “support” that relate to motivation could be issues and concerns.  Others include the messiness and frustration emerging from the participation in forum and discussion boards, when trolling and “tangential discussions”, negative criticisms are present in the forum postings, and the concerns of moderation.

1.2 What factors would determine people’s intention to like/dislike MOOCs?

1.3 How would such likes/dislikes translate into learning in MOOCs?

1.4 To what extent would learning styles impact on one’s motivation to learn in MOOCs (xMOOCs in particular)?

1.5 How would each of the factors, likes/dislikes and learning styles relate to the four archetypes of MOOCs – lurkers, passive learners, active learners and drop-ins?

2.  Teaching, social and cognitive presence are often cited as the most important factors in successful online presence.  To what extent are these presence contribute to the successful learning in xMOOCs?

3. What are the goals and motivation of xMOOCs participants?

In this article on 6002x-data-offer-insights-into-online-learning (full article here):

It is noteworthy that:

Participation and performance do not follow the rules by which universities have traditionally organized the teaching enterprise:  MOOCs allow free and easy registration, do not require formal withdrawals, and include a large number of students who may not have any interest in completing assignments and assessments.

This finding aligns with what have been found in previous research:

As our research on PLENK (cMOOCs) revealed, many participants of cMOOCs are putting assessment as (lowest) in priority. This is different from the xMOOCs where assessment is given a high priority by the instructors (professors), and may be some students, especially the undergraduate students who would like to use that to improve their performance with their own courses. Besides, there are lots of graduates and adult learners and educators in cMOOCs who are more interested in learning about the pedagogy, the different learning theories, and the emergent tools and technology. They may already have got their qualifications, or that they aren’t keen in being assessed, or being “instructed” under a “mastery learning approach”. There are also professors, experts, professionals who wish to know how MOOCs are designed and run, and how they might be used in their own fields. These all “contradict” to the initial design of xMOOCs, though could be easily accommodated in cMOOCs, as that is exactly what cMOOCs are designed for.

It should be stressed that over 90% of the activity on the discussion forum resulted from students who simply viewed preexisting discussion threads, without posting questions, answers, or comments.

This is not surprising at all, as such pattern of involvement in discussion forum has repeatedly appeared in previous cMOOCs (see Rita and her colleagues’ research publications on MOOCs).  It is typical to note a highly active participation or posting on the discussion forum at the start of a MOOC followed by an exponential drop in the later part of the course.  Such pattern of engagement may vary from cMOOCs to xMOOCs though as the xMOOCs have numerous assessment components (like homework, examinations) which may lead students to post questions in the discussion forum.

Discussions were the most frequently used resource while doing homework problems and lecture videos consumed the most time.

There are also differences in the cohort of students, with xMOOCs more likely consisting of younger students compared to that of those in cMOOCs.  A more in-depth analysis of the student populations would be needed to compare the xMOOCs and cMOOCs students’ populations.

In xMOOCs, success has been defined by the research authors as the grades students earned.  Measure of success as “achievement”.

In cMOOCs, success has yet to be defined, though many researchers and educators have proposed it to be defined as the achievement of personal goals as set forth whilst participating and engaging with cMOOCs.

“This is also noteworthy that majority of students (75.7%) did not work offline with anyone on the MITx material.”  and that those who did work offline with others have achieved 3 points higher than those who didn’t.  This again illustrates that many students of xMOOCs would likely learn on their own, without resorting to the “help” or “support” from others, especially with a technical course such as MITx- 6002x.

This pattern of online learning seems to coincide with the current mode of learning in an online learning environment, where most students are still learning on their own, with or without the use of PLE/PLN.

Would this pattern of engagement be typical for xMOOCs humanities courses?

These questions posted in the article are interesting for further exploration.

What are students’ goals when they enroll in a MOOC? How do those goals relate to the interaction with various modes of instruction or course components? What facilitates or impedes their motivation to learn during a course? How can course content and its delivery support students’ self-efficacy for learning? Similarly, how can online environments support students’ metacognition and self-regulated learning? Do interventions such as metacognitive prompts and guided reflection improve student achievement or increase retention?

Transformation of Higher Education – Why is it so hard?

Is transformation of Higher Education possible?  My reflections:

Relating to the ideas on transformation of  Higher Education with improved teaching and education reform as discussed in this article, I reckon this is similar to the adoption of a connectivist approach in Higher Education.  There are still long roads to cross, due to the enculturated values of teaching and research that have been embraced by both professors and administrators for decades.  Besides there are demands of stability under an education system in Higher Education, it would be difficult to transform Higher Education without changing the pedagogy.  Transformation of Higher Education through improved teaching requires a review of the pedagogy adopted in HE.  I would reflect on this important aspect in another post relating to MOOCs.

Carl envisions and demands better teaching, with push backs from other academics due to challenge of traditional values and cultures that have been in the education system for decades.  I think many professors do know what could be done to improve & innovate teaching.  Higher Education values research over teaching, and that wouldn’t be changing as research “creates” & generate new knowledge, whilst teaching would at best transmit knowledge, as generally perceived by professors and students.

For those very smart & talented students, wouldn’t they just need minimum guidance and would then excel as Carl has cited in the article, under an apprenticeship model, with graduates?  For under-graduate students, only the top and talented students would learn most effectively with such model, as they are self-motivated and regulated.

For most other students, there are still needs for close support and mentoring, that are obviously absent if the only way to learn is the 50 min mass lecture method.

If I were to ask Carl: Is your Nobel Prize based on research or teaching?  If the answer is teaching, then would professors be considering how to improve teaching in a deeper way?

Besides, all PhD and Doctorate programs are still focusing on research as a principal means to gauge and evaluate a persons’ achievement in scholastic and research in the field.  How would we expect  professors to spend time in “teaching” their students when such PhD students are already good enough to learn with technology and network affordance?

But would this be an over-simplification of what teaching of under-graduate programs are all about?  Teaching concepts or correcting misunderstood or incorrect concepts in science is important.  However, would the use of MC and T/F or short answer questions be good enough to inculcate the values and applications of science in real life?

Some students would still prefer lecture method, and so many professors would continue to do so (and I think I would practice it too), as any negative comments or feedback from students would only lead to professors adopting more teacher-centered approach, when they are reminded that these are what the students want – to know the answers to the examination, tests, quizzes and assignments straight away, instead of spending time exploring themselves.

Some students are uncomfortable with this approach—even if it’s more effective. “I remember getting an evaluation from one
[UCSD] student who had just finished my course,” says Simon, a pioneer in the use of peer instruction within her field. “I loved
it. It read, ‘I just wish she’d have lectured. Instead, I had to learn the material myself.’ ” See above article.

Numerous researches have hinted that students want simple and effective means of learning, not complicated or complex tasks which are both time-consuming and difficult to perform.  That is the reality and challenge that most educators and professors are facing Higher Education.  Isn’t it?

The old motto: “Tell them what you want to tell them, tell them, and tell them what you have told them” have now been “transformed” into various formats of video lectures (both mass video lectures and short video lectures with quizzes – like those on Youtube), teaching posts or artifacts, or a combination of face-to-face lectures with online tutorials/quizzes – MC, T/F, and short answer questions, or peer assessments, or eportfolios.

Purpose and Quality of MOOCs

Thanks to Grainne Conole for the post on MOOCs and a New Classification of MOOCs, where she has critiqued thoroughly on quality assurance of MOOCs from a course and instructional design perspective, suggesting an alternative pedagogical framework on both c and xMOOCs.

In this post, I would explore the purpose and quality of MOOCs, relating to some of the quality aspects as discussed by Grainne.

What is the purpose of creating MOOCs?

In this corner: MOOC enthusiasts, envisioning how these large, online courses will increase access to higher education, reduce costs, and reinvigorate teaching and learning.  In the other corner: MOOC critics, anticipating how MOOCs will eliminate meaningful interaction between faculty and students, reduce the quality of learning, and decimate the professorship.

If the purpose of introducing MOOCs is to increase access to higher education, reduce costs and reinvigorate teaching and learning, how would such purpose relate to the mission and vision of Higher Education Institutions?

The mission and vision of institution could be viewed as I shared:

The 3 Ms of MOOCs are Mission, MOOCs and Money.

The fundamental questions boards should be asking include:

  • Why are we online? Is the movement to or expansion of online education consistent with the institutional mission? Does and will it serve and advance the institutional mission? Or is the key issue in the discussion about online education—including any conversations about MOOCs—money?
  • How do we assess quality—that of our own online offerings and those of others, including the MOOCs?
  • What will it take to achieve our objectives in terms of online learning—including human and financial capital, content expertise, the political will to change, and many other concerns?

How is quality defined in MOOCs?

Quality in online education, in particular MOOCs might be defined differently from those quality in classroom education, with a face-to-face teaching environment. What is quality of MOOC from the perspective of educators, learners, and employers?

Quality is defined as conformance to requirements (Philip B. Crosby) (slide on Cost of Quality as Driver of Quality Improvement).  Have the cost of conformance and non-conformance been examined and analysed in MOOCs?  What are the “true cost” of MOOCs in the quality equation?

The questions relating to quality in a MOOC are:

1. Whose requirements are most important to be met?  If MOOCs are for the learners, then the conformance to requirements would likely be decided by the learners.  This is quite challenging, in case of institutions, where quality is defined in terms of the institutional requirements.  From a historical perspective, there are always differing requirements from institutions, employers, educators and learners, and so what is best in quality is seldom agreed upon, especially under an open education and learning environment.

2. How is quality determined in MOOCs? What may be viewed as quality may need to be re-examined in light of changing circumstances, as those requirements, purpose (if quality is defined also as fitness of purpose) are changing rapidly in a complex educational landscape.

As mentioned above, the definition of quality is conformance to requirements, under P. Crosby. The concept of Quality Assurance (QA) is to assure quality through quality prevention, with the goal of zero defects in the system (P. Crosby’s quality concepts).  Quality assurance (QA) has been a subject based heavily on quality policy, quality system and procedures, and instructions in formal education institution. The cost of quality relates to the cost of non-conformance (and conformance) in a quality system.  Whilst all these principles of quality are based on an industrial and business model, they had been transferred to education when institutions started to adopt a strategic management approach towards business education, where quality and values become the business driver and hallmark towards excellence in Higher Education.

The quality audit is part of the QA procedure designed to provide an opportunity to enhance the quality of the system through self, second and third party audits and reviews.  However, such quality audits are normally applicable to formal institution education system, and are seldom used for informal learning or non-formal networks, due to the complexity of open social media and the associated technology used.

QA as applied to MOOCs would then relate to the quality design, delivery and audit based on an institutional model education.  Is QA good enough to ensure and assure quality in education and learning?

Starting in late 1980s, there had been a movement from Quality Assurance towards Total Quality Management in education and Higher Education institutions.

The adoption of Total Quality Management (TQM) has been widely used in industry in the late 70s to 90s in industry, and was considered by Higher Education Institutions back in the 1980s to 2000s.  TQM has since been considered and adopted in some of the Higher Education Institutions as both a philosophy and strategic framework to support the overall vision and mission of the institutions, in the deployment of education policies and procedures, and the involvement of everyone (including administrators, professors, instructional designers, support staff, and even customers (students)) in creating and building quality in the system, thus adding value to the customers and stakeholders.

The challenge with TQM in Higher Education however is apparent (especially with MOOCs) when quality is always defined differently by different customers (both internal and external customers), and the standards or requirements that are stipulated by the education provider and authority.  Such quality becomes nuanced and complicated, when customers’ needs and expectations are changing in a complex education and learning environment such as MOOCs.  Indeed, the original quality definition of: Fitness for purpose could be challenging to both the education providers, stakeholders (Venture Capitalists), education authority, educators (professors, course designers), and learners, as the purpose of designing or delivery of the course could be defined and perceived differently for each of the MOOCs (including c and xMOOCs).

Quality is defined quite differently when it is viewed under the network model, with openness, autonomy, diversity and interactivity being the hallmarks of sustainable networked learning based MOOCs.  Recent researches have highlighted the tensions that are inherent when social networks are fused into a formal education system, as the concept of quality would be dependent on the degree of openness and autonomy one would perceive.

In business settings, quality could be achieved through quality planning, control and improvement.  In the MOOCs, quality is achieved through the design planning, course delivery and continuous improvement.


If we are to apply such business model in Higher Education, the purpose behind the introduction of MOOCs would then be to add value to the education, in particular the learning for the students, community, in order to maximize the social capital, while driving down the total costs of education.  This is a bold, ambitious, risky and ground breaking “innovation” and transformation that no education authority in the history of mankind had tried in breaking through.

We are still yet to agree on what quality is, when it comes to MOOCs (x and c MOOCs), as quality could be defined differently under a formal institutional framework (xMOOC) as compared to an open networked education framework (cMOOC), and that different stakeholders (education authority, administrators, professors, educators, learners, and parents etc.) could also perceive quality based on their learning and experience.

Recommendation for future research

Further research is required to explore how quality, quality assurance and total quality management is achieved when social networks and the technology mediation is fused into formal education system with MOOCs and the sort of pedagogy that would foster the learning in a wide spectrum of MOOCs (i.e. ranging from c and x MOOCs).

Postscript: See this book chapter by Grainne Conole.