Measurement of effectiveness of cMOOCs

Here is my response to Christina’s post on difficulties researching cmoocs.

How to measure the effectiveness of a cMOOC?

There are 4 semantic conditions of networks that Stephen Downes has proposed. As Stephen has commented, those properties – openness, diversity, autonomy and connectedness & interactivity is not perfect in cMOOCs. Besides Connectivism as applied in cMOOCs could likely best be based on an informal learning, rather than a traditional institutional model.

I have reiterated that the constraints typically imposed with an institutional model would be huge challenge for administrators and educators to adapt, as is witnessed even in xMOOCs, where a totally new approach (such as flipping the class or flipped learning) as perceived by professors would be at odds with the mass lecture approach typical in mass-education, with a broadcasting model. How to overcome those challenges, and ensure learning is more effective, when cMOOCs are embedded in an institutional model?

Here is my response  that I perceive as a way to measure the effectiveness of cMOOCs – in its

1. awareness of Networked Learning and Connectivism as an “informal learning paradigm”,

2. an adoption and leveraging of the 4 properties- openness, diversity, autonomy and connectedness & interactivity when networking,

3. an achievement of personal goals with immersion in the network and community (and community of practice) on personal basis,

4. adoption of Personal Learning Environment and Network PLE/PLN in pursuit of life-long learning, and

5. a shift of frame of reference and paradigm from knowledge transmission to knowledge sharing and creation model under a knowledge ecology.

John Mak

29 thoughts on “Measurement of effectiveness of cMOOCs”

  1. Pingback: Measurement of effectiveness of cMOOCs | CUED |...

  2. Pingback: Measurement of effectiveness of cMOOCs | cMOOC ...

  3. Pingback: Measurement of effectiveness of cMOOCs | Person...

  4. Pingback: Measurement of effectiveness of cMOOCs | Metodo...

  5. Hi John:

    Thank you for responding to my post helping me to think through my questions! I’m very sorry for being so late in replying. You caught me in the middle of a busy month of travel and moving!

    I am having a little trouble finding what to focus on in your long post linked here are your response. There are many things there about connectivism and cMOOCs, but I’m not sure which points you want to use to help explain how we might determine if a cMOOC has been effective. Or, perhaps, are you suggesting that we should look at points 1-5 in this post and ask if participants in a cMOOC are achieving those? If so, how do you think we might do that?

  6. Thanks Christina,

    May I re-post part of my post here?
    “The Connectivist approach to MOOCs is not about content knowledge that is static, but connectivity to the experts, masters, artifacts, peers, networks, either mediated via tools and networks, or in direct contact if one is having face-to-face facilitation and mentoring. It is not about knowing something that makes us an expert of a field, but the ability to construct and navigate the networks with the design of coaching and navigating activities, and configuration of navigation tools (PLE/PLN) etc.

    In my other post:

    At what point could this become connectivist if the actions are on the internet and interaction/development of ideas takes place? https://suifaijohnmak.wordpress.com/2011/10/27/change11-connectivism-and-constructivism-whats-similar-and-different/ We are all connected with our local communities and networks in certain ways, patterns, but with technology as media, and social media as “catalyst” and agents, we are now able to reach different corners of the world, beyond the traditional closed walls (schools, classes) or local groups or communities. The tyranny of space and time could also be overcome with such a connectivist approach. So, whilst constructivist approach addresses the construction of meaning between agents (mainly human, or actor networks), connectivist approach goes beyond that through multiple agents, multiple actor networks, technology and tools, and most important of all, with a basis of openness, autonomy, diversity and connectedness (properties of networks) in order to strengthen the learning.

    For an elaboration on the characteristics of early MOOCs, you will find them in George andStephen’s various posts and other papers here and here. You could also find some papers inmy publications on the right hand side of this blog menu, which documented how the MOOCs were designed, delivered and developed. The MOOCs are evolving and emerging and so they are based on adaptive, self-organising and emergent learning principles, rather than the static prescriptive instrumental learning principles.

    An ideal MOOC to me would likely be distributed over different learning spaces, which again would align with learners’ different and changing needs and goals. As Stephen mentioned the product of learning is the learner, and so the learning is based on a growth model where learner’s growth of “knowledge” and wisdom with the navigation and construction of networks upon time. This also requires pruning of obsolete network patterns (outdated concepts, information, knowledge etc.), with the growing and nurturing of new and emergent network patterns.

    This is also one of the most difficult and challenging part of education and learning, as it challenges the values of traditional canonical knowledge often prescribed in books and are determined by authorities, and are confined to be “delivered” in a closed wall settings. With the rapid changes in information and knowledge landscape, such ways of “transmitting” information and knowledge limited the discourse and inquiry, reducing knowledge to a set of memorable known facts, information, or procedures which, if understood would constitute learning.”

    Measuring the effectiveness of cMOOCs using an institutional framework with learning outcomes could be challenging, though we need to see what sort of knowledge we are focusing on – prescriptive and or emergent knowledge. In my past few researches, I have attempted to evaluate the effectiveness of cMOOCs, and the findings revealed that it is related back to the achievement of goals and “positive” educational and learning experience of the learners, not merely the goals or learning outcomes set off by the education providers. Please see the researches in my publication list of this blog for details.

    This again could be different from the xMOOCs where prescriptive learning outcomes are pre-determined by the xMOOC provider, and so the effectiveness of the course is determined by the achievement of individuals who have passed the course, based on the assessment criteria set by the course provider and professor.

    So, are we trying to measure “emergent knowledge” as typically common in cMOOCs with a “prescriptive knowledge” tool? Would it be like using a round peg (tool for measuring standard learning outcomes) to fit into a square hole (emergent knowledge)? Here emergent knowledge arises from the interaction, engagement with the networks, agents, in a complex learning scenario. Such knowledge is not easily predicted or prescribed, as they may be tacit knowledge which could only be transformed into explicit knowledge through critical thinking, creative thinking or actual project work or problem solving.

    Can you really measure such knowledge and learning effectively with cMOOCs? May be you could measure some dimensions of the knowledge and learning through surveys, interviews, as many researchers and I have done in the past.
    John

  7. The following is not too focused.

    Difficult question John. To me, cMOOCs are a necessary space for speculation and uncertainty in a time rapid knowledge growth. Somewhere there has to be an accommodation of those who necessarily step outside the walls of their discipline to build something in the unknown. This sort of activity would at least require the recombining of things we know in unfamiliar patterns and surely would be disorientating to the person journeying outside their familiar realm.

    How do we measure progress in speculative travels? Where have we gotten to when we are “there”? It seems all exploration requires a time of being speechless and likely a willingness to endure uncertainty and we haven’t got a name for this state making it difficult to assess. There must be a point where a person passes from one understanding to another but how do we mark this departure point from what is known into something new?

    I’m not a math person but might there be a proofing process for idea development? Even the realization of abstract concepts can be tracked by logic to the edge of where it diverges so maybe there is some sort of evidence to be had? Alternately, many new ideas are shunned before becoming mainstream. So how do humans absorb and normalize the unfamiliar in order to work with it and have it become a functional part of reality?

    My sense of mastery is the ability to release “the rules” from a fixed order and recombine them in an intelligent and useful way. This process creates a new ordering that may not be apparent but does progress the field it occurred in. After the fact it may be predictable though before we again find ourselves without measurement strategies.

    It must be that humans can accommodate and thrive in new and changing environments or we would be extinct. I wonder if we could look at cMOOCs as a kind of doorway to new thinking. That sounds presumptuous yet there must be some method of releasing the rigid hold of the known for us to move on?

    Scott

  8. Pingback: Measurement of effectiveness of cMOOCs | e-lear...

  9. Pingback: Measurement of effectiveness of cMOOCs | Librar...

  10. Well said, and as you have elaborated: I wonder if we could look at cMOOCs as a kind of doorway to new thinking. Not everything could be measured, and only some ideas would lead to innovation, as we have experienced with the use of emergent tools, whilst other ideas and tools are discarded as we learnt more through collective inquiry and critical analysis in our community. That may go beyond what a typical curriculum course (MOOCs) defines. Thanks Scott for your sharing of insights.

  11. We might not even be having this conversation were education to be seen in multiple dimensions rather than the tendency to rely on one version. For instance, the institutional need to account by numbers to the overseeing crediting and funding bodies seems to clash with the freedom to experiment in realms not directly countable (at this time). Because an institution’s first mandate to is survive each day to work tomorrow exploration becomes an element of risk that might be sensibly avoided by a rational administrator.

    It could be the mechanism we have built to probe and prepare the future can’t be the same one that trains and certifies my cardiac surgeon to work on me today. Can we attend to the right now and risk the consequences of missing something because we are also focused on a speculated future? Alternately, is it useful to be freed of consequence to play as we wish.

    I wonder how free is free enough to grow while still “owning” what we might cause? Because we are responsible the things we do matter to us and do restrain us by choice yet we must also attend to what we can’t predict with certainty. These qualities both hold us and free us but are they measured? By intensity of belief? By commitment? By progress that benefits all while harming none?

    Can we have an easier question next time John?
    Scott

  12. Great questions. Interesting to reflect on risk. Risks are part of the game in online education and MOOCs. When calculated risks are taken, we could come up with plenty of opportunities. Indeed, the risks associated with MOOCs could be one of the areas missing in most conversation, as there still aren’t any “viable” and “sustainable” business model where MOOCs could be based upon. This “procrastination” in having a model of business with MOOCs (xMOOCs) could become a high risk, especially for those HE institutions and faculties who are at risk, either in losing out or missing out in this game of MOOCs, or in leveraging technology, and thus not having enough funding or students for their courses. People are responsible for hosting the MOOCs, and the accountability to top within institutions and MOOCs, and risk management needs to be strategically aligned and developed along with their vision and mission. I still haven’t seen many academic papers dealing with the risks of MOOCs in a thorough and objective way (based on learning analytics). I think I may try writing a paper on this topic, though this may be on the plate of some PhD students, scholars or researchers. Just wonder! Let’s evaluate the risks associated with MOOCs – both c and x MOOCs.
    What sort of questions do you want?

  13. Hi John:

    Thank you for clarifying with your comment, above–I understand better what you’re getting to now. I agree that most current evaluation strategies focus on objectives created by instructors, and don’t easily fit into situations where knowledge is “emergent.” That’s part of why I’m struggling with this issue. I really appreciate your point about this emergent knowledge being possibly tacit for awhile, so even the participants in a cMOOC might not recognize they’re getting it until it emerges more clearly through their work in the MOOC, or even later, outside of it. And I agree that learning in a cMOOC is a very complex process that can’t easily be quantified/measured.

    And yet, I do think it’s important to try to discover what sorts of things help such knowledge emerge for participants, which ways of doing cMOOCs work better for this than others. So I think it’s an important issue, and that’s why I’m asking about it.

    And Scott, I really like your point about cMOOCs being a space (among other spaces that do this, perhaps) for experimenting, for going beyond what one knows, for generating new ideas. And that this can involve being speechless and uncertain for a time. Which is valuable, and yet difficult to measure AS valuable. Indeed, it’s easy to say in a cMOOC that if people are being quiet, not writing blog posts, not doing projects, that somehow they are not engaged and it would be better if they were. So sometimes I see people suggest that if there is not a high percentage of “active” participants in a cMOOC, then this is a problem that could be remedied. And I get that, because really, in a cMOOC the learning comes mostly from participants talking/creating/doing with each other, so if hardly anyone is talking or creating or doing, then there isn’t much happening in the cMOOC. But still, being silent and uncertain can be a good thing, a necessary thing.

    I just find this issue really hard to tackle. I want to be able to say that cMOOCs work, that they are good things and thus convince institutions to support them and people to take them, and I want to be able to have a decent sense of how to run one successfully before I try doing so, but beyond anecdotal evidence from people (like me) who have been active and found them very rewarding, I don’t have much in the way of evidence. And I’m still struggling with how to get it.

  14. Pingback: Comment on Measurement of effectiveness of cMOO...

  15. Hi Christina, I couldn’t agree more on what you commented. May be we are now “awaken” or have “awaken” the giants, and so it seems that our shift of reference and focus have changed from the mere instrumental learning to more “reflective and transformational” learning that forms the basis of emergent learning. It is not easy to convince institutions to support such learning, as every course and curriculum must be prescribed in explicit terms, accredited and approved before they are designed and implemented. There is little to no input from the actual learners in re-defining the course, in terms of the learning outcomes, objectives, performance criteria, as they are all emergent under a complex learning scenario, where many of us might have different set of goals, and learning objectives. So, what are the options? Should we allow c and x to co-exist in a MOOC (in terms of pedagogy, or a higher form of pedagogy that support human learning)? May be our institutions would like to experiment with different approaches to those pedagogy, though still like to have the “management and control” over the course. Perhaps, if the MOCOs are run with the ultimate goal of creating knowledge, then it would look totally different, and may be more aligned with the early schools of enlightenment in learning, helping and supporting people to achieve their highest potential (and learning) without the constraints of a set criteria, but emergent criteria of capability and competency – the literacy and capacity of a future learner.
    Thanks Christina again for your wonderful insights.

  16. In complexity science the flocking of birds and the schooling of fish are not predictable in the individual but emerge in very obvious ways in groups. Could we measure a cMOOC outcome in the form of synchronicity in the behaviours of its participants before and after the course? A form of contagious connection like an outbreak of an infection? Or maybe a before and after on the ability to navigate a maze crafted to indicate a higher level of success in extracting information from a test environment?

    In evidence of learning we are looking for something that wasn’t there before. If it appears in in common form in multiple individual participants in a course there exists a suggestion of some common cause. This suggests to me dance as a metaphor for Connectivist synchronization. Sources:

    Emily S Cross academia. Edu page: http://bangor.academia.edu/EmilySCross
    Link “Research Interests” lower left of page then “Dance and the Brain” goes to:

    Linked from Emily S Cross personal bio site. People in the field of movement study.
    http://www.academia.edu/People/Dance_and_the_brain

    Scott

  17. Helping people to build the capacity to learn by themselves might be seen by current institutional structures and short sighted instructors as “putting themselves out of a job” resulting in refusal to sponsor MOOCs. Yet it has always been the job of education to step in and then fade into the background as people develop skills to carry them through life–and maybe even to exceed the sum of the input they received in the classroom to progress knowledge even further. People becoming more than they were has never been a problem for educators in the past. In fact it may be said to be their goal. So I wonder if the resistance to MOOCs is more a misunderstanding of the teaching role or maybe even a panic over the creation of new disciplines without the birthing services and approval of academia? People are just running off in all directions having unsupervised ideas without permission!

    cMOOCs could represent a leap in the logic of learning past our current ability to explain or measure and, as both John and Christina mentioned above, measurement is important. Measurement is how we know something has been changed or transformed. It represents objective and viewable-by-all evidence that can be discussed, disputed and worked with. I’m tempted to say that measurement in the case of might not be appropriate, except that’s a call to create a whole new argument. So how do we observe emergence in a concrete way? Those of us who support cMOOCs as a learning device need to know this or we’re left with taking it on faith.

  18. Hi Scott and Christina, Emergent learning could be measured, and both Jenny and Roy and their colleagues have developed a model of footprint of emergent learning that captures the essence in their researches. To me, that is a fantastic effort towards featuring the complexity of learning via the lens of emergence. In our previous researches, we did find traces and footprints of emergence in various forms, though it is based on a complex adaptive and dynamic system, evolving and morphing along an ever changing trajectory, as the learners gain more experiences with cMOOCs. It is also based on a different set of learning strategies, like metacognition, distributed cognition, sensemaking and wayfinding, which have all philosophical though pragmatic stances, and could be evaluated with meta-narratives and those “projects and problems” solved or achieved, and the individual goals achieved. I would likely write a post looking into such a model of emergence soon.
    Thanks.

  19. Thanks John, I’ll go back and read the Footprints paper. I understand the difficulty that Christina faces promoting the idea of cMOOCs into the already complex realm of education. By necessity, higher education “packages” learning into time limited curriculum to reach specified qualifications and may find it difficult to present undirected and open-ended discovery. How do we qualify someone who is on an unending journey of self when we seem so comfortable with achievement of ends? Do we have secular language to explain or even allow continuously becoming?

  20. A short response to your questions: yes we do. In our institute, we are having lots of professional development activities offered to faculty staff. They are not designed for the acquisition of qualifications, but are extremely valuable for continuous life-long and wide learning that are relevant to work. Such emergence of learning is ongoing, when we continue our exploration both inside and outside our institutions, through various community involvement and social networking. The notion of a “course” may need to be re-visited.
    John

  21. Unfortunately where I live learning opportunities for staff are usually pre-packaged events chosen by either the poorly trained human relations staff or faculty development which is frequently without staffing. Most faculty have allowances in their pay package for development but maybe 5% use it each year. My impression is that most people are incurious and feel that having been hired is proof they “know enough” and can stop learning. I actually have no interest in this type of person but am very interested in the creating, use and adaptation of knowledge as a vital process of social and personal development that makes cMOOCs attractive to me. Having taken hundreds of credit hours at the college extension level I can’t understand why people would ever even want to declare themselves educationally complete. That attitude of mental maturity stunts growth of the person and I don’t know it exists unless it has been taught into us. Is learning strictly a social phenomenon or is it an evolutionary trait? Maybe life-long-learning is a mutation trying to emerge and only expressed in a few of us?

  22. Hi John and Scott:

    I missed some good discussion yesterday! I’ll just reply to a couple of things.

    John, your point about learning objectives often not being created with the input of learners is an important one. Why should learning always happen on the agenda of the instructor only? Of course, when you’ve got hundreds or thousands of people in a course (hundreds in the case of on-campus or online, thousands in the case of online–unless there are campuses with thousands of students in a room?!), it’s hard to generate an agreed-upon set of learning objectives. So then you’re left with everyone forming their own for themselves. Which I think is a good thing, but institutionally it’s a bit tough when you are being told over and over that you’re supposed to have a set of common learning objectives in your syllabus, for each class meeting, each assignment, etc. It’s hard to argue against this tide. Perhaps more data/research publications would help!

    I haven’t read the footprints paper you’re talking about; is it this one? Williams, R., Mackness, J. & Gumtau, S. (2012) Footprints of Emergence. Vol. 13, No. 4. International Review of Research in Open and Distance Learning.

    If so, I’ll get on that right away. Jenny Mackness left a few references to her work on my blog, and I haven’t yet read them!

    And Scott, I find your point about patterns emerging in groups like movements in flocks of birds very interesting. That’s a possible way to go, though I wonder if we could say that a cMOOC has still been effective even if people do very different things afterwards, and don’t achieve an observable synchronicity? This being because they have different reasons for doing the course, did it in different ways, got different things out of it, etc.

  23. As Stephen mentioned in his post relating to a critique on Connectivism and cMOOCs :”First of all, as I have so often argued, connectivism is not constructivism – there is no obligation for the educator to be absent, and he or she should feel free to ‘teach’, to ‘preach’, or make their presence felt in the course as much as they wish, the only limitation being that this presence is as a participant and not an authority figure.

    But more to the point, if people come to depend in general on educators, and only educators, for reassurance and encouragement, then they will be sorely unprepared for life. One of the ways a connectivist course becomes massive is that it eliminates the psychological dependence on the instructor and encourages participants to learn to depend on each other for these necessary supports.” I have shared that in my post on the differences in learning theory.

    Under Constructivism, the four persons will communicate with each other, and share their understandings, feelings, and knowledge, experience, and then come up with new knowledge based on the re-construction of the knowledge each possesses. Under a constructivist approach, the teacher may become the facilitator, and the four persons are encouraged to interact, exchange views and experience and co-construct meaning and knowledge that is based on their needs (still with the teachers’ intervention) under a learning environment (LMS or e-learning in a course)

    Under Connectivism, the four persons will connect their thoughts, their understanding at neural, conceptual and external, social level with information sources, formally or informally. They will also link with others who have experience with elephants – communities, networks and experts. Under a connectivist approach, the pipe (the connections) is more important than the content (as content may keep changing, and needs to be updated to ensure “correctness” or “validity”). The four persons (may act as peer teachers and learners) encourage each other to be involved in networks, internet surfing and navigating, and make use of their sensemaking (metacognition skills – thinking how to think) , patterning (knowledge recognition), and way finding (identifying their goals and mission through those networks and community involvement) and realising the emergent knowledge (ontology – learning to be) through an integration of informal learning with their formal education. This assumes that the four persons are motivated to learn the skills required to communicate, collaborate and cooperate over the net environment.

  24. Hi Christina, thinking about the flocking of birds it occurs to me that the phenomenon is based on a monolithic population and dancers work under the watch of choreographers who are know to be strict interpreters of performance. One theory down and one level up in complexity we have in MOOCs the added element of a diversity of students all receiving and procession in individual ways in a model that encourages autonomy and doesn’t even try to herd students into the norm of an outcome.

    Reading about mirror neurons as a mechanism of imitation that allows us to synchronize our interaction with others brought me to the dance metaphor. But having a tool for understanding implies that we display objective commonalities to understand each other yet reside in subjective worlds of our own that can’t be viewed by others. To me, the value of cMOOCs is they encourage individual voice and sense making (as John has said) in an often confusing or novel environment where what is already known may NOT help you understand what’s going on. They force you to construct ways of describing the unknown to others–a way of bringing them into an agreed upon reality of social knowing and sharing which I see as an artifact and pay-off for openness. It’s uncomfortable to reside in this state of uncertainty but how else do we progress or see potential in the unfamiliar? In cMOOCs at least we aren’t alone in this process.

    Can we measure the ability to process abstractions? What happens in our brains when we encounter to unknown and chose not to run from it? I like this quote even if it doesn’t help me yet:

    “…much of modern life involves understanding systems that lie well beyond one’s individual context. In these circumstances, inferential strategies that essentially rely on individual knowledge and experience can lead to very misleading conclusions. In the absence of specific experience and knowledge, abstract reasoning can potentially allow someone to evaluate the validity of inferences by being able to understand the logical structure of the relationships in question. In fact, of course, this kind of reasoning is what has allowed the construction of fields of knowledge which lie well beyond specific human experience and action. Attempting to understand even concrete implications of these highly abstract domains of knowledge is essentially futile without some basic advanced reasoning skills.”

    p.73 Development of Abstract Conditional Reasoning by Henry Markovits in: “The Development of Thinking and Reasoning” ed. Pierre Barrouillet and Caroline Gauffroy. Psychology Press Taylor and Francis Group 2013.

    Scott

  29. Pingback: Measurement of effectiveness of cMOOCs | MOOC-C...

Leave a comment