Learning Analytics is a rapidly emerging field, asking the question: Can we discern meaningful learning from the digital ‘vapour trails’ that learners leave behind them? The million dollar question this begs is, of course, what do we mean by “meaningful” learning, and what kinds of learning are important for the 21st Century landscape, whose contours are shifting faster than theory and practice can keep up with?
Here’s the podcast and slides from Learning Analytics: Dream, Nightmare or Fairydust? — my keynote at Ascilite 2011, in which I introduce ELLI as one of the promising signposts to the ways in which we can think about analytics for the new learning paradigms needed to prepare for a complex, uncertain world.
OK, I want to launch a thread here which sets out an issue I’ve pondered for a while now, as I’ve understood more and more of the approach to learning power that ELLI embodies, but bringing in some thinking on information visualization, and critically, the underlying assumptions on which these depend. After some good email exchanges with Tim Small and Ruth Deakin Crick from the U.Bristol/Vital teams, it’s worth moving this into the blogosphere because some good stuff is coming out, and we welcome wider views.
We all know how to read a spider Diagram, right?
Spider Diagrams are used widely, providing a visual overview of multidimensional scores, but providing more shape information for the human eye than a simple bar chart. An experienced eye can even categorise a particular genre of profile based on its shape. Two quick examples:
Since the legs of the spider extend out to increasingly desirable scores, more stretch is better. The right-hand example uses red-amber-green shading to drive home this message.
Moving closer to learning and learning power, following in ELLI’s footsteps, TLO have released a tool called BLP Blaze, which generates a “radar chart” around the Four R’s. You can see that they calculate quantitative scores generated from student answers to online quizzes (Reflectiveness 6.9, Reciprocity 8.0, etc):
Which brings us to the ELLI profile, familiar to many of the Learning Emergence network. As a reminder, here are the key meanings of the 7 dimensions:
Critical curiosity: Effective learners have energy and a desire to find things out. They like to get below the surface of things and try to find out what is going on. The opposite pole of critical curiosity is ‘passivity’.
Meaning Making: Effective learners are on the lookout for links between what they are learning and what they already know. They like to learn about what matters to them. The contrast pole of meaning making is ‘data accumulation’.
Dependence and Fragility: Dependent and fragile learners more easily go to pieces when they get stuck or make mistakes. They are risk averse. Their ability to persevere is less, and they are likely to seek and prefer less challenging situations. The opposite pole of dependence and fragility is ‘resilience’.
Creativity: Effective learners are able to look at things in different ways and to imagine new possibilities. They are more receptive to hunches and inklings that bubble up into their minds, and make more use of imagination, visual imagery and pictures and diagrams in their learning. The opposite pole of creativity is ‘being rule bound’.
Learning Relationships: Effective learners are good at managing the balance between being sociable and being private in their learning. They are not completely independent, nor are they dependent; rather they work interdependently. The opposite pole of learning relationships is ‘isolation and dependence’.
Strategic Awareness: More effective learners know more about their own learning. They are interested in becoming more knowledgeable and more aware of themselves as learners. They like trying out different approaches to learning to see what happens. They are more reflective and better at self-evaluation. The opposite pole of strategic awareness is ‘being robotic’.
Clearly there is a quantitative basis to ELLI: answers to ELLI questionnaire items load onto the 7 dimensions, from which is computed the profile’s ‘score’. However, the points are not labelled quantitatively, but qualitatively in relation to oneself: A little like me, Quite like me, Very much like me:
Quite a lot is made in the journal papers of the fact that this is not a ‘score’, but the basis for a conversation with a trained coach/mentor. This is a mirror, reflecting back to learners what they have said about themselves. There is power in the language, in the shape, and in the size. If they don’t like what they see, then it’s because they answered the questions in particular ways. Now, in the course of the conversation, it might become clear that a learner mis-interpreted a question, or had a particular incident in mind when giving a particular answer (the extent to which ELLI profiles are context-dependent is another intriguing question that I want to explore elsewhere!). So, clearly there is a level of interpretive flexibility built into ELLI spider diagrams that one does not encounter in other contexts, where the profile is a linear function computed from a set of objective ‘facts’.
In sum, many who have worked with ELLI would agree with the view expressed by one practitioner that “I think the power of seeing a post-intervention profile that has changed in the right direction is huge for an individual.”
Because we all know what the “right” direction is, right?
On the other hand…
… I have to confess I’m not so sure.
When we are about to see our own ELLI spider diagrams, hands up anyone who doesn’t secretly hope for a nicely rounded profile, not ‘constricted’ on some dimensions, or spiking out in a worryingly unbalanced way?! Rounded and balanced, not shrunken or spikey: the resonance with our metaphorical language for personalities are striking. Which makes them very powerful, but with power comes responsibility.
How excited should we get about pre- and post- test results that demonstrate the blue-to-red ‘stretch’ on dimensions following targetted interventions? It may be that not too much should be read into individual profile shifts, but that aggregate profile changes for a whole class or team can be treated with much more confidence.
Should we worry that a learner has little self-insight? We all know people whose belief they’re rather great at XYZ isn’t shared quite as widely…
Should we worry about a Hawthorn effect, whereby learners who know they’re working on resilience, say, try to answer questions that seem to be about that in a more positive way? We know that ‘subjects’ in experiments are constantly trying to make sense of what’s going on, and that some learners are keen to please the teacher.
Of course, a learner is ultimately only kidding themselves, making authenticity a primary yardstick, which only comes in a trusting relationship with a coach/mentor, and with a degree of insight into oneself. This is why identity and trust are emphasised so much in this work. But this would be a risk if ELLI profiles somehow got tied into workplace performance management, or summative assessment. Then it’s worth gaming the system.
The Dalai Lama’s ELLI Profile
OK, I’m going to force the issue here, just for fun. Imagine that I have the Dalai Lama’s ELLI profile. But before I show you it, you have to take a guess and draw it. Think about what we’ve been saying. It’s a mirror that reflects what you say. It requires insight into one’s own dispositions. It’s possibly context dependent.
Would you draw a totally max’d out profile — stretched on every dimension?
In fact, what does it mean to say that you’re scoring ‘maximum’ on a dimension? Anyone who claimed this would by default be demonstrating impoverished Strategic Awareness…
Now we’re getting to the heart of it. Maybe this would be a really modest profile, because this learner is really very well self-calibrated. As we grow in wisdom and insight, we realise how little we know, how impoverished we are, how much better we could be.
When I raised this with Tim Small (minus the Dalai Lama character!), we got somewhere interesting. Tim shared a brilliant beating heart metaphor — that a profile can actually ‘constrict’ in a healthy way because the learner’s self-awareness is deeper and their vision more expansive. Yes! Less may be more…
My thought from this is that there may be cycles in which one grows up to a point, like reaching a local maximum, but then to get to the next level, you have to descend into the valley, and begin climbing a new, taller peak. At that point, you are going to constrict on one or more dimensions.
To try and summarise, it’s clearly meaningless to talk about being maximally resilient, creative or strategically aware. You’re never going to get to the point where there’s no room for improvement. The key value of ELLI is in providing a language to name important things and dialogue about them internally and with others. Secondarily, it can provide some clues to one’s trajectory, but there seem to be so many subjective variables impacting how a question might be answered, that I personally still hesitate to read that much into the ‘quantitative’ aspect. However, I am a novice when it comes to diagnosing the shape of a profile, and would love to hear more from the veterans on this 🙂
This is clearly complex stuff to communicate. A closing thought is that given the powerful rhetoric of a spider diagram, inherited from its more quantitative roots, I wonder if it’s the best visualization for a domain focused on self-reflection, self-calibration, and meta-cognition? There is no objective reference point. Feels a bit like placing a mercury thermometer in front of someone, but then cautioning them that if the mercury falls, it might actually be getting hotter…
In parallel, we subsequently learned about the ongoing research programme at Bristol, around ELLI, led by Ruth Deakin Crick. Now we’re exploring ways to build on the language for learning that is part of the school’s life, and introduce ELLI as a way of opening up conversations around learning power at the individual pupil level. An initial step has been a simple exercise in which we mapped the BLP “learning muscles” which our staff and students are now familiar with, onto the ELLI spidergram generated from the self-report web questionnaire hosted by Vital Partnerships. We might add in the icons for learning muscles around the spidergram to add more visual engagement as well, similar to the way in which the ELLI team have engaged young people in diverse cultures through the use of the Simpsons characters, and indigenous Australian animals.
The note in the middle reflects the idea that we might audit the school as a whole on its learning power, not just students. We thought we’d share this with you as we embark on this next phase of the journey 🙂
Andrea Curtis (Head) and Simon Buckingham Shum (Chair of Governors)
We are experiencing an unprecedented explosion in the quantity and quality of information available not only to us, but about us. We must adapt individually, institutionally and culturally to the transition in technologies and social norms that makes this possible, and question their impacts. What are the implications of such data availability for learning and knowledge building — not only in established contexts, but also in the emerging landscape of free, open, social learning online?
This conference will be of interest to Learning Emergence readers since we are unquestionably entering the era of data mining, in which machines will be tasked with helping over-pressed humans to make sense of the data deluge. When this comes to learning, we need to make sure that the richness of authentic, connected learning is not lost through over-simplified indicators of “learning” which are deployed simply because they are the easiest things to formalize.
Full details of the topics, keynotes, and ways to participate on the website…