OK, I want to launch a thread here which sets out an issue I’ve pondered for a while now, as I’ve understood more and more of the approach to learning power that ELLI embodies, but bringing in some thinking on information visualization, and critically, the underlying assumptions on which these depend. After some good email exchanges with Tim Small and Ruth Deakin Crick from the U.Bristol/Vital teams, it’s worth moving this into the blogosphere because some good stuff is coming out, and we welcome wider views.
We all know how to read a spider Diagram, right?
Spider Diagrams are used widely, providing a visual overview of multidimensional scores, but providing more shape information for the human eye than a simple bar chart. An experienced eye can even categorise a particular genre of profile based on its shape. Two quick examples:
 Usability Spider Diagram |
 Lubrication Safety Spider Diagram! |
Since the legs of the spider extend out to increasingly desirable scores, more stretch is better. The right-hand example uses red-amber-green shading to drive home this message.
Moving closer to learning and learning power, following in ELLI’s footsteps, TLO have released a tool called BLP Blaze, which generates a “radar chart” around the Four R’s. You can see that they calculate quantitative scores generated from student answers to online quizzes (Reflectiveness 6.9, Reciprocity 8.0, etc):
TLO's "BLP Blaze" radar chart
Which brings us to the ELLI profile, familiar to many of the Learning Emergence network. As a reminder, here are the key meanings of the 7 dimensions:
Critical curiosity: Effective learners have energy and a desire to find things out. They like to get below the surface of things and try to find out what is going on. The opposite pole of critical curiosity is ‘passivity’.
Meaning Making: Effective learners are on the lookout for links between what they are learning and what they already know. They like to learn about what matters to them. The contrast pole of meaning making is ‘data accumulation’.
Dependence and Fragility: Dependent and fragile learners more easily go to pieces when they get stuck or make mistakes. They are risk averse. Their ability to persevere is less, and they are likely to seek and prefer less challenging situations. The opposite pole of dependence and fragility is ‘resilience’.
Creativity: Effective learners are able to look at things in different ways and to imagine new possibilities. They are more receptive to hunches and inklings that bubble up into their minds, and make more use of imagination, visual imagery and pictures and diagrams in their learning. The opposite pole of creativity is ‘being rule bound’.
Learning Relationships: Effective learners are good at managing the balance between being sociable and being private in their learning. They are not completely independent, nor are they dependent; rather they work interdependently. The opposite pole of learning relationships is ‘isolation and dependence’.
Strategic Awareness: More effective learners know more about their own learning. They are interested in becoming more knowledgeable and more aware of themselves as learners. They like trying out different approaches to learning to see what happens. They are more reflective and better at self-evaluation. The opposite pole of strategic awareness is ‘being robotic’.
Clearly there is a quantitative basis to ELLI: answers to ELLI questionnaire items load onto the 7 dimensions, from which is computed the profile’s ‘score’. However, the points are not labelled quantitatively, but qualitatively in relation to oneself: A little like me, Quite like me, Very much like me:
Spider Diagram generated by ELLI: Effective Lifelong Learning Inventory. Blue profile is is pre-intervention, red profile is post-test.
Quite a lot is made in the journal papers of the fact that this is not a ‘score’, but the basis for a conversation with a trained coach/mentor. This is a mirror, reflecting back to learners what they have said about themselves. There is power in the language, in the shape, and in the size. If they don’t like what they see, then it’s because they answered the questions in particular ways. Now, in the course of the conversation, it might become clear that a learner mis-interpreted a question, or had a particular incident in mind when giving a particular answer (the extent to which ELLI profiles are context-dependent is another intriguing question that I want to explore elsewhere!). So, clearly there is a level of interpretive flexibility built into ELLI spider diagrams that one does not encounter in other contexts, where the profile is a linear function computed from a set of objective ‘facts’.
In sum, many who have worked with ELLI would agree with the view expressed by one practitioner that “I think the power of seeing a post-intervention profile that has changed in the right direction is huge for an individual.”
Because we all know what the “right” direction is, right?
On the other hand…
… I have to confess I’m not so sure.
When we are about to see our own ELLI spider diagrams, hands up anyone who doesn’t secretly hope for a nicely rounded profile, not ‘constricted’ on some dimensions, or spiking out in a worryingly unbalanced way?! Rounded and balanced, not shrunken or spikey: the resonance with our metaphorical language for personalities are striking. Which makes them very powerful, but with power comes responsibility.
How excited should we get about pre- and post- test results that demonstrate the blue-to-red ‘stretch’ on dimensions following targetted interventions? It may be that not too much should be read into individual profile shifts, but that aggregate profile changes for a whole class or team can be treated with much more confidence.
Should we worry that a learner has little self-insight? We all know people whose belief they’re rather great at XYZ isn’t shared quite as widely…
Should we worry about a Hawthorn effect, whereby learners who know they’re working on resilience, say, try to answer questions that seem to be about that in a more positive way? We know that ‘subjects’ in experiments are constantly trying to make sense of what’s going on, and that some learners are keen to please the teacher.
Of course, a learner is ultimately only kidding themselves, making authenticity a primary yardstick, which only comes in a trusting relationship with a coach/mentor, and with a degree of insight into oneself. This is why identity and trust are emphasised so much in this work. But this would be a risk if ELLI profiles somehow got tied into workplace performance management, or summative assessment. Then it’s worth gaming the system.
The Dalai Lama’s ELLI Profile
OK, I’m going to force the issue here, just for fun. Imagine that I have the Dalai Lama’s ELLI profile. But before I show you it, you have to take a guess and draw it. Think about what we’ve been saying. It’s a mirror that reflects what you say. It requires insight into one’s own dispositions. It’s possibly context dependent.
Would you draw a totally max’d out profile — stretched on every dimension?
In fact, what does it mean to say that you’re scoring ‘maximum’ on a dimension? Anyone who claimed this would by default be demonstrating impoverished Strategic Awareness…
Now we’re getting to the heart of it. Maybe this would be a really modest profile, because this learner is really very well self-calibrated. As we grow in wisdom and insight, we realise how little we know, how impoverished we are, how much better we could be.
When I raised this with Tim Small (minus the Dalai Lama character!), we got somewhere interesting. Tim shared a brilliant beating heart metaphor — that a profile can actually ‘constrict’ in a healthy way because the learner’s self-awareness is deeper and their vision more expansive. Yes! Less may be more…
My thought from this is that there may be cycles in which one grows up to a point, like reaching a local maximum, but then to get to the next level, you have to descend into the valley, and begin climbing a new, taller peak. At that point, you are going to constrict on one or more dimensions.
To try and summarise, it’s clearly meaningless to talk about being maximally resilient, creative or strategically aware. You’re never going to get to the point where there’s no room for improvement. The key value of ELLI is in providing a language to name important things and dialogue about them internally and with others. Secondarily, it can provide some clues to one’s trajectory, but there seem to be so many subjective variables impacting how a question might be answered, that I personally still hesitate to read that much into the ‘quantitative’ aspect. However, I am a novice when it comes to diagnosing the shape of a profile, and would love to hear more from the veterans on this 
This is clearly complex stuff to communicate. A closing thought is that given the powerful rhetoric of a spider diagram, inherited from its more quantitative roots, I wonder if it’s the best visualization for a domain focused on self-reflection, self-calibration, and meta-cognition? There is no objective reference point. Feels a bit like placing a mercury thermometer in front of someone, but then cautioning them that if the mercury falls, it might actually be getting hotter…
Simon Buckingham Shum
Nice one Simon! And I like Tim’s heartbeat/pulsing metaphor, very much. And how liberating, in fact, the notion is that it might be getting hotter when the mercury falls: it means, in effect, use your senses, your own judgement and feelings and intuition rather than relegating these powerful elements in response to an external “measure”. If anything, this reinforces all the more the essential humanity of ELLI and its mission of authenticity, contrary to many discourses of personal growth and learning which present images of linear travel. And I also like the fact in disrupting such neatness, ELLI does what it does best and most powerfully: it promotes searching, reflective conversation.
Keep up the fight.
Mark
Perhaps its the challenge of the new renaissance, to be able to move between differing forms of truth claims…so the ELLI spider is a one off reflection of an individual’s responses to questions about themselves. And I agree with Mark that the individual receiving it should author their own response…in fact I think authenticity can have two meanings – authorship – and real and meaningful. One spider diagram, one person, idiosynractic and unique. Truth claims interpreted by the individual – hermeneutical and emancipatory. But what about 50 spider diagrams? or what about the claim on one spider diagram that certain groups of questions reflect something we call ‘creativity’ or ‘critical curiosity’…..well to me that’s where the same data enables us to apply another form of rationality – analytical and strategic. So with over 50,000 individual spider diagrams in the learning warehouse we get the chance to engage in a different rationality – using statistics. So, with ELLI, for example, when an individual reflects on their spider diagram – they can be confident that there is something we can be reasonably sure of (don’t like the word objective – prefer pensiero debole after Vattimo) that is being measured by those questions that is reflected in the population as a whole and we call it crtical curiosity and we can describe it. And this is because of the numbers. And we can make generalisable claims on the back of it – we know, for example, that girls are likely to have more creativity than boys, and underachieving students have less strategic awareness, and Aisian students have higher scores on all dimensions than white English….
Tools like BLAZE, simply don’t have this analytic power – indeed one wonders if you might as well use a horoscope….since analysis has shownt that there is no factor structure behind it. It just reflects someone’s opinion or good idea….and there you have it…Simon’s concerns are really important.
The new renaissance requires us to be able to move between differening forms of rationality, knowing what we can reasonably be sure of in what context….and that’s what I find exciting about raw data that produces feedback that serves four purposes, all emancipatory – the feedback is used for individuals, teams, organisations and system wide analysis.
So I would prefer to recieve a spider diagram which I know has some solid science behind it, even if I choose, for whatever reason, to ignore it.
Hi All,
Thought you might be interested to hear that Toyota use a spider diagram to look at the learning that an individual and team gets during a Kaizen (improvement) project. They have – yes – 7 dimensions, which are:
- Knowledge
- Communication
- QC Tools
- Teamwork
- Quality Mind
- Ownership
- Problem Solving
What was particularly interesting to discover is that the ‘winning’ Kaizen project is NOT the one with the best ROI or impact on safety….but one that ALSO shows that the members of the team participated, contributed – and learned!
How cool is that? It struck me that a number of their points are about meaning-making, learning relationships and strategic awareness. They don’t directly measure resilience on that scale – but they do look at whether people took part in the project and continued when it was difficult. They particularly value it when people find PROBLEMS and continue onwards as a team to overcome the problems. This is seen as very good.
Their view of ‘problems’ is not like the standard (something to hide, brush under carpet view) – but, rather, a reason for an exciting inquiry! Including using the 5 Whys and other useful tools. Which makes me wonder whether any of their ‘stuff’ might be useful in relation to scaffolding inquiries in learning? And reminds me again that we really must get David Langford over to speak soon!
Best,
Ems x
PS Short blogppost about visit to Toyota yesterday is available here: http://progpart.wordpress.com/2011/11/28/dont-bring-me-solutions-bring-me-problems-the-truth-about-toyota/