Scaffolding deep reflection with automated feedback?

We’ve all got used to the idea that computers can understand writing and speech to some degree — Google adverts that match your search queries… asking Siri simple questions… IBM Watson winning Jeopardy! But how does natural language processing fit into learning that goes beyond getting the right answer to a focused question, or matching some key concepts?

Language is clearly front and centre in the way that we learn from others, share our understanding, and narrate to ourselves. However, the idea that computers have any substantive contribution to make to the teaching and assessment of writing elicits strong reactions from educators, and understandably so (learn more from this workshop).

In recent work that we’ve been doing at University of Technology Sydney, we’re exploring to what extent text analytics could support the integrative practices of writing that many are now using to help students reflect on their learning. This is particularly germane to future-oriented pedagogy that seeks to help the learner narrate their own learning journey in relation to their identity (how does my learning change who I think I am becoming?), place them in authentic (often professional) learning contexts (on the ward; in the classroom; in a company), and so forth.

As we explain in this article, there are many pedagogical reasons for valuing deep, academic reflective writing, but it poses significant challenges to educators to teach this well, and to students for whom this is often a new genre of writing. Below is the presentation we gave earlier this year. Current the Academic Writing Analytics (AWA) web tool is available only within UTS as we pilot it, but in the future this may become available to other universities, and beyond that, to schools interested in collaborative research.

Buckingham Shum, S., Á. Sándor, R. Goldsmith, X. Wang, R. Bass and M. McWilliams (2016). Reflecting on Reflective Writing Analytics: Assessment Challenges and Iterative Evaluation of a Prototype Tool. 6th International Learning Analytics & Knowledge Conference (LAK16), Edinburgh, UK, April 25 – 29 2016, ACM, New York, NY. Preprint:

Learning Dispositions + Authentic Inquiry in a Primary School

What happens when you turn a curriculum topic over to 10-11 year old children,  give them freedom to choose their focus, and increasing autonomy to make their own decisions to design, create and run a showcase event? Indeed, how do staff cope with stepping back like this? If Ofsted inspectors were to walk in, how could the school evidence learning? How can you evidence the development of lifelong learning dispositions, and how does this relate to the school’s strategic concerns about the progress of different pupil groups on traditional attainment measures? What roles do social learning tools like reflective blogging have to play?

This movie provides a brief glimpse into a two year series of pilots at Bushfield School, documented in more detail in this report. It represents the convergence of both University of Bristol and Open University research and development into learning analytics that can evidence processes associated with deeper learning, especially dispositional analytics (learn more: replay talk / workshop).

(See Reports for the entire library of school case studies.)

Screen Shot 2014-04-25 at 15.43.46Small, T., Shafi, A. and Huang, S. (2014) Learning Power and Authentic Inquiry in the English Primary Curriculum: A Case Study, Report No. 12, ViTaL Development & Research Programme, University of Bristol. [pdf]

This report documents progress in a two-year action-research programme at Bushfield School, Milton Keynes, with two main purposes: firstly, to build on the School’s success in developing children’s capacity to learn; secondly, to track and measure the impact of its interventions for this purpose. The school combined the Effective Lifelong Learning Inventory (ELLI) with the Authentic Inquiry learning methodology from University of Bristol. Qualitative and quantitative data are combined to examine the impact of the pilots from the perspective of staff and pupils, comparing learning power against a range of demographic and attainment datasets, in the distinctive context of a primary school already experienced in the Building Learning Power approach.

Assessing learning dispositions/academic mindsets

A few years ago Ruth and I spent a couple of days with the remarkable Larry Rosenstock at High Tech High, and were blown away by the creativity and passion that he and his team bring to authentic learning. At that point they were just beginning to conceive the idea of a Graduate School of Education (er… run by a high school?!). Yes indeed.

Screen Shot 2014-02-28 at 16.56.56Now they’re flying, running the Deeper Learning conference in a few weeks, and right now, the Deeper Learning MOOC [DLMOOC] is doing a great job of bringing practitioners and researchers together, and that’s just from the perspective of someone on the edge who has only managed to replay the late night (in the UK) Hangouts and post a couple of stories. Huge thanks and congratulations to Larry, Rob Riordan and everyone else at High Tech High Grad School of Education, plus of course the other supporting organisations and funders who are making this happen.

Here are two of my favourite sessions, in which we hear from students what it’s like to be in schools where mindsets and authentic learning are taken seriously, and a panel of researcher/practitioners:

Week 7 coming up is focused on Assessing for Deeper Learning, and I’m very much looking forward to hearing from the panellists. So in this post, I’d like to share some thoughts about how we go about assessing what some of us call lifelong learning dispositions, while others refer to academic mindsets, and welcome comments from DLMOOCers and others.

First, why do we want to assess mindsets?

In education, assessment serves many purposes, which when confused can lead to dysfunctional assessment regimes. See the Assessment Reform Group resources for a helpful summary of this. So, the reasons for assessing shapes the forms that it should take.

“It is helpful to make a distinction, here, between the intended use, or uses, of assessment data, and their actual uses. Assessments are often designed quite differently to ensure their fitness for different purposes. Or, to put it another way, results that are fit to be used for one particular (intended) purpose may not be fit to be used for another, regardless of whether they are actually used for that additional purpose. We must therefore consider carefully the ways in which assessment data are actually used. Paul Newton has identified 22 such uses.” (Assessment in Schools: Fit for Purpose?, p.7)

[p.8] “For the purposes of further discussion, this commentary simplifies current uses of assessment by clustering them in three broad categories.

  1. The use of assessment to help build pupils’ understanding, within day-to-day lessons.
  2. The use of assessment to provide information on pupils’ achievements to those on the outside of the pupil- teacher relationship: to parents (on the basis of in-class judgments by teachers, and test and examination results), and to further and higher education institutions and employers (through test and examination results).
  3. The use of assessment data to hold individuals and institutions to account, including through the publication of results which encourage outsiders to make a judgment on the quality of those being held to account.”

Improve mindsets to boost conventional outcomes

It might be that the goal is primarily to change teacher practice, in order to then see measurable impacts of a conventional sort (stronger engagement through increased attendance; higher grades; increased college admissions). Mindsets themselves are not the object to be assessed or quantified, they are a means to better outcomes that we all agree are worthy (Types 2 and 3 above).

New ideas like mindsets may need to justify themselves in terms of the current order of merit, but we should note also that they may be in tension. If learners are growing in their curiosity and creativity, they may also be less and less inclined to perform standardized tasks which have no meaning, or sit highly artificial exams. So this perhaps moves us to a deeper rationale and more transformative vision for mindset assessment.

Make mindsets a first class outcome that we want to measure

The argument for mindsets seems to me to go beyond just improving conventional outcomes, and seeks to elevate them to ‘first class’ status as a transferable, improvable set of qualities that learners will need for lifelong learning in a complex world. In education, if you can’t measure it, no-one in policy will take any notice of it as something to care about, so we need robust, evidence-based forms of assessment, and intervention strategies. This might expand mindset assessment into Type 1 above, because we want them to be an explicit object of inquiry and conversation.

If we can develop rigorous, evidence-based ways to assess them, then we could help shift the mainstream educational paradigm, and associated assessment regimes dominating schools (and post-secondary/tertiary) that are constrained significantly by the need to assess only what can be measured at scale, quantitatively. That’s the 1-line version, expanded a bit in the intro movie to a Dispositional Analytics workshop we ran at Stanford last summer.

3 approaches to mindset assessment

Let me propose three approaches to mindset assessment, which have different qualities, and tradeoffs:


  1. Heuristics
  2. Self-Diagnostics (formal and informal)
  3. Behavioural Analytics


Heuristics are what we use as reflective practitioners who are learning to recognize mindsets as they manifest in learner behaviour. They’re situated and grounded in real life practice, and based on our developing understanding of what mindsets look like when they are displayed by a learner whom we know and interact with, in a context we know. We’ve learnt from talking with colleagues, reading articles, engaging in courses like DLMOOC, or perhaps going on more formal training in a specific approach. We may also have developed interventions to help coach them. The knowledge is a mix of personal and social, tacit and explicit, and the quality of diagnosis and intervention necessarily varies from one individual to another. It might be transformative, but it may also be patchy in quality, and critically, it’s not possible to evidence that growth very easily in objective, quantifiable terms—e.g. to sceptical colleagues, parents who question it, policymakers wondering if they should pay attention to this, or (gasp) even the learners themselves who want to know how they’re doing!

Self Diagnostics

Self Diagnostics are based on the premise that the learner is the world authority on how s/he experiences his or her learning, and so self-report is a legitimate window into the learner’s mind. Given the links between mindset and profoundly personal, emotional elements such as sense of identity and purpose, asking the learner how it feels for them seems a legitimate route to take.

Self-Diagnostics divide into what might be termed informal and formal.

Informal Diagnostics

…provide a representation of some sort (physical or digital) inviting the learner to express how they experience learning. A good representation externalises an interpretation of the world. The very process of designing or contributing to it can shape cognition. The affordances of a good collective representation directs the attention of the audience to emerging patterns that can then provoke reflection and conversation.

Here’s a wallchart where pupils at my local primary Bushfield School mark on a scale of 1-4 how they feel they’re doing with respect to Managing Distractions, one of about 30 dispositional ‘learning muscles’ in the Building Learning Power framework. [learn more about Bushfield’s use of Learning Power]

Screen Shot 2014-02-28 at 15.20.50

Screen Shot 2014-02-28 at 15.21.04

The screenshot below from the EnquiryBlogger tool shows the use of learner narrative in blog posts to reflect explicitly on their learning dispositions, as expressed in animals representing the 7 dimensions of learning power in ELLI (see below). [watch a talk about this]

Screen Shot 2014-02-28 at 15.55.45

Formal Diagnostics

Formal Diagnostics are provided by tools which elicit learner self-report through a Lickert scale. Example clips from such tools would include the Stanford PERTS Lab survey:

Screen Shot 2014-02-26 at 17.50.15

Mindset Works:


…and the Effective Lifelong Learning Inventory (ELLI):

Screen Shot 2014-02-28 at 15.29.56

The question banks in such tools should have been validated using well established statistical and qualitative approaches used to design educational survey instruments [example]. Both PERTS Lab and the ELLI programme recognise the potential of such tools to build a large dataset [ELLI infrastructure]. Depending on the approach, the quantitative results from the respondant are then rendered and fed back to mentors, who have a repertoire of interventions to suggest in a coaching conversation.

Powerfully, learners may also receive feedback of some sort, since the whole point of mindset approaches is to catalyse change in the learner’s own dispositions, and get them to take more responsibility for how they engage. The evidence from ELLI is that immediately receiving the 7-legged visual spidergram can have a transformative effect on the learner, making visible for the first time a shape and language for talking about what were—until that point—intangible qualities.

Screen Shot 2014-02-28 at 15.30.08

While Self Diagnostics reflect what learners say about how they behave in different learning contexts, what do they actually do in practice? This brings us to the third approach to dispositional assessment.

Behavioural Analytics

Behavioural Analytics are on the frontier of learning technology research, a subset of the emerging field of Learning Analytics. Since we cannot follow students around with a camcorder recording their behavior in anything other than very small scale, expensive research experiments, this begs two questions:

  1. What kinds of learner behaviours can we track automatically, at scale?
  2. How do we map the contents of those low level (digital) logs to the high level personal qualities we care about in terms of mindsets?

The answers to these questions are only beginning to emerge. To take the first question, technologically, tracking learner behaviours is easy when they’re logged into a digital learning environment through a device: every click, page viewed, comment posted and social interaction can be logged. In a gaming or virtual reality environment, the number and variety of behaviours that can be logged explodes even further: where you go, where you look, who you meet, etc. And in the physical world, our behaviours are increasingly leaving digital traces through location and socially aware mobile devices, digital transactions, facial recognition, and the choice that an increasing number of people make to instrument their lives with other digital measures, such as exercise monitoring bracelets, sleep monitors, and so forth (the Quantified Self movement). See Erik Duval’s slides for a glimpse into this world…

Part 2 is the tricky bit. How do we go from a behavioural trace of low level system events, to an inference that a learner is getting more resilient or curious, or starting to believe that they can get better at learning?

As adaptive learning platforms become available, which closely track a learner’s trajectory through a course, and mastery of the concepts in a curriculum, the possibilities grow for inferring mindsets. Appetite for a challenge can now be quantified at a high fidelity, since the platform knows what may be at the limits of a learner’s understanding. Grit or resilience can be quantified in terms of persistence.

In this slide, I suggest some ways in which online behaviours in a given environment might be mapped to ELLI’s learning dispositions:

Screen Shot 2014-02-28 at 15.02.59

See 50:20 for my explanation of these examples:

If you want to understand what the process might look like of mapping from the user actions possible in a learning environment to dispositional qualities, in this research seminar, Shaofu Huang shared preliminary findings in our efforts to map between the SocialLearn platform and ELLI dimensions (it’s atricky process!):

Screen Shot 2014-02-28 at 16.05.09

In an educational gaming context, Valerie Shute has been developing what she calls Stealth Assessment [pdf]. The graphic below shows some of the qualities that she seeks to assess from the traces that school pupils leave in her gaming platform, which you will see overlap with an academic mindset:

Screen Shot 2014-02-28 at 16.07.37


So to wrap up, these three assessment approaches each have their own strengths and weaknesses, and future work needs to explore how each can inform or enrich the others, as well as where they may differ in what they can tell us. The Venn diagram envisages that in the future, we will combine insights from the three lenses to develop an increasingly rich learner profile, which does justice to the complex, deeply personal qualities of dispositions (they have integrity as proxies for the student’s state of mind), but which can also serve as a new form of evidence-based assessment to serve the needs of the different stakeholders in the educational system (see the ARG quotes).

I look forward to seeing how DLMOOCers are thinking about assessment, and whether the above framework is helpful, or lacking in important respects.

Dispositional, Complex Systems Analytics @ LASI


SkypeScreenSnapz010We’ve just got back from an intensely busy, creative and enjoyable week at LASI13, Stanford University, where Ruth Deakin Crick, Chris Goldspink, Rebecca Ferguson, Nelson Gonzalez and I (online!) contributed to the programme. Ruth and Rebecca were both blogging, so you can get their in-situ reflections, and I’m sure there’ll be more to follow as we unpack the many ideas.

SkypeScreenSnapz007Following a successful Dispositional Learning Analytics workshop, on which we’ve had very positive feedback, awareness should grow of Learner Dispositions (or Mindset as Dweck calls it), Teacher Dispositions, the potential of a multi-level Complex Systems approach to educational system design, and from Nelson’s work at Declara, the emerging potential of applying machine learning techniques to such datasets.


Ruth and Rebecca also participated in a panel on analytics for “21st century skills”, in which they introduced the wider audience to the learning power framework, a systems approach, and a particular implementation of these ideas as visual analytics within EnquiryBlogger. (Just in: we have a fresh set of data from a very exciting deployment of these ideas in a primary school over the last month, which we look forward to reporting on!)

Microsoft WordScreenSnapz083All the morning sessions from Stanford are, or will soon be replayable, and Simon coordinated a global network of LASI-Local institutes, and the online participants, whose blogs are automatically gathered in the LASI-Aggregator, and provide a rich account of a momentous week.

lasi-groupImmediate thoughts on next steps were pooled from Stanford and global online participants in a Google Doc, to give you a feel for where people felt they’d got to by the end of Friday. Ruth documented and reported back on her breakout group’s discussion.

Watch the Society for Learning Analytics Research for news of where next, and get your head going for how you might contribute to LAK14 conference next Spring…


Schools at the hub of community engagement

This seminar given by Tess McPeake of the Smith Family in Australia, describes a programme which engages parents in learning for themselves and their families. Tess described the evaluation of the project and drew on extant research which identifies important elements of learning in the community, which enhance student well being and engagement. Lea 2012 Evaluation of Smith Family Girls inthe Middle UoS Emerson Fear Sanders parental-engagement-in-learning-and-schooling

A video about Parent Yarns can be viewed here……