A Framework for Evaluation & Benefits Management

A while back Jisc provided us on the Transformations programme with a request to complete an evaluation spreadsheet regarding our work.  This included the following information:

  1. Current issues/problems that you face/Why are you carrying out the project?/What needs to change?
  2. Stakeholders
  3. Indicators
  4. Expected benefits
  5. Baseline (current state)

Having gone through this I thought it would be interesting to look in some more detail at evaluation, impact and benefits.  I did this by using the above Jisc framework but also including elements of the Cranfield School of Management “Benefits Management” approach.  The third angle I came to this from was to adapt some of the work I did a few years ago (Reference 1. below).

Relevant elements of the Cranfield approach include the “Benefits Dependency Network” and supporting Benefits Workshops.  This is useful to those thinking about change in all areas as put simply it makes you think about linking what you want to do to how you get it.  I’ve found this good to do in a workshop with relevant people, a stack of post-it notes, an empty wall and the aim of identifying each of the following and the relationships which link them:

  1. Your Drivers (Investment Objectives)
  2. The Business benefits
  3. Business Changes required
  4. Enabling Changes required to support this, and
  5. IS/IT Enablers.

This of course nods towards the well versed tendency for organisational projects and developments with technologically deterministic leanings where the risk is that the starting point is a technical solution in search of an organisational objective and change to fit to it.  Yes, of course it sounds like an elderly cliché but lucky are those that work where it doesn’t happen to a lesser or greater extent.

The perspective I bring to this from my previous research (I was working within mental health services, developing IT and clinical systems, analysing psychiatric outcomes data etc.) is consideration of how we understand organisational change (notably that which involves IS/IT) and the human and organisational aspects of change.  Here there is need to challenge preconception about how individuals, groups and organisations think and behave, in relation to what may be seen as prevailing assumption about change, based on what can be seen as organisational logic.

To cut a long story short this led me to challenge understanding of change, to consider reasons why things are more difficult than you would expect.  I did this within the context of mental health services and IT projects and development – a particularly challenging environment to think about why things are complex, why things are very difficult to get right, why unexpected things happen, and to what extent they are related to the organisations and people involved rather than the technology supposedly at the heart of change.  More than most, in this context theory and practice demonstrated that implementation of IT without a clear understanding of human and organisational issues, and a strategy to deal with them, was likely to considerably lessen the chances of success.

One of the outputs I produced to understand and apply this was link between an individual’s:

  1. Presupposition
  2. Understanding
  3. Interpretation
  4. The Development of attitude and attitude in concept
  5. Attitude in context
  6. Behaviour, and
  7. Practice (as formalised behaviour)

Applying these 3 elements I have produced a tool which requires three steps:

1.  Definitional

  • Issues/Problems
  • Deliverables
  • Benefits
  • Measurables

2.  Analysis of Context

For each Issue/Problem identify related:

  • Aims/objectives
  • Issue impact
  • Output – what needs to change
  • Output – how this will change
  • Outcome – attitude change
  • Outcome – behaviour change
  • Outcome – improvement
  • Benefit
  • Measurement of benefit

3.  Match 1. and 2.

  • Compare 1. and 2., revising the analysis of context against the items defined
  • Refine the items created in the Definitional stage
  • Document where in the analysis of context each of the defined items relates.

This is something I will be working on, and may or may not prove generally applicable.  So far however I have found it to be a useful means of looking more deeply at what it is we are trying to do, why we are doing it, and develop realistic understanding of where we may actually get to, what problems we may encounter, why we may need to temper ambition.  The result is that we may be better at anticipation of issues, should think very early about this, and early on develop tactics to achieve benefits we are after.

I’d be interested to hear thoughts on this and hear from anyone who has been looking at the issue.

Reference 1.: Burnham, A. “Developing an Inclusive & Balanced Approach to the Implementation of (Mental Health) Information Systems: A Critique of the Theory and Practice Dialectic of Systems Implementation”