Dr Paul Duignan on Outcomes:
Outcomes Agony Aunt
One of the cool things about being an outcomes theorist is when people tell you about their outcomes problems. It's a bit like being an Agony Aunt columnist but you're providing advice on outcomes rather than people's romantic problems. The moment you mention you're involved in measuring outcomes people start telling you about their difficulties. You can usually identify how their problem is arising - it will be because an outcomes theory principle is being violated in some way. Using the conceptual framework provided by outcomes theory makes it easy to diagnose and provide fixes for most of the common outcomes problems people describe to me.
A touch of theory - levels within outcomes models
To make it easier to talk about the particular problem I want to discuss in this article, we need a little bit of theory. From an outcomes theory point of view, when people are talking about developing lists of outcomes, whether they know it or not, they're talking about setting up an outcomes system. An outcomes system is any system that is attempting to do any of the following: identify; prioritize; improve; measure; attribute; contract; delegate or hold parties to account for outcomes of any type. An important part of any outcomes system is an outcomes model. Conceptually an outcomes model can be thought of as a set of boxes within a visual model showing high-level outcomes and all of the steps it's believed are needed to achieve them.
Outcomes models have three major levels. At the top is where we find boxes that are often described as goals, results or outcomes. At the bottom is where we find boxes often referred to as outputs, activities or projects. It's the area in the middle that I want to talk about here. It's the job of the middle boxes within an outcomes model to detail the logical connection between the boxes at the bottom and the ones at the top of the model, this is often called a 'theory of change'. The need to do this in any outcomes modeling is captured in the Detailing the Middle outcomes theory principle. In cases where the logic of this connection has not been spelt out properly, we're confronting what's called the 'problem of the missing middle'. The function of showing the logical connection between the bottom and the top levels of an outcomes model is central to outcomes modelling - it's the reason that outcomes models are sometimes referred to as logic models.
Effect of constraining the number of allowable high-level outcomes
Recently I was talking to a colleague who had several problems that can easily be formulated in terms of the levels within an outcomes model. As is often the case, my colleague had first been asked to develop a set of high-level outcomes for the work that they're involved in. But, as also often happens, there were constraints placed on how these 'high-level outcomes' were to be itemized and described. My colleague had been instructed that there should only be a limited number of high-level outcomes and for what seemed in essence presentational reasons, they were not allowed to spell out much detail beneath them.
These types of constraints, on how many high-level outcomes there should be and how detailed they should be allowed to be, are often imposed by people setting up outcomes systems. However they have considerable consequences when it comes to working with outcomes. The result of applying these constraints means that the set of high-level outcomes that is produced will inevitably only describe the very highest levels of the relevant outcomes model.
Placing constraints that effectively push the high-level outcomes that are being formulated right to the top of an outcomes model has implications for the 'middle' of the model. In essence, it expands the size of the remaining 'gap' - the 'middle' that needs to be bridged between the bottom (outputs, projects or activities) and the top of the model (results, outcomes or goals). This occurs because the constraints have forced the outcomes that are being specified right up to the top of the model. The gap will be wider than in a case where one is allowed to have as many high-level high-level outcomes as one likes and/or where you're allowed to include as much detail as you like. So the consequence of imposing the constraints is that more detailing will have to take place in the middle. This is because the middle is larger than it might have been if a less restrictive approach had been taken to formulating high-level outcomes.
What should determine the number of outcomes?
The second issue that my colleague was facing was that the area of activity for which outcomes were being developed was a large one undertaken by a single administrative grouping within their organization. This single administrative grouping included a significant number of sub-groups involved in somewhat varied activities. My colleague had been instructed to develop a small set of outcomes, but diversity in what a set of sub-groups within a wider administrative group is trying to archive makes it hard to produce a small set of outcomes which can adequately encompass the work of the group as a whole.
The Real world outcomes principle within outcomes theory states that the number of outcomes struck for any type of activity should be dictated by what is being attempted in the real world, not by the often arbitrary way in which sub-groups that are undertaking different activities have been clustered within an organizational structure. To put it simply, if people are doing a lot of different stuff they're going to need to have sufficient outcomes to cover all of it. In such situations it's a mistake to blindly force people to develop outcome sets which only have a limited number of outcomes.
The constraints that had been placed on my colleague in doing their outcomes work translated into the two problems they talked to me about. The first one was that some of the people from some of the sub-groups within the wider administrative group for which outcomes were being prepared could not 'see' what they were doing within the small set of high-level outcomes that were being been developed. The inevitable result of this was considerable unfruitful argument about the wording of the small number of high-level outcomes my colleague producing. Given the diversity in what the sub-groups were doing, it is unlikely that any wordings could be arrived at which would lead to all the sub-group members being satisfied that what they were doing was adequately represented in the outcome set.
Detailing the missing middle one activity at a time
The second problem that arose from the constraints imposed in this case was regarding detailing the 'missing middle' between the high-level outcomes and the bottom of the model. As I've said, requiring that there only be a small number of high-level outcomes inevitably results in a large 'gap' between the few specified high-level outcomes and the lower-level outputs, activities or projects which are being undertaken. Obviously this 'missing middle' in the outcomes model needs to be detailed in some way in order to comprehensively set out the theory of change that is being attempted. In my colleague's instance a common strategy was being employed to do this. The strategy was to get them to write a block of text for each activity being undertaken within the group. In each instance this block of text was meant to set out the 'rationale' for why the particular activity was being undertaken. The purpose of a rationale of this type is to show how it's believed that each of the activities being done is going to contribute to high-level outcomes. The idea in this approach is for the missing middle to be separately detailed for each of the activities in turn.
My colleague was complaining that they thought that detailing all of the activities in this way was repetitive and inefficient. They were well aware that despite how much work they put in, very few people would ever read the screeds of text they were having to write. Developing this outcomes set was an administrative task that they had to do on top of their normal work duties and was taking up a great deal of their time which though could be more usefully employed.
In addition to the inefficiency of this approach, even when completed, the whole collection of textual rationale statements for each activity would not provide a particularly efficient way for anyone to rapidly overview how well the bottom level of the model is connected to the top - the overall theory of change for the work that the group as a whole was doing.
Using a visual outcomes model to detail the missing middle
The approach being adopted in my colleague's case is a common approach to detailing the middle of an outcomes model. However outcomes theory would suggest that the problem of the missing middle be dealt with in a somewhat different way. First, it would suggest that a comprehensive visual outcomes model should be built for the work of the group. This would be built according to the rules for building outcomes models and would not suffer from the constraint around the number of high-level outcomes and limited detailing of top level outcomes that my colleague was having to confront. Building an outcomes model would allow the identification of as many high-level outcomes as are needed to represent the work that is being done by all of the sub-groups within the wider group. It would also provide as much space as is needed within the visual model to document the full theory of change set out in the steps in the model linking the bottom with the top levels of the model.
In effect this approach would be developing a standardized model of the way in which activities were believed to be influencing high-level outcomes. This allows one to articulate the missing middle once and for all and not have to do it individually many different times for each separate activity. Common pathways for different activities only need to be detailed once within the visual model when using this approach. In contrast, using the individual blocks of text approach to articulating individual rationales for each activity ends up with duplication of all common pathways within different rationale statements.
Showing an activity's logic within a wider context and checking for alignment
Of course using a visual modelling approach, even though you've developed a common model of the logic of what you're trying to do, you still need to be able to identify the particular pathway of an individual activity. If you're using suitable outcomes software (e.g. DoView*), this can easily be done within a visual outcomes model by including a box for each activity and linking it to each box within the higher levels of the model it is focused on.
This makes showing the individual theory of change for any particular activity easy. In outcomes software, clicking on an activity box will show up all of the higher-level boxes within the model which it is linked to. This will show exactly what the activity is attempting to influence within the outcomes model and spell out the particular pathway for that specific activity. This has the advantage of showing the theory of change for the particular activity not just on its own, but also within the larger context of the whole model.
Once the individual pathways for different activities have been mapped onto an outcomes model, this opens up various possibilities for further analysis which are not available through the text-based approach to 'articulating the middle'. Again, with suitable outcomes software which counts the number of activity boxes mapping onto each box within the higher levels of the outcomes model, you can quickly identify gaps and overlaps in the mix of activities that are being undertaken. This is done by using a visual 'line-of-sight' alignment approach where you can see how many activities are, or are not, linked to which boxes within the higher-levels of the model. (For an example of line-of-sight alignment see the screenshots at the end of the page here).
It's not clear how those wanting a block of text rationale to be separately developed for each activity believe that one can efficiently check for alignment between activities and higher-level outcomes. Presumably the idea is that someone reads all of the statements setting out the rationale for each activity and thinks about whether there are any gaps or overlaps. However, the 'cognitive load' of doing this - the amount of mental energy required - is high. As soon as there are a number of activities involved, it is very hard for any reader to remember what was stated in each of the rationale statements and so they cannot make a rigorous judgment regarding whether there are, or are not, any gaps or overlaps.
In conclusion, as I said at the start, when you're being an outcomes Agony Aunt and listening to people's outcomes problems, outcomes theory makes it very easy to diagnose their problems. It is also easy to suggest ways in which their problems could be overcome by applying outcomes theory principles to the way in which they are working with outcomes in their particular setting.
* Full disclosure: Dr Paul Duignan is involved in the development of DoView software.
If you want to comment on this article you can do so on the Linkedin version of it.