Academic references to outcomes theory
Increase your skills with a course in outcomes work and outcomes theory
Signup to the DoView School of Outcomes Tips Newsletter
The status of a decision (or intervention) in a risk management context (‘right’ or ‘wrong’) should not be changed on the basis of later information about desired outcomes being achieved. Whether or not such a decision is the correct decision to make at a particular point in time is determined by what is known at that time, not by the later outcome of that decision. However, final outcomes are relevant information for the overall task of assessing risk over a number of iterations of risk management decision making.
Thinking about whether or not a decision was ‘right’ or ‘wrong’ is complex. The word ‘right’ can have two different meanings here. The first is whether a decision was the right decision at the point in time when it was made. This focuses on whether those deciding have taken into account everything they should have taken into account at that point in time - ‘we were right at the time’.
The second meaning is whether, once the outcome of a decision has become clear, it was the outcome that those making the decision initially desired. So a single decision can be viewed as ‘right’ to ‘wrong’ depending on the timeframe in which it’s viewed.
This creates a problem in risk management where you are wanting to apply sanctions on people who do not factor risk sufficiently into their decision-making. Do they get punished for a decision which at one point in time is ‘wrong’ but later is ‘right’.
The solution to this ambiguity is to be clear about the level of accountability of the parties taking action. If they do not take appropriate risk management action at the time they are making a decision which takes into account what is known at that point in time, then their decision cannot be vindicated by positive outcomes being achieved at a later time.
It should be noted that this principle is not the only principle that is concerned with the relationship between acting and the final outcomes that an action is seeking to achieve. The final outcomes of actions can be relevant in the wider context of working out what level of risk management is required over a number of similar decision-making situations.
In addition, other principles come into play in situations where people are willing to tolerate some failures in decision-making within an overall larger system. For instance in evolutionary, experimental, or entrepreneurial systems, which consist of many attempts to solve problems. These attempts are likely to include some failures as well as some successes - that is how such systems are set up to operate. In such situations, final outcomes and 'success' are more relevant to incentivization than in the type of risk management cases where the When success is not enough principle is more relevant.
Use of visual outcomes modeling
Visual outcomes modeling can be used to embody this principle in a practical tool when working in risk management situations. This is done by using a visual outcomes model to identify the level at which parties will be held accountable. As an example, imagine the case of a group of people hiking in the mountains. They decide to traverse a ridge which is beyond the experience of their party and the equipment they have. Even if they ‘successfullly’ get across the ridge without an incident, their decision to take on the ridge was wrong. The ‘wrongness’ of this decision is not vindicated by the fact that they managed to get lucky and get across the ridge without an accident.
In the illustrative outcomes model below, the steps involved in having a safe hiking trip are spelt out. How the hiking party performed on each step has been traffic-lighted where red is poorly and green is well. Note that the box Appropriate gear for the conditions likely to be encountered has been traffic-lighted green on the assumption that the party did have sufficient gear if it had kept its hike to the appropriate technical level. The group’s errors were all around the risk management boxes traffic-lighted in red within the model below.
The key point is that the hiking group's level of accountability was not at the extreme right box in the outcomes model - Planned trip completed. In this case, the appropriate level to set accountability would be at the Safe hiking trip box where safety is operationally defined in terms of following the right risk management procedures. Because the group failed at this level (the number of red traffic-lights leading up to this clearly shows its failure) it failed in its accountability, despite the green traffic-light on the Planned trip completed in a situation where they actually did ’successfully' get to their destination.
In business decision-making, a visual model like this is a very powerful way of clarifying the level at which parties should be held accountable. In practice, people sometimes print out a PDF of the relevant DoView model showing the level of accountability and attach it to their contracts with providers. This ensures that everyone is clear about the level at which accountability kicks in and it avoids any confusion on the part of any of the parties involved. Without a visual model there often can be a lack of clarity about the level at which accountability is being set.
If you’re interested in downloading this model to play around with it, or to build your own model, get a DoView free trial here and get the DoView file of the model here. You can find the rules for drawing this type of outcomes model here.
A first example is from economics. In an article on the US Federal Reserve’s decision to increase interest rates he called Fed Follies, economist Paul Krugman disagreed with the Fed's decision and made the following comment.
‘ . . . it will be quite some time before we have evidence about whether the Fed’s judgement of the economy’s trajectory was right. (I think this was an ex ante mistake even if it turns out OK ex post . . .’). 
In this quote he’s stating the When success is not enough principle. He thinks that regardless of how the economy actually performs in the future, an analysis of all that’s know about the state of the economy at the current time, it is an incorrect decision to increase interest rates.
A second example is in the case of airline safety. Pilots are punished for deviating from the routines and procedures that they’re required to follow when flying. It doesn’t matter if no crash follows as a result of their behavior. The point is that at the time when they were making their decisions they should have done what is specified as the safetest way of doing things. So their decison to deviate from safe practice is still wrong regardless of the outcome that follows.
 Fed Follies. Paul Krugman, The New York Times Concience of a Liberal Blog http://krugman.blogs.nytimes.com/2015/12/16/fed-follies/