Why do change projects fail?
Dr Andy Milward, PhD, CPsychol, AFBPsS
Management educators, consultants and executives have cited the claim of consulting firm McKinsey for many years that seventy per cent of all organisation change projects fail.
In 2015 McKinsey reiterated:
“Change management as it is traditionally applied is outdated. We know, for example, that seventy per cent of change programs fail to achieve their goals, largely due to employee resistance and lack of management support. We also know that when people are truly invested in change, it is thirty per cent more likely to stick.”
https://www.mckinsey.com/featured-insights/leadership/changing-change-management#/
Other research has also found that the complete change project or partial failure rate is very high. For example, Gartner found that only just thirty-four per cent of organisational change attempts are a “clear success,” fifty per cent are “clear failures”, and sixteen per cent have “mixed results.”
https://www.gartner.com/en/human-resources/insights/organizational-change-management
The research I cite above focuses on distinct change projects. However, change management is an ongoing adjustment process, not only periodic special programmes. Change management is general management.
All change initiatives are behavioural change projects because outcomes depend on how people choose to behave – their decisions and actions. There is no organisation without behaviour.
Consultants, researchers and educators continue to devise methodologies they believe are more likely to yield change project success. However, these approaches tend not to address the underlying problem. In some instances, they may make it worse.
Change initiatives have contributed significantly to the growth of the global management consulting industry, which had revenues of $693 billion in 2022.
https://www.cleartreasury.co.uk/insight/consulting-industry-trends
But is the money spent on expensive change projects a good investment?
As things stand, I suggest not.
Change is hard because people tend to oversimplify the cause of human behaviour in the workplace. There is a tendency to think organisations work through a mechanical cause-and-effect process with the scientific rigour of Physics.
Technically, this cause-and-effect perspective is the scientific doctrine called Empiricism (often known as Empirical Realism to confuse things more). The process of establishing cause and effect is called the Scientific Method.
The Scientific Method – experiments, observations and statistical analysis seeks to identify ‘constant conjunctions of events’. These are situations where one event repeatedly follows from the occurrence of another event.
However, throughout this post, I will refer to this perspective as ‘billiards-ball thinking’ to keep things simple.
If a moving billiards ball collides with a stationary ball, the balls behave as illustrated in figure 1.
Scientists studying the effect of a collision between moving and stationary billiards balls will systematically remove contextual influences on the behaviour of the balls so they can isolate the pure causal relationship.
Through the Scientific Method, scientists can therefore derive a general law applicable to all situations where billiards balls (and similar phenomena) collide.
Because constant conjunctions in the physical world are so robust, they serve as a proxy for causation. The first event is said to cause the second event.
People typically think about change management this way, as reflected in the various methodologies they use to facilitate change. If a person does X, then Y will inevitably follow.
We can see this equation in the various psychological tests and survey instruments that are routinely used to assess patterns of behaviour in individuals and groups.
Once assessed, there follows an attempt to align behaviour in accordance with a model that sets out how people should ideally behave to achieve a desired outcome.
Behave like X and Y will inevitably follow.
The organisational behaviour research literature shows that most theories about change, leadership and management rely on billiards-ball reasoning.
However, people do not, and cannot, behave like billiards balls.
Human behaviour is contextually situated. It is the product of a complex interplay of perpetually shifting influences, within and outside the organisation, often counter-intuitive, frequently random, and non-linear, many of which we do not know about and even more that we can never know about.
There are no laws that we can use to predict how people will behave irrespective of time and place.
Unlike billiards balls, people do not possess stable properties, and accordingly, nor do the diverse groups to which they belong. Instead, people are sentient beings, thinking and feeling entities whose behaviour changes at various rates in response to their ever-changing context, which is usually impossible to predict.
We cannot understand the cause of behaviour in organisations by identifying constant conjunctions. Interactions among people are rarely constant.
In contrast to billiards-ball thinking, where experimenters isolate cause and effect by excluding the impact of context, in organisations, we want to include as much context as possible to explain people’s behaviour. If we do not understand the cause of behaviour, we cannot change it.
Constant conjunctions are correlations between events.
In physics, even if we know that two things correlate, this does not tell us why.
As Elder Vass notes;
“When we have said that A is always followed by B we have not identified a cause at all. Such regularities are not causes or explanations of events, but rather are themselves something that is caused, something that requires an explanation”
Elder-Vass, D. (2010). The Causal Power of Social Structures. United States of America, Cambridge University Press.
However, as constant conjunctions in physics are so robust, we still consider them a proxy for causation.
But although correlations arise everywhere in organisations, we cannot take them as proxy explanations. In organisations, there are likely many explanations for an association between events.
Famously, the sociologist Arthur Stinchcombe once remarked,
“a student of sociology who has difficulty thinking of at least three sensible explanations for any correlation that he is really interested in should probably choose another profession”
Stinchcombe, A. L. (1968) Constructing Social Theories, University of Chicago Press.
To manage change, we must explain how and why people choose to behave in particular ways.
Theories of change management stress the importance of leadership.
Commenting on research on the effect of leadership behaviour on outcomes, leadership scholar Gary Yukl notes,
“Most leadership theories are also limited by a lack of attention to explanatory processes. The theories often propose relationships or causal effects without explaining why they occur.”
Yukl, G. (1998). Leadership in Organisations. New Jersey, Prentice Hall.
Note that the ubiquitous questionnaire method I mentioned above excludes all context.
The model on which the questionnaire is based is imposed on the organisation like a template. Behaviour is forced to fit into the model.
This approach dictates cause, it does not discover it.
The uncertainty of organisational processes (the scarcity of constant conjunctions of events) means we cannot explain outcomes by referencing any generally applicable laws.
People’s behaviour is in a constant state of flux. So the best we can say is that there is a tendency for people to behave in a particular way, although tendencies will dissolve and new tendencies will emerge.
A tendency is not a law but a movement in a particular direction that countervailing conditions can neutralise.
For example, an executive may say something that motivates a group in the morning. However, it may have a different outcome if they say the same thing in the afternoon.
This difference in response is not because the executive has changed (although they could have). Instead, the reaction may be because group members have been affected by some other influence of which the executive is unaware and often cannot be aware, thus rendering the executive’s efforts to motivate them ineffectual.
Change in organisations is inevitably hard to manage if we cannot rely on general laws.
This situation has devastating consequences for change management.
Why are we in this situation?
A quirk in the history of Psychology is responsible for billiards-ball thinking in organisations.
Psychology, the scientific study of mind and behaviour, has always had the ambition to acquire the status of natural science, particularly Physics.
Psychologists have, therefore, always tackled problems according to cause-and-effect reasoning.
In contrast, the closely related fields of Sociology and Anthropology focus on generating explanatory theory, not cause-and-effect relationships.
Billiards-ball thinking has also infused reasoning in sub-disciplines of psychology, including Social Cognitive Psychology, the study of how people perceive, interpret and act in response to the behaviour of others. Social Cognitive Psychology is the bedrock of theory in organisational behaviour.
We all learn billiards-ball thinking in school when we study science. Therefore, when working in organisations, billiards-ball thinking is the default mode of reasoning about the cause of people’s behaviour and how to change it.
There is hope, however!
Psychology is in the grip of what is known as the ‘replication crisis’.
Researchers have found that they cannot replicate the findings of many of the most fundamental theories in psychology, including Social Cognitive Psychology.
Laws in physics are robust and highly replicable throughout time. So we will not suddenly find that billiards balls behave differently from what we expect.
But researchers find it increasingly difficult in psychology to replicate the theories that prevail in the study of organisational behaviour. This development exemplifies that there are no constant conjunctions in human behaviour.
For example,
“In a 2015 attempt to reproduce one-hundred psychology studies from high-ranking journals, only thirty-nine of them replicated. And in 2018, one effort to repeat influential studies found that fourteen out of twenty-eight — just half — replicated. Another attempt found that only thirteen out of twenty-one social science results picked from the journals Science and Nature could be reproduced.”
https://www.vox.com/future-perfect/23489211/replication-crisis-project-meta-science-psychology
If we cannot rely on the theories underpinning organisational behaviour, change management efforts have a low likelihood of success.
I believe that is what we see in the findings of McKinsey, Gartner and numerous other researchers. Therefore, if Social-Psychology is in crisis, organisational science must be in crisis too, as failed change management testifies.
In his 1970 book, ‘The Structure of Scientific Revolutions’ Thomas Kuhn argued that it takes a crisis to generate a shift in a scientific worldview.
https://en.wikipedia.org/wiki/The_Structure_of_Scientific_Revolutions
I believe we are facing such a crisis.
The increasing world turbulence acting on organisations makes understanding the cause of organisational behaviour imperative. Climate change, geo-political events, war and financial volatility, require organisations to adapt to their context rapidly.
Adaptation depends upon understanding causation.
The greater the turbulence, the more dangerous billiards-ball reasoning becomes.
If we cannot replicate fundamental theories about behaviour in psychology and organisation science, we need an alternative scientific worldview. We need a perspective that emphasises explanation, not cause and effect.
There is also a significant financial imperative to adopt an alternative scientific mindset.
If organisations do not adopt an explanatory approach to change, they will waste even more precious financial resources on pursuing management approaches and change projects that are set up to fail.
The future
Researchers in Social Cognitive Psychology and the Philosophy of Science have started to integrate various analytic methods with an alternative scientific perspective that organisations can adopt to understand behaviour better and solve problems.
This perspective emphasises explanation, not mere correlation.
The approach is an alternative scientific mindset, a new way of thinking about behaviour, not a methodology such as Agile, Scrum or Kanban, although such methods and many others will complement the perspective.
This emerging perspective is a response to more awareness of the limitations of billiards-ball thinking. The replication crisis and the world situation is accelerating this development.
An explanation of these developments is far beyond the scope of this post.
This Article was originally published at Why Do Change Projects Fail and Why Do Change Projects Fail: LinkedIn Article
