Managing Complexity: The Battle between Emergence and Entropy

by Julian Birkinshaw

The business news continues to be full of stories of large companies getting into trouble in part because of their complexity. JP Morgan has been getting most of the headlines, but many other banks are also investigation, and companies from other sectors, from Siemens to GSK to Sony, are all under fire.

It goes without saying that big companies are complex. And it is also pretty obvious that their complexity is a double-edged sword. Companies are complex by design because it allows them to do difficult things. IBM has a multi-dimensions matrix structure so that it can provide coordinated services to its clients. Airbus has a complex process for managing the thousands of suppliers who contribute to the manufacturing of the A380.

But complexity has a dark side as well, and companies like JP Morgan, IBM and Airbus often find themselves struggling to avoid the negative side-effects of their complex structures. These forms of “unintended” complexity manifest themselves in many ways – from inefficient systems and unclear accountabilities, to alienated and confused employees.

So what is a leader to do when faced with a highly complex organisation and a nagging concern that the creeping costs of complexity are starting to outweigh the benefits?

Much of the advice out there is about simplifying things – delayering, decentralising, streamlining product lines, creating stronger processes for ensuring alignment, and so on. But this advice has a couple of problems. One is that simplification often ends up reducing the costs and benefits of complexity, so it has to be done judiciously. I have written about this elsewhere.

But perhaps the bigger problem is this advice is all offered with the mentality of an architect or engineer. It assumes that Jamie Dimon was the architect of JP Morgan’s complexity, and that he, by the same token, can undo that complexity through some sort of re-engineering process.

Unfortunately, organisational complexity is, in fact, more complex than that. To some extent, organisations are indeed engineered systems –we have boxes and arrows, and accountabilities and KPIs. But organisations are also social systems where people act and interact in somewhat unpredictable ways. If you try to manage complexity with an engineer’s mindset, you aren’t going to get it quite right.

I have been puzzling over complexity in organisations for a while now, and I reckon there are three processes underway in organisations that collectively determine the level of actual complexity as experienced by people in the organisation.

1. There is a design process – the allocation of roles and responsibilities through some sort of top-down master plan. We all know how this works.

2. There is an emergent process – a bottom-up form of spontaneous interaction between well-intentioned individuals, also known as self-organising. This has become very popular in the field of management, in large part because it draws on insights from the world of nature, such as the seemingly-spontaneous order that is exhibited by migrating geese and ant colonies. Under the right conditions, it seems, individual employees will come together to create effective coordinated action. The role of the leader is therefore to foster “emergent” order among employees without falling into the trap of over-engineering it.

3. Finally, there is an entropic process – the gradual trending of an organisational system towards disorder. This is where it gets a bit tricky. The disciples of self-organising often note that companies are “open systems” that exchange resources with the outside world, and this external source of energy is what helps to renew and refresh them. But the reality is that most companies are only semi-open. In fact, many large companies I know are actually pretty closed to outside influences. And if this is the case, the second law of thermodynamics comes into effect, namely that a closed system will gradually move towards a state of maximum disorder (i.e. entropy).

This may sound like gobbledegook to some readers, so let me restate the point in simple language: as organisations grow larger, they become insular and complacent. People focus more on avoiding mistakes and securing their own positions than worrying about what customers care about. Inefficiencies and duplications creep in. Employees become detached and disengaged. The organisation becomes aimless and inert. This is what I mean by entropy.

The trouble is, all three processes are underway at the same time. While top executives are struggling to impose structure through their top-down designs, and while well-intentioned junior people are trying to create emergent order through their own initiatives, there are also invisible but powerful forces pushing the other way. The result is often that everyone is running very fast just to stand still.

So let’s return to the leader’s challenge. If these three processes are all underway, to varying degrees, in large organisations, what should the leader do? Well, sometimes, a sharply-focused and “designed” change works well, for example, pushing accountability into the hands of certain individuals who are much closer to the customer.

But more and more the leader’s job is to manage the social forces in the organisation. And in the light of this blog, it should be clear that this effort can take two very different forms:

1. Keeping entropy at bay. This is the equivalent of tidying your teenager’s room. It involves periodically taking out layers of management, getting rid of old bureaucratic processes that are no longer fit for purpose, or replacing the old IT system. It is thankless work, and doesn’t appear to add any value, but it is necessary.

2. Inspiring emergent action. This is the equivalent of giving a bunch of bored teenagers a bat and ball to play with. It is about providing employees with a clear and compelling reason to work together to achieve some sort of worthwhile objective. It isn’t easy to do, but when it works out the rewards are enormous.

And here is the underlying conceptual point. The more open the organisation is to external sources of energy, the easier it is to harness the forces of emergence rather than entropy. What does this mean in practice? Things like refreshing your management team with outside hires, circulating employees, making people explicitly accountable to external stakeholders, collaborating with suppliers and partners, and conducting experiments in “open innovation”.

A lot of these are initiatives companies are trying to put in place anyway, but hopefully by framing them in terms of the battle between emergence and entropy, their salience becomes even clearer.