Agile is not working: the IT industry is a “patchwork of contradictions and confusions”

Agile is past its rebel teenage years and are well into adulthood with all the responsibility and seriousness that entails. It has become table stakes in the IT industry and one would expect it to be respected and revered as the best way of organising and doing software development. Why is it then that there are so much frustration still, so many unhappy workers, and heated discussions about core elements of what agile is and isn’t? Some even go as far as saying it is passed its prime and are dying. My hypothesis after studying social sciences is that although we have a fairly good take on the tech aspects of the socio-technical system we are part of, we have very little understanding of the implication of the social. The technical imperative and a predominantly mechanistic worldview has blindsided us to the core human needs and it has become obvious to me that we as technologist are not equipped to design organisational structures and processes involving human interaction. Socio-technical systems thinking is about joint-optimisation of the technological system with the social and doing that without a deep understanding of both lead to the problems we see in our industry today. Agile is too important to be ruined by ignorance and poor execution, an opinion that is now supported by research in the social sciences.

“The basic unit of society is not the individual, it is an individual-in-a-group because it is only in a group an individual can grow." —Merrelyn Emery

Disclaimer: The research presented is taken from Dr Merrelyn Emery’s latest paper “A Patchwork of Contradictions and Confusions: Inside the Software Industry” available freely at socialsciencethatactuallyworks.com/current-affairs

My study of socio-technical system design (STSD) and Open Systems Theory (OST) was trigged by a growing worry that when designing IT systems we seem to miss the importance of organisational design. When we design service-oriented systems for example we know that the team ownership is critical and we see again and again what problems we run into when there is no congruence between the team boundaries and the service boundaries – it is impossible to fight Conway’s law. Some even speak of  “inverse Conway’s law” as a technique to use the team design to enforce the technical design in a wanted direction, then often throwing the people into the deep end just to get a service topology that we want. That nonchalant treatment of people has never sat well with me. We treat technology as more important than human needs and well-being.

My growing interest in systems thinking, specifically the soft kind like the socio-technical framing and later open systems, confirmed this worry and made it obvious that one would never get a good system by optimising for the one aspect alone – it could even break it completely. That perspective also shone a light on the work processes we use in IT, specifically agile, where we again overfocus on a technology, namely the process of software development. What I initially had experienced as a grass-root movement of experimentation and collaboration had become industrialised and enforced top-down by managers designing teams and controlling them. Where was the attention to people’s need and intrinsic motivators? People are treated as resources to be used and controlled as parts of the large industrialised software producing machine. Not out of menace but necessity. How has this come to be and what can be done about it? If anything?

This is when I got in touch with Dr Merrelyn Emery, one of the original trail blazers of sociotechnical thinking and co-designer of OST. She was very interested in what was going on in the fairly new IT industry, an industry she knew little about but which appeared to need help. She suggested running a survey they had used many times before when analysing the situation at companies that wanted help redesigning their organisation to survive in a unpredictable environment. We adjusted it a bit to work as an industrial survey of IT and made it available online. The questionnaire was shared on social media and ran for about a year with input from all over the world, which noew has culminated in the release of the first analysis of the data by Merrelyn, freely accessible at her webpage.

I recommend reading the paper yourself as it is a through and very detailed treatment of the results, but also then a bit of a read with a lot of detailed statistics. I will therefore try to summarise her main findings here, with my layman's glasses. But, first I would like to highlight her analysis of the agile industry (“Agile: from the frying pan into the fire”), which with her background as a social scientist and open system practitioner makes for interesting reading. It is an eye-opener in itself.

What she notices immediately is the focus on the individual. Which may be a bit surprising since we seem to focus a lot on teams, collaboration, and communication. Firstly, there is little evidence that agile had any grounding in social sciences or any soft/socio-technical systems thinking. If it had the social aspects would be front and centre. Further, in counteracting the waterfall approach, they ended up removing structure and replacing it with individuals working on small projects we know as cross-functional teams. Cross-functional is not what is called multi-skilled in STSD/OST as there often is very little functional redundancy in those teams. “…right from the beginning of the agile movement it can be seen that the prevailing philosophy was individualism.”

«Agile is the perfect example of a noble ideal coming to grief on total ignorance of how to attain that ideal. With the jumble of inconsistent, often contradictory and problematic fragments revealed by the literature, it is easy to see why some who work in the industry are concerned. —Merrelyn Emery

The teams frequently also have explicit leadership roles, some formally, some by expertise, which clearly show that the participative democratic approach from OST is missing; it is very much still a formal autocratic structure of personal dominance however hard the focus is on informal collaboration. The individualism is also apparent at the teams level, where autonomy is a prevailing focus, with no formal structure for how teams come together as a coherent whole. Scaling frameworks try to compensate for this, filling in this missing coordination in a very autocratic manner, but they were never part of agile and is ad hoc. Merrelyn makes all this abundantly clear by using the comprehensive social science framework OST, showing how maladaptive agile actually is for organisational design. Agile becomes the naked emperor.

The study itself has confirmed much of what was described in this literary analysis, “exposed the fragmented and confused landscape of the software industry which the origin and history of the agile movement explain perfectly.” The analysis cover one of the two parts of the survey, exploring how the industry is doing, not the output, but rather how people are doing, how we are organised, and how well we collaborate. The results are surprisingly problematic. Overcoming what initially seemed like a very strange dataset, using rigours statistical methods, especially a method called Causal Path Analysis developed by Fred Emery, as well as looking at the data from four perspectives, Merrelyn managed to paint a pretty clear picture of what is going on in the IT industry. The details of the paper is probably a field day for any statisticians, but as I am definitely not one so I will focus on the outcome and not the way there.

The result confirms the suspicion that agile is individualistic and has little focus on the second design principle (DP2) where organisations is built on self-managing groups. There are attempts at creating self-organising teams as described in the agile manifesto, but only at the lower levels within the existing autocratic and bureaucratic structure (DP1), and the teams are still very much focused on the individual as the basic unit. What agile has done is remove the old supervision  structure that handled control and coordination, but missed placing that at the group level. That is, control was taken over by the individuals, while coordination was more or less left out. This brings about the classic confusion known as laissez-faire, where the is no formal agreement who supervise, controls, and coordinates what. “…it is quite clear how pervasive the individualistic and laissez-faire approach has been in this industry.“

What the data also show is that this lack of coordination actually makes a more powerful contribution to low motivation than control. What we personally gain in motivation by individualism is more than lost to the lack of coordination between people and teams. Put in another way as described in OST, the balance between autonomy and homonomy, between need for self-control and the need to have a sense of belonging, is unbalanced. The mechanistic belief in the independent individual is closed systems thinking, while open systems thinking is stressing that we need structures that simultaneous meet our needs as humans as well as the needs of the organization. “Adequate levels of coordination are required for any sort of sustainable success and its absence is almost a guarantee of poor outcomes.” Agile is simply not producing highly motivated productive employees as promised.

Although agile set out to democratise work, moving the decision power closer to the coal face, so to speak, to the people doing and knowing how to produce the software needed, “most agile applications have merely imposed the new ideas on quite conventional dominant hierarchies meaning they have mainly failed to remove or even replace DP1 structures.“ A fair response to this could be “so what?” – isn’t this better than a full DP1 and all their waterfall projects? The problem is the dominance of laissez-faire which has been proved to not only produce worse results, it has all the negative effects of DP1 as well as confusion. «As these modern forms of laissez-faire are inherently unstable and short-lived and given the importance of this industry into the future, there is a need for a rapid educational campaign about the design principles so the process of organizational redesign can begin to redress its current failings.»

So let us not throw the baby out with the bath water by getting rid of agile, let us mend it by creating a structure that adheres to DP2 in the whole organisation. Like the mere 3% of the respondents to the survey seem to have managed. It can be done when there is a will at all levels of the organisation and knowledge of how.

"Based on misunderstandings of what people and organizations require for good functioning, the industry has emphasized control over coordination. Subsequently, many good intentions are not being fully realized and there are some very poor results for both individuals and their organizations. The industry is recoverable with an education in the design principles and redesign of the structural inadequacies." —Merrelyn Emery