Failure on demand – Scenes from an agile transformation

In this blog post, we want to show why agile transformations fail, illustrating various situations that unfortunately still occur far too often in reality today.
More and more often, we notice that the company culture lived by the management and the employees is crucial for a successful implementation of an agile transformation or agile development. Particularly in large companies, onboarding employees and motivating them tends to be neglected. For example, there are initiatives to transform the company, but they falter and fail due to the attitude of the workforce and/or management, i.e. the company’s culture.
When a healthy agile culture can’t be established in an organization, the following situations may appear:

“But, we are agile! We are doing Scrum!”

Agile has become an overused term. In this article, we look at agile from the perspective of corporate culture. A healthy agile culture has to be lived by employees and their superiors (who act as role models). The transformation to it is laborious, because old structures and behaviors need to be broken up.
Often, Agile is equated with Scrum. It is assumed that if Scrum is practiced, agile software development follows. Working according to Scrum sounds like an easy task, and yet many mistakes happen in the implementation of Scrum in terms of a good agile practice. We have noticed that Scrum does not have to be that important at all. It is more important to live agile in the project culture.
An agile culture includes a focus on transparency, e.g. about all project decisions made, and strengthening the flow of information both within the team and externally. This allows each team member to contribute fully to the success of the project by being involved in the planning, design and improvement of the project. Agile methods (e.g. Scrum) are frameworks to enable this to happen continuously and iteratively.
In the following, we will address common agile anti-patterns related to Scrum.

The “agile waterfall” project: It is often observed that agile projects are planned similarly to a waterfall project at the beginning: a time and project plan is defined and the budget must be determined. Deadlines are determined for the individual features. Requirements are packaged into user stories and these are unleashed in the project in one big batch. In the worst case, the user stories are of poor quality, e.g. because they are sliced into too large chunks or are soon outdated.
What follows is that features are implemented, but validation or feedback from the actual end users takes place far too late, if at all. The project falls further and further behind the planned schedule and ultimately fails for the same reasons as usually waterfall projects fail: the planned launch is postponed several times, and when the release finally arrives, it lacks user acceptance. The textbook example of a Big Bang – not at all in line with an agile culture.

“Chief programmer” sets the direction: Scrum is used as an excuse to pretend to be Agile. Example Pattern: a product owner (e.g. a former project manager, now retrained) tells each individual team member exactly what tasks to do and when to do them. Scrum tools and meetings are only used to micromanage the team. It is impossible for the team to work independently. Priorities are often changed during the sprint, e.g. because features are not prioritized according to maximum benefit for the product, but according to political aspects (e.g. because a certain feature was promised to a manager for the next release). As a consequence, the team’s decisions are influenced by the “chief programmer” on actual team-internal decisions, therefore it gets impossible for the team to commit to a (sprint) goal.
One thing needs to be made clear: Scrum was not developed to control teams better. It should be clear that not everyone can know everything and therefore a solution must be worked out with the team. If everyone in the team works together and everyone has a good overview of the project’s goals and the future product development, perhaps a better solution can be found.

Lack of dedicated scrum master: Good scrum masters are expensive, but they are important. We often hear: “But they don’t even produce code! Well, then a team member can just take over the whole thing.”
This often leads to frustration and impediments within the team, because, for example, retrospectives are poorly conducted or omitted. There is no dedicated person who takes care of removing impediments. This is followed by a decreasing development speed and the previously saved costs are gone.
There is only one solution for this: employ dedicated scrum masters! They take care of protecting the team from external influence and pressure, organize the scrum meetings and solve internal problems. The team must be able to work undisturbed!

Overly large teams: Teams should not be too large, otherwise some negative effects can be noticed: meetings drag on due to the number of participants, decision-making becomes increasingly difficult due to many parties to consider, and it becomes more complicated to maintain psychological safety in the team.
Also, knowledge transfer between team members is more challenging. In addition, the team spends 30-50% of the sprint time in meetings, and the time for development becomes increasingly scarce.
In large projects, a meeting is called for pretty much every decision or blocker, and preferably everyone has to be there, because it could affect everyone. In the worst case, this leads to minutes of silence and a lot of wasted time.
Teams should be able to act as independently as possible and they should not be too large. Shared dependencies and interfaces should be discovered and teams should provide a “product” to the respective other team. This leads to a decoupling of the individual teams and the teams can work in a self-determined manner again. Larger meetings between the teams should only take place with individual “emissaries” of the respective teams. The “community of practice” pattern has also proven itself here.

“Well, we are agile! We are doing DevOps!”

DevOps (as a buzzword) has arrived in management. Nevertheless, fundamental concepts of a healthy DevOps practice are not or only partially implemented. When DevOps practices do not work, this is also often due to a lack of alignment of habits, processes and methods by management and employees.
Among other things, teams are not encouraged to live a DevOps culture. Instead, siloed departments or teams continue to exist, with different responsibilities, budgets, and therefore their very own political goals.
For example, DevOps is mistakenly established as a department within the company rather than a practice. A DevOps team as an enabler within the organization would not be a bad thing in this regard, but in reality this department does not provide support for teams, but is hidden behind ticketing systems and lengthy processes. The teams cannot get away from the constraints of the old plan-build-run culture. For example, getting the much-needed CI pipeline doesn’t require one click within a tool, but rather lengthy application forms must be filled out and signed off on by multiple higher-ups so that the pipeline can then be created by the so-called DevOps team.

These processes have grown in companies over a long time, perhaps for good reasons. Yet they prevent developers from working effectively on the project. The credo must be: Everything and everyone in the company must work toward enabling developers to work effectively and without disruption. Only then can developers focus on creating value for the company and advance the company’s product.
So for our example, this could mean that a DevOps team provides a working self-service platform for CI/CD, which allows projects to create new build and deployment pipelines without much effort. They act as an enabler team and train and support other teams to quickly integrate their platform into their work.
An agile culture and a functioning DevOps practice form a strong symbiosis. DevOps supports development according to agile methods enormously. Only when teams can handle resources independently and in a self-determined manner are they able to also quickly bring features into production.

“Digitization project XY has the highest priority!”

Focus is important. Very often, we see that teams are distracted from their current project by other tasks. How can the important digitization project be successfully completed if team members still spend 50% of their time firefighting for legacy systems or are dragged into pointless meetings?
Good management keeps these team members on task and propagates the project goals to keep all employees focused on a common goal. Goals need to be clear to each team member and management decisions need to be transparent. Only then will team members work effectively and with great intrinsic motivation.

“You said two months ago that it would take X days, now get on with it!”

In addition, teams fall into another trap: they are nailed down to schedules and budgets that were arbitrarily set months ago, but are absolutely untenable with the current knowledge of the product or domain. At the beginning of the project, many technological and architectural decisions were made based on now outdated assumptions when the project had been running for a month or two. The team now tries to meet these deadlines at all costs, even though it is clear that they can no longer be met: motivation drops and the belief in success is lost. Huge amounts of overtime accumulate and the technical debts become bigger and bigger. A downward spiral begins.
It would be more important to measure teams not on deadlines and budgets, which in most cases cannot be met, but on the basis of delivered features. Big decisions should be made as late as possible in order to keep the solution space open and thereby achieve results as quickly as possible, which can then be evaluated and tested. On the other hand, small portioned decisions should be made quickly in order to be validated quickly and to receive feedback fast.

“Are we at a science fair?”

This statement is extreme but we have heard it more often than we would have liked to. It really astonishes us. For us, an explorative approach is a matter of principle and a continuous development of solutions out of hypotheses and their validation through user tests is indispensable.
In reality, it often looks like this: the exact requirements for the product seem to be clear. So we get started (but without requirements specifications, because we are agile!) and user stories are created and a schedule is worked out. For example, features are then taken from (to-be-replaced) legacy systems which are obviously absolutely needed. In addition, it is agreed that the old system was great, but must now be replaced by something better, but the new system must do exactly the same things as the old one.
One wants to do everything right from the start. The way to get there is often unclear and we observed that the customers themselves are not so sure which features really produce the greatest business value for the new product. But instead of starting from scratch, a canonical data model is usually developed (or adopted), the architecture is defined, and instead of figuring out what will create the most value in the new product, a large feature catalog is worked through.

Again, a downward spiral starts, because the opposite is usually the case: the requirements are anything but clear. What usually happens: the project’s goals are not achieved, there is an ever-increasing delay to the deadlines and the end users are not asked for feedback. The developers’ motivation declines and nobody believes in the success of the project. The project will fail sooner or later.
An explorative approach can prevent this, but it does not come without pain. After all, at the beginning it looks like there are no successes made and that there is a slow development speed. Assumptions are made, features are developed and later much is discarded. However, the earlier assumptions can be validated or invalidated, the better and the higher the development speed will be later on.

It is important to be open to change from the start! This is part of an agile culture. Change must be reflected in the architecture and implementation of the software. It will happen that the data model will change again after each iteration and the code will have to be heavily refactored each time. Nevertheless, such an approach allows for implementing the actually important features and for getting the necessary feedback from the users early on. It is important to measure the success of the team not on the basis of deadlines met, but on the basis of tested and validated decisions.

“Yes, we want to be in production as soon as possible!”

Unfortunately, we hear this statement far too often in projects that are usually already delayed, e.g. due to the situations already mentioned above. This sometimes means that a product is already months or years behind the actual schedule. Yet, during development, it has not even been successfully rolled out to production.
One thing must be clearly stated: if the feature/product has not deployed to production, no value has been created. Yet, we see it time and time again that teams continue to be bombarded with feature requests: the focus is not on getting existing features into production and testing on end-users and getting feedback. Development goes on and on, money is burned, and in the end customers are presented with a fait accompli.
When the time comes, and the product can be deployed to production, the team lacks the knowledge of which company-specific compliance rules and processes must be adhered to in order to even deploy the new “cloud-native digitization project” to production. For example, cloud teams are deployed as enablers within the company (similar to DevOps teams described earlier) to introduce the respective major cloud providers within the company, but later on these teams are either not designed to support other teams on their way to production and/or are completely busy with other political tasks.
As with a poorly executed DevOps practice, the lack of production readiness again shows a poor “as a service” mindset towards a project. Furthermore, it should be natural for DevOps-practicing teams to also deploy to production several times a day. Only in this way is it possible to obtain a continuous development flow with constant feedback.

TL;DR

This blog post highlighted various situations and scenarios that can occur in agile projects and then doom them to failure.
Let’s summarize briefly:

• “Agile” is often an empty word and it does not permeate the entire corporate culture. Scrum is introduced and equated with agile. A cultural change does not take place.
• DevOps practices are important to enable agile projects, but are not implemented properly and may drown in a jungle of application forms and politics.
• Teams cannot work autonomously, independently and with focus.
• Continuous learning, feedback, and improvement do not take place, or take place only infrequently.
• It is rarely or not at all deployed to production, experiences cannot be gathered and no lessons can be learned from them. No value has been created as a result either.