Maintenance hell? No, thanks!

Challenges of and solutions for the maintenance of custom made software

Custom made software solves individual business problems. The development of the software is already quite challenging. In addition to this, all successful – in terms of deployed to production – custom made software products have a crucial thing in common: due to changes they require continuous maintenance and additional development over a long period of time. Unfortunately, the issue of maintenance is often not given the necessary attention in a timely manner and the maintenance team, the client and the users of the system all end up in maintenance hell together. In this article I would like to alert you as my readers to pay increased attention to this topic. After pointing out quite a few things that can go wrong, I will describe some possible solutions in the remainder of this article.

From an organisational perspective, there are two different, prevalent options for handling software maintenance. Firstly,  a larger project (or product) team could permanently manage and maintain the software by itself. This is often referred to as “You build it, you run it “. Secondly,  the project team could hand the software over into the care of a maintenance team after the main development phase has ended. The second option quite often includes the additional challenge of transferring the responsibility for maintenance from an IT service provider to the client company or vice versa. In particular the second case shows: If maintenance is not considered early enough, similar problems are always encountered. Changes to the software running in production are necessary and these changes can often be described like this:

• They are exorbitantly expensive.
• They are associated with very high risks.
• They tend to take extremely long.

In the end the changes cause pain and frustration for everyone involved: developers, customers and users. Costs are high, frustrated employees may quit and a loss of reputation results in lost business opportunities and endangers the company’s general success.

But does it have to be like this? Does it have to hurt so much? Can we find a way to at least ease the pain and lessen the unpleasant consequences? Could we even completely avoid them?

Let’s ask a first important question: What exactly is going wrong?

Unfortunately, often enough I experience that at the beginning of product development – in other words at the beginning of the actual software development project – neither the client nor the project leader think enough about the future maintenance of the developed software. Involved teams and stakeholders do not consider this topic in an appropriate manner. This is also true for the procedures used to develop the software.

Fig. 1: Software product lifecycle

Software products follow the product lifecycle shown in the image above. In the introductory phase a project team consisting of many developers creates the software. Until going live the team is fully focused on the completion of the product. The project team will often initially take care of maintenance and further development during the transition into the growth phase after going live. Sooner or later the development team is withdrawn from the development of the product and maintenance is transferred to another organizational unit and other developers. They will then take over maintenance and also further development of the product in the maturity phase.

As long as the original project team is still responsible for maintenance and further development of the software we seldom encounter any problems. Productivity will in general decrease slightly because the attention is now divided between developing new features and solving maintenance issues. Well-defined processes help to mediate this and keep the losses due to friction between the different types of tasks low. The project team knows the proprietary software inside out. Errors are found quickly. Risks while fixing those errors are well understood and can be adequately taken into account.

Problems begin when responsibility for the product maintenance is – more or less planned –  transitioned to a maintenance team:

• The software is not really maintainable.
• It lacks sufficient and automated testing.
• The documentation is not suitable to provide the information needed by the new developers.

The list can be continued almost infinitely. The following observation makes the situation even worse. Most maintenance work is usually done during the long-lasting maturity phase. The introduction and growth phase of the custom made software often take no more than one to two years. But it is not uncommon to use and operate the software for many years after these two phases are over. So after one to two years of fun during the development of the software, a significantly longer and frustrating period of maintenance and further development is to follow? It doesn’t need not be like this!

So let’s ask the next question: What can and should be done?

There are no universal recommendations and best practices as everything is customized  and individual:

• The software to be developed is unique (custom) by definition.
• The organizations that create and use the software are differing.
• The vertical integration and types of cooperation between customers, business departments, the IT and the service providers are very diverse.

Still one thing in general remains true: All parties involved should think about software maintenance in  the early stages, when the software development project is envisioned and planned. I frequently experience that these considerations are not made at all. People involved often justify and explain this by claiming that it is not possible to envision what will happen in one or two years. Although accurate predictions are indeed difficult if not impossible, this should not prevent the customers, the project sponsors and the project team to at least make rough assumptions. These are to be checked afterwards and adjusted continually in the sense of an agile planning.

All project participants should consider the following questions in any case:

• How much change is expected?
  Is the environment more stable or are frequent changes expected due to changes in business models, business processes or new legal requirements?
• What organizational approach to maintenance is taken?
  “You build it, you run it” by the development team or a transition after the main development phase to a maintenance team? Or maybe a combination of both approaches? Does the maintenance team already exist? If not, when and how is it established? Is a transition of responsibilities to or from a service provider planned?
• How can knowledge be preserved and passed on?
  How can you prevent that knowledge is lost during transfer of responsibilities? What documentation should be build in order to ensure subsequent maintenance of the software? How can the documentation be kept up to date over time?
• What actions must be taken in order to develop a software that is “easy to maintain”?
  Which tools and infrastructure can be provided that make maintenance easier?
• What regulations and agreements have to be made in terms of service level agreements concerning the operation of the software?
  How important is the final product for the business of the sponsor and how and with what usage patterns will it be used? What impact does this have on organizational structures, contracts to be written and maintenance budgets to be provided?

In my experience it has proven useful to examine these issues in a structured way with a checklist starting as early as during the envisioning of the software development project. First rough assumptions and determinations can be made already at this early stage. If necessary and possible measures and follow-up activities are defined.

Fig. 2: Extract of a checklist used at codecentric

At regular intervals and at certain milestones during the product life cycle, such as the product launch, the project team gets back to the checklist and updates the measures based on the new and almost certainly changed situation. And of course during the transition of the responsibility for the product to a maintenance team the checklist and the measures will be checked and adjusted.

One last question: What specific measures can people responsible for the product take?

How can the maintainability of software be increased? What actions ensure that you avoid the maintenance hell or that at least the temperatures can be kept bearable? I will now present a few exemplary solutions without going into too much detail. It should again be stressed that due to the high individuality of each situation there can be no universal set of best practices.

Create maintainable and well-structured software:

Comply to  the principles of software craftsmanship  while developing the software. Take care of regular updates regarding libraries and tools and keep on refactoring the software in order to control technical debt.

Use automation:

Try to achieve a high level of automation for testing, provisioning of the environments, CI and CD. Adhere to DevOps approaches and invest in a flexible and scalable infrastructure. Infrastructure monitoring has become standard, but think about adding application monitoring to your system.

Aim for a good distribution of knowledge:

Distribute knowledge continuously in the team and avoid head monopolies. Don’t try to pass knowledge via some theoretical training events, but by letting the experts and knowledge holders work on specific tasks together with their “trainees” . Produce adequate documentation (“as much as necessary, as little as possible”) and keep it up to date.

Define clear responsibilities:

Give some thoughts to team structures and processes to be followed. But please find lightweight and flexible solutions. Consider using Kanban to enable a flow based approach and to enforce continuous improvement.

Create a capable and motivated maintenance team:

Invest in your employees. See value in education and training of the team members. Rely on employees with a high problem solving ability (“T-Shaped People”) and give them access to specialists to solve particular problems.

Conclusion

Unfortunately, there is no universal set of best practices to succeed in the field of maintenance of custom made software. However, this should not be an excuse to ignore the subject until it is too late and we literally find ourselves in maintenance hell. There is no need to define some global processes and tools for a whole company. Rather I would like to encourage you to consider this important topic individually for each project (product) at an early stage and to start questioning and updating the insights continually. Aspects of the maintenance of custom made software should be taken into account on time and with a holistic view. Only this can lead to reduced pain and frustration in the maintenance phase for your software products. Only this way you can successfully do maintenance and even have fun doing it.

By the way: This blog post first was published in German in a special edition („Digitalisierung“ ) of the codecentric Softwerker-Magazine. Order this issue for free .