Sunday, 12 January 2014

Technical Debt - Breaking the Poor Development Cycle

What is Technical Debt

In nice simple terms technical debt is the cost of not doing software development in a sustainable, best practice approach. In terms of the actual development work it can range from poor coding practices including hard coding in dates, poorly named variables, sloppy SQL to poorly designed tables and stored procedures. In the wider SDLC in can be a lack of business understanding, poor requirements, heavy intensive manual testing, no automated testing, slow manually intensive migration processes and non-existent operational monitoring. Technical debt is often the outcome of a poorly planned and executed projected, time and budget constraints, an ever changing scope and a management philosophy of "get it done".
This article focuses on the development driven technical debt. How the development practices can be changed and improved so the debt can be paid and no new debt is incurred. Like your finances, you need to avoid the habit of taking out a consolidation loan to pay back credit card debt. Its just one loan to pay another. You need to save money or earn more.

Is there an Actual Cost

Yes. Its call incidents, BAU, break/fix, or the need to constantly make manual changes to keep the application running or the business needs to change it behaviours to work with the application in a less efficient manner. Incident work can be captured in a time tracking application. Operational and process costs are likely to be less visible. Just like finance debt (see the US Debt Clock) technical debt incurs an interest cost. This cost does not repay the debt. It gets higher with more technical debt. Just like financial debt you need to decide what level of "interest rate" you can live with. If your spend 35% of your team's time fixing incidents you can assume this is too high. If you need to spend 5% of your team time, this may be ok to live with. Like a mortgage interest rate at some point you do need to pay it off.

Technical Debt Cycle

Technical debt can easily become a cyclical process. Its not a good cycle. Debt tends to add to more debt and efforts to resolve it can only add to it. Its a vicious circle and an easy one to get into.

Poor Practices

It starts with poor quality design and coding. This can occur for a number of reasons, the Developer many not be familiar with the programming language, they may not know best practice, they are under time pressure to get the code written so they hard code, they may not be disciplined to follow best practice, there are no code reviews, no unit tests and no quality control until it hits the test environment or worse production environment.  
This is start of a cycle of events which the following diagram highlights.


Frequent Incidents

Poorly written code will break and it will break often. Worse of all it breaks in production when someone or (something) is trying to use it. This leads to incidents being logged - by users or by operational monitoring. These incidents need to be fixed, so the Developer needs to drop what their working on and work on the incident. This is called context switching. If the incident happens in some code you just worked on then you can fix it pretty quickly. Most happen in code you worked on a few weeks, months or even a year ago. You then have Knowledge Decay - and you need to time to re-familiarize yourself with the code. Even worse if its somebody else's code and an application you have never looked at before it can take several days to resolve an incident. Plus the incident will need testing and migrating. Its not just the Developer's time that it being used to fix the incident. If you are getting frequent incidents this will be happening a lot. Its frustrating for all involved.

Low Recoverables\Less Project Time\More Pressure

If your Developers are working on incidents then they are not working on projects - building new applications or enhancing existing ones. This tends to lead to projects falling behind as they are getting less resource time to complete tasks by an agreed deadline. It also leads to lower billable hours for the team. Long project meetings and email chains also do not help. Not a specific development or SDLC practice, but the sign of a poor and non-collaborative culture. Less time for projects tends to lead to more pressure to get them done. More pressure can easily lead to poor practices and our cycle is now three-quarters complete

Touch & Go

All of the previous three elements add up to a Touch and Go mentality. Developers are under pressure to fix incidents, work on projects, they also need training and over time begin to build up high leave balances. As time is precious a "death march" approach of getting tasks completed takes hold, even the most skilled and competent Developers will just get tasks done for the sake of it. They may be aware that their code is poor, that they are rushing, the become to expect interruptions, changing priorities, confidence drops and morale sags. 
This completes the cycle. The poor practices are further cemented and will get worse - like a hurricane over a warm ocean - it begins to spin faster. 

Breaking the Cycle

Once you have identified all the elements that make up the Technical Debt cycle and how it reinforces and spirals, you can start planning to break it. This will require process, procedural, coding changes and most importantly a cultural changes. You need to be prepared to do things properly and to defend the good habits you are trying to form.

Good Practices

In order to ensure that best practice is followed you need to define best practice. Establish coding standards. Include peer reviews. Make it clear what is expected of the Developers. Code which is poorly written will not pass a review and will require the Developer to re-write. Also the Technical Debt cycle encourages a silo-ing of Developers. They stop talking to and helping each other. This needs to be broken down, Daily stand-ups are very useful for this. Get the Developer to stand and talk, ask them about progress, issues, problems, etc. Offer them support and assistance. Ensure that robust and thorough unit tests are prepared, they are run, they pass. They are updated as required. Check the results as part of the review. Break the tasks down into manageable pieces of work. 
This will ensure that good quality code is written. It is more robust and hence less likely to break. This starts the cycle in a positive way - see below.


Less Frequent Incidents

Looking at a list of incidents can be a bit daunting, where do start and what do you do? Firstly, establish the priority order. Fix the ones that cause the most problems. Also look for like incidents or incidents where the same underlying problem causes multiple incidents to be raised. Have a plan and update it regularly. Stick to the tasks which are high priority. If possible look at including the incident fixes in planned project work. Accept there is a risk here of having to test fixes to existing code as well as new code. Remember each release requires a test migration and a test resource. It may be worth the risk - time and cost - to do all the fixes and enhancements in one go. This will also require some skilled negotiations with the business, they might not see the need to reduce technical debt as a priority. They may want only "new stuff" built. They might not realize the doing this just adds to the debt.
Also it may take sometime to see the benefit of less frequent incidents, especially if there is a long list. Plan frequent releases. Seek to build trust by fixing the higher priority and releasing to the business. The more you fix and the less of the new code that breaks the happier they are.

Improved Recoverables

Just like any good investment, good coding and good practices pays more over time. Less incidents, less fixes and good quality code means more time for building and enhancing applications. The "good stuff", what Developers enjoy. Stick to daily stand ups with short and to-the-point meetings. Ensure that the Developers and project team as a whole stand up and talk to each other. Visualize the results - be it on a whiteboard or a JIRA board - so the team (and others) can see progress.

Be Thorough

None of the above will sink in and be repeated if the unacceptable pressure to deliver is still present. This is where project and team leadership becomes important. Firstly, the Developers. Make sure they are supported as they continue with these practices. If you use external resources which are costly contractors make sure they cross-skill and train the team. Establish internal training sessions, make time for a couple of half day sessions a month in a training room. Identify the knowledge gaps and resolve them. Again, this will take time. However at this point the Developers know what they need to do properly. They are supported by their Project Manager and their Line Team Leaders/Managers. Priorities don't change frequently. Developers know they can sit down, focus and complete a piece of work where quality is the first priority. Once you are good at something, you can get faster at doing it and still be good. The other way around rarely works. 

Project Managers & Team Leadership

Doing stuffproperly - the first time, second time, third time - can take time and cost money, until your processes, coding, etc. matures to the point where you are fast, efficient and still producing quality code. Until then the Project Managers and Team Leaders must protect the team. Give them the space to complete tasks properly, run interference from customers, senior managers, etc. all with different priorities and time frames. Team Leadership is vital, even the best PM will be tempted to accept a little less quality for quicker outcome - and this has a habit of spiralling. Establish a good relationship with the PM, ensure they know of the standards and methods you follow, why you follow them and the expected outcome. Its important that the Developers do not get dragged into project or organisational politics. That customers, senior managers, etc. can distract and change priorities causing frustration and starting of the technical debt cycle again. Be prepared to take the time to explain the above approach to your customers, senior managers, etc. Once you get runs on the board with good code and good robust applications, there will be less pressure and less distractions.  

But People Yell at Me with Their Priorities

No matter how logical, sensible, best practice your approach is there will also be others seeking to distract and send you spiralling back into the bad old ways. You can't fix every problem at once for everyone. Drive a stick in the ground and do it. Be prepared to face escalations from those you are not first on the list to have their applications fixed. Be prepared for tough conversations with those whom just want to get tasks finished and close off projects, without consideration for whom is going to fix the left over project issues. At the end of the day you will have to attend meetings, respond to emails and escalations defending this approach. Regardless of this noise, make sure they team are not exposed to it. They have space, they get things done properly (I think I have said this twice, this is good, its important).


At the end of the day you may find you still have technical debt. You may realize that you don't have time and resources to fix all of it. It comes down to make can you live with. Can you live with paying a mortgage. Most homeowners do and they accept it. Can I live with a high credit card balance with a high interest rate? Probably not. Be prepared for some distractions, some fixes will take to take place, some context switching will happen soon escalations will need dealing with. This is ok, as long as they not frequent or hugely time consuming.  Breaking the cycle can often take a lot of time, several months would be great, but realistic it may take a year or more. Always look for the opportunity to include debt reduction in any work you do. 

Avoidance Is Best

The best and most efficient way of resolving technical debt is to avoid it. When you get the opportunity to build a brand new application, sit down and plan, establish the best practice standards, roles & responsibilities, what code to reuse, what new components to use. Challenge requirements, model and test before you build. Visualize building your application with quality code, robust unit tests, reliable automated deployments, automating testing, good test scripts, engaged users, a pragmatic and enthusiastic business owner. The outcome will be worth the effort you put in.

Key Learnings

Work out what practices and behaviours makes up your technical debt, how you got into debt and how you keep getting deeper into debt. This is the first step in breaking it.
Plan how to get out of debt. Don't accept going in cycles or getting deeper into debt. Trumpet the wins and successes you achieve, this helps build the trust and time you will need to break the cycle.
Avoidance is best. 

No comments:

Post a Comment