The Water-Scrum-Fall Guide to Requirements

 Recently I have been involved in a number of conversations in relation to more efficient and visible requirements gathering. The issues relate to Business Analysts and the length of time it takes them to complete requirements and when are finally complete, the high level of detail and complexity that is captured - way beyond the scope of the original request. It’s a very easy trap to fall in but there is a way out. If you are in a Waterfall type project then there are some agile principles that can help.

Shut the Gate

One suggested approach to resolve this issue is to have more weekly reports and gate reviews. Fill in a template, email it off, give a weekly update and so on. To infrequent, too impersonal and too easy to paint a picture that is different from what is actually happening. This won’t work. It actually slows the process down. Which in turn means more Gate Reviews, more formal “template” updates and more smoke when you are trying to find the fire.

Think!

What do I need for day one production, use it to refine your scope. This is rapidly becoming my mantra. If I had to drive between Auckland and Hamilton what would I need? A car, licensed, registered, roadworthy car and petrol. Do I need a car radio? No. Do I need Air Conditioner? No. Do I need a Coffee Holder? No.  Think the same way when capturing requirements.

Manage

The second part of the solution is being managed, both the person and the tasks. No one should be allowed to work without reporting progress and having visibility of what they are doing. This is not an SDLC problem, it is something that Agile will not resolve, this is a person management problem. 

Be Agile-ish

What I have recommended is to add some good Agile practices to the Requirements phase. You don’t need  to wait for the coding to start to use some of the good agile practices.

• Cross functional team, involve the functional, technical and test experts.
There is often a surprising amount of business logic in the heads of the Developers and other technical experts. Their input into the requirements is vital to ensuring they are sensible, manageable and realistic.

• Daily stand-ups
Discuss progress, issues, upcoming events each day, do not wait for weekly reports, gate reviews, etc. Surface and deal with issues on the day.

• Visible tickets
What ever you are working on can go into a card, sticky note, JIRA ticket. Define the task and whom it is assigned to and hence whom is responsible. Completing requirements still means working through a series of tasks and having them visible means progress can be tracked.

• Define done before you start
Decide on what you need for day one production readiness. Think the basics. Keep the requirements you capture as simple as possible and for each one define what is done. This makes life significantly easier for the developer, tester and those doing the user acceptance testing. Also it helps prevent scope creep as it prevents different uninformed and undefined perceptions of what is done.

Build Frequently & Demo Often

Once the requirements are signed off, sprint (literally) to get the first build in front of the customer. At the completion of the first demonstration the customer(s) will provide feedback which will mean changes. Don’t think of these as scope creep or changes, but the natural evolution of the software. 

Rinse & Repeat

Again capture this feedback, discuss, socialise, make it visible, define done and track it. Be pragmatic, if you find a method or approach that works then use it. Mix methods to find the best results. Never stop trying something new to solve a problem. 

Why Projects Succeed

Over the past year I have worked on or managed a number of projects which have been quite successful, they have complete on time, close to, just over or under budged and with happy stakeholders.  Good luck or good project management, here are my thoughts:

Project Management

Defining bad project managers is easy, they are the ones that ask you how their project is going cause they don’t know and/or don’t care, or ask you if they should turn up to their own project meetings. Disengaged, disorganised, no milestones, no updates, no idea of issues and risks, no chance of success. Go too far the other way and you can have a micro managerial fanatic bogged down in detail and making no progress.

A good PM has a firm, but subtle hand on the tiller. They guide the project with informed decisions, communicate clearly and often, build and use relationships to get work done, they delegate, trust, monitor, measure and have contingency plans. They do not seek to offload their responsibilities or hide from the difficult projects. They know how to get things done and who to go to.

Managing Changing Requirements

Unless you are running a very short, very limited scope project, requirements will change. It’s how you manage this change that is important. You can push back and remind the senior user whom signed the high level requirements that they cannot change their minds. Or you can expect changes, the first few demonstrations will also reveal changes to the original requirements and design. As Senior Users begin to use the iteratively built features of your software they will give you feedback which may invalid or ideally fine tune the original requirements. This is great, this is what you want your users to tell you. No point sticking to the original requirements and design if no one wants the end product.

Roles & Responsibilities

I have sat in several meetings and asked “Who is the Project Manager”, only to receive silence or “Who is the Scrum master?”  and even once “Who is the customers/senior user?”. Who is doing what and from that when. This forms the basis of a plan. Otherwise you’re like a group of outfielders running around under a fly ball all yelling “yours”.  Without R&R you cannot have a plan and without a plan, even a high level one, you chances of succeeding are low.

Define whom you and what you need to do, add in a timeframe and bingo, the elements of a good project start coming together.

Communication (previously known as Co-location)

About a year ago I would have sworn that co-location was a key element in a project’s success. But since then I have completed 5 projects working with a company based in another city, doing all the work remotely.  Having clear R&R (as above) and communications is the catalyst to co-location success. Since the person (or persons) is not sitting next to you, you have to put extra effort into communicating in a clear and concise manner. Start with a good plan and work from there, ensuring that everyone knows what they are doing when. Actually I believe that Non-Location maybe better for a project as it prevents you from assuming good communications will come from simply sitting together.

Technical Factors (infrastructure & environments)

This used to be such a headache. I know I have an Applications basis, but how many times has a project been delayed by waiting for infrastructure to be built, configured, ports opened, firewalls changed and soon on. Even in the days of virtual environments and clouds it can still take time. Then there is the time and frustration cost of a failed migration from a System Admin whom cannot follow instructions. Having a good Operations person on the project is invaluable, days of waiting for database and file access turn into minutes and migrations become seamless and hassle free. Frequently – and often without realising it – they become one of the key project resources and unsung hero. I can think of several where I currently work.

Human Factors (context switching & resource planning)

Managing a team of 20+ people includes a strong element resource planning and management. Projects start, stop, pause and take off suddenly. Though a variety of formal and informal communications channels we are made aware of current or planned projects. We plan and resource accordingly. We also had contingency based on project process. The aim of this is to ensure a Developer(s) can stay on a project until completion and avoid distractions. Which leads into context switching. In simple terms it’s an overload of tasks (be it project or incidents) that prevents an individual focusing on completing their work in quality and timely manner. Carefully resource planning and project priorities  will avoid this from happening. The culture of the organisation you work for can assist or hinder this process. I have heard senior managers tell me it’s ok to have a Developer assigned to three active projects – maybe they should learn/remember how to code and try it.

Catalyst & Gelling (Soft Human Factors)

Have you ever notice that there are certain individuals on projects that others will listen to, whom are knowledgeable, willing to support and help others and though not team leaders or managers, others will naturally follow their instruction.  Project teams will naturally bond and gel around these people and their wisdom and knowledge becomes invaluable. I have worked with a few people like this, they are naturally intelligent, calm, patient and helpful. They will also often be very modest and step back from the limelight and move onto the next project. They know who they are, good project managers want them and good people managers retain them.

Conclusion

Go through all the above factors and you come to the conclusion that a successful project is not the result of good luck, but one of good management . Good communication and the skills of an air traffic controller juggling several planes in and outgoing also helps. that it all involves people – whom understand technology  - but not technology itself. I will leave you with my favourite quote of the year so far.

The major problems of our work are not so much technological as sociological in nature. (PeopleWare Productive Projects and Teams (DeMacro & Lister).

The Basics of Good Application Management

 

  Recently I completed the corporate Wiki upgrade, bang on time and at one-third of the planned budget.  When I presented the closure report I was asked by our Accountant  why I was so under budget. I  replied “I am a really really really good Project Manager. [Happy Face]”, followed by a humorous exchange.  I had a similar result   one month ago with the JIRA upgrade, on time and only half the budget.

However it did get me thinking why it was such a successful project. It all went to plan and the final production upgrade was uneventful.  It comes down to the basics of good Application Management. I have read many application management articles, some are too high level and many can be overly complex,  so based on my experience and the theme of “simple is good” here are the basics:

Strategy & Processes

Have a vision for your application, have plans to get it there. Measure those plans and change them if they are not working. After the upgrade to the newest version last year I had a vision of the Wiki usage increasing, more spaces, more content, more views and more users. The application Wiki statistics have proven this.

Process, the how, what and why – in detail. During the our big upgrade we realised that a lot of processes we needed existed in email folders or did not exist. Discussing, proposing, socialising and finalising processes around support, upgrade paths, plugins, macros, etc. meant we can move from project to operational mode without losing momentum and knowledge.

Treat DEV and TEST like PRODUCTION

One of the issues we faced with our three year catch-up upgrade project last year was the mess that the Development and Test environments were in. Development was a partial failed install of an old version and was dead, Test was running, but was a failed install of yet of another version and no one could log on and Production was stable, running and out of date and was been used like development and test. Plus there were issues with disk space, missing files, missing images, permissions, firewalls, ports, backups, user numbers, licensing, plugin versions, performance all of which needed to be fixed.

After considerable effort to align all three environments it was made very clear to those whom access these environments they are not a play pen for bored Developers or Operations staff. All changes on all environments need my approval. I regularly check them and cycle through changes. If anything is broken I want to know why and I want to know whom.  A discussion ensures followed by corrective action.

We now have an ecosystem which is  easy to cycle your upgrades and changes through.

Use Experts

The corporate Wiki is not our core business, we are not interested in customising the application or developing any form of intellectual property or specialist knowledge. We are happy to outsource.

So when we commenced planning last year we sought the help of experts in this particular field. Especially as previous efforts had failed through lack of internal resource availability and expertise. Our external partners are experts, they  are also fast, efficient, effective and  thorough.

Partners not Vendors

This is a deliberate choice of words. I want to work with another company that sees our working relationship as long term and beneficial to all involved. They go beyond the boundaries of our agreement to assist and we do likewise. This is a benefit to both of use. It still amazes me how many companies treat their clients as an invoice recipient and  if you are lucky - a biannual coffee visit.

Learn from Your Lessons Learnt

This sounds obvious, but how often is it done. There can often be a “Groundhog Day”  affect with IT Projects. Understand how and why previous projects did not work and plan differently. In our case, we used an external expert, listened to their advice and respected their expertise. This made a huge difference in the quality of technical work done. One of the weak points in our previous project was communication, during the planning of the most recent upgrade we developed a communications strategy, a communications log and followed the communications plan. The outcome was a much better informed and happier user community.  The lessons from our last upgrade will go into the next upgrade,  this is to prevent the good points for being forgotten as much as to remediate the bad points.

Understand Your User Community

You don’t want to release “New Coke” thinking it’s the next big thing.  Find your experts and champions and get them on-board, work with them and get close to them. Listen and learn from their experiences. They will offer good advice and make great user acceptance testing Testers.  They will championing your application to others and spread the word.

Benefits

Define and measure. If you don’t have a good reason to do something, then don’t do it. This comes back to your strategy. If you are doing something and planning to get a benefit and do not, go back and think again, you are doing it wrong.

Closing

All the above can take some hard work and some good planning (let’s not confuse simple with easy).  Many of the above areas overlap, lessons learnt with communication, expertise and change control on all your environments. Above all – have a plan. Be realistic, define, measure and realise your benefits. It gets easier when the elements fall into place.

Prince2: The Simple Approach is Best

 

After years of informal project management – running projects with no formal qualifications or more often, being asked to "assist" after the project is stalled, struck, and in deep do-do. I decided to undertake some formal PM training, namely Prince2. Having some formal training and an understanding of a project methodology would help round out my knowledge in this space. Part of a Development Manager role is to be heavily involved in projects so it’s good to know.

So here are my thoughts.

Roles and Responsibilities. One of key decisions in the earlier stages (starting up or initiating) of a project phases. Yet so easily often forgotten. How can you have a project without a project manager? The answer is you cannot (if only this was a Prince2 exam question). But also whom is the technical lead, BA, Tester, who is doing what. This is a common failing of many projects. Not having clearly defined R&R. So you don’t know who is doing what, what is being delivered when? Whom is signing off of and so on. Confusion reigns.

Planning.  One of the overriding themes of Prince2  was – be prepared.  You will need to setup a lot of management products and they are there for a reason. Issue Register, Daily Log, Lessons Learnt, Risk Register and so on, then there are the governances, Risk Management Strategy, Communications Strategy and so on. But a well prepared project, is a ready for anything project, when risks and issues occur, you are respond in an organized way. Like a well drilled ship’s crew you response to each alert and emergency as per the plan. The plan gives you confidence it what you are doing and that  you can resolve the situation. Everyone knows what they’re doing and when they’re doing it.

But is it Simple, Tailor it to what you need.

What struck me as important was how simple many of these management products where. The project briefs were just that, brief, one to two pages. Many of the strategies were one to two pages for simple projects. They were concise, to the point and useful. You went to your Risk Strategy when a risk occurs, you use your Communications Strategy to guide your communications to who, when and what message. Keep it simple.

 How many times do you work with a Program or Project office and when you go to prepare these products, they send you long, painful, tortuous templates with equally useless examples of ones already done. By the time you fill in there SO much detail the purpose is lost. The document becomes unusable, unused, useless and you manage off the seat of your pants. Ever wonder why an aircraft evacuation  card is simple? Do you want a 10 page document on how to get off the plane after it crash lands? No, where’s the door and how do I get there. Done.

Takeaways form Inspiring Project Managers or Middle Aged Development Managers.

So going forward. I like the structure, preparedness and planning that Prince2 dictates. Be ready for the expected and unexpected and have a plan to deal with it. Keep the plans simple, keep the management products simple. There are some nice and simple learnings, keep Lessons Learnt Logs, future projects will praise your foresight and refer back to them for wisdom and learnings. Define, measure and capture your benefits, years down the track and project is still delivering, state it, measure it, publicize and celebrate it.

Most of all if you’re the stick in the mud that practices planning and preparedness, your successes will stand out from other stuck in the mud.

The Inherent Quality of Quality

There is a particular section – especially two sentences – from the PeopleWare eBook that has become quite literally lodged in my head.

“In some Japanese companies, notably Hitachi Software and parts of Fujitsu, the project team has an effective power of veto over delivery of what they believe to be a not-yet-ready product.”

And

“Enough of a quality culture has been built up so that these Japanese managers know better than to bully their workers into settling for lower quality. Could you give your people power of veto over delivery?”

My Japanese wife will simply point out that being a hairy white barbarian I naturally have no concept of good quality or culture, hence I cannot marry the two concepts together. Going beyond the humor of our cross cultural marriage these are quite profound statements.

Imagine being on a project team and having the final say. No pressure from the project manager, your manager, senior managers, directors, they all understand that quality is first and if the product is not a quality product you do not deliver. 

Quality initiatives can come in any forms. Often they are driven from the ground up. The truly professional motivated IT people will brand together, set standards, establish principles and seek to eliminate poor practices. However without management support these ideas are likely to last more than a week, maybe a month if you are really keen. The constant and often overbearing demand for an output – be it any output – overrides the ideal of quality. How many times has a senior manager pursed their lips and said “I don’t care, it needs to be done”.

Managers themselves – often aware of the technical debt or far from complimentary feedback on their teams’ products – may also embark on quality initiatives. The more senior ones will have enough authority and budget to essential to drive these initiatives. Teams will respond to this. Managers may lack the details of implementing quality. At a detail level they will need to trust and empower their team members. However such great ideas will get derailed by others whom will politic, escalate and literally yell to get things done.

Therein lays the obvious yet challenging secret to the Japanese success. Culture. They all accept it. It is an establish professional protocol. Not a fad, not a CIP, not a fancy presentation from a suite wearing smooth latté slipping consultant. You don’t need to sell an ideal to a group that have in established from birth. The entire organization buys, they all accept it. And therein lies the reason for sustainable success – the entire organization buying in.

It makes you think of John Lennon’s famous lyrics “imagine all the projects, with quality outcomes…”

Agile (and we didn’t even know it)

Back in 2003 I was involved in the resurrection of the building consents project. This was the third bite in the fourth year of a very prolonged and unsuccessful series of projects to replace a legacy system. The first failed as it placed a poorly coded ASP application on top of the legacy system –which would not accept multiple web users and crashed on go live day, the second failed largely due to the PM being so disinterested they did not attend their own project meetings and the technical lead going off at angles meaning that most of the code was half done. So it was my turn, using the leftover code form the second attempt. There was me, Mickey the band new Developer and Ruth, the expert from the business, whom became the ultimate champion and unsung hero of the project.

So we kicked off. The list of current defects was extensive. Virtually most of the application did not work and you could not login. We decided to focus on each functional area, get it working before moving to the next. So we started with the login screen, security, administration options and the basic consent summary landing page. Each day Mickey would code away,  while I would peer review and test (between being the BA, Trainer and PM). On a daily basis we would pop up and see Ruth in her quiet little office and spend an hour having her test the application and get feedback. We would then do a bit coding in the afternoon and so on.

After the consent summary it was onto the name and address module, building inspections, consent tracking, fees, conditions and finally notes. Each section was focused on and worked on until Ruth was happy that it was working as required. We then demonstrated and tested the code with a smaller group of key users. We then worked on creating a consent, printing and reports (which were really basic). Finally after two months of daily code, demo, test, code and repeat we were ready to begin more formal testing.

 We made some very pragmatic decisions as we went. The layout and logic of the consent tracking was not ideal, but we did not have the time to re-write it, the name and address register was unnecessarily complex (and ironically probably still is 10 years later),  the inspection business logic was too complex to code within the time frame. Each of these issues was prioritized into post-production releases – of which we had planned three before we finished.

Finally into UAT, training and release. Go live day was entertaining. As the previous aforementioned version 2 had died on go live day all the users jumped on and started searching and looking up consents, waiting for it too fall over. After about half an hour I popped up and asked if some would like to create a new consent – and actually use it. Off they went, no issues.

There are many agile-ish elements to this project. Sprints (off sorts), multi-functional closely located team, frequent demos, peer programming, retrospectives and so on.  At the time (and for many years later) the waterfall approach was standard at Council – despite efforts to introduce more agile practices. There were many common sense elements to why we succeeded - a dedicated team, engaged focused and extensive business knowledge, committed senior management whom wanted a successful outcome, a wider group of enthusiastic  users, limited but visible project governance. Also at the time using the corporate property system as the sole and automated source of property data as opposed to manual entry was new, but lead to a vast improvement in accuracy i.e. we went from having 7% of consents not matching to valid properties, to 0%.

The application is still in use, 10 years plus, parts of the original code and a lot of the original data model are still in use. We never did get around to upgrading to .NET and it’s still an ASP application. Due for decommission shortly as a new corporate wide solution is adopted.  Mickey left council two years later and two years after that Ruth retired. Over those four years myself and Ruth worked through all the backlog and made numerous and regular changes based on the feedback she received. Every year we would have a cake to celebrate the birthday of the application, this continued right until 2010 when I left, interestingly enough it was often over a mouthful of cake that some great ideas came out. This project remains the one I am most proud of as a Developer and I do miss working with Ruth and Mickey.

Standup My Friend

Recently I was asked what were some ways  to create a positive workplace culture, the organization in question had established a social club and it was not working. No one was showing up at the events and it was about to be axed.  I had a chat with my peer and told him social clubs only work when people work. Let me explain:

Organizations that are challenged with the need to develop or enhance culture will decide that one of the best ways is to organize a social club. In the ideal classless society such as New Zealand this is a seemingly great idea, grab the BBQ, butter the bread, squeeze the sauce and by virtue of wrapping a sausage around a piece of bread we are great mates. How many times at the social club BBQ do people come, eat, say hi and vanish with the sausage in a napkin back to their desk? How many times to they sit down and chat long after the sausages have slowly brunt or gone cold, the flies have landed on the bread and the salad remains untouched?

To want to socialise together you first need to work together. You come to work (noun) to work (verb). Simple. You need to complete tasks that are assigned to you. So far so good. Nobody said you needed to like your colleagues, this is assumed that it will happen. So your neighbour three desks down the row and on the same project team. What do you know about them?

Daily standups are an effective communications mechanism for projects. The team get to know what’s going on every 24 hrs. But there are also a great social mechanism. Inevitably small and occasionally candid pieces of personal information slip into conversations. Suddenly the annoying tester three desks down isn’t the jerk who seems to enjoy failing your code, but a guy with a pregnant wife who spent the weekend shopping for baby clothes. He seems more interested in his wife’s blood pressure than yours.  But that’s understandable, after all you have two small children. You were once that naive person that though antenatal classes taught you everything you needed to know about childbirth. He also seems like an okay kind of guy, so why not go and have a chat to him – offline – about his testing approach, expectations, ask for his opinion and feedback before submitting the code.

I am not suggest daily standups become a substitute social club, but they are a good start to social interaction. Communication is why humans evolve to where we are on the food chain, we can co-ordinate, retain and spread oral knowledge, entertain, empathize and so on with these skills. Otherwise we would be another struggling homo- species and cat food on the plains of Africa.

And antenatal classes, you forget everything at the first contraction.

The Mistake is Not Making a Mistake

There is a quite thought provoking section in the book PeopleWare: Productive Projects and Teams (DeMarco & Lister) called “A Quota for Errors”. In starts with:

For most thinking workers, making an occasional mistake is a natural and healthy part of their work. But there can be an almost Biblical association between error on the job and sin. This is an attitude we need to take specific pains to change.

This is quite a profound statement. How many organisations have methodologies that have become so rigid that – in an effort to drive out errors – they have killed off innovation, free thinking and creativity. But when is an allowance for an error a sign that standards are slipping and code reviews are not robust?

Good question. If you tell the team it’s okay to make errors. Can you see the number of failed code reviews reaching new highs and performance reviews becoming less pleasant. But the more I think about it there is an important principle here.

Why would you create an environment for people to make mistakes? An environment to take risks, think of new ideas and approaches. Otherwise:

Fostering an atmosphere that doesn’t allow for error simply makes people defensive. They don’t try things that may turn out badly.

Another great quote, they won’t try anything new, the risk of failure outweighs the risk of success.

 With errors, there is the good and bad. Cutting corners – no error handling, no rollback, not closing connections, basic code indentation – these are lazy errors. Mistakes someone makes when they are not being careful. These errors have cowboy written all over them.

Good errors occur when the Developers are thinking, trying something new, a different approach, a different library, code module, technology, whatever. They see a gap in the current approach or an opportunity to improve a methodology, the go for it. If they fail. They tried and failed. As opposed to did nothing, learnt nothing, gained nothing and won’t try again. If they succeed, what’s the outcome, the saving, the benefit, the payback. What’s the return on the risk now, next month, next year, next piece of software that needs building.

The average level of technology may be modestly improved by any steps you take to inhibit error. The team sociology, however, can suffer grievously.

So a somewhat late New Year’s resolution, allow the team to make errors, allow them to be human – not a component in a production line – and see the benefits.

Holding Auditions for Interviews

Over the past few years I have conducted a number of interviews. I take pride in having a robust process for interviewing and selecting the right candidate. I often say that 90% of the manager’s job is done if you hire the right staff. I look for great team fit, excellent problem solving skills and solid technical skills. We go through two interviews,  a technical assessment  and a final socialisation interview with the team.

I feel it’s important to include the team in the process. But thinking about it further the last step is a little informal. Ok, if you cannot chat about work, projects, code, social activities with someone then there is not a good chance you cannot work together. But you need something more solid that will start the bonding process between the team and the newcomer. A more robust process, something that challenges the candidate and gives the team confidence in the selection.

Reading through PeopleWare: Productive Projects & Teams (DeMarco & Lister) – I found the chapter Holding Auditions quite enlightening, here are some extracts:

“The business we’re in is more sociological than technological, more dependent on workers’ abilities to communicate with each other than their abilities to communicate with machines. So the hiring process needs to focus on at least some sociological and human communication traits. The best way we've discovered to do this is through the use of auditions for job candidates.

The idea is simple enough. You ask a candidate to prepare a ten- or fifteen minute presentation on some aspect of past work. The date is set and you assemble a small audience made up of those who will be the new hire’s co-workers.

At the end of the audition and after the candidate has left, you hold a debriefing of those present. Each one gets to comment on the person’s suitability for the job and whether he or she seems likely to fit well into the team.

Even more important, any new person hired is more likely to be accepted smoothly into the group, since the other group members have had a voice in choosing the candidate.

It soon became clear that the audition process served to accelerate the socialization process between a new hire and the existing staff members. A successful audition was a kind of certification as a peer.”

What I Like

You are assessing the all-round presentation skills and ability of the candidate. During the presentation the energy, enthusiasm and passion for what they do will come out. The keenness for the role will be self-evident. Telling a team in a monotone voice that you are “passionate about IT”, isn't going to cut the mustard.

You are assessed by your peers. We do this already. I like this idea. As a manager with a team of 24 people realistically you are not going to spend a lot of time working closely with your new hire. Others in your team will. During the presentation they may pick up and strengths and weaknesses that you did not notice. 

The last paragraph of the audition process is a great statement. A good presentation starts the bond between the existing team and the newcomer. The team are confident that the right person has been hired. They have had a say in the process. Come day one they – and the new person – will be less nervous, they will both know what to expect.

What I Don’t Like

Nothing. It might be the first time a candidate has been asked to present in this manner, so there might be some culture shock. This is good. A robust interviewing process is a sign of an organised and well lead team.

What’s Next

I have PeopleSoft Developer role I need to take to market, so test of all I can trial this new formalised process in the not too distant future. 

Technical Debt - Breaking the Poor Development Cycle

What is Technical Debt

In nice simple terms technical debt is the cost of not doing software development in a sustainable, best practice approach. In terms of the actual development work it can range from poor coding practices including hard coding in dates, poorly named variables, sloppy SQL to poorly designed tables and stored procedures. In the wider SDLC in can be a lack of business understanding, poor requirements, heavy intensive manual testing, no automated testing, slow manually intensive migration processes and non-existent operational monitoring. Technical debt is often the outcome of a poorly planned and executed projected, time and budget constraints, an ever changing scope and a management philosophy of "get it done".

This article focuses on the development driven technical debt. How the development practices can be changed and improved so the debt can be paid and no new debt is incurred. Like your finances, you need to avoid the habit of taking out a consolidation loan to pay back credit card debt. Its just one loan to pay another. You need to save money or earn more.

Is there an Actual Cost

Yes. Its call incidents, BAU, break/fix, or the need to constantly make manual changes to keep the application running or the business needs to change it behaviours to work with the application in a less efficient manner. Incident work can be captured in a time tracking application. Operational and process costs are likely to be less visible. Just like finance debt (see the US Debt Clock) technical debt incurs an interest cost. This cost does not repay the debt. It gets higher with more technical debt. Just like financial debt you need to decide what level of "interest rate" you can live with. If your spend 35% of your team's time fixing incidents you can assume this is too high. If you need to spend 5% of your team time, this may be ok to live with. Like a mortgage interest rate at some point you do need to pay it off.

Technical Debt Cycle

Technical debt can easily become a cyclical process. Its not a good cycle. Debt tends to add to more debt and efforts to resolve it can only add to it. Its a vicious circle and an easy one to get into.

Poor Practices

It starts with poor quality design and coding. This can occur for a number of reasons, the Developer many not be familiar with the programming language, they may not know best practice, they are under time pressure to get the code written so they hard code, they may not be disciplined to follow best practice, there are no code reviews, no unit tests and no quality control until it hits the test environment or worse production environment.  

This is start of a cycle of events which the following diagram highlights.

 

Frequent Incidents

Poorly written code will break and it will break often. Worse of all it breaks in production when someone or (something) is trying to use it. This leads to incidents being logged - by users or by operational monitoring. These incidents need to be fixed, so the Developer needs to drop what their working on and work on the incident. This is called context switching. If the incident happens in some code you just worked on then you can fix it pretty quickly. Most happen in code you worked on a few weeks, months or even a year ago. You then have Knowledge Decay - and you need to time to re-familiarize yourself with the code. Even worse if its somebody else's code and an application you have never looked at before it can take several days to resolve an incident. Plus the incident will need testing and migrating. Its not just the Developer's time that it being used to fix the incident. If you are getting frequent incidents this will be happening a lot. Its frustrating for all involved.

Low Recoverables\Less Project Time\More Pressure

If your Developers are working on incidents then they are not working on projects - building new applications or enhancing existing ones. This tends to lead to projects falling behind as they are getting less resource time to complete tasks by an agreed deadline. It also leads to lower billable hours for the team. Long project meetings and email chains also do not help. Not a specific development or SDLC practice, but the sign of a poor and non-collaborative culture. Less time for projects tends to lead to more pressure to get them done. More pressure can easily lead to poor practices and our cycle is now three-quarters complete

Touch & Go

All of the previous three elements add up to a Touch and Go mentality. Developers are under pressure to fix incidents, work on projects, they also need training and over time begin to build up high leave balances. As time is precious a "death march" approach of getting tasks completed takes hold, even the most skilled and competent Developers will just get tasks done for the sake of it. They may be aware that their code is poor, that they are rushing, the become to expect interruptions, changing priorities, confidence drops and morale sags. 

This completes the cycle. The poor practices are further cemented and will get worse - like a hurricane over a warm ocean - it begins to spin faster. 

Breaking the Cycle

Once you have identified all the elements that make up the Technical Debt cycle and how it reinforces and spirals, you can start planning to break it. This will require process, procedural, coding changes and most importantly a cultural changes. You need to be prepared to do things properly and to defend the good habits you are trying to form.

Good Practices

In order to ensure that best practice is followed you need to define best practice. Establish coding standards. Include peer reviews. Make it clear what is expected of the Developers. Code which is poorly written will not pass a review and will require the Developer to re-write. Also the Technical Debt cycle encourages a silo-ing of Developers. They stop talking to and helping each other. This needs to be broken down, Daily stand-ups are very useful for this. Get the Developer to stand and talk, ask them about progress, issues, problems, etc. Offer them support and assistance. Ensure that robust and thorough unit tests are prepared, they are run, they pass. They are updated as required. Check the results as part of the review. Break the tasks down into manageable pieces of work. 

This will ensure that good quality code is written. It is more robust and hence less likely to break. This starts the cycle in a positive way - see below.

 

Less Frequent Incidents

Looking at a list of incidents can be a bit daunting, where do start and what do you do? Firstly, establish the priority order. Fix the ones that cause the most problems. Also look for like incidents or incidents where the same underlying problem causes multiple incidents to be raised. Have a plan and update it regularly. Stick to the tasks which are high priority. If possible look at including the incident fixes in planned project work. Accept there is a risk here of having to test fixes to existing code as well as new code. Remember each release requires a test migration and a test resource. It may be worth the risk - time and cost - to do all the fixes and enhancements in one go. This will also require some skilled negotiations with the business, they might not see the need to reduce technical debt as a priority. They may want only "new stuff" built. They might not realize the doing this just adds to the debt.

Also it may take sometime to see the benefit of less frequent incidents, especially if there is a long list. Plan frequent releases. Seek to build trust by fixing the higher priority and releasing to the business. The more you fix and the less of the new code that breaks the happier they are.

Improved Recoverables

Just like any good investment, good coding and good practices pays more over time. Less incidents, less fixes and good quality code means more time for building and enhancing applications. The "good stuff", what Developers enjoy. Stick to daily stand ups with short and to-the-point meetings. Ensure that the Developers and project team as a whole stand up and talk to each other. Visualize the results - be it on a whiteboard or a JIRA board - so the team (and others) can see progress.

Be Thorough

None of the above will sink in and be repeated if the unacceptable pressure to deliver is still present. This is where project and team leadership becomes important. Firstly, the Developers. Make sure they are supported as they continue with these practices. If you use external resources which are costly contractors make sure they cross-skill and train the team. Establish internal training sessions, make time for a couple of half day sessions a month in a training room. Identify the knowledge gaps and resolve them. Again, this will take time. However at this point the Developers know what they need to do properly. They are supported by their Project Manager and their Line Team Leaders/Managers. Priorities don't change frequently. Developers know they can sit down, focus and complete a piece of work where quality is the first priority. Once you are good at something, you can get faster at doing it and still be good. The other way around rarely works. 

Project Managers & Team Leadership

Doing stuffproperly - the first time, second time, third time - can take time and cost money, until your processes, coding, etc. matures to the point where you are fast, efficient and still producing quality code. Until then the Project Managers and Team Leaders must protect the team. Give them the space to complete tasks properly, run interference from customers, senior managers, etc. all with different priorities and time frames. Team Leadership is vital, even the best PM will be tempted to accept a little less quality for quicker outcome - and this has a habit of spiralling. Establish a good relationship with the PM, ensure they know of the standards and methods you follow, why you follow them and the expected outcome. Its important that the Developers do not get dragged into project or organisational politics. That customers, senior managers, etc. can distract and change priorities causing frustration and starting of the technical debt cycle again. Be prepared to take the time to explain the above approach to your customers, senior managers, etc. Once you get runs on the board with good code and good robust applications, there will be less pressure and less distractions.  

But People Yell at Me with Their Priorities

No matter how logical, sensible, best practice your approach is there will also be others seeking to distract and send you spiralling back into the bad old ways. You can't fix every problem at once for everyone. Drive a stick in the ground and do it. Be prepared to face escalations from those you are not first on the list to have their applications fixed. Be prepared for tough conversations with those whom just want to get tasks finished and close off projects, without consideration for whom is going to fix the left over project issues. At the end of the day you will have to attend meetings, respond to emails and escalations defending this approach. Regardless of this noise, make sure they team are not exposed to it. They have space, they get things done properly (I think I have said this twice, this is good, its important).

Pragmatism

At the end of the day you may find you still have technical debt. You may realize that you don't have time and resources to fix all of it. It comes down to make can you live with. Can you live with paying a mortgage. Most homeowners do and they accept it. Can I live with a high credit card balance with a high interest rate? Probably not. Be prepared for some distractions, some fixes will take to take place, some context switching will happen soon escalations will need dealing with. This is ok, as long as they not frequent or hugely time consuming.  Breaking the cycle can often take a lot of time, several months would be great, but realistic it may take a year or more. Always look for the opportunity to include debt reduction in any work you do. 

Avoidance Is Best

The best and most efficient way of resolving technical debt is to avoid it. When you get the opportunity to build a brand new application, sit down and plan, establish the best practice standards, roles & responsibilities, what code to reuse, what new components to use. Challenge requirements, model and test before you build. Visualize building your application with quality code, robust unit tests, reliable automated deployments, automating testing, good test scripts, engaged users, a pragmatic and enthusiastic business owner. The outcome will be worth the effort you put in.

Key Learnings

Work out what practices and behaviours makes up your technical debt, how you got into debt and how you keep getting deeper into debt. This is the first step in breaking it.

Plan how to get out of debt. Don't accept going in cycles or getting deeper into debt. Trumpet the wins and successes you achieve, this helps build the trust and time you will need to break the cycle.

Avoidance is best. 

The Application 100 Day Review

Introduction

In 2013 Group Applications introduced the "One Hundred Day Review" concept. The idea was to have an approach to deal with "phase 2" items from a green fields application development and feedback post go live feedback on the application. To do this in a structured and organised approach so the original project closes, the project team moves to new work and there is not a list of outstanding tasks, improvements, on hold tickets that just falls into limbo. It was first used on the Volunteer Application. This went live in March 2013, in early July we went back to the business and asked them what they wanted changed and fixed. This was captured and carried out. These reviews are also planned for Study Options (February 2014) and FindAThesis (July 2014). As such they appear on the Group Applications Resource planning for 2014.

Background

Name and Address Register & the Yellow Line

Over 10 years ago I worked as the PM, BA, Co-Developer, Test Lead, etc with one other Developer and a Business SME (Subject Matter Expert) to build the Building Consents application for Manukau City Council. As we were pushed for time as a small team working on an existing code base and database that was left over from the first failed implementation, we made some very pragmatic decisions. One of the areas of the original application we did not change was the name and address register. We had used a model borrowed from another project that was not used in any other applications. The outcome was the process by which names, addresses and roles were added, edited and deleted seemed slow and clumsy to us. We thought users would hate it. However we did not have time to rebuild it and placed it first on our list to fix post go live. We went live on the 13th November 2003. When I left council in mid-2011 we still had not changed it. There was no need, no one complained, the users all found it easy to use and no one ever asked us once in 8 years to change it.

One of the modules we developed was the consent tracking page. Users could see the location of the consent and who was working on it. Users complained that they could not see the location without having to look at it in detail and read the entire screen. So we added a css and code change that highlighted the current location of the consent on the page with a bright yellow background. This background was ugly, it was so bright you could see it standing three feet back from the PC. The users loved it. It was so easy to see and use. So the term "the Yellow line" entered into both the Business and IT vocabulary.  We referred to the current location of a consent as the consent the yellow line was on. The Current Location report became known the Yellow Line report. It was simple and effective.

What's the Lesson?

The lesson is until you use an application thoroughly and frequently you won't fully understand what works and what does not. Deciding in test phases what needs to be changed, enhanced or improved may not be needed post go live and other areas which seemed okay may suddenly be a pain point. The example above is interesting as the Name and Address Register was infrequently used so no one minded the extra step or two to use. The consent tracking page was used heavily every hour of the day. In needed to be more user friendly. Ironically "fixed" with a bright yellow line.

Once an application is released there is a real temptation to ask for early changes. If there are urgent fixes this is a differently story, these need to be resolved. Non-urgent fixes may not need to be resolved or may become unimportant and can wait. Give the application and users time to use the application and ask them to capture feedback and discuss among themselves.

What's Included?

• Any tickets that have been left over from the original development, inclusive of bugs, improvements and wish-list items.
• Any current non-urgent fixes.
• Any changes which will make the application easier to use and maintain. 

Plus any requests from the teams responsible for support and maintenance. They may have ideas on how to improve it.

A lot of the requests from the first Volunteer Project 100-Day review were changes such as adding new fields to pages, adding a validation rule in, resolving an issue with an alert appearing in the log files, make some fields and input boxes bigger. Mainly small items which make the application a lot easier to use.

What's the Process?

The process is relatively simple.

• Before you commence development of the original application explain this concept to the business. Any ideas, improvements, etc that occur late in the build and in UAT can wait for 100 Day Review for inclusion.
• Complete and release the application, flag tickets remaining from the project in JIRA as required to be reviewed. Inform the business to start their 100-day review list.
• About 3 months later arrange a meeting with the business and discuss the list. There will need to be conversations around funding, priority against other activities and timing.
• Discuss what items need to be built, what bugs need to be fixed and if any of the existing bugs are so trivial the tickets can be closed. Be pragmatic.
• Stand up the review as a small project and apply normal SDLC rules and standards until completion.
• Close out the project as per normal.

What Happens if there are LOTS of Changes and Bugs?

This is likely to be outcome of the original project not capturing the requirements correctly, poor development, poor testing (including UAT) and possibly an unrealistic 100-day list. The projects stood up by 100-day reviews are meant to be small. If the volume of change is significant then you may need to consider a refactoring type project or build a completely new application.

Further Changes

Further requests for changes will come as the application is further used. Business processes are not static and the application will need to change to meet any changes in process. Users will find further ways to improve the application and may uncover bugs that are deeply buried in the application. There is no hard and fast rule to standing up projects to deliver these changes. This is something that should evolve over the life cycle of the application.

Closing Thoughts

The review does not have to be exactly 100 days, especially if you release 100 days before Xmas. I would not make it a shorter period, but a longer period of one to six months would be okay.

System Warrant of Fitness

Introduction

---

 A number of projects have kicked off and somewhat completed over the past 18 months looking to significantly enhance a number of applications. However many of these projects have failed to acknowledge the current technical debt of the applications that they are attempting to change. The end result is often a project which runs well over time and over cost, with unhappy project members, stakeholders, managers and customers.

 Moving forward, systems must be assessed - i.e. Warrant of Fitness - before attempting to add functionality, to determine whether they are in a fit state to be enhanced. If not, the project must include the payback of technical debt or scale back on such ambitious plans or not even commence.

Technical Debt Definition

---

 Technical debt (also known as design debt or code debt) is a metaphor referring to the eventual consequences of poor or evolving software architecture and software development within a codebase. The debt can be thought of as work that needs to be done before a particular job can be considered complete. "Interest payments" are both in the necessary local maintenance and the absence of maintenance by other users of the project. On-going development in the upstream project can increase the cost of "paying off the debt" in the future. One pays off the debt by simply completing the uncompleted work. It also may be considered as outstanding tasks to complete a project - to the customer's satisfaction - even after the project has "completed" and "closed". The OSCH Project is a good example of this.

 Another measure of technical debt interest is the number of support calls logged after the project closes. Ideally projects complete and no calls are raised; the Volunteer Project is a good example of this. Technical Debt Avoidance is far more efficient and cost effective than paying the debt off - as at the point of payment you include interest. A number of steps have been taken to reduce debt which involve solid adherence to SDLC principles. Inclusion of code reviews, early and frequent involvement of testers in analysis and development, thorough, sound, robust requirements, frequent regression testing and so on. As a result current projects are leaving behind little or no technical debt. Again the Volunteer Project is a good example of this.

Technical Debt Responsibility

---

 Payment of technical debt is not solely the responsibility of IT, but includes the application owner, steering committee members and stakeholders - all must take a level of responsibility. Business pressures, where the business considers getting something released sooner before all of the necessary changes are complete, builds up technical debt comprising those uncompleted changes. However poor coding, review, test, design practices i.e. - SDLC - are largely the responsibility of IT. 

Background

---

 The Admissions\Expressions project - in simple terms the merging of Expressions functionality into the Admissions application - was a deliverable from the FOC Programme of Work (POW). The project was focused on introducing significant change to the Admissions application. At the time the project was scoped there were a number of known issues with the Admissions application, inclusive of the obsolescence of the version of GRAILS the application was written in, the need to modularise and refactor the application, open tickets with support issues. Though known to the project team and business they were not considered necessary to resolve. Though the upgrade and modularisation was completed, the outstanding tickets were not scoped for inclusion in the project. However, as the project required regression testing it is almost certain that current known and unknown issues would be raised in testing. This then places the project in the position of apparently uncovering a range of "bugs' - which are in fact current issues. At this point a decision then needs to be made to fix or ignore. Some issues cannot be ignored as the inclusion of new functionality means a minor issue now becomes a critical issue. Effort spent on resolving these issues incurs a time, cost and opportunity spend. This pushes out the project timeline and costs and places the project team in an uncomfortable position. 

What is the SWOF?

---

At the business case stage of a project a systems warrant of fitness needs to be carried out. The system - and this is inclusive of infrastructure, database engines, network, current test scripts, automated testing, etc. (not just code) - needs to be assessed to determine its state of readiness for change. Necessary steps to check
• Known issues captured in JIRA. Outstanding tickets that are require action i.e. to be fixed; need to be included within the project. Tickets which won't be fixed ideally need to be closed flagged as "Won't Fix".
• Current versions of the code base and libraries.
• Current state of the code. This may require a deeper dive and review of the time. Code refactoring and modularisation are effort and cost intensive. However if necessary needs to be stated.
• Versions of underlying infrastructure including physical and virtual servers, database engines and versions, state of backups, restores, disaster recovery plans. This could be quite comprehensive and a decision would be needed on how deep to go. However there may be serious underlying issues - such as obsolete and poorly database - which impede application performance.
• Current state of test scripts and automated testing
• Last known run of regression testing, coverage of regression testing i.e. 100% of application functionality and results of regression testing
• Last known run of performance testing
• Current state of technical and functional documentation

 Desired Outcome

.The outcome of the assessment should determine whether the project is technically feasible and should flow into the future stages of the project, particularly the design phases. There are significant economies to be achieved if technical debt tasks are combined with functional enhancements i.e. regression testing of a code upgrade as well as functional change as opposed to two separate regression tests.