Agility is not Informality

Agility is about following the Manifesto for Agile Software Development, not the sloppy and crazy implementations of Scrum, XP, or other home-grown schemes out there. Sometimes I feel like people are using Agility as an excuse to be informal.Agility is about discipline and being formal for key things.

Two of the key principles of the Agile Manifesto are:

  • Working software over comprehensive documentation
  • Responding to change over following a plan

chaosSomehow some developers have interpreted this as meaning that there are no formal processes to be followed. But just because you are putting the priority on working software does not mean that there can be no documentation; just because you are responsive to change doesn’t mean that there is no plan.

Agile development when properly implemented can accomplish great things.  When it is just an excuse to be undisciplined and cover-up cowboy development then a so called ‘Agile’ process will work very poorly and the organization will just spin its wheels.

Agile development in general raises productivity by 18.2% and quality by 27.1%

Extreme programming (XP) specifically raises productivity by 12.8% and quality by 17.8%

Agile software development does have formality and it requires quite a bit of discipline to stick with it. A few formal practices for Scrum include:

  • Stand-up meetings
  • Work burn down charts
  • Sprint demos
  • Simplicity
  • Improvement through reflection

Stand-up Meetings

Stand-up meetings are an integral part of the Scrum process and is a key discipline of Scrum, when they are done properly. This meeting is a coordinating mechanism of a self-organizing team where each member states:

  • What have they finished
  • What are they working on
  • What external issues do they need help on

The scrum daily meeting raises productivity by 16.4% and quality by 23.5%

Believe it or not, there are some Scrum teams that sit down for the stand-up meeting. What the heck is this? The purpose of standing up is to be slightly uncomfortable and keep in mind that the meeting is supposed to move quickly. The meeting will be short if you do it standing up.

You state what features that you have finished so that team mates waiting on your deliverables know where you are at. Finished means that you have unit tested your code (i.e. TDD) and that black box testing can start if there is a tester in your Scrum team.

Automated unit testing raises productivity by 16.5% and quality by 23.8%

You state what you are working on so that other team mates know what you are doing and can organize their work accordingly.  Don’t bother to state how you solved problems, that was an issue for planning with team mates before you started coding.

You state what you need help on that is external to the Scrum team so that the Scrum master can resolve the issue.  This is not a design meeting, if there are issues that you need to resolve with team mates then you can resolve this issue right after the meeting.

Burn Down Charts

Work burn down charts is a formal discipline and essential to success (see Who Needs Formal Measurement?). The work burn down chart is the heart of accountability in Scrum.  For the chart hours to get to 0 on a regular basis, it relies on the principles of: 1) commitment, 2) execution, and 3) estimates.

The point of a self-organizing team is that it commits to work each sprint to get the work done.  If it turns out that they over committed to a sprint then they must do whatever it takes to get the job done.  That may mean extra hours or it may mean being extra creative, but once you commit to work then you have got to get it done.  That means execution, it means doing what you say that you will do.

You may have a sprint or two where the work burn down chart does not go to 0. If this is happening on a regular basis then you have either an estimation problem or a team commitment problem.  If engineers are regularly underestimating their tasks then it means that you really don’t know how long things take.  You should consider some of the techniques in the Personal Software Process (PSP) to improve your ability to estimate.

PSP development raises productivity by 21.3% and quality by 31.3%

Another cause of work burn down charts not going to 0 is that the team is not really self-organizing.  The team needs to commit to the work to be done, it will never work if the organization assigns the work to be done in the sprint.

If you lose discipline during the sprint and don’t keep the work burn down charts up to date, then you are not tracking progress correctly.  Remember, you can’t manage what you don’t measure.

Inadequate progress tracking reduces productivity by 16.0% and quality by 22.5%

Sprint Demos

The sprint demo is not optional, it is a mandatory part of the process.  The sprint demo is to demonstrate near-production code and demonstrate a working system.  The team needs to put pressure on itself that the code must be complete by the end of the sprint; this forces a self-organizing team think through the amount of work that they can commit to. No demo = no team commitment.

Remember the whole point of Agile development is to put the emphasis on working software.  If you are not producing working software every sprint, then what are you doing? Whatever you are doing it is not Agile development.

The Agile Manifesto puts emphasis on working software.  If your sprints are not producing working software then you are not doing agile development.

Simplicity

The best designs are the simplest designs because they are easier to understand and maintain.  In a previous article No Experience Required! I discuss how years of experience do not make for better programmers.  You might expect that the best developers have more years of experience, but 8 studies over 30 years show that this is not the case (you might even want to get rid of poor performers with many years of experience, but see the article 🙂 )

In fact two persistent facts about the differences between the best and worst developers is that:

  • program execution speed about 10 to 1
  • program size 5 to 1

Much of this results from simplicity. The best developers plan their code and write simple and straight forward routines. The are rewarded by having smaller programs that execute quickly. The worst developers do not plan their code and shoot from the hip. They write convoluted code that is much larger and has much worse execution time.

Agility is about simplicity. It is spending 80% of your time thinking and 20% of the time writing simple and elegant code. It is not about spending 100% of your time writing complex and convoluted code that even you can’t understand.

Simplicity does not happen, it is planned; it takes tremendous effort to write simple code. The best developers plan their code (see PSP above) and are continually refactoring their code to make designs simpler to understand and maintain.

Automated restructuring raises productivity by 8.0% and quality by 11.7%

Code refactoring raises productivity by 4.3% and quality by 6.7%

Improvement through Reflection

A key discipline in Agile development is to reflect on what you can do better as individuals and as a team. There is always room for improvement, there is often a better way to do things.  Professional developers are keen to keep learning and improving their craft, and the best way to do this is by reflecting on the sub-optimal choices that we have made in the past and trying to do better in the future.

Conclusion

Agile development is about following the Agile Manifesto, it is not about informality.  True agility is developed by having the discipline to be formal in many things:

  • Team synchronization
  • Learning to estimate
  • Tracking progress through burn down charts
  • Committing to create production code
  • Planning simple code
  • Improving by not making the same mistakes

Agility is about being disciplined and formal in processes that count


References

N.B. All productivity and quality percentages were derived over 15,000+ actual projects


Articles in the “Loser” series

Want to see sacred cows get tipped? Check out:

Moo?

Make no mistake, I am the biggest “Loser” of them all.  I believe that I have made every mistake in the book at least once 🙂

VN:F [1.9.22_1171]
Rating: 0.0/5 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Who needs Formal Measurement?

We all know the expression “You can’t manage what you can’t measure“, but do we really understand it?

After execution, feedback is an essential part of all processes.  Just think about how difficult it would be to drive from home to work wearing a blindfold.  Without your sense of sight to give you feedback on the traffic signals and the locations of other cars you would crash your car.  Yet we develop software systems without instituting formal measurement programs all the time and wonder why we succeed so rarely?  (for success rates see Understanding your chances of having a successful software project)

You can’t manage what you can’t measure

No measurement means no feedback, which means your chances of success are minimized. Success is possible without formal measurement but it is much easier with formal measurement.

Formal measurement raises productivity by 20.0% and quality by 30.0%

A best practice is one that increases your chance of succeeding, it does not guarantee it. It has been established that formal measurement is a best practice, so why do so few people do it?

Measurement has a cost and organizations are petrified of incurring costs without incurring benefits. After all what if you institute a measurement program and things don’t improve?  In some sense managers are correct that measurement programs cost money to develop and unless measurement is executed correctly it will not yield any results.  But is there a downside to avoiding measurement?

Inadequate progress tracking reduces productivity by 16.0% and quality by 22.5%

Failure to estimate requirements changes reduces productivity by 14.6% and quality by 19.6%

Inadequate measurement of quality reduces productivity by 13.5% and quality by 18.5%

So there are costs to not having measurement.  Measurement is not optional, measurement is a hygiene process, that is, essential to any process but especially to software development where the main product is intangible.

A hygiene process is one which can prevent very bad things from happening. Hygiene processes are rarely fun and take time, i.e. taking a shower, brushing your teeth, etc.  But history has show that it is much more cost effective to execute a hygiene process than take a chance of something very bad from happening, i.e. disease or your teeth falling out.

There are hygiene practices that we use every day in software development without even thinking about it:

  • Version control
  • Defect tracking

Version control is not fun, tracking defects is not fun; but the alternative is terrible.  Only the most broken organizations think that they can develop software systems without these tools.  These tools are not fun to use and virtually everyone complains about them, but the alternative is complete chaos.

Formal measurement is a best practice and a hygiene practice

The same way that developers understand that version control and defect tracking is necessary, an organization needs to learn that  measurement is necessary.
Is Formality Necessary?

The reality is that informal measurement is not comprehensive enough to give consistent results. If measurement is informal then when crunch time comes then people will stop measuring things when you need the data the most.

When you don’t have enough formality then processes take longer and by extension cost more.  When you have too much formality then you have process for processes sake and things will also take a long time.  Any organization that implements too much formality is wasting their time, but so is any organization that does not implement enough.

When you suggest any formal process people immediately imagine the most extreme form of that process; which would be ridiculous if it is implemented that way. We have all been in organizations that implement processes that make no sense, but without measurement how do you get rid of these processes that make no sense? For every formal process that makes sense, there is a spectrum of implementations. The goal is to find the  minimum formality that reduces time and costs. When you find the minimum amount of formal measurement you will accelerate your development by giving yourself the feedback that you need to drive your development.

What to Measure

It seems obvious, but incorrect measurement and/or poor execution leads to useless results.  For example, trying to measure productivity by measuring the hours that the developers sit at their machines is as useful as measuring productivity by the number of cups of coffee that the developers drink.Another useless measure is lines of code (LOC), in fact, Capers Jones believes that anyone using LOC as a measurement should be tried for professional malpractice!Measuring the the three things mentioned above will improve productivity and quality because there will not be a negative effect on your organization:

  • Measuring progress tracking (productivity +16.0%, quality +22.5%)
  • Estimating requirements changes  (productivity +14.6%, quality +19.6%)
  • Measurement of quality (productivity +13.5%, quality +18.5%)

Other things to measure are:

  • Activity based productivity measures
    (productivity +18.0%, quality by 6.7%)
  • Automated sizing tools (function points)
    (productivity +16.5%, quality by 23.7%)
  • Measuring requirement changes  (productivity +15.7%, quality by 21.9%)

So to answer the question: who needs formal measurement?

We all need formal measurement

 


References

N.B. All productivity and quality percentages were derived over 15,000+ actual projects


Articles in the “Loser” series

Want to see sacred cows get tipped? Check out:

Make no mistake, I am the biggest “Loser” of them all.  I believe that I have made every mistake in the book at least once 🙂

VN:F [1.9.22_1171]
Rating: 0.0/5 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)

Not planning is for Losers

Only the ignorant don’t plan their code pathways before they write them.  Unless you are implementing classes of only getter and setter routines code needs to be planned. We talk about The Path Least Traveled The total number of pathways through a software system grow so quickly that it is very hard to imagine their total number. If a function X() with 9 pathways calls function Y() which has 11 pathways then the composition function X() ° Y() will have up to 9 x 11 = 99 possible pathways. If function Y() calls function Z() with 7 pathways, then X()  ° Y()  ° Z()  will have up to 693 = 9 x 11 x 7 pathways. The numbers add up quickly, for example a call depth of 8 functions each with 8 pathways means 10.7 million different paths; the number of possible pathways in a system is exponential with the total depth of the call tree. Programs, even simple ones, have hundreds if not thousands (or millions) of pathways through them.

Negative vs Positive Assurance

Quality assurance can only come from the developers, not the testing department.  Testing is about negative assurance, which is only a statement that “I don’t see anything wrong”; it doesn’t mean that everything is correct, just that they can’t find a problem.  Positive assurance which is guaranteeing that the code will execute down the correct pathways and only the developer can do that. Quality assurance comes from adopting solid practices to ensure that code pathways are layed down correctly the first time

Any Line of Code can be Defective

If there are 10 pathways through a function then there there must be branching statements based on variable values to be able to direct program flow down each of the pathways. Each pathway may compute variable values that may be used in calculations or decisions downstream. Each downstream function can potentially have its behavior modified by any upstream calculation. When code is not planned then errors may cause execution to compute a wrong value.  If you are unlucky that wrong value is used to make a decision which may send the program down the wrong pathway.  If you are really unlucky you can go very far down the wrong pathways before you even identify the problem.  If you are really, really, really unlucky not only do you go down the wrong pathway but the data gets corrupted and it takes you a long time to recognize the problem in the data. It takes less time to plan code and write it correctly than it takes to debug complex pathways.

Common Code Mistakes

Defects are generally caused either because of one of the following conditions:

  1. incorrect implementation of an algorithm
  2. missing pathways
  3. choosing the wrong pathway based on the variables

1) Incorrect implementation of an algorithm will compute a wrong value based on the inputs.  The damage is localized if the value is computed in a decision statement, however, if the value is computed in a variable then damage can happen everywhere that value is used.  Example, bad decision at node 1 causes execution to flow down path 3 instead of 2. 2) Missing pathways have to deal with conditions.  If you have 5 business conditions and only 4 pathways then one of your business conditions will go down the wrong pathway and cause problems until you detect the problem.  Example, there was really 5 pathways at node 1, however, you only coded 4. 3a) The last problem is that the base values are correct but you select the wrong pathway.  This can lead to future values being computed incorrectly.  Example: at node 10 you correctly calculate that you should take pathway 11 but end up going down 12 instead. 3b) You might also select the wrong pathway because insufficient information existed at the time that you needed to make a decision.  Example:  insufficient information at node 1 causes execution to flow down path 3 instead of 4. The last two issuses (2 or 3) can either be a failure of development or of requirements.  In both cases somebody failed to plan…

What if it is too late to Plan

Whenever you are writing a new section of code you should take advantage of the ability to plan the code before you write it.  If you are dealing with code that has already been written then you should take advantage of inspections to locate and remove defects.  Don’t wait for defects to develop, proactively inspect all code, especially in buggy modules and fix all of the code pathways. Code inspections can raise productivity by 20.8% and quality by 30.8%

Code Solutions

The Personal Software Process (PSP) has a specific focus that every code section should be planned by the developer before it is implemented.  That means that you sit down and plan your code pathways using paper or a white board before using the keyboard.  Ideally you should spend the first part of your day planning with your colleagues on how best to write your code pathways.  The time that you spend planning will pay you dividends in time saved. PSP can raise productivity by 21.2% and quality by 31.2% If you insist on writing code at the keyboard then you can use pair programming to reduce errors.  By having a second pair of eyes looking at your code algorithmic mistakes are less likely and incorrect decisions for conditions are looked at by two people.  The problem is that pair programming is not cost effective overall. Pair Programming can raise productivity by 2.7% and quality by 4.5% Studies confirm that code sections of high cyclomatic complexity have more defects than other code sections.   At a minimum, any code section that will have a high cyclomatic complexity should be planned by two or more people.  If this is not possible, then reviewing sections of high cyclomatic complexity can reduce downstream defects. Automated cyclomatic complexity analysis can raise productivity by 14.5% and quality by 19.5% Design Solutions All large software projects benefit from planning pathways at the macroscopic level.  The design or architectural planning is essential to making sure that the lower level code pathways will work well. Formal architecture for large applications can raise productivity by 15.7% and quality by 21.8% Requirements Solutions Most pathways are not invented in development.  If there is insufficient information to choose a proper pathway or there are insufficient pathways indicated then this is a failure of requirements.  Here are several techniques to make sure that the requirements are not the problem. Joint application design (JAD) brings the end-users of the system together with the system architects to build the requirements.  By having end-users present you are unlikely to forget a pathway and by having the architects present you can put technical constraints on the end-users wish list for things that can’t be built. The resulting requirements should have all pathways properly identified along with their conditions. Joint application design can raise productivity by 15.5% and quality by 21.4% Requirements inspections are the best way to make sure that all necessary conditions are covered and that all decisions that the code will need to make are identified before development.  Not inspecting requirements is the surest way to discovering that there is a missing pathway or calculation after testing. Requirement inspections can raise productivity by 18.2% and quality by 27.0% Making sure that all pathways have been identified by requirements planning is something that all organizations should do.  Formal requirements planning will help to identify all the code pathways and necessary conditions, however, formal requirements planning only works when the business analysts/product managers are skilled (which is rare 🙁 ). Formal requirements analysis can raise productivity by 16.3% and quality by 23.2%


Other articles in the “Loser” series

Want to see more sacred cows get tipped? Check out

Moo?

Make no mistake, I am the biggest “Loser” of them all.  I believe that I have made every mistake in the book at least once 🙂

References

VN:F [1.9.22_1171]
Rating: 0.0/5 (0 votes cast)
VN:F [1.9.22_1171]
Rating: 0 (from 0 votes)