Not using UML on Projects is Fatal

Posted on December 17, 2014 by Dalip Mahal

The Unified Modeling Language (UML) was adopted as a standard by the OMG in 1997, almost 20 years ago. But despite its longevity, I’m continually surprised at few organizations actually use it.

Code is the ultimate model for software, but it is like the trees of a forest. You can see a couple, but only few people can see the entire forest by just looking at the code. For the rest of us, diagrams are the way to see the forest, and UML is the standard for diagrams.

They say, “A picture is worth a thousand words“, and this is true for code; even on a large monitor you can only see so many lines of code. Every other engineering discipline has diagrams for complex systems, e.g. design diagrams for airplanes, blueprints for buildings. In fact, the diagrams need to be created and approved BEFORE the airplane or building is created.

Contrast that with software where UML diagrams are rarely produced, or if they are produced, they are produced as an after thought. The irony is that the people pushing to build the architecture quickly say that there is no time to make diagrams, but they are the first people to complain when the architecture sucks. UML is key to planning (see Not planning is for losers)

I think this happens because developers, like all people, are focused on what they can see and touch right now. It is easier to try to code a GUI interaction or tackle database update problems than it is to work at an abstract level through the interactions that are taking place from GUI to database.

Yet this is where all the architecture is. Good architecture makes all the difference in medium and large systems. Architecture is the glue that holds the software components in place and defines communication through the structure. If you don’t plan the layers and modules of the system then you will continually be making compromises later on.

In particular, medium to large projects (>10,000 function points) are at a very high risk of failure if you don’t consider the architectural issues. Considering only 3 out of 10 software projects are successful only a fool would skip planning the architecture (see Failed? You get what you deserve!)

Good diagrams, in particular UML, allow you to abstract away all the low level details of an implementation and let you focus on planning the architecture. This higher level planning leads to better architecture and therefore better extensibility and maintainability of software.

If you are a good coder then you will make a quantum leap in your ability to tackle large problems by being able to work through abstractions at a higher level. How often do we find ourselves unable to implement simple features simply because the architecture doesn’t support it?

Well the architecture doesn’t support it because we spend very little time developing the blueprint for the architecture of the system.

UML diagrams need to be produced at two levels:

the analysis or ‘what’ level
the design or ‘how’ level

Analysis UML diagrams (class, sequence, collaboration) should be produced early in the project and support all the requirements. Ideally you use a requirements methodology that allows you to trace easily from the requirements onto the diagrams.

Analysis diagrams do not have implementation classes on them, i.e. no vendor specific classes. The goal is to identify how the high level concepts (user, warehouse, product, etc) relate to each other.

These analysis level UML diagrams will help you to identify gaps in the requirements before moving to design. This way you can send your BAs and product managers back to collect missing requirements when you identify missing elements before you get too far down the road.

Once the analysis diagrams validate that the requirements are relatively complete and consistent, then you can create design diagrams with the implementation classes. In general the analysis diagrams are one to many to the design diagrams.

Since you have validated the architecture at the analysis level, you can now do the design level without worrying about compromising the architectural integrity. Once the design level is complete you can code without compromising the design level.

When well done the analysis UML, design UML, and code are all in sync. Good software is properly planned and executed from the top down. It is mentally tougher to create software this way, but the alternative is continuous patches and never ending bug-fix cycles.

So remember the following example from Covey’s The 7 Principles of Highly Effective People:

You enter a clearing where a man is furiously sawing at a large log, but he is not making any progress. You notice that the saw is dull and is unable to cut the wood, so you say, “Hey, if you sharpen the saw then you will saw the log faster”. To which the man replies, “I don’t have time, I’m too busy sawing the log”.

Don’t be the guy sawing with a dull

UML is the tool to sharpen the saw, it does take time to learn and apply, but you will save yourself much more time and be much more successful.

Bibliography

Covey, Stephen. The 7 Habits of Highly Effective People
OMG, Unified Modeling Language™ (UML®) Resource Page

Rating: 0.0/5 (0 votes cast)

Rating: 0 (from 0 votes)

Why User Stories Rarely Work

Posted on July 11, 2014 by Dalip Mahal

All tools are useful when used appropriately, and User stories are no different.

User stories are fantastic when used in small teams on small projects where the team is co-located and has easy access to customers.

User stories can quickly fall apart under any of the following situations:

the team is not small
the project is not small
the team is not in a single location
customers can be accessed in a timely fashion
project end date must be computed

User stories were introduced as a core part of Extreme Programming (XP). Extreme Programming assumes that none of the above are happening; relax any of these constraints and you can end up with a process out of control. XP, and hence user stories, works in high intensity environments where there are strong feedback loops.

User stories need intense communication

User stories are a light-weight methodology that facilitates intense interactions between customers and developers and put the emphasis on the creation of code, not documentation. Their simplicity makes it easy for customers to help write them, but they must be complemented with timely interactions so that issues can be clarified.

Large teams make intense interactions between each pair of developers difficult; intense interactions keep everyone on the same page. Most organizations break teams into smaller groups where communication is through email or managers — this kills communication and interaction.

Larger projects have non-trivial architectures. Building non-trivial architecture by only looking at the end user requirements is impossible. This is like only having all the leaves of a tree and thinking you can figure out all the branches and the trunk must be, good luck.

User stories don’t work with teams where intense interaction is not possible. Teams distributed over multiple locations or time zones do not allow intense interaction. You are delusional if you think regular conference calls constitute intense interaction.

When emphasis is on the writing of code then it is critical that customers can be accessed in a timely fashion. If your customers are indirectly accessible through product managers or account representatives every few days then you will end up with tremendous latency.

Live weekly demos with customers are necessary to flush out misunderstandings quickly and keep you on the same page

User stories are virtually impossible to estimate. Often, we use user stories because there is a high degree of requirements uncertainty either because the requirements are unknown or it is difficult to get consistent requirements from customers.

Since user stories are difficult to estimate, especially since you don’t know all the requirements, project end dates are impossible to predict with accuracy.

To summarize, intense interactions between customers and developers are critical for user stories to be effective because this does several things:

it keeps all the customers and developers on the same page
it flushes out misunderstandings as quickly as possible

Diluted

All of the issues listed initially dilute the intensity of communication either between the team members or the developers and customers. Each issue that increases latency of communication will increase misunderstandings and increase the time it takes to find and remove defects.

So if you have any of the following:

Large or distributed teams
Project with non-trivial architecture
Difficult access to customers
Projects in new domains
Projects where knowing the end-date is necessary

Then user stories are probably not your best choice of requirements methodology. At best you may be able to complement your user stories with storyboards, at worst you may need some form of use case.

What the Heck are Non-Functional Requirements?

Posted on December 2, 2013 by Dalip Mahal

What the Heck are Non-Functional Requirements? Simply put, if functional requirements create code that will address the needs of the end-users (customers), then non-functional requirements address the needs of the people who install, operate, and configure the code.

Those people are the operations personnel and help desk personnel in whatever organization that uses your software. Every developer needs to be aware of what those non-functional requirements are and why operations personnel and help desk personnel are customers that are just as important as the end-users.

Functional Requirements

Functional requirements are baked into the code that developer’s deliver (interpreted or compiled). Events from input devices (network, keyboard, devices) trigger functions to convert input into output — all functions have the form:

This is true whether you use an object-oriented language or not. Non-functional requirements involve everything that surrounds a functional code unit. Non-functional requirements concern things that involve time, memory, access, and location:

Performance
Availability
Capacity
Continuity
Security

Non-functional requirements are slightly different between desktop applications and services; this article is focused on non-functional requirements for services.

If you have any knowledge of ITIL you will recognize that the last 4 items deal with the warranty of a service. In fact, the functional requirements involve the utility of a service, the non-functional requirements involve the warranty of a service.

Availability

Availability is about making sure that a service is available when it is supposed to be available. Availability is about a Configuration Item (CI) in the environment of the operations center that specifies how the code is accessed. Availability is decided independently of the code and is at best part of the Service Design Package (SDP) that is delivered to the operations department, at worst it is simply code dumped on the operations personnel.

Developer’s need to be aware of the difficulty of creating the CI for the operations personnel. If a CI is manually created then there will always be a potential for an error when the service is installed or updated. The requirement to create a CI is a non-functional requirement and the ability to minimize errors is another non-functional requirement.

Developer’s need to be aware of single-points of failure (i.e. services hard-coded to a specific IP) which causes fits in operations that are not running virtual machines (VM) that can have virtual IPs . The requirement to create code that is not reliant on static IPs or specific machines is a non-functional requirement. Availability is simplified in operations if the code is resilient enough to allow itself to easily move (or be replicated) among servers.

Availability non-functional requirements include:

Ability to easily make the CI
Automatic installation of CI or mechanisms
Ability to detect and prevent manual errors for a CI
Ability to easily move code between servers

Capacity

Capacity is about delivering enough functionality when required. If you ask a web service to supply 1,000 requests a second when that server is only capable of 100 requests a second then some requests will get dropped. This may look like an availability issue, but it is caused because you can’t handle the capacity requested.

Internet services almost always can’t provide enough capacity with a single machine and operations personnel need to be able to run multiple servers with the same software to meet capacity requirements. The ability to run multiple servers without conflicts is a non-functional requirement. The ability to take a failing node and restart it on another machine or VM is a non-functional requirement.

Capacity non-functional requirements include:

Ability to run multiple instances of code easily
Ability to easily move a running code instance to another server

Continuity

Continuity involves being able to be robust against major interruptions to a service, these include power outages, floods or fires in an operational center, or any other disaster that can disrupt the network or physical machines.

Where availability and capacity often involve redundancy inside a single operation center, continuity involves geographic and network redundancy. Continuity at best involves having multiple servers that can work in geographically distributed operation centers. At worst, you need to be able to have a master-slave fail over model with the ability to journal transactions and eventually bring the master back up.

Security

Security non-functional requirements concern who has access to functions and preventing the integrity of data from being corrupted.

Where access is concerned, how difficult will it be for operations personnel or help desks to set up security for users? Developer’s build in different levels of access into their applications without considering how difficult it will be for a 3rd party (help desk or operations) to set-up end users. The ease of setting up security is a non-functional requirement.

Data integrity is another non-functional requirement. Developer’s need to consider how their applications will behave if the program encounters corrupted data due to machine or network failures. This is not as important an issue in environments using RAID or redundant databases.

What Happens When You Forget Non-Functional Requirements

Commonly start-ups are so busy setting up their services that the put non-functional requirements on the back burner. The problem is that there are non-functional requirements that need to be designed into the architecture when software is created.

For example, it is easy to be fooled into building software that is tied to a single machine, however, this will not scale in operations and cause problems later on. One of the start-ups I was with built a server for processing credit/debit card transactions without considering non-functional requirements (capacity, continuity). It cost more to add the non-functional requirements than it cost to develop the software!

Every non-functional requirement that is not thought through at the inception of a project will often represent significant work to add later on. Every such project is a 0 function point project that will require non zero cost!

Generally availability, capacity, and continuity is not a problem for services developed with cloud computing in mind. However, there are thousands of legacy services that were developed before cloud computing was even possible.

If you are developing a new service then make sure it is cloud enabled!

Operations People are People Too

Make no mistake, operations and help desk personnel are fairly resourceful and have learned how to manage software where non-functional requirements are not handled by the code. Hardware and OS solutions exist for making up for poorly written software that assumes single machines or does not take into account the environment that the code is running in, but that can come at a fairly steep cost in infrastructure.

The world has moved to services and it is no longer possible for developers to ignore the non-functional requirements involved with the code that they are developing. Developer’s that think through the non-functional requirements can reduce costs dramatically on the bottom line and quality of service being delivered.

The guys that run operational centers and help desks are customers that are only slightly less important than the end-user. Early consideration of the non-functional requirements makes their lives easier and makes it much easier to sell your software/services.

It is no longer possible for competent developers to be unaware of non-functional requirements.

Understanding your chances of having a successful software project

Posted on July 11, 2012 by Dalip Mahal

We have been building software systems for over 50 years, and yet success rates remain extremely low, see Dan Galorath for some information. Different reports put the success rates at different levels but successful projects are rarely higher than 30-40%.

Report	Year	Successful	Challenged	Failure
Standish Chaos Reports	2009	32%	44%	24%
Saur & Cuthbertson	2003	16%	74%	10%
Tata Consultancy	2007	38%	62%

It might seem strange, but we don’t all have the same definition of success for software projects. Success is when a project delivers the expected benefits within 10% of cost and schedule. For me, a project is NOT successful when:

It does not deliver what was promised
It has cost or time over runs of over 10%
It has its scope dramatically reduced so that victory can be claimed
It does not have a positive net present value, i.e. it never breaks even

Under those conditions I’m guessing that there are even fewer successful projects out there. Let’s make software success extremely concrete. Imagine that you are on a street corner watching people cross the street to the other street corner. Imagine that out of every 10 people trying to cross the street only 3 people cross successfully, the other 7 get maimed or killed.

How interested would you be in crossing the street?

Sun Tzu wrote:

War is of vital importance to the state; hence it is a subject of inquiry which can on no account be neglected.

In our modern world, software is of vital importance to your organization. If you can solve your business issues by building software consistently and reliably you will gain a tremendous advantage over your competition.

One misconception is that software projects fail because “I am surrounded by idiots!” Just because we get frustrated at being unable to get software built does not make this statement true. In fact, the exact opposite is true, the average IQ of Computer System Analysts is 111.3[1], and any IQ above 110 is considered to be Superior Intelligence[2].

Don’t get me wrong; you might have a bunch of developers from the shallow end of the gene pool, but that does not explain how thousands of organizations fail to build quality software. The point is that software does not fail because there are not enough smart people looking at the problem.

There are plenty of consultants, a.k.a. snake oil salesmen, that are willing to sell you “silver bullet” solutions that will solve your every problem. Have you ever seen any of these really work? Each of these solutions will generally solve one aspect of your problem and leave you with a larger one to fix later; they will also leave big holes in your budget. Unfortunately, we often succumb to “silver bullet” solutions because they tell us what we want to hear.

To really fix your software development problems requires better understanding of basic principles; after all, there are software projects that succeed out there. Not surprisingly, the companies that have figured out how to develop software consistently and reliably tend to have the fewest failures. Learning how to develop software consistently and reliably requires that you learn how the following 5 elements intersect and affect each other:

Requirements
Project Management
Principles
Developers
Executives

Each of these elements will intersect with all of the others. You will continually stumble through software development until you get the minimum level of execution and synchronicity between these 5 rings.

Organizations that do not understand these 5 rings will create organizational structures and processes that are doomed to fail. Poor organizational structure and processes will create systemic problems that will lead to the following problems:

Constant fire fighting (blog)
Inflexible software (blog)
Poor architecture (blog)

Problems from poor organizational structure and processes will lead to failed software projects because of the SYSTEM, not the PEOPLE. However, people always assume that someone is to blame; they rarely look for problems inside the system. This will lead to severe morale problems and the loss of competent personnel.

Fixing your software development is a matter of understanding the principles of good organizational and process design. Once you understand how to balance the 5 elements you will begin to experience success in building software.

Appendix: Modern IQ Ranges for Various Occupations

According to modern IQ ranges, computer system analysts have one of the highest intelligence quotients of all professions.

[1] Average IQ by occupation (estimated from wordsum scores), January 22, 2011. Available from http://anepigone.blogspot.ca/2011/01/average-iq-by-occupation.html

[2] What Different IQ Scores Mean, April 12, 2004. Available from http://wilderdom.com/intelligence/IQWhatScoresMean.html