Software projects are notoriously difficult to complete on time, on budget and meeting user requirements. According to a study by The Standish Group (2009) only 32% accomplish that.
Is it wise to use the waterfall methodology for software development?
Before the term “waterfall model” was even coined, that model was outlined by Winston Royce in “Managing the development of large software systems” (1970). However, Royce didn’t advocate for its use in software development. Rather he said that this model was “risky and invites failure”! Iterative, and more specifically Agile development is now wildly recognized as a better approach for software development. It was defined in 1996 by the Agile manifesto (see annex). Among the different Agile methodologies, Scrum is now one of the most popular.
So why software project are failing to be delivered on time, on budget, answering the users needs? According to Charette (2005) there are 12 main reasons. We are going to analyze them one by one in the light of the Scrum methodology afterward.
Inaccurate estimates of needed resources
Hofstadter’s Law: “It always takes longer than you expect, even when you take Hofstadter’s Law into account”
By actually delivering a working piece of tested software at the end of every iteration, Scrum makes it easier to estimate the time required to complete the project. Since every iteration involves all the steps, including testing and debugging, it is easier to get realistic estimate of the velocity of the development.
Badly defined system requirements
The user stories allow the developers and client to discuss requirements throughout the project lifetime. In a Scrum project, the requirements are not defined months before implementation, but rather just before they are needed. Why is that better?
When one’s put the requirements in writings, some knowledge is lost. Even more is lost when those requirements are handed to the analyst, and more again to the programmers. (Poppendieck & Poppendieck, 2006). Scrum avoids most of these losses of knowledge because the requirements are defined “just-in-time” and also by encouraging direct communication to the project owner or the client itself.
Also, we could also argue that requirements are easier to define when the software is already taking shape before your eyes, using the feedback from previous released. From experience, I could add that developers tend to have a lot of news ideas when there are in front of a working piece of software.
Poor reporting of the project status
Scrum avoids poor reporting of the project status in many ways. First, the delivery of frequent working software deliverable is even better than status report because there’s no way to lie about it, either a feature is there or not!
Also, user stories are not as abstract as classic software requirements and therefore they are easier to understand by all stakeholders.
Also, the product backlog, i.e. the list of features left to be developed, offer a clear and updated image of what’s left to be done in lay terms. A sprint retrospective is also held after each iteration. It’s a kind of status report, but stakeholders are looking at a working piece of software.
Sloppy development practices
How Scrum can help reduce sloppy development practices? First let’s talk about motivation. Lack of motivation from software developers is notorious and Scrum can help a lot in that area.
First, because the team members together define what is realistic to do in a sprint, they are more motivated to achieve their objectives.
The team is also self-organizing, which means they make commitments to their peers.
Each members usually pick work-items themselves, usually based on their own expertise and taste, so they are more likely to excel at them and to have fun working on them.
It’s been reported that job satisfaction is higher (Melnik & Maurer, 2006), probably because of a higher intrinsic motivation. Higher job satisfaction lead to a higher productivity and lower turnover (Barney, 2009). Turnover is usually very costly so it’s a big improvement.
It’s also worth mentioning that turnover has less impact when using an Agile approach with Scrum because tasks are typically assigned on a daily basis. Intrinsic motivation also increase performance in tasks requiring creativity, cognitive flexibility and cognitive understanding (Barney, 2009). This could be one of the source of the higher quality output found in Agile project.
Creating a working piece of software as soon as possible gives clear idea of the risk involved in a project. In term of technology, Agile also promote a risk-driven development. The rule is to choose the riskiest, most difficult elements first, and also in earlier iteration.
“For example, maybe the client says « I want the Web pages to be green and the system to handle 5,000 simultaneous transactions. » Green can wait. In this way, the highest risks are surfaced and mitigated early rather than late.”(Larman & Basili, 2003)
Use of immature technology
One of the main risks in software development is that it is very difficult to foresee the technical problems ahead. By actually trying to use the technology, the developers can see quickly if the technology is mature enough for their project and get an idea about the learning curve involved.
Poor communication between customers, developer and users
“In busy downtown areas, flagging down a taxi often works better than having one dispatched to your hotel” (Poppendieck & Poppendieck, 2006)
One of the most important characteristics of Scrum is to actively involve the client for functionalities definition and prioritization for the entire development phase. One of the team members has to play the role of product owner. This person is physical present with the team and represents the client of the project when he’s not present. (I had the opportunity to play that role) The product owner is not a project manager. However, he needs to understand the business plan and the vision to make sure the software being developed answer the business needs.
Developers hate doing useless documentation. Doing the a complete requirements document at the start of the project often means that these requirement will either become outdated very quickly, or that the developers are going to spent a lot of time updating them (instead of coding). And it’s also important to mention that a document is useless if nobody reads it. Scrum emphasis direct and constant communication regarding the requirement. In other words, the discussion about requirements should be timely.
Unrealistic or unarticulated project goals
With frequent releases and greater involvement of the client (or the product owner who represent the client), the risk of having unrealistic goals his greatly reduced. In scrum, a project is made of user stories, i.e. tasks that will need to be accomplished using the software. Ex: “As a customer representative, I can search for my customers by their first and last name”. And the product owner (or the client) can add new stories himself. This avoids the problem of having developers interpreting abstract goals.
Inability to handle the project’s complexity
By delivering a working piece of software every iteration, the project becomes less abstract. Also, 45% of the feature of the software end up not being used at all (Larman & Basili, 2003), so by postponing the decision to implement a functionality or not, it help reduce the complexity.
Also, in the product backlog, you will find what we call Epics, i.e. big chunk of functionalities not yet broke down into user stories. Because they might never reach the sprint backlog, Scrum recommends waiting before breaking down epics into smaller user stories. For the same reason, Scrum also strongly discourages the use of a PERT chart. (Larman & Basili, 2003)
Poor project management
The role of the Scrum master is not to manage the project per se. The Scrum master has much less to do than a project manager because the team workload is self-manage. The scrum master will often be part of the development team.
He however has to enforce the rule of the Scrum and motivate the team. Also, since he doesn’t have to do a huge job of planning upfront, it reduces his importance. Therefore, he could actually be replace quite more easily than a PM.
The project owner represents the main stakeholder, the customer. Since he is physical present within the team, stakeholders are more represented than in a traditional software project. Also, Scrum prioritizes features delivery based on their business value, and the stakeholders can reprioritize them every month. Therefore we could argue that stakeholders are potentially more listened.
Again, because Scrum allows the requirements to be changed and prioritized based on their business value; this approach is more adapted to respond to commercial pressure quickly.
Scrum isn’t a panacea. There is some potential pitfalls or risk associated with this methodology. First, Scrum presupposes the participation of competent developers. Scrum therefore exposes incompetence more than other methods, which can be a good thing in the long run. Second, it is difficult to apply use Scrum to large teams. Also, because it requires face-to-face communication, a Scrum team works optimally when everybody is working in the same physical space.
While the waterfall model usually result in the development of features that are useless, an Agile methodology facilitates the delivery of features based on their business values. Also, in a traditional approach, changes are feared instead of being welcomed, and that fear is a sure way not to answer the clients’ needs. Since answering the business needs is part the definition of success itself, no wonder the failure rate is so high.
Acuña, S., Gómez, M., & Juristo, N. (2009 йил 1-Jan). How do personality, team processes and task characteristics relate to job satisfaction and …. Information and Software Technology .
Barney, H., Moe, N., Dybå, T., & Aurum, A. (2009 йил 1-Jan). Balancing Individual and Collaborative Work in Agile Teams. Agile Processes in Software Engineering and Extreme … .
Charette, R. (2005). Why software fails. IEEE spectrum .
Gagné, M., & Deci, E. (2005). Self-determination theory and work motivation. J.of Organizational Behavior
Larman, C., & Basili, V. (2003). Iterative and incremental development: A brief history. Computer .
Melnik, G., & Maurer, F. (2006 йил 1-Jan). Comparative Analysis of Job Satisfaction in Agile and Non-Agile Software Development …. Lecture Notes in Computer Science .
Poppendieck, M., & Poppendieck, T. (2006). Implementing Lean Software Development: From Concept to Cash (The Addison-Wesley Signature Series).
Royce, W. (1970 йил 1-Jan). Managing the development of large software systems- Concepts and techniques(Large …. csa.com .
Schwaber…, C. (2007 йил 1-Jan). The Truth About Agile Processes. thoughtworks.ca .
Sutherland, J., & Schwaber, K. (2008). The Scrum Papers: Nuts, Bolts, and Origins of an Agile Process.
Takeuchi, H., & Nonaka, I. (1986). The new new product development game. Harvard Business Review , 64 (1), 137-146.