Assumptions behind No Estimates

No Estimates does not currently have a single, clear definition. I hope the upcoming No Estimates book will change that. While we wait for the book, I think we need to outline some of the underlying assumptions behind No Estimates:

1. Software is complex. Fred Brooks (of Mythical Man-Month fame) described the complexity very well in his seminal article No Silver Bullet, and concluded that there is nothing in sight that can make software significantly easier to work with within the next decade — in the years since, we still haven't seen any "silver bullet", and there is no hope on the horizon. In theory, software is deterministic and thereby "merely" complicated, but the advent of multi-processing has reduced this argument. There are also ways of using formal logic to identify and categorize each and every possible state of a program, but this is not feasible for anything beyond a few thousand SLOC.
2. People are complex. People are definitely not machines. Given the same input on two different days, they can react very differently, for no apparent reason. For an in-depth discussion, you can dip a toe into the field of social complexity.
3. Coding is design, not manufacturing. The blueprint of software is not the design spec, but source code. From the source code, the manufacturing process — compiling, linking, smoke testing, packaging, deploying etc. — is very deterministic and can be carried out quickly and cheaply by computers. Translating the design spec into source code is detailed design, and requires creative thinking as well as a sense for details and, in any non-trivial project, collaboration skills.
4. Software development is not a repeatable activity. If you give the same requirement to the same coder twice, he will either copy-paste the previous code and be done in no time, or he will rewrite the code as it should have been written the first time. If you give the same requirement to two different coders, the difference in time can be 3x (some people say 10x). 
5. Software estimates are uncertain. This should be clear from points 1-4. Also check out Barry Boehm's work on the "cone of uncertainty" and Steve McConnell's books on estimation. To be sure, some technological domains are more predictable than others, but few domains can be predicted more than say 4 months ahead, and the general trend is towards lower and shorter predictability, i.e. higher uncertainty. (There are ways to cope with this, including working with estimate ranges, or using the wisdom of crowds.)
6. Having targets lowers your performance. When estimates have inherent, non-trivial uncertainty, you are unlikely to ever be on schedule: in other words, you are either early or late compared to target. Running late, some people take risks and cut corners in order to meet the target, while others wear themselves down by working overtime. Both strategies result in mistakes and defects that will have to be fixed later, making people even more late. (People are seldom early: this is because if you deliver ahead of time, you will be punished with tighter schedules in the next project. Smart people don't repeat that mistake.)
7. Software can be split in smaller, still valuable pieces. You are not forced to implement a software system bottom-up. In fact you can take a piece of end-user functionality and write only the code necessary to make that piece work. The smaller piece you work on, the higher the overhead though, as you may have to implement large pieces of functionality below the surface. However there are methods for identifying user needs and then splitting the functionality so that the most important needs are covered first (e.g. story mapping, feature injection and impact mapping). This means that the early end-user functionality pieces, which are costly, also can have high value. This is supported by the Pareto principle: 20% of the work gives 80% of the value.
8. Estimates are not goals. (We are for the moment disregarding the question of whether estimates are accurate or not.) Estimates are a means for managing and synchronizing assumptions and expectations. By giving and taking estimates, we might get closer to a joint understanding of the system to build and of its complexities. Our goal should however be to get as much value (features) as we can in the shortest possible time (lowest cost), and in this we should be limited only by the laws of physics, not by the agreements of man.
9. Managing scope is easier than managing time or cost. This is because of how the iron triangle (cost, scope, time) fits together. In a typical software development project, costs are mainly wages and other personnel-related costs, and they are inexorably linked with time. Adding or removing people (changing the cost) has an impact on time, as the throughput of the team will drop (see The Mythical Man-Month by Fred Brooks). Similarly, changing the time constraint has an impact on the cost. Thus the only remaining variable is scope. Luckily, as per point 7, managing the scope is easier than one might think.
10. Flow is your friend. The law of large numbers says that when you get enough samples, the variations tend to cancel out. If you can split your work down to the level of < 1 day, you are in the green.

This is a preliminary list of assumptions, based on a couple of hours of work. I reserve the right to update them in the future. Maybe some of them will be proven wrong, who knows.

Getting to the "practices of No Estimates" is not something I will attempt today.

However, it can be noted that these assumptions are in direct contrast with some of the traditional assumptions, to wit:

• Developing software is to translate design specifications into source code
• Software estimates are essential
• Setting targets is the only way to get stuff done
• Software is valuable only when it's (almost) complete - there is no value in getting halfway
• Estimates help define the price tag, which is then written into a contract
• Buying software is much like buying a car or a house - "this is what I want, what's the price tag?"
• Software development is engineering, and it is repeatable
• People with similar skill sets (roles) are interchangeable
• A competent engineer can pick up a new project in a matter of days

I think these differences in assumptions are what makes it so difficult to hold a conversation about No Estimates. No Estimates is only a few years old, it's still emerging and doesn't have a Body of Knowledge to rely on. We may know something more when the book is done.

The iron triangle

This is an updated repost of a blog entry I wrote in June 2010 on our internal blog, combined with some material I wrote earlier this week as a result of a good discussion with @duarte_vasco. Enjoy!

An interesting thought occurred to me last Thursday. You all know about the "iron triangle": out of scope, time and quality, you can pick any two. The reason why you can't nail down all three is the unpredictability inherent to software development.

If you have read your Fred Brooks, you know that software is invisible. Walk into a factory and you perceive noise, activity, movement. But walk into a software development office and you perceive silence, inactivity, stillness. A lot of things are happening, a lot of work is being done, but it is invisible to the naked eye. Fred Brooks says in his infamous essay "No Silver Bullet" that software is an abstract construct of interlocking data sets, data item relationships, algorithms and functions. It is "pure thought-stuff, infinitely malleable", complex, changeable, and invisible. These essential and inherent attributes makes software virtually impossible to visualize, model and understand.

Orthodox plan-driven methods attempt to nail down all three parameters. Since quality is the least visible of the parameters, it will be the one that fluctuates. Therefore in a plan-driven project, you set a Quality Manager to "manage" the problem of quality. Unfortunately quality management is a post-facto task; there's nothing much relevant a Quality Manager can do himself to improve the quality of the product. It's not like he's going to go and fix the bugs himself, right? He can try to prioritize change requests and bugs and increase visibility into where the product is. Another problem is that depending on how long your development cycle is i.e. how often you integrate your product and how long the system testing takes, the picture that emerges doesn't show where the product is, but where it was.

In Scrum and other Agile methods, we do it the other way around. We require high quality in a short time and let the scope float. The quality is defined and controlled using automated tests, and the timebox is defined as the maximum time we can keep change out of the project (or even better, as the minimum time needed to make something we can demonstrate and invoice the customer for). But we don't send out the scope to sea to float freely around, oh no! The key is that instead of letting the scope fluctuate randomly, we set a scope manager (actually, let's call him e.g. the Product Owner) to actively manage it. And this must be done ahead of implementation.

You see where this is going? Paradoxically enough, the Product Owner also prioritizes change requests and bugs (read: backlog items) and increases visibility into where the product is going.

So what's the difference between the PO and the QM? The difference is that the QM works in a system based on predictability, while the PO works in a system based on unpredictability. Since software development is fundamentally unpredictable, the QM tries to salvage the product afterwards, while the PO is effectively managing the product one step ahead.

And that, my friends, is it.