Thursday, April 23, 2009

"Worse is Better"

When I was new in my career, I read a lot of books about programming craft. One common theme of the books was perfectionism - and analysis paralysis. That a team would want to get the requirements (or design) "right", and so, would not actually product anything. I never saw it; I thought it was a myth.

Then I went to work for a big company. Oh. My. Goodness.

I met people who talked a great game yet, in a period of years, had never actually shipped any software. I'm talking about /nothing/. Years later, some of them are senior managers and executives. I'm still not quite sure I understand how that works. I have a clue, but that's not what this post is about.

I realized that -- somehow -- my value system was different. I wanted to get something out there, learn from it's failings, adjust it, and make it better - instead of trying to get everything right "up front." Reading things like Peopleware or the Agile Manifesto convinced me that I was not the only person that felt this way.

The best (short) explanation of this I have ever read is "Worse Is Better" by Richard Gabriel; here is an excerpt:

I and just about every designer of Common Lisp and CLOS has had extreme exposure to the MIT/Stanford style of design. The essence of this style can be captured by the phrase ``the right thing.'' To such a designer it is important to get all of the following characteristics right:

* Simplicity-the design must be simple, both in implementation and interface. It is more important for the interface to be simple than the implementation.

* Correctness-the design must be correct in all observable aspects. Incorrectness is simply not allowed.

* Consistency-the design must not be inconsistent. A design is allowed to be slightly less simple and less complete to avoid inconsistency. Consistency is as important as correctness.

* Completeness-the design must cover as many important situations as is practical. All reasonably expected cases must be covered. Simplicity is not allowed to overly reduce completeness.

I believe most people would agree that these are good characteristics. I will call the use of this philosophy of design the ``MIT approach.'' Common Lisp (with CLOS) and Scheme represent the MIT approach to design and implementation.

The worse-is-better philosophy is only slightly different:

* Simplicity-the design must be simple, both in implementation and interface. It is more important for the implementation to be simple than the interface. Simplicity is the most important consideration in a design.

* Correctness-the design must be correct in all observable aspects. It is slightly better to be simple than correct.

* Consistency-the design must not be overly inconsistent. Consistency can be sacrificed for simplicity in some cases, but it is better to drop those parts of the design that deal with less common circumstances than to introduce either implementational complexity or inconsistency.

* Completeness-the design must cover as many important situations as is practical. All reasonably expected cases should be covered. Completeness can be sacrificed in favor of any other quality. In fact, completeness must sacrificed whenever implementation simplicity is jeopardized. Consistency can be sacrificed to achieve completeness if simplicity is retained; especially worthless is consistency of interface.

Early Unix and C are examples of the use of this school of design, and I will call the use of this design strategy the ``New Jersey approach.'' I have intentionally caricatured the worse-is-better philosophy to convince you that it is obviously a bad philosophy and that the New Jersey approach is a bad approach.

However, I believe that worse-is-better, even in its strawman form, has better survival characteristics than the-right-thing, and that the New Jersey approach when used for software is a better approach than the MIT approach.

Of course, you can read the entire section yourself - it's taken from a much longer paper titled LISP: The Good News, The Bad News, And How To Win Big.

Now, the mental attitude of perfectionism is entrenched; you can't just forward someone a link to a Gabriel paper and expect them to change their attitude or behavior. Asking them to change behavior would be, in effect, asking them to change their value system.

The one approach I have had some success with is waiting six months and asking "So, how's that working for you?" then asking how long it would take if the team just ... did it and adjusted?

It's hard to argue for six more months of designing the process for a reporting and analysis package when you could create the reports and put them in the hands of the customer in less than a month.

Failing at that, I talk about fail fast, lean thinking, and waste.

UPDATE: I've found that one section I quoted, treated historically, to be a some of the greatest prose ever written on software development dynamics. Jamie Zawinski, the creator of Netscape Navigator, quotes just that section on his website. Certainly, the LISP folks actually shipped things. I did not mean to compare them to other people of a different character who were /totally/ stuck in analysis paralysis.


  1. Matt,

    Excellent post ! I think the silver bullet is to find the right balance between both approach. I think if you have the right feedback (ie. tests in any of its form) you're not too bad with the worse-is-better approach.

    I'm keeping your article on my pile and might comment more either here or on my blog ...

    Fred @

  2. you need to be aware what design in this context means. design of Common Lisp is the design of the language features and how they work together. A lot of work has been done for it. It took the years from 1983 to 1994 from start to publishing the standard. The design process was not waterfall, instead a the language evolved with implementations and feature implementations. Several first implementations were available soon after the first book describing the language was available. Later a second edition was published and even later the final standard. Common Lisp was over-engineered for a demanding customer: the U.S. department of defense. So, even the MIT 'the right thing' approach produced actual results early and improved those in several groups with customer feedback. It is not really describing a waterfall approach, where the design takes a long time and no product is produced. In the CL case the design took a long time, but there were many shipments. Programmers had the language implementations from 1985 on and provided feedback and helped testing and designing the language over the years until the standard was published.

    Thus the design perfection meant by Gabriel was more about the qualities of the desired 'end' product - not so much how to arrive there (the 'process').

    You should be aware the Gabriel also wrote several other versions of this paper where he argued from different angels (worse is worse, ...).