There have been a number of posts recently from both the Real Story Group and JBoye about the use and abuse of spreadsheets in the process of selecting software products such as ECM, WCMS and search applications. The consensus view seems to be that they have little value and that in the end there is a strong tendency to manipulate the scores to end up with the product that intuition (usually the consultant's) suggests is the best fit. In one respect I agree with those who raise issues about the value of spreadsheets, and that is that any process that involves basing a decision on the sum of all the scores is flawed.

It is easy to fall in love with numbers and forget the basis on which they were generated. Many years ago, when I was in the market research business, I had an important annual review meeting with the head of market analysis at IBM Europe. At the time I was running the European market research group for International Data Corporation, which had, and still has, a very high reputation for the quality of its market research. My IBM customer started off by telling me that they never believed the forecasts my team were making. To IBM the value was that we were challenging their own forecasts, and in making the comparisons IBM often came up with very important insights into the way that markets were going to develop.

The fundamental issue with any software product selection is that multiple trade-offs are involved, balancing functionality against price against development capability against ...the list is almost endless. Trade-off analysis is a well-respected approach to support decision making, and there is far more to it than just putting numbers against features, adding them up and then applying a fudge factor. Amazon currently offers over 2000 books with references to trade-off analysis, but initially take a look at an excellent (and open source) introduction to the subject published by the US Army Institute for Water Resources in 2002.  Although the examples are about the trade-offs in water resource management the basic principles are applicable to the selection of software products.  This 155 page manual should be required reading for anyone involved in selecting CMS and search applications, though some of the sections on techniques such as Multiple Attribute Utility Theory can probably be skipped over. There are also now many software packages on the market, mainly aimed at the market research community undertaking conjoint analysis of field surveys.  

The simplistic and simple summing of scores on spreadsheets is never going to work, but using Excel to build some models that assist with the initial process of deciding what clients actually want can be very helpful. This enables some sensitivity analyses to be undertaken to illustrate just what the impact might be of changing various factors in the model. A CMS or search selection project could well result in the need to develop multiple spreadsheets and decision matrices, and as consultants we should be familiar with trade-off analysis techniques so that they are become another tool in our collection. Our experience and intuition as consultants both have important roles to play in the process, but to brush spreadsheets aside because they seem to give the wrong results is a rush to judgement. They give the wrong results because perhaps we are using them in the wrong way.

Martin White