July 29, 2004

Response to the META Group White Paper, “Messaging Total Cost of Ownership: Microsoft Exchange 2003 and Lotus Domino in Small and Medium Organizations”

Michael Sampson, michael.sampson@shared-spaces.com

Shared Spaces Research & Consulting

www.shared-spaces.com

I was surprised to see another poorly done white paper on Exchange vs. Domino so close to the Radicati fiasco of last week. However, this one is from The META Group (www.metagroup.com), which has traditionally published very good work. I’m greatly surprised at the lack of thoroughness and due diligence in preparing this paper. It is not available on the META Group web site (so it is probably not written by one of the META analysts, eg, Matt Cain), but is available from Microsoft’s competitive intelligence site on Exchange, at http://www.microsoft.com/exchange/evaluation/compare/METAEx2k3vNotes.asp.

Why am I writing this?

· It has always been difficult, if not impossible, for an analyst to compare Microsoft Exchange and Lotus Domino.

· Comparative TCO studies are very important, but are hard to do well. Many different items across the two comparative products need to be looked at the same. For the comparisons to be valid, the author must apply the same standards and use a consistent approach with both products.

· The META group has a good reputation in the market, so its work carries weight.

· The author has prefaced the report with specifically chosen words about how careful he or she has been. A casual reader would take that at face value.

· I believe there are fundamental flaws in this TCO study, and they should be taken into consideration when reading it.

Michael’s Take

Here’s the points at which I think the author has displayed a “lack of thoroughness and due diligence”:

1. Comparing apples with oranges. Why is it a comparison of Exchange 2003 and Lotus Domino R5 and R6? The sample size is very small -- only 9 organizations – so why could The META Group not have found five Lotus customers running R6.5.x, rather than a mixed bag running versions dating back to 1999? That’s inconsistent.

2. The report makes it sound as though Microsoft and IBM were involved. Why? Because it says that Oracle was “invited to participate with its Oracle Collaboration Suite, but declined to do so” (p1). Hence, I reason therefore that Microsoft and IBM were invited to participate and agreed to do so. But, I am assured through IBM channels (Ed Brill) that this is not the case. If I combine that data point with the fact that the paper is not available from the META Group’s web site, this leads me to believe that the white paper is the latest in the string of Microsoft-sponsored research that parades as a publication from an independent research house. Remember the fiasco Forrester faced last year after its Microsoft-funded “study” on Linux?

3. p.2 “META Group conducted this study with 12 organizations. Of these organizations, nine were chosen for the final analysis”. Chosen? On what basis? By whom? Why were the other three dropped? Were they Domino users or Microsoft ones? This partial disclosure raises lots of questions. Eg, were the three not chosen because they made the Microsoft figures look bad?

4. p.2. “The companies were selected from each of the target messaging and collaboration platforms that have seat counts between 500 and 6,000 users”. Fine ... but then in listing the actual customers on pp5-6, the actual seat count for each of the nine is not mentioned. it should be. Also, that’s a big spread on which to try to do a comparison. It would be more statistically valid and defensible if 10 organizations were compared, each with 450-550 users, half running Domino 6.5.x and the other half Exchange 2003.

5. The data collection dates appear to have been carefully “managed”. Exchange 2003 shipped in August 2003, and the data for this report was collected between October 2003 and June 2004 (p). Why was that? So that the five Microsoft accounts could get Exchange 2003 fully implemented prior to the collection of cost data on specific line items? This has a direct impact on at least one cost element ... p24 compares “Annual Disposition Task Costs per Seat”. If my supposition is correct, then of course the costs for Exchange 2003 would be significant less than for the Domino shops ... Exchange 2003 was already installed, and was a brand new product so wouldn’t be disposed of. Bad comparison.

6. More apples and oranges. Three of the four Lotus shops have decentralized multi-site architectures, whereas only two of the five Microsoft shops do. Of course the costs for Lotus shops will be higher. Companies with common architectures should have been chosen and compared. One of the Microsoft shops (M4, p6) runs only two servers, whereas the smallest number run by a Lotus shop is six servers (L1, p5). Bad comparison.

7. The “Server and Miscellaneous Hardware” costs vary greatly ... $45.17 for Lotus shops, and $10.11 for Microsoft shops. Why?—possible reasons: (a) The Lotus shops have much more decentralization in the messaging backbone architecture, thus having higher costs. That would be true. (b) One of the Lotus shops uses Sun servers ... which are more expensive to purchase and operate. (c) The Lotus servers also provide hosting for collaborative applications, and thus are up-provisioned in comparison to the Microsoft ones. True, p1 of the report says that “the costs of Lotus/Domino application components were not included in the study”, but that could just refer to the *applications* themselves.

Interestingly, what cost element was left out of the study? P1, “… note that participants’ messaging system performance and uptime levels are not reflected in the findings”. Given the poor record that Microsoft has delivered to the market in terms of security (and thus ongoing patching, prevention, etc), I would not be surprised that if those elements were included, the Microsoft figures would be much worse. With respect to hardware costs, uptime has a definite impact on the overall cost structure. A Microsoft shop will require more servers and other infrastructure to deliver the same level of uptime as a shop running Sun servers. As the author says, this isn’t taken into account.

8. The biggest cost variance is the “Personnel” line item … p13 … $208.59 per year for Lotus and $93.11 per year for Exchange. This is broken down on p20, into 8 different line items. Since this represents the major cost difference, I ask myself “why are the two so different”? The author does an admirable job of listing the 36 different tasks that an administrator might do, and of grouping them into eight task categories.

Questions that I ask myself as I read the descriptions in light of the 9 respondents are:

· Administration … they are about the same, with Exchange higher. For some of the items, however, such as the tracking and monitoring of assets, I would have thought that the Lotus shops would have had a greater cost exposure, due to the greater number of servers and other hardware deployed over a greater number of data centers. Strange.

· Deployment … again, very similar.

· Disposition … over twice as much for Lotus shops, but duh!, all of the Exchange 2003 installations were new, whereas the Lotus ones had a mixture of servers at different points in the lifecycle. Inconsistent approach.

· Disaster Recovery … given the description of the task (p10), it is not immediately evident why the Lotus shops would get an average of $8.34 per seat and the Exchange shops only $0.55. Perhaps the reason is that the Lotus shops worked out and tested a plan whereas the Exchange ones didn’t? Insufficient information is provided to draw a definitive conclusion.

· Operations … $72.96 for Lotus and $15.23 for Exchange … big spread. But why? Given the list of tasks on p10-11, I don’t understand how the Lotus shops would spend such a huge amount compared to the Exchange ones. Again, perhaps the Lotus shops did work in this area and the Exchange ones didn’t. Or alternatively, perhaps the Lotus shops had higher paid individuals working on these tasks rather than the Exchange ones. Given the larger footprint of the Lotus shops from the perspective of decentralization and multi-data centers, that leads me to conclude that the Lotus administrators were more likely to be “full time administrators” rather than “generalists” (p31) at the Microsoft shops. For at least three of the operations tasks – backup and recovery, messaging security, and monitoring – I would expect the costs for the Microsoft shops to be much higher. And again, given that most of the Lotus shops had a decentralized architecture, “performance tuning and load balancing” would most likely be higher than for a shop that follows a single data center strategy with Exchange. There are too many points at which this analysis could be flawed to place any credence in the data. More data needs to be presented to the reader.

· Maintenance … again, there is a big variation between the two, but insufficient data to know why. As with operations, was it the higher cost of the Lotus staff due to them being full-time administrators vs. lower-paid generalists who had other responsibilities at the Microsoft shops? Or alternatively, what does “Application development and integration” (task 1, p11) mean within the context of “email and collaboration” (p1) for Domino and Exchange Servers? Has the author been sneaky in slipping in some Domino development time, vs. nothing for Exchange?

· Procurement … $18.72 for Lotus, and $4.32 for Microsoft. Why? Because, firstly, the Lotus shops had a decentralized architecture and greater number of servers and other hardware infrastructure to procure and organize, and so their costs for procurement were higher than for Microsoft. Secondly, any procurement tasks for a Domino implementation would consider issues around application development to meet business requirements, and the necessary infrastructure for that. Of course it will be higher than for Microsoft, where, given the nature of the messaging backbone architectures implemented by the five Exchange shops, the procurement decisions were much easier. Invalid comparison.

· Upgrade … $20.09 for Lotus, and $2.15 for Microsoft. Again, why is this? Is it the greater decentralization of the Lotus infrastructure, and thus the greater coordination required and decision points needed for considering the upgrade? I was surprised, given Microsoft’s poor record in the security realm, that the costs for implementing security patches in the Exchange shops was therefore so low.

Recommendations

To the META Group I recommend:

· Revamp the study. Clearly say who sponsored it and who didn’t. Don’t make the study appear to have been sponsored by both parties. Ensure that Microsoft removes the current study (because it is wrong in multiple places) and that it posts the new version.

· Be consistent in comparisons. It is wrong to compare two products that are deployed using different architectures, and are at different points of the lifecycle. Nothing intelligent or helpful will result.

· Include the name of the author on the report.

To readers of this white paper I recommend:

· Anyone reading a TCO study from an independent research house should be told (a) who is was written by, and (b) where the funding for it came from.

· Read TCO studies from multiple research houses. The different methodologies and different assumptions taken by different research houses create very different outcomes.

About Michael Sampson

Michael Sampson has been an active researcher, analyst and consultant in the messaging and collaboration market since 1994, working with an international client base in the United States, the United Kingdom, Europe, Australia, and New Zealand, his country of residence. Michael is passionate about helping businesses and government agencies leverage the power of collaborative technologies in the context of their day-to-day work processes for business success and outstanding government results.

Michael established Shared Spaces Research & Consulting to work closely with interesting businesses and government agencies working to develop an all-encompassing collaboration infrastructure by leveraging the power of shared spaces to overcome the limitations of today’s inadequate individual communication technologies.

Contact Michael at michael.sampson@shared-spaces.com, or phone +64 3 317 9484.

About Shared Spaces Research & Consulting Ltd

Shared Spaces Research & Consulting fulfils two complementary objectives in the market: to publish thoughtful self-funded vendor-neutral independent research on how businesses and government agenices can leverage the power of shared spaces for enterprise collaboration, and to provide consulting services on shared spaces. Key technologies explored include:

· Collaborative Team Workspaces

· Real-Time Interaction Technologies

· Collaborative Business Portals

· Presence & Availability in Business Applications

· Wireless Collaboration & Messaging

· Collaboration Auto-Discovery.

Visit www.shared-spaces.com for daily coverage of interesting happenings in the messaging and collaboration industry.