A critical element of many web development engagements is the selection of a Web Content Management System (CMS). However, the large number of products and the rapid rate of change in the web CMS market, which combined with a customer’s budget and schedule requirements lend themselves to pursuing a two phased approach: i) Candidate Identification (covered in part 1); and ii) Candidate Evaluation. The remainder of this post will address the Candidate Evaluation phase by outlining some of the activities that can be used to help compare and contrast CMS candidates, and ultimately lead to the selection/recommendation of the strongest CMS for the project at hand. The goal of the Candidate Evaluation phase is to perform a detailed comparison of the CMS candidates that were the output of phase 1, and provide a framework that will result in the identification of a winner. The breadth and depth of evaluation activities undertaken, and relative weights (if any) afforded each, will vary depending upon the nature of the engagement (including budget and schedule considerations) and the characteristics of the customer (such as their degree of sophistication and overall knowledge, and their preference for level of involvement). However, the following a la carte selection provides a useful starting point:
- Vendor Presentations – one of the best ways to get an overview of the CMS products is to request a demonstration. Vendors are generally only too willing to take an hour or two to show off their product, and it is often helpful to schedule two sessions with each. The first, call it an initial screening session, provides an opportunity for the vendor to provide a more freeform tour of the product, and can result in the decision to drop a candidate very quickly. The second session, which I recommend inviting a small number of your customer’s organization to, can be more focused on the task at hand by providing each vendor with a list of requirements or requirement categories, that should be covered by the demonstration. In the event that one or more people from the customer organization attend, ask them to give each vendor a high, medium or low rating for each of the requirements (categories) demonstrated, as well as for their overall impression.
- Vendor Literature and Websites – most CMS vendors have a significant amount of product literature available on-line and a thorough review of this material can help to understand the products strengths and weaknesses and overall alignment with the requirements of the project at hand.
- First Party Feature Comparisons – develop a spreadsheet or other table that identifies and categorizes individual requirements, and rate each product on its ability to meet each one
- Third Party Feature Comparisons – although it is always important to consider the source, to the extent that any current and detailed feature/performance comparisons of CMS products are available, they can be helpful in identifying any significant gaps and/or differences
- Request Architecture Diagrams and Associated Pricing from each Vendor – not only do these help to determine the degree to which the solution can be accommodated within the existing customer technical, not to mention financial, environment, but they provide insight into how interested/motivated the vendor is as a consequence of demonstrating their level of understanding of the customer and project in question
- Check References – ask each vendor to provide contact information for existing customers and speak with them about their level of satisfaction with, and enthusiasm for the product. Note: references must be treated with some caution since each customer’s experience is influenced by the competence of the organization that implemented the solution, as well as the version of the product implemented
- Other Considerations – beyond the ability to meet feature, user experience, technical, schedule, and budget requirements, there are often other key, albeit less tangible considerations. Other considerations may include things such as: fit with the customer’s existing technology and vision; scalability and extensibility of the product, i.e., will the product be able to satisfy the customer’s future requirements; the stability of the vendor and clarity of its mission and strategy, i.e., is it a leader in the web CMS market that is likely to be a going concern for the foreseeable future; (for Open Source products), Open Source Momentum (see additional description below)
In the event that one or more Open Source products are in the running, I encourage the consideration of what might be described as Open Source Momentum. Open Source Momentum is a characteristic that addresses legitimate concerns about the stability and viability of Open Source products. Open Source Momentum is a function of things like: a products history in the market place; the number of sites using the product; and the size, rate of growth, and degree of activity of the developer community. The higher the level of Open Source Momentum, the less an organization should be concerned about using a particular Open Source product. By considering the results of these activities, it will be possible to identify the strongest of the CMS candidates for the project at hand. In addition, by explaining the different activities and the results to the customer, the rational for the recommendation should be crystal clear. In closing this post, I’ll pose the following questions for your consideration:
- Are there other activities that you use to perform a detailed evaluation of candidate CMS products?
- Are there other examples of ‘Other Considerations’ that you have used to evaluate CMS products?
- Does the concept of Open Source Momentum ring true to you?