CHAPTER 6

Selecting the Right LMS for Your Organization

If you don’t have an LMS, or you use a product that has not been upgraded for several years, your organization may be missing some beneficial features that are available in current products. If you already have an LMS and are not satisfied with its features and capabilities, it may be time to consider a more suitable product. The LMS marketplace is always changing, with new players emerging all the time. A product that is well suited to your organization’s needs is probably out there.

Finding the LMS that best meets your organization’s needs is not easy. Hundreds of LMS products are available. An organization’s investment in an LMS and related technologies is considerable, both financially and in the time spent choosing one. All this combines to heighten the possible risk of selecting the wrong solution. LMS implementation can involve a significant disruption to your organization. Once the LMS is in place, your organization is likely to use it for the next three to five years, or more. You certainly don’t want to pour your efforts into a product poorly matched to your needs.

A study I conducted with the eLearning Guild showed that the process an organization uses to evaluate and select an LMS can have a significant impact on an organization’s satisfaction with the solution (Foreman 2013a). Figure 6-1 shows that, of survey respondents who thought that their evaluation and selection process was ineffective, less than one quarter are satisfied with their LMS product. As the respondents’ view of their evaluation process’s effectiveness increases, so does their satisfaction with the LMS product.

So, if your organization wants to evaluate LMS products and vendors effectively—and come away satisfied with the decision—what should you do?

How to Evaluate Products and Select an LMS

LMS evaluation and selection is a process that involves five major steps, as shown in Figure 6-2: Analyze your needs, define your requirements, vet products, evaluate suitable products, and select a product. These steps reflect a proven practice for LMS evaluation and selection. Although this process may take more time and effort than you anticipated, it helps ensure that you end up with an LMS that is best suited to your needs. Ultimately, following this process will save you from wasting valuable time, money, effort, and sponsorship support on a failed LMS project.

Step 1: Analyze Needs

Your first step is to analyze the needs of your organization. Without a needs analysis, you can only make assumptions about what your organization really needs from an LMS. Your assumptions are likely to be based on your perspective and, perhaps, those of the people you work with most closely. I have seen groups start listing features they want in an LMS without giving much thought to the needs of other key stakeholders. This “head in the sand” approach can result in a solution that meets some of the organization’s needs—but not all. It is critical that you ask others in the organization what they need to manage learning. By exploring the perspectives of your stakeholders, you are likely to discover important needs that you would not have identified on your own.

There are other benefits to this approach. You will be connecting with your users to gather input that will inform your LMS selection and may also help you better align your training strategy, initiatives, and priorities with those of the organization. People who are involved in evaluation activities tend to have more ownership of the resulting decision and become acclimated to the product in advance of implementation. Your sponsors will have increased confidence in the decision because of the due diligence involved in the process. Through this evaluation process, the learning organization builds relationships with its customers, sponsors, stakeholders, and IT, all of which help set the stage for implementation.

You will need to start by identifying and talking with stakeholders. These are the people in your organization who will be affected by the new LMS. Some of your key stakeholders are people in leadership and management roles, who are concerned with ensuring the people in their departments have the needed skills and knowledge to be productive. Other stakeholders include people who will use the LMS, such as e-learning developers, training schedulers, instructors, registrars, and data analysts. You may not be able to speak with every single stakeholder; a representative sample from each key group will be fine. Your goal for these discussions is to collect data that reflect the entire organization’s needs and goals so that you can define a comprehensive set of requirements that your LMS must meet.

So, what is a requirement? Requirements describe the capabilities the LMS must have to be useful to your organization. Your requirements should cover all aspects of what all your stakeholder groups need. LMS requirements are similar to course learning objectives. They describe what you need to be able to do once you have the LMS.

Explore the Leadership View

People in leadership roles are likely to provide a strategic point of view. Whenever possible, start here. Awareness of the strategy informs some of the more tactical and technical questions that should be explored with management later in the analysis process.

Contact these leaders and let them know that you seek a strategic viewpoint as part of the needs analysis you are conducting for purchasing an enterprise LMS. To gain the most information you can in a short amount of time, you will want to meet one-on-one with each leader. Ask for a 20- to 30-minute interview.

When you sit down with the leader, ask the big, open-ended questions like, “What do you see as the key human performance challenges for our organization over the next three to five years?” You may be surprised at how much useful information and insight leaders will share, from which you can form LMS requirements.

For example, I once met with the chief financial officer of a $12 billion company and asked two questions: “What are the key strategic goals of the company over the next several years, and what do you see as the most critical human performance challenges related to achieving those goals?” He responded that the company was expecting rapid global growth through mergers and acquisitions. This would require the effective assimilation of people from other companies, who spoke different languages and were accustomed to different corporate cultures, business processes, systems, and tools.

I then met with several executive vice presidents who expressed concern about the company’s need to meet the regulatory compliance requirements of all the countries in which it manufactured and sold its products. One emphasized her belief that although English tends to be the primary language in which global business was conducted at the company, people need to be given the opportunity to learn in their native language.

This collection of leadership viewpoints generated a set of strategic drivers for LMS requirements. The drivers were global growth, regulatory compliance, and a need to learn in one’s native language. These strategic goals drove requirements for multilingual support; tracking, reporting, and management of compliance training and retraining; and coordination with human resources on the timing and processes for adding acquired employee groups to the human resource management system, which would share data with the LMS.

Explore the Management View

Your next step is to meet with middle management and other key stakeholders. People in management roles are likely to provide an operational point of view. This group should include managers of people in each of the departments and functional areas that will use the LMS to get training. Other key stakeholders may include people in satellite training groups, such as customer training, supplier training, sales training, or regional training.

I like to meet with managers in focus groups. I find that groups of managers tend to react to one another’s comments. When a single manager brings up an issue and no one responds, I make a note and move on. When a manager brings up an issue and a lively group discussion ensues, I know I’ve found a concern that is more broadly experienced and therefore carries more weight.

When organizing focus groups, I suggest selecting managers at the same management level in similar functional areas. Mixing higher- and lower-level managers in the same group can result in some people not wanting to speak up and risk judgment by their supervisors or subordinates. Mixing functional areas can create a lack of focus. Managers from one area may dominate parts of the conversation, which the other functions may find irrelevant.

I subscribe to a few additional guiding rules for assembling focus groups. First, plan for just enough time to gather all the data you need. I find that 60 to 90 minutes is usually sufficient. Second, keep the group size manageable to ensure that everyone has a chance to speak. I prefer a size of eight to 12 people. Third, bring a colleague to help conduct the discussion and share the note-taking. You need to capture the salient information that is discussed, while steering the conversation, pursuing issues, and knowing when to move on. A colleague can help by picking up lines of questioning, hearing things, and taking notes that you may have missed. If you cannot find a colleague to help you, consider bringing a recorder to minimize your note-taking so that you can focus on leading the discussion. Finally, do not bring so many colleagues that you outnumber the people in the focus group. Doing so can be quite intimidating to focus group participants.

If you are planning to replace your current LMS, you might want to kick off the discussion by asking people what they like most about the current solution and what they would most like to see improved. If your organization is starting this process for the first time, focus your questions on how learning is managed currently and the key problems areas—operationally and financially.

Explore the IT View

When you meet with people from IT, you may find that a number of discussions need to take place. First, IT has its own training needs and, in that sense, will be an end user of the LMS. Second, and perhaps more important, IT is tasked with managing the organization’s technology infrastructure, so they are likely to be concerned about how the LMS will fit into the broader IT architecture and what technical support will be required of them.

Consider exploring these two IT perspectives in separate focus groups. For IT’s training needs, you will need to speak with the IT people who are responsible for training. This discussion will be similar to those you had in the management focus groups.

The IT infrastructure discussion will be focused on the technical standards, processes, and constraints that will drive your LMS technology requirements. This discussion will encompass IT’s strategy related to cloud-based solutions, mobile access, network demand, security, privacy, systems integration, scalability, reliability, and support. From this discussion, you will generate a set of technical requirements. Consider including someone from your team or a consultant who understands the technical side of the LMS and is well acquainted with the perspective and typical concerns of IT.

When Are You Done With the Needs Analysis?

A rule of thumb is to keep meeting with people until you start to get mostly repetitive information. Once you have collected sufficient input from the interviews and focus groups, you should be able to identify strategic, operational, and technical drivers from which you can define your LMS requirements. Table 6-1 lists some examples.

The strategic, operational, and technical drivers you discover during needs analysis will bring your LMS needs and requirements into focus. This approach will ensure that your LMS selection criteria are in alignment with your organization’s goals and priorities.

Step 2: Define Requirements

Requirements form the basis of your LMS selection criteria. The clearer and more complete your LMS requirements, the easier it is to evaluate products. Try to focus on what learners or administrators must be able to do with the LMS.

When writing requirements, keep in mind that LMS vendors will need to accurately interpret each requirement to respond to it or demonstrate that it can be met by their product. Even more important, you will need to test and verify that the products you’re considering can meet each requirement.

There are four rules to follow when writing LMS requirements:

Sometimes it helps to organize your requirements into categories, such as functional and technical. Functional requirements describe what the system must be able to do from a learning management perspective, whereas technical requirements describe how the system must fit into the broader IT enterprise infrastructure. Table 6-4 lists some examples of requirements in each of these two categories.

You can also define subcategories under each of the two general categories. For example, under functional requirements, you might use administration, general course management, classroom learning management, online learning management, personalization, navigation, enrollment, and learning results as subcategories. Under technical requirements, you might list online learning interoperability, accessibility and security, systems integration, and system performance and management.

It is important to prioritize your requirements. By prioritizing your requirements, you can use a weighted scorecard to evaluate how well various LMS products meet each requirement and then calculate overall scores that indicate which products align best with your requirements and priorities. Some organizations use a simple method for prioritizing requirements, like high, medium, and low. I prefer to use a statistical approach called analytic hierarchy process (AHP), which assigns a weight to each requirement, because its results reflect priorities more accurately. AHP identifies the importance of each requirement compared with other requirements and assigns it a weight. AHP is very effective when used with a group of people because it drives discussion, consensus, and a shared vision of what features are most important and what the organization can expect from an LMS. A web search on AHP will provide lots of additional information and even some Excel templates, which you can customize for use in your LMS project.

Step 3: Vet Products

Once you’ve analyzed your needs and defined your requirements, your challenge is to narrow the hundreds of LMS choices to a short list of products. To get started, decide which of the three major types of LMS products will be your focus: corporate LMS, academic LMS, or LCMS-LMS. Then, decide whether you need a stand-alone LMS or an app or plug-in for your existing WordPress, Salesforce, or SharePoint sites. Consider whether you want to explore LMS products that specialize in your industry, or broaden your search for products to encompass more general-purpose solutions.

Next, identify eight to 10 requirements that can help you rule out nonqualifying products. For example, not all LMS products support multiple languages. If this requirement is critical for your organization, then it is a good vetting criterion. Not all products comply with specific accessibility and security regulations. Some LMS products support online courses only. Others support instructor-led, but not online. Some have e-commerce features, and some do not. Some support SCORM; others do not. Some support cmi5, while others do not. These can all be good vetting criteria.

Once you have defined a set of vetting criteria, you can research product websites and contact vendors to inquire about their product’s support for your vetting criteria. Select a short list of products, perhaps eight to 12 that meet all or most of your vetting criteria. This gives you a manageable number of products to evaluate in more detail without being overwhelmed.

Step 4: Evaluate Products

The product evaluation process works like a funnel, as shown in Figure 6-3. With each evaluation activity, you rule out nonqualifying products and continue to evaluate those that best meet your requirements. The list of products you are considering gets smaller as you complete each evaluation activity.

You will need to find the appropriate balance between thoroughness and efficiency. I have found that the following evaluation activities yield optimum results. The first two activities—request for information (RFI) and use case demonstrations—should be performed in order. Activities 3 through 6—trial version, IT review, customer referrals, and vendor health—can be performed concurrently. Activity 7, request for proposal (RFP), should be last. However, I have at times streamlined this process by combining or omitting some of the activities when working with organizations with time or logistical constraints.

Request for Information

A good activity to start with is an RFI. It contains all your requirements, with two to six questions per requirement for vendors to answer. Avoid asking yes or no questions; ask open-ended questions that will require vendors to describe how they meet the requirement. For example, instead of asking the vendor, “Can your product provide a report on enrollments?” ask the vendor, “Please describe your product’s reporting capabilities, and include a list of all available reports.”

The vendors’ written responses to your RFI provide valuable information that will help you gain insight into their products, their ability to interpret your requirements, and their responsiveness to your requests. It is a great way to gain an in-depth view of the capabilities of the products you are considering, outside the context of the vendor’s typical sales process.

When you’ve put together your RFI document, contact your vetted list of vendors. Let them know that you are in the process of evaluating and selecting an LMS and that you have selected their product as a candidate. Ask if they are willing to respond to an RFI. If not, they do not value your business enough to invest the time. This is a strong indicator that they will not view you as a high-priority customer and may be difficult to work with. Cross them off the list and move on. If they agree, send them the RFI along with a due date for their response.

You may need to set ground rules to keep the process fair for all vendors. For example, you may decide that no vendor will be told who the other vendors are, answers to any questions asked by one vendor will be distributed to all vendors, and no RFI responses will be accepted after the due date.

Decide who in your organization will be involved in reviewing and scoring RFI responses. Depending on the number of requirements, it may take one to three hours to evaluate each vendor’s response. It is important that all reviewers agree to review all the vendor responses. Using different reviewers for different vendors, or accepting a reviewer’s evaluation of some of the vendors but not all, will skew your results. This would be unfair to the vendors, who have put a good deal of time and effort into responding to the RFI. More important, it would be counterproductive to your LMS evaluation and selection efforts.

Provide a scorecard for reviewers to use as they evaluate the vendor responses. The scorecard should allow reviewers to rate how well the vendor meets each requirement. I like to use a five-point scale, such as unsatisfactory, suboptimal, acceptable, strong, and optimal. Table 6-5 shows an example of an RFI scorecard.

When reviewing an RFI response, focus on one requirement at a time. Read the vendor’s responses to each question related to the requirement and rate the product based on what you read. Ideally, the vendor responses should be clear, concise, thorough, and pertinent. However, keep in mind that you are not rating the vendor’s grammar or writing style. You are rating how well the vendor’s product matches up with the requirement, based on what you have read.

Once all the reviewers have completed their scorecards, you can aggregate the scores to calculate an average score for each requirement, for each vendor. If you have applied weights to prioritize the requirements, you can incorporate the weight into the score for each requirement. You can then total the scores for each requirement category and subcategory, as well as a single, overall score. These scores will reflect how well each of the LMS products matches up with your requirements, based on what is most important to your organization.

After scoring the responses to the RFI, you can rule out some of the lower-scoring vendors and continue evaluating the top contenders. Send a message to the vendors you plan to drop from consideration while thanking them for their RFI responses. Let them know that despite its strengths their product did not match up with your requirements as well as some of the others you reviewed.

Use Case Demonstrations

Meet with your review team to brainstorm a list of 10 or 12 use cases. A use case is a scenario or goal and set of parameters that would typically be performed by a learner or administrator. Use cases should reflect the way your organization manages learning. Select some common use cases as well as some that are complex and challenging. This will enable you to compare all vendors based on the same demonstration criteria.

Next, ask the vendor to demonstrate some of your common use cases. These might include scheduling training events, administering annual compliance training, finding and registering for a course, or generating an important report.

Don’t stop there. Go beyond the common scenarios and think about some of the trickier things you may need. I worked with an organization that offered an instructor-led course that took place over a three-week timeframe with a “checkerboard” schedule. A class would meet in week one on Monday from 8 a.m. to 12 p.m., and on Thursday from 1 to 5 p.m. at a different location with a different instructor. The sessions on weeks two and three met on different days and times. This was a very good use case to try with LMS vendors. The sidebar shows an example of how you might document a use case for your vendors to demonstrate.

Invite each of the top contenders to meet with your organization, either physically or virtually, to demonstrate how their product handles each use case. Send the use cases to the vendors in advance so that they can prepare. Let them know that their demonstrations will be evaluated by the attendees and compared with demos from other vendors. Some vendors will want to spend time giving you a sales presentation telling you about their customer base, market share, history, and so on. I like to allow the vendors to manage their use of the allotted time, but emphasize that the meeting will end on time, they must demonstrate all the use cases, and they must clearly state to participants which use case they are demonstrating.

Again, you can distribute a scorecard to participants so they can evaluate the vendor demonstrations. Tally the scores and rule out those vendors whose products did not perform well in the demonstrations. At this point, you should have a short list of qualified “finalist” products.

Trial Version

As you continue to evaluate the finalists, you may ask each of the vendors for a trial version of the LMS software or access to a “sandbox” installation in the cloud where you can explore the product. Hands-on exploration will give you a better sense of the user interface design, features, and capabilities of the product.

Invite members of the review team, learning management administrators, people who need to run reports, and others who will use the LMS directly and frequently, to try out the product. Set a date and time to meet with the reviewers to discuss their experiences with and opinions of each product. Use a flipchart to list positives and negatives for each product. Decide whether any of the products should be dropped from consideration.

IT Review

Set up one or two meetings between your vendor’s technical staff and your IT organization for some “under-the-hood” discussions. The purpose of these discussions will be to satisfy IT that the vendor’s technology, security, and integration capabilities are acceptable; set the vendor’s expectations about what will be required, technically, to implement their product in your organization; and establish a professional rapport between the vendor and your IT team, which will benefit your organization later if you decide to implement the vendor’s product.

Customer Referral

Ask for customer referrals from each vendor. By checking with existing customers, you can find out more about the product. Ask the vendor’s customers about how quickly and effectively the vendor responds to questions and problems with the product. Also ask how long it took them to implement the product and what they might do differently next time. Finally, ask how their last upgrade went and what they would most like to see improved in the product. Be aware that vendors are likely to give you the names of their most satisfied customers. However, every customer has experienced bumps with their products. So, be inquisitive.

Vendor Health

You may also want to check the vendor’s financials. The strength of your vendor’s financial position and customer base helps ensure that the vendor will support your product well and will continue to evolve and improve with sufficient funding for product research and development.

Financial reports for public companies are readily available online. For private companies, you may find information by checking with Dun & Bradstreet. You may also want to check information from industry analysts on the financial health and market share of the products you are evaluating. Some well-regarded learning-technology industry analyst groups include Gartner Group, Forrester Research, Brandon Hall Research, and Bersin by Deloitte.

Request for Proposal

Once you’ve narrowed your group of finalists down, send a request for proposal to each of your finalists asking for pricing quotes, implementation timeframes, and support options. If you are considering a cloud solution, be sure to ask for hosting options, service guarantees, and pricing. For more information, see the section labeled “Product Costs” in Table 1-1.

Work with your purchasing or procurement departments when creating the RFP. They may have standard RFP templates and processes for you to follow, or they may assist directly in preparing and administering the RFP. Decide how long you want to allow for vendors to prepare their RFP responses. A reasonable period is two weeks. Compare the RFP responses, and don’t be afraid to go back to each vendor to ask them for a better price or more feature options comparable to incentives the other vendor finalists are offering.

For clarity, the comparison chart in Table 6-6 explains the differences between an RFI, which you use in the product evaluation phase, and an RFP, which you use in the product selection phase.

Step 5: Select a Product

The final step is to settle on a product. If possible, you can place your organization in a strong position to negotiate the best pricing and terms by selecting two or three finalist products, any of which will meet your needs. Don’t be afraid to negotiate; remember that LMS vendors are eager to gain your business. To an LMS vendor, each new customer represents a multiyear revenue stream.

You have much more negotiating power when your evaluation results in more than one product finalist and you can reap the benefits of a vendor bidding war. But even if you have decided that a single vendor’s product is the best fit for your organization, you can still negotiate for more than what the vendor has proposed. To gain your business, a vendor may offer you a discounted price, a free set of feature options that normally cost more, a better support package, a quicker start date for implementation, or other perks.

Always remember to build adequate time into your schedule to accommodate the contracting process. Contracting often involves the legal departments in both your organization and the vendor’s and can sometimes take several weeks to complete. Once the contract is signed, you are ready to get busy with implementation.

Key Takeaways

This chapter described a proven process for analyzing your organization’s learning management needs, defining requirements, and evaluating and selecting a product. The key takeaways are:

• A thorough needs analysis engages stakeholders and sponsors throughout your organization to identify your learning management system needs. Make sure to explore the viewpoints of your executive leadership, management, and IT.
• Clearly define and document your requirements and use them as the basis of your product selection criteria.
• Derive vetting criteria from your requirements; that is, distinctive requirements that will help you select a short list of vendors to evaluate.
• A funnel approach to your product evaluation activities helps you narrow hundreds of products down to the handful that are best matched to your requirements.
• Getting bids from two or three vendors whose products are equally good candidates for your organization positions you to negotiate the best terms and pricing.

The next three chapters explain how to implement your new LMS. Chapter 7 focuses on how to plan for your LMS implementation and configure your new product. Chapter 8 focuses on integrating your LMS with other systems and migrating your course content and learning-related data from your old LMS to the new one. Chapter 9 focuses on testing your LMS to make sure it is ready and going live.