CHAPTER 16
Compatibility Testing

At this point in the book, the new features of Windows Server 2016 have been presented and discussed in depth, as have the essential design considerations and migration processes. The goal of this chapter is to examine the process of testing the actual applications that rely on the Windows Server 2016 infrastructure.

This chapter provides insight into the steps necessary to gather information before the testing process begins, how to actually test the applications and document the results, and how to determine whether a more extensive prototype testing process is needed. Going through this process is vital to ensure the success of the project and to avoid a displeased end-user community. The application testing process is intended as a quick way to validate the compatibility and functionality of the proposed end state for the upgrade.

Currently, many companies are seeking to optimize their network environment, whether that is getting the best use out of servers on-premise or migrating server services to the cloud. Migrating to Windows Server 2016 with its higher capacity limits and multitenant capabilities is a chance to optimize servers within the network infrastructure. At the end of the process, fewer servers will handle the same tasks as before, and new functionality and cloud-based services might have been added, making the configurations of the environment that much more complex, and making it even more important to thoroughly test the mission-critical networking applications for compatibility. For example, Windows Server 2016 introduces a tremendous number of new technologies that enhance failover clustering, web applications, virtualization, security, Remote Desktop Services, improved branch office deployments, cloud services with Azure, and much more, prompting some organizations to replace existing Windows systems with Windows Server 2016. Therefore, it is even more important to test this configuration to ensure that the hardware and software are compatible, the performance meets user expectations, and the everyday features used by the employees to share knowledge and collaborate are in place.

The results of the application compatibility testing process will validate the goals of the project or reveal goals that need to be modified because of application incompatibility or instability. If one key application simply will not work reliably on Windows Server 2016, the legacy Windows system might need to be kept as part of the networking environment, which changes the overall design. As discussed in Part II of this book, “Windows Server 2016 Active Directory,” a variety of different combinations of Windows Server system configurations can be combined in the end configuration, so the chances that there will be a way to keep the troublesome applications working in the new environment are good.

The differences between a prototype phase and an application testing phase can be dramatic or negligible based on the nature of the upgrade. A prototype phase replicates the end state as completely as possible, often using the same hardware in the test lab that will be used in the production rollout.

Most network users do not know or care which server or how many servers perform which task or house which application, but they will be unhappy if an application no longer works after a migration to Windows Server 2016. If the organization already has Active Directory in place and is running Windows Server 2008/2008 R2 systems, the risk of application incompatibility is likely to be less than if the organization is migrating from an older operating system, such as Windows Serve 2003, Windows 2000 (or earlier) or a competing operating system, such as Novell NetWare.

A significant advantage of following a phased design methodology, as presented in Chapter 2, “Planning, Prototyping, Migrating, and Deploying Windows Server,” is in the planning discussions that take place and in the resulting statements of work, design, and migration documents that are created as deliverables. Often, companies contract with migration experts or Microsoft partners—such as Convergent Computing, also known as CCO—to help companies avoid classic mistakes in the upgrade process. By the end of this planning process, it will be clear why the project is happening, which departments need which features and capabilities, and what budget is available to perform the work. The timeline and key milestones also will be defined.

If a phased discovery and design process hasn’t been followed, this information needs to be gathered to ensure that the testing process addresses the goals of the project stakeholders, and that the right applications are in fact tested and verified by the appropriate people.

A key issue to discuss at this point is whether it is acceptable to have multiple versions of the Windows Server operating system in the final solution. Some organizations want to control standards on both software and support services and require just a single network operating system such as Windows Server 2016 across the board.

Third-party applications should be identified as well. The applications most often used include tape-backup software modules or agents, antivirus software, fax software, and voicemail integration products. Additional third-party add-on products might include the following:

Administration

Antispam

Backup and storage

Customer relationship management (CRM)

Log monitoring

Line-of-business applications

Migration

Reporting

Security and encryption

The hardware to be used should be listed, as well, to ensure that it is available when needed. Ideally, the exact hardware to be used in the upgrade will be ordered for the application testing process, but if that is not possible, hardware with specifications similar to that of the servers that will eventually be used should be allocated. Although processor speed and amount of RAM will most likely not make a difference to whether the application functions properly on the server platform, certain hardware devices should be as similar as possible. Tape drives, for example, should have the same features as the ones to be used in the production environment because this is one of the most critical components. If an autoloader will be used in the production environment, one should be made available for the application testing process. If faxing from the Outlook Inbox is required, the same faxing hardware should also be allocated. Another example is implementing clustering with a storage-area network (SAN) back end. If a SAN will be used in production and the test criteria of the lab is to validate clustering functionality, the same production SAN should be used in the test environment. By using the same SAN solution, clustering test criteria and clustering functionality can be validated and guaranteed.

Some applications require clients to be present for the testing process, so at least one workstation class system should be available for this purpose. Connectivity to the Internet might also be necessary for testing the functionality of remote access products and antivirus software.

Table 16.1 shows a sample checklist of requirements for summarizing the scope of the application testing phase.

This process should not take a great deal of time if previous planning has taken place. If the planning phase was skipped, some brainstorming will be required to ensure that the scope includes all the key ingredients required for the application testing. The goals for the application testing process will also affect the scope, which is covered in the following section.

One useful way of looking at the goals for the project is to treat them as the checklist for successful completion of the testing. What conditions need to be met for the organization to confidently move forward with the next step in the Windows migration? The next step might be a more complete prototype testing phase. For smaller organizations, it might be a pilot rollout, where the new networking environment is offered to a select group of savvy users.

These goals are separate from the business goals the company might have, such as a more reliable network infrastructure or improved security. A more complete prototype phase could seek to address these goals, while the application testing process stays focused on the performance of the specific combinations of the operating system and embedded and connected applications.

A convenient way to differentiate the goals of the project is to split them into key areas, as described in the following sections.

If there is very little time available to perform the testing, this limits how much time can be spent on each application and how many end users can put each through its paces. It also necessitates a lesser degree of documentation. Remember to include time for researching the applications’ compatibility with the vendors as part of the timeline. A quick project plan might be useful in this process as a way of verifying the assumptions and selling the timeline to the decision makers.

Contingency time should ideally be built in to this goal. Resources assigned to the testing can get sick or might be pulled back into the office for production support, or applications might require additional testing when problems are encountered. Vendors might not provide trial versions of the software as quickly as desired, or new versions of software or even the hardware itself can be delayed. With many companies seeking to consolidate the number of servers in use, it is not uncommon to see labs evolve through the testing process. Different versions of the Windows operating system are used, as are different versions of various application software programs.

Of course, there might be no budget allocated for testing, but it is better to know this as soon as possible. A lack of budget means that no new hardware can be ordered, evaluation copies of the software (both Microsoft and the third-party applications) need to be used, and no external resources will be brought in. If the budget is available or can be accessed in advance of the production upgrade, a subset of the production hardware should be ordered for this phase. Testing on the exact hardware that will be used in the actual upgrade rather than a cast-off server will yield more valuable results.

More and more virtualization technology is being utilized in test labs for reducing costs associated with hardware procurement. Virtualization is an excellent way to reduce capital expenditures. Keep in mind that hardware-specific prototype testing cannot be achieved when using virtualization as the guest operating system. In addition, performance metrics might get skewed when running more than one guest operating system on a virtual server.

Often, the internal network administration staff is too busy with daily tasks or tackling emergencies that spring up (which might be the reason for the upgrade in the first place), and staff personnel should not be expected to dedicate 100% of their time to the testing process.

If an outside consulting firm with expertise in Windows Server 2016 is going to be used in the testing process, it can be a good leverage point to have already created and decided on an internal budget for the testing process. This cuts down on the time it takes to debate the approaches from competing firms.

This goal might be set for different types of applications where some mission-critical applications would need to have extensive testing, whereas less-critical applications might have more basic testing performed. A short time frame with a tightly limited budget will not allow for extensive testing, so basic compatibility will most likely be the goal.

Although the IT resources performing the testing will learn a great deal by going through the testing process, the organization might want to provide additional training to these individuals, especially if new functionality and applications are being tested. If external consultants are brought in, it is important that the organization’s own resources are still involved in the testing process for training and validation purposes. The application testing phase might be an excellent time to have help desk personnel or departmental managers in the user community learn more about new features that will soon be offered so they can help support the user community and generate excitement for the project.

Again, the budget and timeline for the testing will affect the answer to this question. Many organizations require a paper trail for all testing procedures, especially when the Windows infrastructure will have an impact on the viability of the business itself. For other organizations, the networking environment is not as critical, so less or no documentation might be required.

The application testing phase is a great opportunity to document the steps required for application installations or upgrades if time permits, and this level of instruction can greatly facilitate the production rollout of the upgraded networking components.

If there are applications—such as customer relationship management (CRM), document routing, voicemail or paging add-ons, or connectivity to PDAs and mobile devices—a higher level of user testing (at least from the power users and executives) should be considered.

Organizations decide to keep labs in place after their primary purpose has been served for a number of reasons. Whenever a patch or upgrade to Windows Server 2016 or to a third-party application integrates with Windows Server 2016, it is advisable to test it in a nonproduction environment. Even seemingly innocent patches to antivirus products can crash a production server. Other items might require user testing to see whether they should be rolled out to the production servers.

By taking the time to document these constraints, the testing process will be more structured and less likely to miss a key step or get bogged down on one application. The individuals performing the testing will essentially have a checklist of the exact testing process, and are less likely to spend an inordinate amount of time on one application, or “get creative” and try products that are not within the scope of work. After the testing is complete, the stakeholders will also have made it clear what is expected in terms of documentation so the results of the testing can be presented and reviewed efficiently.

This summary document should be presented to the stakeholders of the project for review and approval. The organization is then ready to proceed with the research and testing process for Windows Server 2016 compatibility.

For systems that are part of the network domain, the devices should be identified by which network operating system they are running. Another item that should be captured is whether the server is physical or virtual. Table 16.2 shows a sample system device inventory sheet.

The primary applications that need to be upgraded will be obvious, as will the standard services such as data backup and antivirus software. However, in most organizations, additional applications hiding on the network need to be identified. If System Center Configuration Manager 2007/2012/2012 R2 (ConfigMgr) is in use, or another network management tool with inventory capabilities, it should also be able to provide this basic information.

It is also helpful to separate the Microsoft and non-Microsoft applications and services. The Microsoft applications that are to be upgraded to the new Windows Server 2016 environment are likely to have been thoroughly tested by Microsoft. Possible incompatibilities should have been identified, and a great deal of information will be available on Microsoft TechNet or on the Microsoft product page of its website. For non-Microsoft applications and services, however, weeks could pass after a product’s release before information about any compatibility problems with the Microsoft operating system surfaces. This holds true for service packs and product updates, as well, where problems might be made public weeks or months after the release of the update.

Furthermore, many organizations that create custom applications will find that little information is available on Windows Server 2016 compatibility, so they could require more complex lab tests to validate compatibility.

A sample product inventory sheet includes the following categories:

Vendor name

Product name

Version number

Application or service?

Mission-critical?

Compatible with Windows Server 2016 (Y/N)?

Vendor-stated requirements to make compatible

Decision to migrate (update, upgrade, replace, remain on existing OS, stop using, proceed without vendor support)

Additional items that might be relevant could include which offices or departments use the application, how many users need it, and so on.

Any notes from the vendor, such as whitepapers for migration, tip/trick migration steps, upgrade utilities, and any other documentation should be printed, downloaded, and kept on file. Although a vendor might state that a product is compatible on its website today, you might find that by the time an upgrade occurs, the vendor has changed its statement on compatibility. Any backup information that led to the decision to proceed with the migration might also be useful in the future.

There is no need to treat all applications and utilities with equal importance because a simple utility that does not work and is not identified as a critical application can be easily upgraded or replaced later and should not hold up the migration. However, problems with a mission-critical business application should be reviewed in detail because they might affect the whole upgrade process.

Remember that certain utility applications should be considered critical to any network environment. These include tape backup (with the appropriate agents) and virus-protection software. In organizations that perform network and systems management, management tools and agents are also essential.

For early adopters of certain application software programs, more research might be necessary because vendors tend to lag behind in publishing statements of compatibility with new products. Past experience has shown that simply using the search feature on the vendor’s site can be a frustrating process, so having an actual contact who has a vested interest in providing the latest and greatest information (such as the company’s sales representative) can be a great time-saver.

Each vendor tends to use its own terminology when discussing Windows Server 2016 compatibility (especially when it isn’t 100% tested); a functional way to define the level of compatibility is with the following four areas:

Compatible

Compatible with patches or updates

Not compatible (requires version upgrade)

Not compatible and no compatible version available (requires new product)

When possible, it is also a good practice to gather information about the specifics of the testing environment, such as the version and service pack (SP) level of the Windows operating system the application was tested with, along with the hardware devices (if applicable, such as tape drives, specific mobile devices, and so forth) tested.

Vendor name

Product name and version number

Vendor contact name and contact information

Level of criticality: Critical, near-critical, or nice to have

Compatible with Windows Server 2016: Yes/No/Did not say

Vendor-stated requirements to upgrade or make application compatible

Recommended action: None, patch/fix/update, version upgrade, replace with new product, stop using product, continue using product without vendor support

Operating system compatibility: Windows Server 2012/2012 R2, Windows Server 2008/2008 R2, Windows Server 2003, other

Processor architecture compatibility: 64-bit compatible?

Notes: Conversation notes, URLs used, copies of printed compatibility statements, or hard copy provided by vendor

It is a matter of judgment as to the extent of the notes from discussions with the vendors and materials printed from websites that are retained and included with the inventory sheet and kept on file. Remember that URLs change frequently, so it makes sense to print the information when it is located.

In cases where product upgrades are required, information can be recorded on the part numbers, cost, and other pertinent information.

1. The application version currently in use is compatible with Windows Server 2016.

2. The application version currently in use is compatible with Windows Server 2016, with a minor update or service patch.

3. The application currently in use is compatible with Windows Server 2016, with a version upgrade of the application.

4. The application currently in use is not compatible with Windows Server 2016 and no upgrade is available, but it will be kept running as is on an older version of Windows Server (or other network operating system) in the upgraded Windows Server 2016 networking environment.

5. The application currently in use is not compatible with Windows Server 2016 and will be phased out and not used after the upgrade is complete.

6. The application currently in use is not compatible with Windows Server 2016 per the vendor, or no information about compatibility was available, but it apparently runs on Windows Server 2016, so the organization needs to determine whether it will run the application on an operating system potentially not supported by the application vendor.

Each of these states is discussed in more detail in the following sections.

It is up to the organization to determine whether testing is necessary to verify the vendor’s compatibility statement. If the application in question is critical to the integrity or security of the Windows Server 2016 operating system or provides the users with features and capabilities that enhance their business activities and transactions, testing is definitely recommended. For upgrades that have short time frames and limited budgets available for testing (basic testing as defined earlier in the chapter), these applications might be demoted to the bottom of the list of priorities and would be tested only after the applications requiring updates or upgrades had been tested.

A clear benefit of the applications that the vendor verifies as being Windows Server 2016 compatible is that the administrative staff will already know how to install and support the product and how it interfaces with Windows Server 2016 and the help desk; end users will not need to be trained or endure the learning curves required by new versions of the products.

During the testing process, the service updates and patches are typically quick and easy to install, are available over the Internet, and are often free of charge. It is important to read any notes or readme files that come with the update because specific settings in the Windows Server 2016 configuration might need to be modified for them to work. These updates and patches tend to change and be updated themselves after they are released, so it is worth checking periodically to see whether new revisions have become available.

These types of updates generally do not affect the core features or functionality of the products in most cases, although some new features might be introduced, so they have little training and support ramifications because the help desk and support staff will already be experienced in supporting the products.

The amount of time required to install and test these upgrades is greater, the learning curve steeper, and the danger of technical complexities and issues increases. Thus, additional time should be allowed for testing the installation process of the new products, configuring them for optimal Windows connectivity, and fine-tuning for performance factors. Training for the IT resources and help desk staff will be important because of the probability of significant differences between the new and old versions.

Compatibility with all hardware devices should not be taken for granted, whether it is the server itself, tape backup devices, or SAN hardware.

If a new version of the product is required, it can be difficult to avoid paying for the upgrade, so budget can become a factor. Some vendors can be persuaded to provide evaluation copies that expire after 30 to 120 days.

Although it might sound like an opportunity to skip any testing because the server configurations aren’t changing, connectivity to the new Windows Server 2016 configurations still needs to be tested, to ensure that the functionality between the servers is stable.

Again, in this scenario, the application itself is not upgraded, modified, or changed, so there will not be a requirement for administrative or end-user training.

Take care during the testing process to note the differences that the administrative staff, help desk, and end users will notice in the day-to-day interactions with the networking system. If features are disappearing, a survey to assess the impact can be very helpful. Many users will raise a fuss if a feature suddenly goes away, even if it was rarely used, whereas the complaints could be avoided if they had been informed in advance.

If the organization chooses to keep the application, it might be kept in place on an older version of Windows or moved to the new Windows Server 2016 environment. In either case, the administrative staff, help desk, and end users should be warned that the application is not officially supported or officially compatible and might behave erratically.

This document can be expanded to summarize which applications will be installed on which network server if there are going to be multiple Windows Server 2016 servers in the final configuration. In this way, the document can serve as a checklist to follow during the actual testing process.

Some of the goals might now be more difficult to meet, and require additional budget, time, and resources. If essential network applications need to be replaced with version upgrades or a solution from a different vendor, additional time for testing and training might also be required. Certain key end users might also need to roll up their sleeves and perform hands-on testing to ensure that the new products perform to their expectations.

This might be the point in the application testing process at which a decision is made that a more complete prototype testing phase is needed, and the lab would be expanded to more closely, or exactly, resemble the end state of the migration.

The Microsoft Assessment and Planning (MAP) toolkit can assist with a migration or upgrade to Windows Server 2016 by conducting inventory, assessments, and reporting on servers throughout the infrastructure. In addition, unlike other tools, it can gather information without installing agents on servers.

You can download the toolkit from http://technet.microsoft.com/en-us/library/bb977556.aspx. After the toolkit is installed, you can create a server inventory report, which will identify currently installed operating systems. The report will also include detailed analysis of software and hardware readiness and compatibility with Windows Server 2016.

The testing process can yield unforeseen results because the exact combination of hardware and software might affect the performance of a key application, but it's far better to have this occur in a nonproduction environment where failures won’t affect the organization’s ability to deliver its services.

During the testing process, valuable experience with the installation and upgrade process will be gained and will contribute to the success of the production migration. The migration team will be familiar with—or possibly experts at—the installation and application migration processes when it counts, and are more likely to avoid configuration mistakes and resolve technical issues.

If the testing process is to exactly mirror the production environment, this would be considered to be a prototype phase, which is generally broader in scope than compatibility testing, and requires additional hardware, software, and time to complete. A prototype phase is recommended for more complex networks in which the upgrade process is riskier and more involved and in which the budget, time, and resources are available.

Don’t forget to allocate a representative workstation for each desktop operating system that is supported by the organization and a sample remote access system, such as a typical laptop or mobile device that is used by the sales force or traveling executive.

When configuring the servers with the appropriate operating systems, the company standards for configurations, based on industry best practices, should be adhered to, if they have been documented. Standards can include the level of hard drive redundancy, separation of the application files and data files, naming conventions, roles of the servers, approved and tested security updates, and security configurations.

Next, Windows Server 2016 should be configured to also meet organization standards and then for the essential utilities that will protect the integrity of the data and the operating system, which typically include the backup software, antivirus software, and management utilities and applications. After this base configuration is completed, it can be worth performing a complete backup of the system or taking a snapshot of the server configuration in case the subsequent testing is problematic and a rollback is necessary.

So many different combinations of applications might be installed and tested at this point that the different permutations cannot all be covered in this chapter. As a basic guideline, first test the most essential applications and the applications that were not identified previously as being compatible. By tackling the applications that are more likely to be problematic early on in the process, the testing resources will be fresh, and any flags can be raised to the stakeholders while there is still time left in the testing process for remediation.

Thorough testing by the end users is recommended, as is inclusion of the help desk staff in the process. Notes taken during the testing process will be valuable in creating any configuration guides or migration processes for the production implementation.

During the analysis phase of whether existing applications will be compatible with Windows Server 2016, it is a best practice to validate that the applications do carry the Certified for Windows Server 2016 logo by contacting the manufacturer. By having the logo, application testing and additional analysis of a specific application is minimized when upgrading to Windows Server 2016.

You can find the ISV Application Readiness and Certification program at www.windowsservercatalog.com. The program provides guidelines and validation that an application meets the standards for application compatibility and integration support expected in a tightly integrated Microsoft-based solution.

At a minimum, a summary of the testing process should be created, and a final recommendation for the applications to be included in the production upgrade or migration should be provided to the stakeholders. This can be as simple as the upgrade decision matrix discussed earlier in the chapter, or it can be more thorough, including detailed notes of the exact testing procedures followed. Notes can be made available summarizing the results of end-user testing, validating the applications, and describing results—both positive and negative.

If the testing hardware is the same as the hardware that will be used in the production upgrade, server configuration documents that list the details of the hardware and software configurations can be created; they will ensure that the servers built in the production environment will have the same fundamental configuration as was tested in the lab.

A more detailed build document can be created that walks the technician through the exact steps required to build the Windows Server 2016 system, in cases where many network servers need to be created in a short period of time.

The level of effort or the amount of time to actually perform the upgrade or the migration of a sample subdirectory can be recorded as part of the documentation, and this information can be very helpful in planning the total amount of time that will be required to perform the upgrade or migration.

The following is a checklist that will help your organization make the decision:

Is sufficient budget available for a subset of the actual hardware that will be used in the upgrade?

Is sufficient time available for the configuration of the prototype lab and testing of the software?

Are the internal resources available for a period of time long enough to finish the prototype testing? Is the budget available to pay for external consulting resources to complete the work?

Is the Windows networking environment mission-critical to the business’s capability to go about its daily activities and generate revenues, and will interruption of Windows services cost the company an unacceptable amount of money?

Does the actual migration process need to be tested and documented to ensure the success of the upgrade?

Do resources need to be trained on the upgrade process (building the servers, and configuring the network operating system and related applications)?

Do the applications that will be tested need to be compatible with 64-bit processor architecture?

If you find that the answer to more than half of these questions is yes, it is likely that a prototype phase will be required.

The first steps include identifying the scope and goals of the project to ensure that the stakeholders are involved in determining the success factors for the project. Then research needs to be performed, internal to the company, on which in-place applications are network-related. This includes not only Windows Server, but also tape backup software, antivirus software, network management and monitoring tools, add-ons, and inventory sheets created summarizing this information.

Decisions as to which applications are critical, near-critical, or just nice to have should also be made. Research should then be performed with the vendors of the products, tracking sheets should be created to record this information, and the application should be categorized in one of six states of compatibility. Next, the testing begins, with the configuration of the lab environment that is isolated from the production network, and the applications are loaded and tested by both administrative and end user or help desk staff. The results are then documented, and the final decisions of whether to proceed are made.

With this process, the production upgrade or migration is smoother, and the likelihood of technical problems that can harm the business’ ability to transact or provide its services is greatly reduced. The problems are identified beforehand and resolved, and the resources who will perform the work gain familiarity with all the products and processes involved.

Take the time to understand the goals of the project (what will the organization gain by doing the upgrade?) and the scope of the project (what is included and what is excluded from the project?).

Understand all the applications that connect with Windows Server 2016 and whether they are critical, near-critical, or simply nice to have.

Accelerate a migration to Windows Server 2016 by using the MAP toolkit.

Document the research process for each application because this will prove to be very valuable if problems are encountered during the testing process.

Create a lab environment that is as close to the final end state of the upgrade as possible. This reduces the variables that can cause problems at the least opportune time.

Test applications for compatibility with both typical end users of the application and application administrators who support, maintain, and manage the application.

Leverage virtual server technology to minimize the cost associated with procuring hardware for a test lab.

Ensure that applications have been tested for compatibility with a 64-bit operating system, such as Windows Server 2016.