CHAPTER 6
Optimizing the Service Manager environment

Microsoft System Center 2012 Service Manager is a complex product that requires Windows Server, Internet Information Services (IIS), Microsoft SQL Server, SQL Server Reporting Services, SQL Server Analysis Services, and Microsoft SharePoint. Although Service Manager planning and deployment guides make installation straightforward, configuring SQL Server, defining views, and configuring workflows and SharePoint is more complicated. However, these tasks are no less important since improper configuration of any of these components can lead to performance issues. In fact, analysts who use Service Manager often criticize the console’s sluggishness. For example, customers who use Service Manager commonly complain about having to wait for a window to close and about seeing messages like "This item cannot be updated because it has been changed by another user or process." Performance issues like these can be frustrating, especially because the console is the only workspace analysts can use to manage service requests (apart from using third-party web-based consoles). Fortunately, most of these issues can be resolved through Service Manager customizations or by the analysts simply changing the way they use the console.

The official sizing guidelines for Service Manager (available from http://www.microsoft.com/en-us/download/details.aspx?id=13605) indicate that Service Manager can support up to 50,000 users and computers. This is not a hard limit, however. It simply reflects the scenarios that the product group has tested and documented. Several large customers with environments far beyond 50,000 users successfully use Service Manager. A core SQL Server infrastructure configuration with a focus on performance and optimization, together with a well-planned configuration of Service Manager itself, make it possible to scale well beyond 50,000 users.

This chapter provides guidance on optimizing the Service Manager environment for better performance of both the console and backend components such as the SQL database, workflow, and Portal. The chapter addresses some common mistakes and provides recommendations for avoiding pitfalls. These guidelines are mainly targeted for large environments (over 20,000 users), but some of the recommendations apply to all organizations regardless of size.

Service Manager management server

The Service Manager management server houses the main software portion of a Service Manager deployment. The first Service Manager management server to be installed acts as the workflow server and is critical to the environment. This role can be moved or failed over for disaster recovery. The following guidance applies mostly to larger environments:

▪ If supporting more than 40 to 50 Service Manager consoles simultaneously, add one or more dedicated Service Manager management servers and place these in a network load balancing (NLB) cluster. This will isolate the workflow management server and ensure the workflows are executed regardless of the number of users utilizing the Service Manager console.

▪ The most important resources to the management servers are RAM and CPU. Make sure to monitor the servers to ensure they are sized adequately, following the recommended specifications found in the official sizing guidelines.

▪ Always try to avoid adding other roles to the management servers, for example SharePoint or SQL Server. This is because the management server is the primary engine supporting the Service Manager infrastructure.

Service Manager console

The Service Manager console is the way that analysts interact with Service Manager. The perception they have concerning how Service Manager performs is therefore based on their experience when using the console. The following guidance is important to consider:

▪ Never maximize the Service Manager console because it impacts performance due to a Windows Presentation Foundation (WPF) issue. Instead, minimize the console and extend the window as needed, even to fill the entire screen; just don’t use the maximize button. This is a one-time task since the Service Manager console size is stored and reused at next launch.

▪ If the Service Manager console is left running for a long period, for example overnight or longer, it can consume a large amount of memory. Because the Service Manager console is written in C# (a managed code language) it relies on the .NET Framework garbage collector to manage memory. Therefore, when there is available memory on the machine, the console will cache data in memory to improve performance. As memory demand increases from the operating system or other applications, the garbage collector will release the least amount of memory needed for the Service Manager console to function. Do not worry, therefore, about whether the console is consuming too much memory; it will be released when other applications require it. For more information on this issue, see http://msdn.microsoft.com/en-us/library/0xy59wtx.aspx.

▪ Each view the user selects when using the console increases the console start time because every view is cached. For example, selecting the All Incidents view has a huge pull on the database, and therefore it takes a long time to load. Instead of many views with similar criteria, encourage users to use the filter and search functions to find the objects they are interested in. You can also chose which views will be available to each role when you configure user roles. And if a view is not needed at all, you can delete it from the General view.

▪ When creating views, use the smallest type projection possible. (If you're not sure what type projection means, see http://blogs.msdn.com/b/jakuboleksy/archive/2009/01/20/getting-started-with-type-projections.aspx). The following classes exist for Incident view:

• Incident

• Incident (Typical)

• Incident (Portal)

• Incident (Advanced)

▪ Depending on the class chosen, more components may be included in the view. The Incident (Advanced) type projection is the most inclusive, but it is also the most performance impacting class. Views based on the Incident (Advanced) class will thus have an impact on the console performance. As a result, you should always use the smallest projection possible. For more information, see http://blogs.technet.com/b/servicemanager/archive/2010/12/02/faq-why-is-my-custom-incident-view-so-slow.aspx where you will also find a link to a management pack with many useful type projections.

▪ Consider using the Global Operators group when assigning tickets to limit the number of users loaded when searching for anything assigned to Analyst. Doing this can result in a drastic performance improvement. (See http://blogs.technet.com/b/servicemanager/archive/2012/12/14/faq-why-does-it-take-so-long-to-find-users-in-the-assigned-to-and-primary-owner-fields.aspx.)

Service Manager databases

A Service Manager environment consists of two database roles: the Configuration Management Database (CMDB) and the Data Warehouse. The CMDB is a centralized, model-based repository for all service management information managed by the Service Manager server. This database contains three types of information:

▪ Configuration items (CI) such as knowledge articles and components from organizations imported via the Service Manager connectors

▪ Work items (WI) such as incidents, change requests, problem records, and so on

▪ Configuration settings for the product itself

The CMDB is where the console retrieves data to display in the views. It's also where changes or updates to any work item are written. The database is therefore very I/O intensive and is a critical component when it comes to performance. Follow this guidance for better CMDB performance:

▪ For optimal performance, place the Log database, TempDB database, and the Service Manager database on separate high performance logical unit numbers (LUNs) to optimize I/O.

▪ Configure the SQL Service to use 1 to 2 GB less than the total RAM available in the server to ensure the SQL Server does not exhaust the operating system.

▪ Use the official Service Manager sizing tool (http://go.microsoft.com/fwlink/p/?LinkID=232378) to estimate the database size, then configure the database for the expected size and disable autogrow. This will ensure the database isn't extended during normal working hours; instead, SQL administators can manually extend the database if needed.

▪ The TempDB performance is critical to Service Manager performance. Use Multiple TempDB files, for example one per two cores (this is not necessary for the other databases). For more information, see http://technet.microsoft.com/en-us/library/ms175527(v=SQL105).aspx.

▪ The default retention of work items in the CMDB is 90 days for incidents and 365 days for the others (change, release, service, and problem). You can decrease these settings and still comply with the requirement to have enough work items available in the console views. Decreasing the retention keeps the CMDB to a reasonable size. Remember that work items that are closed (not resolved, completed, and so on) are removed from the CMDB, but they are still available in the Data Warehouse for reporting purposes. For more information, see http://blogs.technet.com/b/servicemanager/archive/2009/09/18/data-retention-policies-aka-grooming-in-the-servicemanager-database.aspx.

The Data Warehouse acts as the long-term repository for the service management information collected by Service Manager. It can also be configured to store a subset of information from System Center 2012 Operations Manager and System Center 2012 Configuration Manager. When the Data Warehouse management group is installed and all features are enabled, seven databases are created:

▪ DWDataMart

▪ DWStagingAndConfig

▪ DWRepository

▪ DWASDataBase

▪ OMDWDataMart

▪ CMDWDataMart

▪ ReportServer

A Data Warehouse management group for a Service Manager environment is not a requirement and provides only reporting and long-term storage. If you are working in an environment where reporting is not needed or required, you can safely leave out the Data Warehouse. Not installing the Data Warehouse won’t affect the usability of either the Service Manager console or the Service Manager Portal. If you plan to use reporting at a later stage, install the Data Warehouse at the start, not only to avoid losing needed data before reporting as required, but also to avoid impacting the performance of the Service Manager environment when too much historical data is synchronized at one time.

Some configuration changes can be made to improve the performance of the Data Warehouse and reporting. Consider these recommendations if the Data Warehouse is having performance problems, which is often the result when synchronization jobs from the CMDB fail. It's important to remember that the performance of the Data Warehouse has no impact on the console user's performance experience.

▪ Split some of the Data Warehouse databases across more SQL servers. However, keep in mind:

• The DWStagingAndConfig and DWRepository databases must be on the same SQL Server instance.

• The DWDataMart, OMDWDataMart, and CMDWDataMart can each be placed on different SQL Server instances. (There is no reason to place these databases on different SQL Server instances because the quantity of data and access to the data stored in the OMDWDataMart and CMDWDataMart databases are not large.)

▪ Place the SQL Server Analysis Server instance on a different SQL instance than the Data Warehouse databases. This will ensure that cube processing does not affect the extract, transform, and load jobs or the execution of reports. Processing of cubes is handled differently in SQL Server Standard than in SQL Server Enterprise; Enterprise can do incremental processing instead of just full. With SQL Server Enterprise, you can move the Analysis database to another drive. This is not an option in SQL Server Standard. If possible, use the Enterprise edition for the Data Warehouse. See the table in the next section for more information on SQL Server editions.

▪ By default, data in the Data Warehouse is retained for three years. You can modify this retention setting, but it requires some complicated changes. For more information, see http://blogs.technet.com/b/servicemanager/archive/2011/06/07/how-much-data-do-we-retain-in-the-service-manager-data-warehouse.aspx.

SQL Server editions

SQL Server 2012 and SQL Server 2008 are available in both Standard and Enterprise editions. Service Manager will function with either edition, but SQL Server Enterprise includes additional features that enhance your experience with the Service Manager Data Warehouse. For the Service Manager Database (CMDB), however, the Enterprise edition of SQL is not needed.

The following table lists the key differences between SQL Server Enterprise and SQL Server Standard for Service Manager environments. Carefully consider which SQL Server edition is best for your deployment.

Workflows

Incorrectly configured workflows heavily impact customer environment performance. If a workflow runs too frequently, performs large queries or bulk updates to work items, and so on, performance will suffer. Some Service Manager environments have thousands of workflows, resulting in slow and unreliable performance.

As described in Chapter 3, sometimes the processes in an organization need to be redesigned to align with Service Manager. Many times, customers want Service Manager to look and behave exactly like their current service management tool and do not consider it from a people/process perspective. Configuring Service Manager to mimic the incumbent tool usually results in installations with too many workflows, and a poorly performing Service Manager environment typically results. Therefore, to get the best out of Service Manager, you might need to redesign some of your processes to better align them with the tool.

Service Manager Self-Service Portal

The Service Manager Self-Service Portal, shown in Figure 6-1, provides end-user self-service and activity management. Service Manager Portal is based on SharePoint Server and is where end users can view the published service catalog, create service and incident requests, search for knowledge articles, and view, update, and approve review activities.

Deploying Service Manager Portal for end users usually reduces the number of calls to your service desk. This means your service desk has more time for other tasks such as incident and request management.

FIGURE 6-1 Service Manager Self-Service Portal

The following are some best practices for planning a deployment of Service Manager Portal:

▪ Service Manager Self-Service Portal for System Center 2012, 2012 SP1, and 2012 R2 is supported only on SharePoint 2010. SharePoint 2010 is supported to run only on Windows Server 2008 or 2008 R2, not on Windows Server 2012. Service Manager Portal is the only Service Manager component that cannot run on Windows Server 2012.

▪ Self-Service Portal can be installed in an existing SharePoint 2010 farm. If you're not running SharePoint 2010 or if you have already deployed SharePoint 2013, you can install Service Manager Portal on the free SharePoint Foundation 2010 edition.

▪ When the Web Content Server role is installed, make sure to verify that Application Pool Recycling is disabled. The default is nightly recycling, which results in a slow initial load for the users first accessing Portal. Also make sure you do not set recycling for high memory usage. For more information, see http://blogs.technet.com/b/servicemanager/archive/2011/05/11/faq-why-is-the-self-service-Portal-so-slow.aspx.

▪ When designing an environment that needs to support more than 20,000 users, you should add more Web Content servers and place them in an NLB cluster. This will ensure better performance when the Web Content servers are reading or writing data to or from the CMDB.

▪ When designing request offerings, aim to limit the number of custom icons used. Too many custom icons can have an impact on the time it takes to load the Portal main page.

Connectors

Connectors allow Service Manager to communicate with other products and technologies such as Active Directory and other System Center products. Connectors also allow Service Manager to import and store Configuration Items (CIs) in the CMDB. Using multiple connectors allows you to deploy a complex CMDB for the infrastructure. The following connectors are included out of the box in Service Manager:

▪ Active Directory connector

▪ Configuration Manager connector

▪ Operations Manager Configuration Item connector

▪ Operations Manager Alert connector

▪ Virtual Machine Manager connector

▪ Orchestrator connector

▪ Exchange connector (supported on Service Manager 2012 SP1 and later.)

General considerations

Some general considerations concerning connectors include:

▪ For both out-of-the-box and third-party connectors, always use a unique run-as account for each connector. Doing this will create a separate MonitoringHost.exe process on the workflow management server for each connector running on the server, making it easier to see which connector is currently running and how much memory/CPU it is consuming. It also makes it easier to isolate one connector from other connectors so that it can be terminated without affecting other connectors or workflows.

▪ Configure your connectors to avoid having them pull large amounts of data into Service Manager during the day, thereby affecting performance. All of the connectors are configured via the Administration tab in the console, except the Active Directory and Virtual Machine Manager connectors, which are configured using management packs.

Active Directory connector

The Active Directory connector is used to synchronize data between Active Directory Domain Services (AD DS) and the Service Manager CMDB. Service Manager can synchronize the following types of directory objects:

▪ Users

▪ Computers

▪ Printers

▪ Groups

Customers often configure Active Directory connectors to import all objects from the root of the domain. However, this means that all users (enabled and disabled), groups, and computer objects are imported, causing the Service Manager database to bloat with objects that are not needed. So, the first thing to consider when configuring the Active Directory connector is whether you need all directory objects in the CMDB.

Computers

If you are also using the Configuration Manager connector, there really isn't a need for the Active Directory connector to import all computer objects since it would only mean that Service Manager would have to import, merge, and maintain two sources. All relevant information about computers in the environment is delivered by the Configuration Manager connector.

Groups

Work items are often assigned to support groups first and then to individual user accounts, so they don’t contain membership information. Therefore, groups are often not used in Service Manager. So instead of importing all Active Directory groups, filter to import only the relevant groups as needed.

Users

The Active Directory connector imports all users in a domain, regardless of whether they are enabled or disabled. If contacts are created as domain users in the directory, these are imported as well. It is therefore important to consider which organizational units (OUs) to import users from and also whether to import both enabled and disabled users. Typically, depending on your OU structure, it is best to create separate Active Directory connectors to avoid populating the CMDB with unnecessary data.

For example, if you want to configure a connector to import only enabled user accounts, you can use the LDAP filters that were introduced in Service Manager 2012. Launch the Active Directory connector wizard and, at the Select Objects option shown in Figure 6-2, select Users Or User Groups and enter the following LDAP query:

(&(ObjectCategory=User)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))

This LDAP filter will ensure only enabled user accounts are imported.

FIGURE 6-2 The Active Directory Connector Wizard

The following are some best practices for designing the Active Directory connector:

▪ For the most optimal planning and performance of the Active Directory connector, first perform a thorough cleanup of any obsolete objects in the directory and identify which OUs to import into the CMDB. Some environments include thousands of obsolete objects that can be mistakenly imported into the CMDB. Having large numbers of obsolete objects in the CMDB will impact console performance since some user information is cached in the console at startup. Importing obsolete objects will also impact Service Manager performance. In addition, obsolete and irrelevant user objects will impact analysts during a user lookup.

▪ For the Active Directory connector, decide which directory objects need to be part of the CMDB. Set up one or more Active Directory connectors that import only relevant data. A poorly managed directory can contain thousands of stale objects that aren’t needed in the CMDB.

▪ When designing the Active Directory connector, consider using LDAP filters to import only relevant types of objects. For more information on LDAP filters, see http://msdn.microsoft.com/en-us/library/windows/desktop/aa746475(v=vs.85).aspx.

▪ If your design includes several Active Directory connectors that daily import large numbers of changes, consider scheduling the time each connector runs, taking into account the duration that each connector needs to complete, so no more than one Active Directory connector runs at a time.

Operations Manager connector

There are two different Operations Manager connectors. Operations Manager Configuration Items (CI) connector is used to synchronize CI data from Operations Manager to the Service Manager CMDB. Operations Manager Alert Connector is used to synchronize Alerts to and from Service Manager. The Operations Manager Alert connector is a two-way connector, meaning that when an incident related to an Operations Manager alert is resolved, the Alert connector will also close the same alert in Operations Manager.

The following are some best practices for designing the Operations Manager Alert and CI connectors:

▪ If you plan to use the Operations Manager Alert connector to have alerts automatically created as incidents in Service Manager, do not create the connector until you have tuned the Operations Manager environment to report only relevant actionable alerts. Some customer environments forward all alerts from Operations Manager to Service Manager despite many of them being false positives. Besides affecting the performance of Service Manager, this also affects the usefulness and trustworthiness of Service Manager since not all incidents need to be acted upon.

▪ Begin with only a few management packs from Operations Manager to import alerts into Service Manager. Create incident management templates with predefined values for Support Group and Incident classification per management pack to start off slowly. Export all the management packs in Operations Manager and provide this list to the IT infrastructure team and ask: Who needs this alert assigned to them? Then create incident templates using the information that IT provides to streamline the assignment of alerts to the groups that are responsible for them.

▪ Be very selective concerning which Operations Manager monitored objects need to be added to the CMDB. There is no requirement for importing all Operations Manager monitored objects into the CMDB. The more objects that are imported into the CMDB, the more objects need to be maintained by the connectors, workflows, and so on. For more information, see http://blogs.technet.com/b/servicemanager/archive/2010/02/26/managing-the-allowed-list-for-the-operations-manager-ci-connector-with-powershell.aspx.

Configuration Manager connector

The Configuration Manager connector is used to synchronize data between the Configuration Manager database and the Service Manager CMDB. Service Manager synchronizes hardware and software inventory as well as asset intelligence inventory attributes. In addition, Service Manager provides a workflow for creating incidents when managed machines are reported as non-compliant by using the Desired Configuration Management (DCM) feature in Configuration Manager.

The following are some best practices for designing the Configuration Manager connector:

▪ If you are not using DCM in your environment, you should disable the DCM rule in Service Manager. For more information, see http://blogs.technet.com/b/mihai/archive/2012/11/30/configuration-manager-connector-s-dcm-rule-can-cause-massive-performance-issues-in-service-manager.aspx.

▪ Always consider which collections in Configuration Manager to import because selecting All Systems might import data from systems that aren’t relevant to store in the CMDB. See Figure 6-3 for more information.

FIGURE 6-3 The Configuration Manager Connector Wizard

Orchestrator connector

With the Orchestrator connector, you can synchronously invoke Orchestrator runbooks from within Service Manager by using workflows. This capability integrates Orchestrator automation capabilities and the Service Manager Portal, as well as business modeling capabilities.

Activities that make up a service request can be mapped to runbook activities, which in turn are mapped to an Orchestrator runbook. For example, the parameters that are necessary for a custom start activity to invoke a runbook in Orchestrator, such as a computer name, can go into Service Manager as objects. You start this process by importing runbook objects into the Service Manager database using the Orchestrator connector.

If you expect to run many Orchestrator runbooks that will involve Service Manager and you have identified that this might degrade the performance of your environment, add an additional management server and target all Service Manager-related runbooks to that server. This server's sole role will then be to reply to Orchestrator requests in isolation from other Service Manager workflows or to support console access, which is housed on other management servers.