Total Cost of Ownership

Up to now, this chapter has focused on the types of servers, OS, and DB platforms supported with SAP software, including some details about server technologies. The subject of Total Cost of Ownership (TCO) of these server platforms remains. A detailed cost analysis of each server and OS platform is clearly beyond the scope of this book, primarily because it is difficult to collect accurate and reliable data in a rapidly changing marketplace. However, the costs of the server, OS, and DB platforms in relation to the overall project costs are presented in this section, based on studies of ERP/SAP projects. Important factors that make up the TCO of the infrastructure requirements are also presented.

An important fact about SAP projects is that the ongoing operational costs, regardless of operating system, are a significant factor in the long-term TCO. To help reduce the ongoing operations costs, the subject of server consolidation is the primary topic covered in this section.

Overall Project Costs

An SAP project can take anywhere from several months to several years to fully implement, depending on the size of the organization and on the application customization requirements. Once a project is implemented, there are ongoing costs to be incurred. The acquisition of server and storage for the SAP systems are usually only accounted for at the beginning or during the implementation phase. The ongoing support of the servers and storage infrastructure may be a bigger cost, however, than their initial purchase price.

Comparing the prices of server platforms, whether Intel-based servers, the Open Systems Unix servers, or the mainframe hosts, is not discussed in this book. Published benchmarks attempt to address the price/performance ratios of the major server platforms (ex: TPC-C). However, these typically only address the cost of the servers and storage acquisition and maintenance. They do not include the remaining SAP project costs, which are usually much larger, nor do they consider all factors or reasons for choosing different systems.

The major categories of SAP project cost factors include the hardware, the software, the configuration, and the operations and support aspects, as shown in Table 2-3. Studies have been done to show the contributions to the total cost each of these categories and project cost factors makes. In general, the server and storage costs are relatively small (less than 5%) given the total implementation costs. In many projects the network infrastructure and the end-user PCs alone cost more than the server and storage requirements.

Table 2-3. Categories of SAP Project Cost Factors

Major Categories	Project Cost Factors
Hardware Infrastructure	Servers and Storage
	Network—LAN and WAN
	End-User Devices (PCs)
Software Licenses	mySAP.com Business Applications and Roles
	Database, OS, Backup/Archiving, and HA Software
	Management Tools, Middleware Apps, and Others
Application Development and Configuration	Consulting
	Internal Implementation
Operations, Support, and Administration	Production Operations
	User Support, Training, and Service (Help Lines)

There are essentially two major phases of any SAP project: implementation and production (ongoing). Although the particular time-period for each phase differs, every successful SAP project does eventually go into production where the ongoing support costs apply. According to a META Group study, the ongoing support costs for ERM systems can account for nearly 40% of the total cost of ownership during the initial three-year period. Some of the details that make up the infrastructure category, with focus on servers and storage, are included in Table 2-4.

For the implementation phase, the IT infrastructure costs are a smaller part because the application configuration (SAP customization) costs represent a large percentage in the beginning. For the ongoing support costs, however, the IT infrastructure costs can represent up to two-thirds of the ERP project (with application configuration and software license fees making up the rest). One of the ways to manage the ongoing server and storage costs is to use a common infrastructure whenever possible.

Thoughts for Choosing a Server Platform

Given the ERP project cost analysis shown in Table 2-3, the initial cost of the server platform is only one of many factors. The decision for a particular server, OS, and database plat-form should be driven less by the up-front acquisition costs, but more by the ongoing costs, given their impact on the project costs. Some of these downstream ongoing costs for the server platform include:

• Server support/maintenance costs and availability in all needed regions;

• Ease of upgrading server's processors, memory, and disk systems;

• Speed of upgrading to meet increased capacity (i.e., instant capacity on demand);

Table 2-4. IT Infrastructure Cost Factors, by Phase

Implementation	Ongoing
• Server hardware	• Server hardware maintenance
• Storage hardware	• Storage hardware maintenance
• Server utilities software (including HA software)	• Disaster recovery management (including backup and restore processes)
• Network infrastructure	• Minor bug fixes and patches
• Desktop PCs for end-users	• Facilities
• Professional services for infrastructure	• Upgrades of servers, storage, and network infrastructure
• Internal staff costs for infrastructure	• Operating system administration
• Infrastructure training	• Database and middleware administration
	• Network management
	• Data archiving
	• Help desk
Source: Based on the META Group study, Enterprise Application On-Going Support Costs Unveiled, 10/99 (www.metagroup.com).

• High-availability and disaster recovery features and support from server platform vendor (including advanced clustering implementation and support offerings);

• Integration of high-availability, disaster recovery, and network management solutions for the SAP systems into the existing IT infrastructure, if already established;

• Support of server consolidation, together with high-availability configurations, to help reduce the physical number of boxes to administrate;

• Training requirements of the OS and database platform (i.e., can in-house IT administration knowledge be leveraged for the SAP systems?);

• Remote administration capability, especially in outsourced or co-locations; and

• Reliability, maturity, and application availability of the server, OS, and database platform, especially with 64-bit implementations.

Cost/Performance for the SAP Server Choice

A common approach to choosing server systems for SAP is to consider the following:

• For the database server and storage, spend the money and purchase the most capable and flexible system the business can justify. This is the most critical system in the landscape and cannot easily afford to be down due to failure or performance issues.

• For the application servers (in a multi-tier environment), however, the sweet spot in price/performance seems to be with standard, 4-, 6-, or 8-way mid-range SMP systems (or partitions). These systems are more cost effective because they get more performance from each additional processor. Consider choosing standard server types for categories of usage to create a common and flexible server infrastructure.

System Consolidation

System consolidation is undoubtedly one of the mainstreams in business computing. The reasons for this are, however, not technological. Driven by globalization, acquisitions, and mergers, organizations are experiencing a tremendous increase in user numbers, SAP R/3 modules, and mySAP.com components, customizing, and so on. It is common for multinational organizations to have multiple production SAP R/3 systems installed for different countries or subsidiaries. Particularly in a mySAP.com environment, a multitude of components with up to seven different databases and application and special purpose servers may each need to be set up with development, test, and production configurations. These systems must support thousands of users around the world on a 24 x 7 x 365 basis.

The result is a mass proliferation of servers, which causes an increase in floor space, complexity, administration effort, and costs as the result. System consolidation has emerged as the preferred solution to these problems. The total costs of ownership for a few large systems are naturally lower than for a multitude of smaller systems having the same capacity. The economic reasons for consolidation have been presented in detail by several studies (Gartner Group, etc.). However, all studies emphasize that the main contribution to TCO comes from operations. UC Berkeley claims that “for every $1K a company spends on a server it spends $9K on management.” The effort associated with managing an increasing number of servers with their respective operating systems leads to costs higher than in a consolidated setting. A recent case study by Gartner Group shows that consolidation from six distributed to two central systems with 800–1000 users resulted in:

• 38% reduction of hardware/software costs,

• 54% reduction of storage/backup management costs, and

• 60% reduction of administration/support costs.

The aim of this section is not to deliver the justification for consolidation. This has already sufficiently been done by others. A multitude of consolidation opportunities exists, spanning from server, storage, and platform consolidation up to data center and management consolidation. The implementation of SAP standard software itself can be seen as a consolidation approach. In one customer case, SAP was introduced to consolidate over 150 different applications.

However, consolidation to one large server will not necessarily result in the highest degree of TCO savings. Placing more than one SAP system on one physical server results in a more complex software configuration, which comes to bear during installation and upgrades, and also when adding additional third-party software to help operate or manage this environment. Technically, many configurations are supported, but to keep support simple, several SAP Notes recommend placing the SAP systems on separate physical servers. Deciding which systems to consolidate can be tricky due to various dependencies. It is important to understand the release upgrade process before deciding which systems to consolidate or to combine on one server. Experienced organizations prefer a solution with a standardized environment of equal sized servers. This results in common tools and procedures together with simple configurations to eliminate variances and errors. Depending on the specific situation, an optimal balance has to be found between the ease of configuration and the number of servers. After all, the ultimate goal of consolidation is to reduce the administration and operations costs.

Server consolidation is about which combinations of SAP and RDBMS components can be placed on one server. Some of the essential components that make up one SAP system are

• The SAP (instance) kernel files or binaries, which are a small set of OS dependent executables responsible for starting up an SAP instance, whether the central instance or an application server instance.

• The database (instance) kernel files or binaries, which are the set of OS dependent executables responsible for starting up and managing the relational database.

• The SAP release, which is the business application itself. Much of this is either ABAP code or data within the SAP R/3 Repository—stored in the database tables.

• The user data, which includes business transactions and master data.

• The operating system (OS), which is also considered an instance, but it is not included in the core definition of an SAP system.

• The server, which is considered as the physical computer box only.

In this section, we introduce the technology concepts and implications of large-scale consolidation in respect to mySAP.com infrastructures.

Phase One: Server Consolidation

From a technical standpoint, there are no SAP architectural or software design reasons preventing consolidation, and some advantages may result from it. In the past, common SAP R/3 installations used individual application server(s) for the different modules and a dedicated database server. This was primarily because of the limited CPU resources available on a single server box. The famous three-tier architecture was introduced because even the biggest servers did not have enough CPU resources to support a sufficient number of users. The benefit of this approach is scalability—when you need more power, just add on another application server. However, this approach has a few drawbacks:

• Each server needs a separate operating system instance, which must be managed and maintained. Upgrades and patches must be applied to all servers.

• The SAP R/3 scheduler enables load balancing between application servers at logon time only. When the load distribution changes within the workday, the only way to use free resources on another application server is to have users to log out and log back on again.

• A three-tier system generates a lot of network traffic between the database and application servers, potentially impacting the overall system performance.

The first step to mySAP.com system consolidation is to reduce the numbers of servers per mySAP.com component. The proliferation of superior computing power per single server is the enabler for reduction of the number of servers from many to a few or even to a single one. For medium workloads, the three-tier architecture can be replaced by a simple two-tier architecture, as shown in Figure 2-10. This way, database and business applications are consolidated onto one single server. SAP named such an approach a centralized system. Central systems have always been fully supported by SAP and are similar to the host environment from which SAP R/2 came. The software (logical) remains three-tier or more, but the server (physical) layers are reduced whenever possible.

Several benefits can be expected from a centralized system:

• Only one operating system instance needs to be managed and maintained,

• Only one SAP R/3 instance (or very few) needs to be managed and maintained,

• Ideal load balancing between all processes is achieved,

• CPU interrupts caused by network traffic are reduced,

• Response times improve due to lower IP stack latency time, and

• Reduced footprint is achieved by reducing the number of application servers to zero.

The elimination of the server network handling large data transfers between the database and business logic layers can save CPU resources and improve response times. In a consolidated central system, data is exchanged faster, directly from process to process. Moreover, the complexity of the whole setup is reduced. To implement a realistic consolidated environment, however, it is highly recommended to use 64-bit versions of the operating systems, databases, and SAP applications.

Phase Two: mySAP.comSystem Consolidation

A straightforward second step in consolidating mySAP.com infrastructures is to reduce the number of servers for the environment as a whole. From this point of view, system consolidation is the strategy to consolidate multiple SAP applications onto fewer physical server boxes. This may be interesting for outsourcing data centers, as well as for application service providers, who want to leverage their investments to multiple customers. The same need applies to many enterprises as well, because they deploy more than one production SAP R/3 system. For example, it's common practice to execute the human resource application on a separate SAP R/3 system. This way the frequent legal updates necessary for HR do not affect the main SAP R/3 system.

As time goes by, the number of SAP systems begins to grow in the data center due to subsidiaries, mergers and acquisitions, new business functionality, and so on. With the new license model introduced by mySAP.com , the number of SAP instances and databases will increase. Even if the implementation of all mySAP.com application components is not mandatory, there is a natural tendency to move specialized functions of a business process, like reporting or strategic planning, to specialized SAP systems (SAP BW, SAP APO, SAP SEM, etc.).

In many cases, these additional instances are relatively small. A consolidated one will fill only a small percentage of the performance available on a state-of-the-art midrange server. From the standpoint of TCO, these systems are natural candidates for a SAP system consolidation. A precondition for system consolidation is that the version of the operating system is compatible within the releases of the mySAP.com components.

SAP System Consolidation Levels or Variations

From a technical perspective, there are several levels (or types) of configurations running more than one mySAP.com component on one physical server. The levels zero through three designations are only used as labels for keeping the various configuration possibilities apart. They are not official names given by SAP.

Level Zero: Common OS, DB, and SAP

SAP's standard approach to consolidation is the multi-client system. In such a system, multiple clients share a single SAP R/3 kernel and database. The isolation of client data is enabled within the SAP R/3 system via a number assignment (ex: 001, 100, 200, etc.) The advantages of a multi-client system are ideal load balancing of memory and processor resources and only a single operating system, database, and SAP R/3 kernel to administrate.

In this configuration, only one set of SAP and database executables or binaries is used (see Figure 2-11). This is actually a single SAP and database system and instance. Therefore, this concept also has some drawbacks:

• Because all clients share the same repository, they share the same set of ABAPs or customizations of business transactions.

• The same SAP release level is mandatory for all clients.

• Unexpected resource consumption of one client may impact the response time of the other clients.

• There is one single and very large database to administer and back up.

• Any downtime for backup, maintenance, and so forth must be coordinated with all clients.

Level One: Common Database

Some of the drawbacks of the Level Zero consolidation configuration are avoided by using separate SAP systems with one common database. On some OS and DB platforms, this is the only additional level of consolidation possible. Each SAP system has its own repository and can be upgraded or treated separately. However, all SAP systems use one common set of database kernel files, a factor that must be considered during upgrades (see Figure 2-12). Upgrades of mySAP.com application components are often dependent on using a particular version of OS and RDBMS platforms. A large difference in SAP releases on the same server may not always be possible due to limitations of database releases supported.

Level Two: Independent Databases

This configuration uses separate SAP systems and databases. The mySAP.com application components are executed independently of each other. They are all kept isolated from one another and can reside on different disk volumes. Different SAP R/3 releases with different database kernels can be run on one server (see Figure 2-13). This allows a flexible upgrade path for the SAP releases. One can be upgraded without affecting the other. The databases (including the kernels) are also independent or multi-homed, therefore relatively small and straightforward to back up independently. Even different database platforms (ex: Informix, Oracle, etc.) can be deployed on the same server. The only item in common is the operating system (i.e., one OS instance). Because each mySAP.com component system can be stopped individually, no coordination of downtime is necessary. However, some upgrade dependencies may still cause downtime for the entire system. Care must be taken during installation, of course, to ensure unique directory paths to all the installed system kernel files, and for Windows NT/2000, the registry must be adapted. To avoid memory limitations or configuration hassles, it is recommended to use 64-bit for everything, including the OS, database, and SAP R/3 application. For Windows 2000, use 36-bit memory addressing with AWE support, or Windows 2000 64-bit.

Another big benefit is that the SAP versions can be different, if needed. This can be necessary for certain industry solutions (IS). For example, an SAP R/3 4.0B and 4.6C system can be placed on the same server and upgraded independently. Most other mySAP.com applications are also based on an SAP R/3 kernel but have different data and application programs in the database, as well as different SAP R/3 kernel releases. HP has demonstrated, in cooperation with SAP's software factory, a setup of ten mySAP.com application components and seven database instances, consolidated to a single HP-UX N-Class server and an Intel Windows NT-based HP NetServer in one rack. Figure 2-14 shows some example mySAP.com application components consolidated, but this makes sense only in a development, test, or demo/training environment, or in a very controlled and low-usage production environment where security concerns are not an overriding factor.

Any number of SAP application packages can be executed on one OS instance (or server). It's simply a matter of server hardware resources, as shown in Figure 2-14. Some customers stack over 10 SAP R/3 systems on a single server. Typically, using a 64-bit operating system and 64-bit applications is necessary to gain the full benefits of running multiple SAP systems on one server. To avoid negative impacts on resource consumption between the independent mySAP.com components, a process resource or workload manager (described later) is recommended.

There are some restrictions for multiple SAP R/3 systems with multiple DB instances on Windows NT/2000 servers. The primary restriction is that some important environment variables, such as the directory PATH to the database kernels, are set globally in the Windows registry. These settings are specific to each installation and database version, making it tricky to install and support with third-party add-on tools. Many of these tools, including some backup tools, may expect only one database home directory, not two or more. Newer versions of the databases may have solved some of these issues. For example, Microsoft SQL Server 2000 includes independent registry configurations for each database instance.

Level Three: Independent OS

This configuration uses separate SAP R/3, database, and OS instances. There are two ways to implement this: with hard partitions or with virtual OS partitions. There are no common files used, not even the OS kernel files, as shown in Figure 2-15. Using hard partitions requires a specific hardware configuration but is also the most flexible, theoretically. This may be an attractive level of consolidation for large data centers due to its flexibility, scalability, and higher levels of security.

Static Hard Partitions use multiple, discrete SMP server nodes with independent OS instances. They can be connected together by a high-speed link to be managed as one scalable system. An example would be a rack full of eight-way servers; each configured to support up to 100 users. This, however, requires specially tuned applications where parallel DBs are implemented.

Virtual OS Partitions, within a single node, allow multiple OS instances to be run on one system or within one processor partition. This is the lowest cost version of consolidation. Only the latest Unix OS versions support this relatively new feature (ex: HP-UX 11i). It can be used on any supported SMP server, including mid-range systems. Care must be taken to avoid security issues by configuring each virtual OS to use its own I/O (network and disk) resources.

Configurable Hard Partitions running multiple OS instances on one physical server node can be reconfigured under operator and not application control. This is a system-within-a-system concept of collapsing multiple smaller servers using processor domains or partitions to separate the workload. The OS partition can be reconfigured to use more processor, memory, and I/O resources by simply restarting it, and there are no security issues from sharing resources. Each configurable hard partition can be combined with virtual OS partitioning for further flexibility (ex: HP-UX 11i). Sun's UE 10000 server was the first on the market to use this approach. HP's SuperDome has similar functionality, as well as Compaq's Wildfire server. Repartitioning is useful for handling peak loads, such as month-end processing or other seasonal peaks, or simply as the number of users grows.

Dynamic Hard Partitions, such as those of the mainframe, are also possible. They can be built, resized, merged and deleted under application control. Given the cost and complexity of this technology, however, some organizations have found that configurable hard partitions, when coupled with virtual OS partitions, can provide similar capabilities, often with more flexibility and at a lower total cost of ownership. However, dynamic hard partitioning is planned for most of the high-end server offerings.

Soft Partitioning, in comparison, allows application resources to be allocated (load balanced) according to policy-based prioritization. This is available on a single OS instance on most platforms (level two consolidation), including Unix servers and Windows 2000 Data Center.

Process Resource Management

In any scenario where multiple mySAP.com application components are executed under a single instance of the operating system, the applications compete for the available system resources. In case one of the systems comes under heavy load, the other applications will face higher response times unless the resources are managed. This situation can be resolved by soft partitioning with a workload management tool like HP's Process Resource Manager (PRM) for HP-UX.^[*] In such a scenario, each application is given its own partition, the size and capacity of which can be dynamically changed (in seconds) to guarantee specific service levels. PRM partitions ensure that performance isolation exists for as many applications as necessary, while the applications are running, without the need to reconfigure the system or to reboot the application.

^[*] Most Unix vendors offer a similar tool.

PRM is a workload management tool used to control the amount of resources that processes are granted to utilize. A PRM-Group contains any processes dedicated to an application. In the case of a mySAP.com application component, a PRM-Group can consist of the SAP instance together with the database instance. This way, the component will behave within its PRM-Group exactly the same way it would on a central system. All users and applications of one PRM-Group share the resources assigned to the group. The assignment of resources to the particular groups can be achieved by two methods:

• Soft Limits: Every group gets a minimum set of assigned resources. If the system is partially idle, the free resources can be requested and used by any PRM group.

• Hard Limits: Every PRM-Group gets only its associated part of resources. This implies that free resources cannot be used. Hard limits, or capping, are used to set guaranteed assignments.

Using hard limits helps IT administrators adjust the system resource during initial production phases so that users don't get used to or dependent on excellent response times. Resources are purposely cut back so that the response times at the beginning phases of a system rollout are similar to the end phases.

All limits are dynamically changeable without reboot! To manage CPU resources, PRM will simply extend the standard time slot algorithm of the operating system with an interface to adjust the otherwise fixed time slot distribution. This way PRM is absolutely transparent to the application and consumes no additional resources by itself.

In contrast to Figure 2-16, a standard OS scheduler will grant equal processor time slices to each of the groups of application resources, or 33.3% each.

Customer Examples: SAP System Consolidation

Stacking multiple SAP R/3 systems on a single server is nothing new or special. The most prominent examples are the development and training systems of SAP, in Walldorf, Germany, as well as the systems managed in the data centers of many SAP outsourcing providers. A few of the reasons for SAP system consolidation are provided here for two example customers with production SAP systems.

Large Catalog Retail Company, Germany

Some of the business drivers for consolidating system and servers at a large catalog-based retailing company in Germany included:

• Dramatic rise in computing power and storage needs for business applications,

• Proliferation of servers through x servers = 1 project/application,

• Better load balancing of resources between projects,

• High availability of all business applications,

• Existing systems needed in decentralized environment,

• Low ratio of administrators to servers,

• Physical space for adding new servers is restricted, and

• Better utilization rates for the consolidation servers.

An example of the German catalog retailer's consolidation environment is shown in Figure 2-17. It used multiple applications on one server, and included multiple systems in a clustering environment.

Oil Products Company

Hewlett-Packard also helped a Scandinavian oil company trim the physical number of servers needed for its production SAP environment, as shown in Figure 2-18. When asked what the reasons for application stacking were, the answers included the following:

• It just evolved,

• Saved operational costs for additional servers,

• Fully deployed the available resources, and

• Achieved flexibility to move around SAP systems.