5.5. Staffing for Strategic Projects
As I said previously, staffing for projects that fall out of the realm of new implementation or operational changes tend to fall into the category of strategic changes. Three core types of strategic changes include the following: (1) discrete technology “refreshes,” (2) larger scale SAP infrastructure projects or consolidation efforts, and finally (3) SAP functional upgrades, all of which are tackled next.
5.5.1. Discrete Technology Migrations in the Real World
Perhaps my favorite general stress-testing projects stem from a planned discrete upgrade to a customer's SAP system. I like these projects because there are fewer dependencies to manage and an excellent baseline—the current system—is readily available. Further, discrete technology refreshes, by virtue of their, well, discreteness, are very focused projects. The staff necessary to pull off a technology-specific stress test fall comfortably into a particular layer of the SAP Technology Stack, and thus the needed skill sets are by default well understood and quantifiable. Consider the following projects I've worked on, and you'll understand what I mean by discrete:
Given my background in SQL Server-based SAP implementations, I'm no stranger to SQL Server updates and upgrades. In moving from SQL Server 6.5 to 7.0, and then to 2000, a number of my long-time customers leveraged my Microsoft colleagues and me to help them plan for and execute SAP application layer or pure database-focused stress testing. The current move to 64-bit SQL already finds us engaged in simple “delta throughput” testing (32-bit versus 64-bit technology stacks) as well as full-blown stress testing. In terms of staffing, the key was and will continue to be identifying one or more resources skilled in both the “before” and “after” database releases (especially when it comes to deploying specific database releases within a specific SAP environment or release), along with the usual technology stack, business process, and test-tool players. I prefer to leverage the RDBMS vendor itself, like Microsoft, to provide the “best” resource, because this gives the project what I believe to be the best chance of succeeding. After all, who knows Microsoft's products better than Microsoft? Or for that matter, Oracle 9i or 10G better than Oracle? Who has more skin in the game—more to lose and more to gain—than the vendor? And who has better lines of support when things invariably go south once in a while in a stress test or similar performance-tuning project?
I have helped my clients migrate between many different physical disk subsystems over the years. From various flavors of direct-attached Small Computer Systems Interface (SCSI) disk subsystems to fibre channel arbitrated loop (FCAL) solutions, to the first SANs, and more recently to network-attached storage (NAS) and virtual SANs, I've supported a large number of SAP-specific disk subsystem–based stress tests. Without such testing, I'm confident that the level of throughput enjoyed by my customers would never have been close to what they finally enjoyed in their production systems—these types of discrete hardware migrations really benefit from the iterative testing and tuning of which I'm so fond. Note that when it came to staffing, identifying the “before” and “after” technology specialists made all the difference.
Stress testing various server and OS combinations has consumed much of my time as well, especially in the last 3 years, as commodity-based solutions pushed much bigger but more expensive iron out of the data center. Variety has made these projects nothing if not fascinating. Projects have included server migrations at the SAP database layer, the SAP application (online and batch) server layer, the supporting middleware/integration layer, and even the client access layer (in the form of ITS stress testing, to identify the maximum number of supported HTML-based WebGUI front-ends). In some cases, the technology stacks have been completely different (IBM AIX/Oracle versus HP/Intel/Microsoft, or pure Microsoft stacks versus Linux/Oracle). Most of the time, though, these stress tests are focused more on just the impact that a new server model has, or just a particular version of a new OS. Staffing varies based on what's being compared, but is still focused—in this case, on the server/OS (and to a lesser extent, database) combination.
Official “SAP Migrations” sometimes fall into the realm of discrete technology migrations, too, though quite often such a migration is anything but discrete. For instance, I consider moving from Oracle to SQL Server a fairly discrete technology project. Yes, it's a significant change, as is any database migration between different RDBMSs, but it's a lot less significant that throwing a new hardware platform and OS into the mix!
5.5.2. SAP Infrastructure Projects in the Real World
I have participated in a number of infrastructure load-testing projects as well, which are also arguably “discrete technology projects” in and of themselves. When I think of the term infrastructure as it pertains to SAP environments, I typically envision the physical data center hosting the SAP solutions. Thus, network infrastructure and server platforms come to mind first. Network infrastructure in particular can drive very interesting stress-test projects, requiring network-savvy personnel typically not found in the core SAP support organization; on separate occasions, I've been asked to
Identify the network load that a T1 network link hosting 600 online and concurrent SAPGUI users can comfortably carry (before the days of EnjoySAP, 600 SAPGUI users would consume only 10%–15% or so of a T1; today, it's closer to 30%–40%, not including print and other traffic)
Compare and quantify 100-Mb versus 1-Gb network architectures when it came to supporting both the database-to-application server backend network and the front-end client network
Quantify the performance deltas between maintaining a single “shared” network for all SAP traffic—both back-end and front-end—versus the performance observed by clients when a tiered network architecture was put in place
Beyond the servers and network infrastructure tying everything together, however, there are other “shared resources” that may drive a stress test in your particular case. I was involved in a short-term keyboard/video/mouse (KVM) stress test once, for example, where I helped my client validate the server and user load that a remote KVM-over-IP (Internet Protocol) solution could support. Granted, it was not an SAP project per se, but the results were applicable to their entire enterprise environment. In a similar fashion, one of my colleagues and I helped quantify the thermal load a particular data center could handle comfortably before heat thresholds were exceeded; this aided my customer in sizing and building out their new data center and later served as a baseline when they migrated to new server and disk subsystem platforms, each generating different, though generally greater, thermal loads (in smaller and smaller packages!).
In the case of the KVM project, testing without benefit of direct access to the vendor would have made things much tougher. We didn't require an onsite resource, but the time we took to identify a phone-based support person prior to doing the testing paid off. As for the data-center thermal testing, though, we never engaged an “expert” from any outside vendors per se, other than myself. But without my data-center rack-and-stack knowledge, my knowledge of HP and Compaq server and disk subsystem platforms, my general knowledge gained from supporting and working in data centers for a number of years (I got my real start in IT as an IBM mainframe computer operator in the USMC), and access to online materials regarding Liebert cooling equipment, we would have spent much more time iteratively trying different rack/server combinations, power combinations, and so on.
5.5.3. SAP Consolidation Efforts
Consolidation is hot everywhere in IT, from customer-facing Web-server farms to back-end data warehouses and core transactional systems. The world of SAP is no exception. In fact, given the “largeness” of most SAP system landscapes and their supporting environments, it's no wonder that IT consolidation for SAP is discussed in many different circles today—CIOs want to better understand the TCO ramifications of consolidation, IT managers want to get a handle on performance and manageability considerations, the business wants to better understand how “all your eggs in fewer baskets” might actually be a good thing, and so on. Indeed, the topic is broad enough and compelling enough to warrant its own book. For our purposes here, though, let's take a look at common consolidation scenarios and how each impacts staffing a T3.
First of all, consolidation for SAP impacts every single technology layer, including even the fundamental data-center/infrastructure layer (e.g., collapsing many data centers into fewer). But the most common form of SAP consolidation I've seen hinges on reducing the number of SAP application servers servicing a database server. The second most common consolidation scenario involves bringing together expensive storage and tape subsystems into a few well-architected SANs (as an aside, avoid collapsing everything into a single SAN—you still need a place to test changes to the SAN fabric that connects all servers, storage, and tape components, without the possibility of impacting your production environment). Consolidation is also becoming more common at the Web AS and ITS layers, especially in nonproduction environments where the need for a whole server dedicated to a particular mySAP component or product server simply wastes resources in most cases. Finally, consolidation of multiple database instances onto fewer Database Servers (and still maintaining the same number of instances—not MCOD, but something close) seems to be growing in popularity, too.
What does this all mean in regard to staffing? Simply put, you'll benefit from having both an SAP sizing expert and a true SAP performance-tuning expert on your team—and a person as comfortable with SAP Basis as he or she is with writing and executing business process scripts that touch multiple SAP systems. Outside of this, you'll also want an expert in the core technology stack layer undergoing the consolidation. That is, if you're collapsing many servers into fewer servers, you'll want to get the most performance and highest levels of availability out of those fewer servers and therefore should bring an expert in that particular server model into the T3.
For example, are you trying your hand at SAP's MCOD approach to consolidation by combining CRM, R/3, and EBP together on one database? If so, you'll certainly need not only an SAP technology expert in each of the core components (CRM, R/3, and EBP), you'll also need an SAP technical sponsor, someone with whom you can immediately work to help you understand or internally escalate your MCOD test results. And, when it comes to MCOD, the requirement to understand peak business loads within each component/business area cannot be underestimated— get your business representatives on board early to determine the true peak load that you might place on a single database configured for MCOD. That is, be sure to analyze the workdays for each component so as to identify the busiest combined peak (e.g., the third or sixth workday of the month). And then take care to size and test the solution, to ensure it can indeed support such a load.
Similarly, if you're collapsing Web AS or ITS servers, you could do with an expert in these areas as well, someone familiar with installing and managing multiple instances on a single-server platform. The same is true in the case of using “virtual partitioning software” to effectively slice and dice a single physical server into multiple logical servers, each running its own nonproduction SAP instance. I'm a big fan of VMware's various products, Microsoft's Windows System Resource Manager (WSRM), and HP's various partitioning products as well—if you choose to pursue one of these consolidation approaches and are already an expert in deploying them for SAP environments, I suggest giving the respective vendors a call and arranging for some customized or formal training and onsite support. It's safe to say that the time and money you spend in doing so will be far less than the time you waste underdeploying or underoptimizing these products. And, with SAP's support for particular portioning solutions varying across the board, you want to be sure you are deploying a software solution that is indeed supported, especially in production environments.
5.5.4. SAP Functional Upgrades
Many of my customers have confirmed that their SAP functional upgrades were successful because they staffed them heavily with their own employees, augmented by consulting and contracting agencies to fill in niche or small-impact needs. Further, well-known SAP patrons have said over and over again that underestimating the tools and methods necessary to deploy and test SAP's products will only get you in trouble. Sure, the tools can be daunting at first blush, and the methods complex. But, with no supported alternatives, your choices are limited. With all of this said, then, it should be clear that SAP functional upgrades—where a core release or version of a particular SAP product is upgraded to a newer release—are major projects. Very major! Besides their inherent complexity, functional upgrades impact everyone, too, from your technology-focused IT staff to business process and other functional and development folks, to the end-user community tasked with using the system day to day. The last thing you therefore want in an upgrade scenario is to be ill prepared from a staffing perspective—just like an SAP implementation, a lot of people's jobs are at stake.
Staffing should therefore address every layer of the SAP Technology Stack, from developing contacts with the Data-Center's SAP Operations teams all the way up to working with core business-focused representatives tied to each SAP module or functional area. You don't need full-time resources, necessarily, but you do need full-time T3 representatives from each area. And, because the business is likely reinventing itself to some extent, in preparation (or as a result) of the functional upgrade, tight links into your change management organization are critical, too.
Challenges will be many, though. First and foremost, the business processes leveraged by your team for baselining may very well change as a result of a functional upgrade. The change could be minor, or very significant. Either way, the value initially realized through the baseline will diminish and must be factored in to subsequent testing/tuning engagements. In a similar manner, the underlying technical infrastructure and perhaps even the system's architecture may change, simply because SAP has never released a new version of its software that would support the same precise workload as its predecessors. Accordingly, it's not uncommon to add 10% to 30% more processing power, memory, disk, or network bandwidth just to host the same number of online users and batch processes previously hosted by an older SAP release. This complicates measuring performance against your baselines, and so must be factored in alongside the functional changes occurring at the SAP application layer.
SAP functional upgrades are by no means short-lived, too, often consuming 6 to 9 months as they ebb and flow between pure functional upgrade testing, migration strategies, risk analysis, functional testing and tuning, change management processes, technical platform refreshes, and so on. What does this all mean in regards to staffing? Bottom line, you'll need a very experienced, capable, and adaptable team to tackle load testing for SAP functional upgrades. To top it off, you'll need a team that is flexible in terms of locating the best resources for supporting a particular technology layer or functional area; the team will represent a diverse mix of internal employees, contractors, partners, and third-party consultants. Links into other organizations will need to be established and managed, from the database and network teams to the DR organization responsible for business continuity, to any other technical or functional team that integrates with or otherwise touches SAP. And finally, you'll need to establish and manage a solid “feedback loop,” so that the value you receive from iterative testing/tuning may actually be captured, tracked, and maintained as testing progresses (a full-time task in itself!). My advice is to seek management's unequivocal sponsorship and then give yourself plenty of time to find the best people, engage with the functional teams early, and tie off with your internal IT teams.