The Other Issues
Lack of Metrics
Not one of the companies we studied had organization-wide metrics. About half of the companies had some sort of help desk metrics, and the 75 percent that had mainframe systems had some form of metrics for them.
There is a lack of metrics to measure effectiveness in the client/server environment. That old saying holds true here, if you can't measure it, there's no way you can manage it effectively. It is also important to establish internal metrics for each area of the operation, as well as quarterly or semi-annual ratings to compare the results based on cost and performance efficiencies. When these companies get their infrastructure in order it's imperative that they establish metrics. For more information on establishing metrics see our book entitled Building the New Enterprise.
The general use of metrics is imperative to the overall running of the IT department. Using a service such as Gartner Group to evaluate your organization is a constructive use of budget dollars. This is not to say that it is an annual expense, yet it is good for the IT department to compare its cost and performance efficiencies to other similar operations in and out of their industry.
Benchmarking Information Technology Services
The next step is to document the extent of services and their related costs so that you can compare them. The idea is to compare the cost of System Administration support or Network support with vendors providing this type of service. Tear your infrastructure apart in pieces. Then go out and benchmark. Ask them what they would charge to support your mission-critical servers, network, and so on. This is only a benchmark exercise, as outsourcing any part of your client/server mission-critical environment is not recommended. Once your infrastructure is cost-effective, take a look at your competition. For additional information on benchmarking your environment read our book titled Building the New Enterprise.
Poor Communications
With the evolution of client/server computing and the torrid pace of change with Information Technology, maybe this section should read "No communications." In the spirit of doing more with less—who has the time? And if by some small miracle you find the time, to whom do you turn? In this networked era there are no clear demarcations as to who does what to whom and when. And this is only internally within IT. What about the poor users? Are we effectively communicating with them? You can answer that one. Very few IT shops are even measuring customer satisfaction. Only a couple of the 40 companies we analyzed actually attempted to measure customer satisfaction. This is atrocious! But we understand the frustrations with the old process of sending out survey forms every quarter. Scheduled survey forms aren't popular and the usual response rate is less than 30 percent. There are ways to reengineer this entire process (see our book Rightsizing the New Enterprise). It is a must!
Not Fully Implemented Systems Management Tools
Systems management is critical to successful daily operations of the client/server environment. It is also an important resource for the planning of future activities. Systems management tools gather data that can be saved and used to determine trends. So, they are not only valuable to daily monitoring of system activities, but they provide other benefits.
On a daily basis, systems management can save the organization. Monitoring tools are as important today as they were with standalone mainframes. The difference is that with so many computer complexes running programs and passing data from platform to platform, systems management tools help isolate the problem. Remember that client/server means distributed programs and data. What better way to isolate a problem than with systems management tools?
In every company we studied, systems management tools are not being fully implemented. The gap between their potential use and their actual use ranged from 90 percent to 30 percent, meaning that the tools weren't being fully used. The systems management tools in the majority of these companies (80 percent) are not implemented by as much as 50 percent. This is a real sad statistic. One of the primary reasons the mainframe environment is so successful in maintaining RAS is because systems management tools are fully implemented, customized, and maintained.
Systems management tool vendors haven't changed much. They promise the world with each and every product, regardless of the platform. We all know it's hogwash that the vendors claim their products require very little effort for installation or maintenance. The reality is that these tools take a heck of a lot of effort to implement, customize and maintain. As we discuss throughout this book, it's the organization structure that causes this problem. The current structures do not allow the proper time for senior technicians to effectively do what they're good at doing, which is providing enterprise-wide, system management solutions. To be successful, you're going to have to devote time and resources to make these tools work.
Lack of Senior Technical Resources
The number of technologies in the computing environment today is unprecedented. You need senior technical staff to implement and maintain these technologies, and to know how they interact.
All of the 40 IT organizations we surveyed complained about not having adequate technical resources. For a quarter of these companies it was a legitimate beef. They were scarce on senior technical resources and finding talented help has become increasingly more difficult. Most of their efforts are being spent on external recruiting. This should continue, but there must be breeding within the organization as well. The current organization structures do not promote effective technical career development. This needs to be a priority.
In many instances, what happens after someone works his or her way up through the ranks, having been trained by the organization, is that he or she seeks free agency out in the marketplace where the rewards are very lucrative for technical personnel. This can never be completely prevented. Our answer is that it's better to have this individual on board for a few years than not to have them at all.
Retaining staff needs to be a focus as well. But as you can see with all the problems highlighted in Table 2-1, especially the daily firefighting routine with no end in sight, this is a monumental task. The sooner that you resolve the organizational issues with three levels of support and an effective problem management process, the better off your IT group will be. This would allow some time for your senior technical staff to perform some architecture and design functions, not to forget about playing with the latest and greatest technology in the marketplace.
Transitioning and Mentoring Legacy Staff
Due to the scarcity of technical resources, transitioning and mentoring existing legacy staff is paramount. You've got to have a developmental plan, if for no other reason than to entice your personnel to stay and not go down the block. This is often a difficult management decision. Does an IT manager spend money on training employees and run the risk of losing them to a competitor for a better employment package, or do we spend the money with the confidence that they will not take the training and run?
Employees will come and go, and if the working environment is challenging and the management is fair, the chances are very good that the employee is going to stay. We found that many IT organizations don't distribute the training and education evenly throughout the employment ranks. Some employees received the lion share of the training and others were neglected.
One of the most important ingredients and biggest challenges in transitioning to client/server distributed computing is motivating the mainframe professionals to adopt and adapt to this new networked environment. They need special guidance and special projects on which to work. They need to be involved in the planning and implementation of your heterogeneous infrastructure while maintaining RAS in the legacy environment.
There are several reasons why this is so important. Mainframe staff are trained and disciplined in supporting a controlled and managed environment. To successfully transition to a client/server environment, you want to include mainframe staff in your new paradigm. It's easier to teach mainframe staff a new technology than it is to teach discipline to client/server technical staff. Their disciplined mentoring is critical to building a world-class client/server environment.
Another key factor in favor of retraining and transitioning mainframe staff versus hiring replacement talent is that they know the business. Last, and probably the most critical, is to sustain morale. It takes everyone working together to make this transition successful. Providing the organization with at least the opportunity to learn is important. The ones with the initiative and drive will take advantage of these programs. The ones that won't will have nothing to complain about as long as they were provided the opportunity. It is imperative that you establish a curriculum to break the mindset barriers between the legacy and newer distributed environments. You may want to consider the following programs:
Vendor-sponsored events on technology trends in the marketplace. Technology is evolving at a torrid pace. Use your hardware or software vendor to share those trends with your organization on a monthly basis.
Like everyone else in the industry, you're probably inundated with dozens of invitations to vendor seminars every month. Neither you nor your staff has the time to attend these events. So, ask the vendors to visit you. We recommend that you use hardware and software vendors to provide monthly lunchtime seminars on the latest and greatest in technology. Make these sessions mandatory for your staff. Keep them relatively short, under 90 minutes, and tell the vendors that you want them to minimize the marketing hyperbole and concentrate on the applicability of their solutions to your environment. In other words, you want them to have to think about how their products can help you solve your problems, not just make their sales figures look good. You might ask the vendor if his or her company uses the products to run their business. If they do, suggest that they send a representative of their IT department to meet with you to talk about how to use the tools or products in a day-to-day operating environment. The vendor will jump at the opportunity to showcase its latest and greatest products. Management should also track in monthly status reports who attends each session.
Get acquainted with the hardware and OS. The mainframe has always received preferential mission-critical treatment, and for good reason. In the seventies and early eighties that type of thinking was warranted. Nowadays that association needs to carry forward throughout the enterprise. We recommend a program consisting of a half-day classroom session to understand the hardware (servers). The second half of the class should be hands-on in the data center (their own habitat) in which teams of three to five students (staff from different functional groups) take apart new servers and then put them back together again. This does wonders for the mindset and promotes the fact that a production system is a production system regardless of the box, not to mention promoting teamwork.
Professional technical training. Pick a minimum set of classes for a particular organization. For your production support personnel, pick a minimum of three courses. Make sure you include classes on systems administration and script programming. Provide this minimal set to all data center employees. If you exclude people, you will start having morale problems and that's something you don't want to experience. The point is to increase morale.
Script programming is included in the equation because you just don't send people to training without having them involved in real projects. Writing and editing of scripts are essential skills for Unix systems management. So that your senior technical staff members can learn to apply their years of experience maintaining and performance-tuning mainframes to Unix-based systems, you should also require several other essential courses for them, including C++ Programming, Advanced Systems Administration, Operating Systems Internals, etc.
Hands-on projects. Look at all the manual processes you have in place today. Put your Unix system administrators and mainframe personnel on projects together to create scripts to automate manual processes and eliminate as much of the bureaucracy as possible. This should be done on their personal time and will provide management with insight as to who has the initiative and drive to move forward.
A good example of a manual process that's still being used today in a mainframe data center, or any fairly large 24×7 environment, is called "shift turnover." This manual process usually is nothing more than a piece of paper on a clipboard explaining special requests for a night's production processing. This process covers anything out of the ordinary the operations staff must know. Once all the special requests are completed and signed off, this paper is put into a file cabinet. This is a good process, but manual and bureaucratic. Why not have your operations staff work on a project to automate this process? Call it shift so that from your desktop a script appears with the night's special processing requirements. When completed, the requirements get e-mailed to the appropriate parties and, once completed, the sequence of work is filed in an online database.
Train the trainer. Your lead systems administrator should be the first person to go through the training programs. Once he or she completes the program, the rest of your mainframe staff will follow suit. They have to be convinced that you are committed to the programs. To make sure the programs are the right ones for retraining your mainframers, who better but your best and most respected systems programmer to help you along the right track and lead the others down the path of client/server computing? It is essential that this person display initiative and drive.
Brown bag lunchtime seminars. This newly-trained individual can also hold weekly lunchtime sessions on different topics. Each topic should have homework assignments that are to be done on staff member's own time. Establish a special lab with servers, where students can complete assignments that can vary from reconfiguring hardware to modifying the operating system. It's important to have one of your own employees do this because it brings the group together. Do not hire a consultant to do this!
Invest in a lab for your staff. You don't want your IT operations staff toying with production servers or even those that are designated for applications development. You want to give your operations staff its own lab with its own servers to test the hardware and OS. The servers don't have to be the latest and greatest. In fact, talk with your vendors about purchasing or leasing refurbished gear. You'll save money and yet have ample hardware to support the versions of the OS you have in production.
Books. Provide staff with a list of books to read on their own time. Pick the top two or three books on networking, database administration, system administration, and so on. Maintain a current book list. One script writing project could be to make the reading list accessible on the Web.
Metrics. Track the students' performance. Include the students' progress through the training courses each month in your status report to senior management. If you've implemented a dashboard of metrics for your organization, include training as one of the gauges. So, if an employee has a performance problem—as determined through other Human Resources-sanctioned evaluations—you could easily see if the cause was insufficient training. It is more of a CYA than anything else. This employee could run to HR and complain of insufficient training programs while the organization is transitioning to support a more heterogeneous mission-critical production environment. As long as you track all assignments, programs completed, homework, books read, etc., there won't be anything to discuss with HR.
Job descriptions. Establish one generic job description for each function, which includes all technologies. (See Appendix A in our book titled Building the New Enterprise.)