Current Organizational Structures
This chapter shows the way many organizations are structured today. You wouldn't believe what we saw out there. Maybe you've seen the same setups, too. The first one (Figure 4-1) was our personal favorite and receives the first DORG (Disaster ORGanization) award. It's a designation that we give to organizations that have intrinsic problems just by the way they are created and the limitations they place on the people in the organization.
Figure 4-1. Infrastructure split in half.
The Infrastructure Split in Half.
In a few of the 40 companies we analyzed, we found the infrastructure was split between infrastructure development (implementing processes and technology) and infrastructure support (production support—fulltime problem resolution). You can tell this idea came from someone in management whose career didn't evolve from the bowels of the data center or, for that matter, from someone who probably never headed up a large infrastructure group.If you've been in the industry and had the opportunity to know at least a handful of IT shops, you know that an increasing number of IT executives are appointed to their posts because top management thinks they have done well in other parts of the company. The net effect is that while IT requirements are growing dramatically, some IT executives lack the experience to really know how an IT organization works.
The structure had merit. They knew they had to focus on implementing processes and tools for their new client/server environment but they forgot about the biggest and most important part of the equation, the people aspect. If you were a technical person, would you like your role to be labeled production support or infrastructure development? We'll answer that one for you. All technical personnel would rather play with the latest widget and gadget than provide around-the-clock support. Playing with new technology is always more fun, so infrastructure development has more elan.
Splitting the infrastructure in half is by far the worst structure we've seen! It causes the following issues:
Security privileges are shared. Security privileges should never be shared.
Put yourself into the shoes of a senior technician. Why would anyone want to provide production support rather than test and implement the latest and greatest technology?
Responsibilities overlap when splitting the organization.
Difficult to turn over projects to support, especially since these projects were initiated and implemented in other organizations.
Problem management becomes difficult—people will go to the expert for faster resolution.
Duplication of system management efforts.
Communication issues are worse than ever before.
Cross-training constraints.
Resource constraints.
There should be a happy medium. The answer is simple: Have just one infrastructure organization which is structured to provide three levels of technical support. This should be done for the Network, Desktop, and Data Center or Production Server room. Table 4-2 identifies three levels of support and the percentage of time required for each of the major functions.
We recommend that you have one infrastructure group.
| Level of Support | Job Function | Percent |
|---|---|---|
| First | Monitoring/Problem resolution | 80/20 |
| Second | Problem resolution/Architecture development and technology analysis. | 75/25 |
| Third | Architecture development and technology analysis/ Problem resolution. | 80/20 |
Focusing on Technology
In over half of the companies we visited, we found that IT organizations were divided to focus on technology, usually on the basis of operating systems. This is one of the most frequently used structures in the corporate world today. Another big-time DORG award. This structure is designed solely to focus on particular technologies. We usually see this structure in mainframe shops (see Figure 4-2).
The original intent was good. You need to start somewhere and heaven forbid if you corrupt your legacy environment with this crazy networked world. The big problem is after designing and implementing this structure, and after management says all the right words by telling the staff that one day the walls will come down, it rarely happens that way. The issues with structuring your organization in this manner are:
Limitations on existing resources.
Morale problems.
Territorial boundaries.
Cross-training constraints.
Duplication of processes/tools.
Extremely poor communication amongst the different groups.
Lack of enterprise-wide systems management solutions.
Limited skills development.
Figure 4-2. Organization structure focused on particular technologies
We recommend these walls come down as quickly as possible. Start mentoring that a production system is a production system regardless of the box. This should be IT's mantra for the new millennium: A production system is a production system regardless of the box, application, or operating system.
The Architecture Function
The architecture function also deserves a DORG. At least 70 percent of the companies we've visited have this problem. Tell us, why didn't we need an architecture function back in the 1960s, 1970s, and early 1980s? Many argue that the infrastructure is so much more complex today than it was in the past. To some extent we will agree. Client/server computing, as we said earlier, is like no other beast. But when we were mainframe computer operators or operations analysts, climbing the corporate ladder to become mainframe systems programmers, we viewed that world as a complex environment and it was. Assembler, TSO, VTAM, NCP, etc., created a complex information technology environment.
When we first started learning client/server, Unix, script programming, etc., it was difficult as well. Writing Clists was much more difficult than writing a shell script in Unix. It is all relative.
The architects in the mainframe era were systems programmers. The Systems Programmers of yesteryear built a darn good infrastructure because they were given the time to do so. There were three levels of support in the mainframe infrastructure:
Level 1 — Operations and the Help Desk
Level 2 — Production control (operations analyst)
Level 3 — System programmer and database administrator
Rarely did systems programmers deal with daily problem resolution. In those days junior technicians (level one and two support personnel) were scared to death to disturb the mainframe gods. Everyone knew they were busy designing a temple. They were paid the big bucks to build an infrastructure that provided RAS, not to chase every problem that came into MIS. Think about it for a moment. It worked!
We have yet to see this function work effectively anywhere in the world. Why didn't the architecture function work when the concept sounded good? The function is supposed to:
Keep abreast of technologies.
Focus one set of eyes and ears to business requirements.
Work with the infrastructure group to understand its needs.
Provide guidance in establishing an architecture/roadmap.
We have yet to see it work as advertised. Do away with this function. Let your senior technical staff architect your infrastructure.
Database Administration
No DORG awards here. The database administration function is the most politically sensitive function for each of the large corporations we visited. We've seen the database administration group in three different areas of the organization. The first is within applications development as depicted in Figure 4-3. The second is within the infrastructure group as depicted in Figure 4-4. The third area is a mixture of both, as shown in Figure 4-5.
Figure 4-3. The database administration function within applications development.
Figure 4-4. The database administration function within the infrastructure group.
Figure 4-5. The database administration function within both applications development and the infrastructure group.
It's a tough call, but look back to our roots once again. In the mainframe world we centralized this function under the infrastructure group. The intention was to:
Pool resources.
Enhance career/skills development.
Design and develop the architecture.
Address performance and tuning issues.
So, help us out. What's wrong with that? One might argue that this group's focus should be on designing the proper database. If it's structured under the infrastructure group the perception would be that the database wouldn't be designed properly. We don't think that's realistic. Isn't the DBA function a service function just like any other function within IT? In the mainframe world the database group was centralized under the infrastructure group and things worked out just fine. As a matter of fact, our IDMS database in the eighties was extremely reliable. It couldn't have been designed any better, and that's because they wanted it perfect from the beginning so there would be fewer heartaches to deal with after it was in production.
We emphasize the recommendation that database administration should be centralized under the infrastructure group.
The Project Office
We just love giving out all these DORG awards. Once again the concept/theory gets some bonus points, but does it work? The project office was established to implement key processes like change control or applications development methodologies. When management conceived this organization their hearts were in the right place (see Figure 4-6). At least they knew that they needed to provide process design and implementation. Unfortunately, processes should only be designed and implemented by the folks in the trenches, not by people who are far removed from the front lines.
Back in the 1970s and 1980s we didn't need a special group to develop some of the most effective processes ever designed. What we had was a group we called production control which had centralized ownership of many of these critical processes. The processes themselves were developed by the Help Desk, Systems Programming, Database Administration, etc., and included:
Figure 4-6. Structure designed to focus on process design.
There's so much more, but we think you're getting the point.
Some of these groups are even responsible for disaster recovery. What's this world coming to? Once again, it's those senior technicians who should be in charge of disaster recovery. They'll figure out how to make this stuff work. Structure the organization properly and give them the bandwidth. Please don't take it out of their hands.
You're wasting company resources! Once again we didn't need it back in the mainframe days and don't tell us the environment was different than it is today. Sure it was, but the best processes (i.e., change control, problem management, etc.) were created in those days. Disaster Recovery was second to none. As we've circled the globe several times and visited with hundreds of Fortune 1000 companies, we have yet to see an effective disaster recovery solution for client/server computing, and not because the technology is so much more complex. It's been taken out of the hands of the senior gurus, those senior system administrators who know how to do this stuff, and put into the hands of people without the experience of knowing the importance of this responsibility.
Global Coordination
We're on a roll giving out DORG awards left and right. Good communication and coordination practices are essential in today's fast paced global environment. Communication is worse than ever in IT due mainly to this networked age of client/server computing and how things are distributed all over the world, becoming chaotic and uncontrollable. Once again the concept/idea of someone in charge of global coordination sounds good, but in reality you're introducing more bureaucracy and complexity into the organization.
We've seen the global coordinator or head of international support report to the CIO as well as the infrastructure group (see Figure 4-7). If you're bent on having such a position then we recommend having this person reporting into the infrastructure group just so their head doesn't swell. But it's also more important to coordinate infrastructure issues (i.e. connectivity, software distribution, hardware installations, production support, etc.) vs. Applications Development issues.
Figure 4-7. Ineffective global coordination.
We recommend that you not have a separate function to handle global coordination. It should be the responsibility of the infrastructure group to deal with their regional management who should be reporting to him/her.
Positioning the Help Desk
It's so critical to structure the help desk properly. It's not only the first point of contact for your customers, but the staff should be the owners of problem management which is one of the top three disciplines in IT. The other two are change control and production acceptance. The help desk needs to be structured at the enterprise level, nothing less and nothing more. This is easier said than done. One of the more commonly placed locations is under the desktop group as depicted in Figure 4-8.
We recommend that the help desk be structured at the enterprise level as depicted in Figure 4-9. The help desk needs to have authority and be able to support all groups equally.
Figure 4-8. Common location of the help desk.
Figure 4-9. The help desk at the enterprise level
Desktop Support
We've seen so many variations of where and how desktop support should be structured. We have to give it a DORG award because people are working very hard trying to make things more confusing than they should be. The diagram in Figure 4-10 seems logical, especially to management, but for the troops in the trenches it's a nightmare. It's extremely difficult to separate hardware, software, and architecture. The help desk has a difficult time figuring out whom to call within the desktop support group because the functions overlap and confusion usually reigns.
Figure 4-10. Undesirable desktop support structure.
Do not separate hardware and software support.
The Tape Librarian
The management of tape libraries will continue to be a critical function in the 21st century. Most IT production support groups today that support a client/server environment leave out the tape librarian function as a cost-reduction exercise. Put it back in! Never compromise data integrity! There must be one person dedicated to this function; it cannot be a part-time exercise performed by the operations staff.
Figure 4-11. The tape librarian as part of the operations structure.
Most client/server operations supporting organizations today have done away with a person dedicated to ensuring data integrity. The Tape Librarian function was, and still is, essential (see Figure 4-11).
As we travel the globe doing our talks and assessments we often ask the question, can you guarantee that you can restore data from your backup tapes? The answer is always, "We think so." If you were to ask that same question to a mainframe organization—well, to tell you the truth, you would never ask them because it would be embarrassing to do so. Mainframe management would never have this type of worry.
Global Technologies
It's like having your own applications development staff for the infrastructure group (Figure 4-12). The only difference is that this group is a small—usually two or three people. Their main purpose in life is to focus on major initiatives (e.g., rolling out Microsoft Exchange around the world). The staff (permanent employees) should remain small. If for any reason the project is much larger in scope than this group can handle, consultants should be hired. In some shops we recommend this, and in others we do not. That's because sometimes the group has a tendency to grow too large. The individuals for this group are very senior technical staff, with a background in systems administration, applications development (Java, C+, etc.), and project planning.
Figure 4-12. The global technologies group with the infrastructure.
Business Requirements First
Once again the intent is good, but we need to give it a huge DORG award. Not to be mean because it's an understandable mistake, but because it's a very costly mistake for the company. This scenario usually occurs when the head of IT comes from the business unit to head up IT. The typical structure is shown in Figure 4-13.
As an example, this structure would be supporting three distinct business units: Manufacturing, HR, and Finance. Each business unit would have a set of applications developers, usually one database administrator, and one system administrator. These are pretty much designed to be autonomous IT organizations. The intent is for the entire organization to work together, but in reality the priority and alliance is to the heads of Manufacturing, HR, or Finance. The operations box is the traditional centralized IT infrastructure support group.
The problems this structure causes are enormous:
Systems management tools are not fully implemented; there is a lack of resources to properly implement tools.
Senior system administrators spend 90 percent of their time on fire-fighting and daily problem resolution. They're left with only 10 percent of their time to plan, architect, and design the infrastructure that includes implementing system management solutions but for their own environment, not an enterprise solution. The goal should be 80 percent to plan, architect, and design enterprise solutions for the entire infrastructure, and 20 percent for problem resolution.
Figure 4-13. "Business requirements first" structure.
Three levels of support not specified for system administration. They have resources for only one level of support.
Lack of a centralized lape librarian function; each group handles its own tapes. There are very few (if any) procedures to assure integrity. This is extremely risky!
Lack of a centralized computer operations function; senior system administrators are spending their time mounting tapes, monitoring systems/peripherals, etc. It is ludicrous for senior technicians to be wasting their time on such mundane tasks when they should be providing analysis on the latest and greatest technologies.
Lack of a formalized applications support team (that role is partly filled by developers, DBAs, other individuals). Very little is formalized in the way of production support.
Lack of a production control function. Once again, due to a lack of resources, this function does not exist, which means production QA, process ownership, second-level support structure is nonexistent.
The help desk is not structured properly within the organization. It should be structured at the enterprise level and given the authority to own problem management.
The organization is structured within silos focusing on particular technologies.
Career development for Unix and database technical support staff is almost impossible in the current structure.
Lack of processes.
There is a general lack of discipline throughout the infrastructure.
Finger pointing between the centralized IT operations staff and the autonomous groups.
Teamwork and communication is less than adequate.
Our perception at first was that customer satisfaction was extremely high—more so than in the traditional IT organization structure. But that was not the case. In all the companies we visited, customer satisfaction was the same whether it was the traditional IT organization structure or this one.
The real proof is in the numbers, the industry average for System Administrators to mission-critical production servers is about 9 to 1 for good environments. The better environments are about 12 to 1. These autonomous IT organizations are averaging about 3 to 1—in some cases we saw 2 to 1. What would the CEO think? How much money is being wasted?