The Organization
Production Control
The first function to which we need to pay homage in the legacy environment was referred to as production control, which provided us with:
Second-level production support. These were—and still are—the primary areas where problems were introduced to MIS after the problems were detected or raised to the first level. The first level was the help desk and/or computer operations. Second-level production support were junior technical staff (i.e., operations analysts in the mainframe world), one level above computer operators. Eighty percent of problem determination was handled by these people first.
The goal was to resolve at least 80 percent of the problems before they reached those computing gods of the seventies and eighties, the very elite mainframe systems programmer (third-level support). We were developing their skill set to become system programmers (third-level support).
Heaven forbid that problems be turned over to them before every possible attempt was made to resolve the problem. They had chips on their shoulders and an attitude bigger than their mainframe. No one wanted to disturb them from designing, building, and maintaining their infrastructure.
Assistance to senior technical staff in the analysis, implementation, and customization of enterprise-wide system management tools. It takes at least one for each type of discipline (tape backup, security, etc.). We would pair a very senior person and one junior person to work together. Once trained, junior administrators can take ownership and provide maintenance on the tool. We can learn a couple of lessons from this arrangement. One, the pairing of senior and junior staff provides a mentoring process so that less experienced staff can learn on the job, which builds overall competency and individual skill sets. Second, systems-management tools aren't that easy to implement and maintain, as we alluded to earlier in the book, despite what the vendors may claim. Remember, our survey showed that most companies fail to use the full potential of the tools, so take advantage of their capabilities. It helps to devote resources to their use.
Assistance to senior technical staff in providing junior system administration functions (i.e., O/S support, hardware setup, on-call support, etc.). Senior technical staff should not get bogged down with day-to-day maintenance functions.
Production quality assurance function. No system was put into production before its time.
Process ownership (e.g., change control, problem management, etc.) There can be only one owner.
Systems Programming
The second area we need to eternally bless is the system programmer function. RAS evolved mostly from this function. Without this elite organization there would be no RAS. These pros provided the high-level planning/architecture function. For decades there wasn't a separate architecture function. The Systems Programmers were also the ones to resolve unique problems which no one else in the organization could figure out.
In today's environment most Fortune 500 companies have a separate architecture function. In the 1970s and 1980s this function was imbedded within the technical services or technical support staff. These people were the visionaries and planners in building that Fort Knox of computing we refer to as the data center. The big difference between the systems programmer of the past and today's architect is that one listens to the users and shmoozes with senior management while the other could not care less. Those ancient yet highly respected system programmers built things the way they saw fit. They didn't do the politically correct things, they just did the right things and did they do a great job of it. OK, they may have aggravated the users. It was not their forte to market and sell IT services, or do/say the politically correct things. But, boy, could they build an infrastructure! Going forward into the new century, the objective should be to combine the background skills of the systems administrators with the business acumen of today's applications development specialists. We want to get the best of both cultures, so to speak, so that new applications will be developed with a wholesome attitude and concern for RAS as well as solving business problems for IT's customers.
Given enough time, many people can build an infrastructure. There's no reason in the world why system administrators today couldn't do the same. But they aren't given the time or opportunity. Unfortunately, this isn't a priority. Priorities have changed. As we mentioned earlier, building the proper infrastructure is secondary to new development initiatives. And, to make matters worse, CIOs don't have a clear understanding of what's going on in the trenches of their own infrastructure. The people and process issues are out of control right in their own back yard and they don't even know it. But it's not their fault. They're too busy working and understanding the business issues, listening and politicking with the users, and shmoozing big time with corporate executives. The outside consultants, for the most part, aren't concerned with the infrastructure either. The big consulting firms' experts focus on applications and business issues, not infrastructure processes. How many big five consultants do you know who came out of MIS or IT organizations where they were responsible for RAS? Not many, we're willing to wager.
Database Administration
The DBA group is responsible for supporting the database functions of applications. They take ownership of database servers and software.
Database administration has typically carried two meanings. One is the design, definition, and support of the logical database. The second is the design, definition, and support of the physical database. In the age of monolithic mainframe computing solutions, a single person or staff filled these roles. This was facilitated by the database design being developed in a hierarchical method. Therefore, relationships between objects were defined in a parent-child type of structure. This allowed for a centralized computing environment with centralized staff. As the technology moves from glass-house computer room environments to distributed server-room installations, the purpose of database administration has also changed. Databases are now predominately in a relational database model. This requires design/definition issues to be an integral part of application development.
This is expected to continue with object oriented technology as well. As such, the logical database administration tasks fall within a development group and the physical database administration tasks belong to an operations/support organization. The following attempts to outline roles/responsibilities/considerations of physical database administration in distributed environments.
Database environments should be distributed to allow information, which is critical to the success of a given organization, to be located as close to the user group as physically possible. This allows the production environment to be more reliable due to the removal of multiple points of failure. For example, if the user community which is primarily responsible for the data is located in Boston, and the server is in Colorado, the users have to traverse a myriad of network connections in order to play their transactions. This means that if a network connection goes out in Chicago, the transaction will either be queued or re-routed through a less expedient path. In the event of a lack of redundancy for every network connection, the user will most probably get a message of failed transactions. Then the transactions will have to be replayed later. For environments that involve a high number of user transactions, a remote location could involve network saturation and a high rate of collisions. This would also affect the response time to the user and that could be detrimental to business success. These are a few of the reasons why production environments should be distributed. However, a distributed environment does not eliminate chaos.
Every distributed server needs to be configured with a minimum set of requirements as well. It is important that production environments adhere to some configuration standards in order to leverage the highest DBA-to-server ratios. In the event of database outages or errors, you want a DBA to quickly access the machines, navigate through the file system structure and obtain information necessary to quickly take corrective action.
If the directory structure for the production machines is not established according to guidelines, there will be a further delay in problem diagnosis and resolution.
The role of a database administrator includes responsibilities in the areas of performance and tuning. Most performance gains can be found in the process of application and logical database design. However, there are tuning procedures that can be done by the database administrator to optimize performance across the physical resources of the production server. For example, allocating data space and index space across separate devices and device controllers can prevent bottlenecks. In order to accomplish this tuning function, the DBA needs access to programs and routines that can collect performance statistics on an as-needed basis. There are programs available that can track logical and physical writes and reads on an object-by-object basis for the popular database environments such as Oracle, Sybase and DB2.
Remember back when the DBA function was centralized under one organization, providing IT related services throughout the entire corporation? Just like any other IT function (networking, operations, etc.), providing service was the primary responsibility. It really didn't matter to which group they provided those services. Their charter was and still is to design, implement, and maintain the corporate databases. It's an atrocity to see what's happened to this function over the past decade. The political atmosphere of being decentralized has caused more bickering than the Democrats and Republicans.
DBAs have been restructured into the business unit, in Applications Development, in Operations, and in some companies they are spread out everywhere. Tables 6-1, 6-2, and 6-3 list the positive and negative aspects of the DBA function being decentralized or centralized.:
| Positive | Negative |
|---|---|
| Priority and focus for the business unit. | Lack of common standards. |
| Faster turnaround for Database implementations. | Lack of enterprise-wide Database Administration solutions. |
| A better understanding of the business. | Usually very thin on resources. |
| RAS compromised—definitely not a priority. | |
| Wasted corporate costs because each BU is doing their own thing—Integration, tools, etc. are a big issue. |
| Positive | Negative |
|---|---|
| Focus is on design and development of the database. | Performance and tuning issues are not a top priority. |
| Following standards is not a priority. | |
| Poor communication with DBAs providing production support. |
| Positive | Negative |
|---|---|
| Adherence to standards. | Business units will not get the same priority as if they had their own DBA. |
| Poolling of resources. | May not have the same understanding for that particular business. |
| Enhancement of career/skills development. | |
| Design, plan, and architect the proper databases. | |
| Cost effective. | |
| Performance and tuning issues addressed up-front. |
We reiterate that this function has to be one of the most politically sensitive in IT. We're well aware of the popular clamor in the industry calling for every development group to have its own DBA or two. From our analysis, seeing operations across the globe, we know that this thinking may be popular but the results are not good; in fact, this whole idea is ludicrous. Just because the perception is that a particular business unit or division is not receiving preferential treatment doesn't mean that IT executives should give up the goal of RAS. Give us a break! The DBA's function is to serve! And let's not forget, those DBAs of yesteryear provided some awesome high-RAS databases (IDMS, IMS, etc.) that served multiple divisions or departments in their organizations.
If centralizing DBAs is not the right answer for your company, or the political atmosphere will never allow this, then what we recommend is for the DBAs to reside in business units but report directly to a centralized DBA group within Operations. The other scenario (not our top choice) is to have the same DBA who reports to the business unit also report to a centralized DBA group on a dotted-line basis.
Three Levels of Support
Whoever came up with three levels of support within the organization was a genius. This is one of the best structures ever designed and probably the single most important reason the mainframe world was so successful. First the roles of this structure:
| Level 1: | Monitor the systems (servers, network, peripheral devices), first-level problem determination, and resolution attempt. After an N number of minutes, as determined by the problem management process, the problem will be escalated to second-level support. |
| Level 2: | Problem determination and attempted resolution. After N minutes as determined by the problem management process, the problem will be escalated to third level. A side note: There was a fear of escalating to the third level. The group's goal was to do everything possible to resolve the problem here before escalating to the senior gurus of the department. Senior System Administrators are worth their weight in gold. The entire organization needs to protect this valuable resource. |
| Level 3: | The buck stops here. If they couldn't fix the problem, then no one could. |
The benefits from this structure were enormous:
First and foremost, this structure allowed senior technical staff the opportunity to architect and design a reliable, available, and serviceable infrastructure. The goal should be for first and second-level support staff to handle 80 percent of the problems before escalation.
Skills for junior and second-level support personnel are enhanced. Organizations today need to breed senior technical staff within the organization as quickly as possible, and continue with their external recruitment efforts.
Better turnaround for problem resolution.
The ability to fully provide analysis and implementation of enterprise systems management solutions.
Computer Operations
Computer operators did so much more 20 years ago than just monitor the computing environment. They:
Assisted in every facet of infrastructure development and support.
System management implementations
Disaster recovery
Data center hardware installs
Print management
Assisted second- and third-level support with facilities management.
Provided a higher level of problem determination and resolution.
Provided a solid tape librarian function. This function played a key role in disaster recovery testing.
What's happened to us? Why are we in IT abandoning all these resources? These were individuals who were mentored to be part of all RAS efforts. They were given responsibilities that would enhance their career development. We used everyone to his or her utmost capabilities!