CHAPTER 9
Documentation
Documentation of the systems is one of the least exciting subjects for any team. If a vendor provides it, the documents are just there to be reviewed as needed. But if it is for interfaces or customized code developed in-house, someone has to write the documentation. This is a task that is frequently pushed off to the end of the project and often never completely finished before time on the project runs out. The pervasive mentality that documentation is low-priority work drives this situation. But have you considered documentation as an effective risk mitigation tool?
Effective project management teaches us that we need to manage risk. Chapter 19, “Risk Management,” covers the subject as it relates to maintenance risk in greater detail. The following four processes relate to maintenance documentation risks.
• Risk Identification
• Risk Analysis
• Risk Response Planning
• Risk Monitoring and Control
So we will approach documentation assuming it is an effective risk mitigation tool. If that assumption is true, what risks are we trying to mitigate? The remainder of this section presents five maintenance risks along with the analysis and response/quality standard for each. This approach changes the focus of Risk Response Planning by including a “quality standard” for the documentation. Each risk is related to the others, but each is broken down for greater insight into the potential problem. Risk Monitoring and Control is addressed later in the chapter.
• Risk Identifications: Risk of losing project team knowledge before the maintenance team can acquire that knowledge.
• Risk Analysis: Portions of the project team may disband before the maintenance team is even set up. There may be nothing that can be done about this timing.
• Risk Response/Quality Standard: Have tasks included in the project team’s project plan for producing, reviewing, and approving all needed documents capturing needed knowledge.
• Risk Identifications: Risk of poor knowledge transfer from the project team to the maintenance team.
• Risk Analysis: The transfer of knowledge from the project team to your maintenance team may occur only “verbally,” but if nothing is written down, what do you have that can be used to transfer this knowledge the next time? Such a situation can arise when project team members transfer into the maintenance team. Their knowledge on the systems is complete, but they may see no reason to write anything down.
• Risk Response/Quality Standard: Set expectations of what is meant by effective documentation. The latter part of this chapter presents the recommended documents and what information they should contain.
• Risk Identifications: Risk of losing key maintenance team members.
• Risk Analysis: You never know when a team member may leave your company. On the positive side, you want to be able to support career advancement opportunities outside your group for your people and don’t want to be constrained by having the group’s success totally dependent on them.
• Risk Response/Quality Standard: You, the manager, need to be disciplined enough to not let the team’s success be totally dependent on any given team member. If such a situation exists, you need to have a plan in place to remove the dependency.
• Risk Identifications: Risk of not effectively training backup team members.
• Risk Analysis: I have seen this happen often: when a specific production problem occurs, everyone seeks the one guru on the subject. When that person is involved, the problem is rapidly fixed. But when he or she is unavailable, the other team member flounders with the problem and becomes stressed, frequently upsetting the customer. Sometimes gurus like being gurus and don’t readily share their knowledge. But it is vital to have depth of knowledge on your team.
• Risk Response/Quality Standard: Give the guru the job task of training other specific team members and providing them with the necessary written documents.
• Risk Identifications: Risk of not doing things consistently.
• Risk Analysis: Similar to the preceding risk, the guru and other team members may not approach a production problem or enhancement task in the same way. Without a documented process, there is little basis for starting to improve the existing process so that it will ensure a consistent approach that is used by all team members. But if the process is documented and followed by all the team members, you will be able to monitor the results and go back to the process to fine-tune it for continuous improvement.
• Risk Response/Quality Standard: Ensure that processes are documented and expectations are set for all team members to follow the processes. If there is resistance from team members who might say that the documented process is inaccurate or too cumbersome to follow, assign that person the task of rewriting the process.
It is essential that you, as the manager, set the standard for quality documentation. You can’t afford to have documents produced that are of low quality. The documentation standard needs to effectively address each of the risks that have been presented.
This chapter is placed in the Planning Processes part of this book so that we can focus your attention on obtaining or producing needed documents early in the forming of a maintenance team. We do not want to treat documentation as an afterthought, because doing that is precisely how important knowledge is lost.
Developing Documentation
So how do you produce effective documents? The best answer to this question is that you don’t produce them—someone else does. Someone other than the maintenance team should write the documents. The best people to write maintenance documents are the members of the project team. They possess the knowledge of what they programmed and configured, how it was compiled, and the limitations of the system.
The project team’s project plan should include tasks for production of the necessary documentation. The plan should have these tasks scheduled to be completed before the system is deployed. The documents can thus be used to help the project team initially support the system after deployment.
The maintenance team should audit the documents written by the project team, because the team will have the ability to evaluate the documents for their usability.
At this point, you may be laughing at what the last three paragraphs said. From your experience, those paragraphs may be fiction. The project teams you have encountered may have personified the philosophy that states: “Throw the remaining project loose ends over the wall to the maintenance team.” The preceding paragraphs deal with an ideal situation. It would be nice if it applies to your situation, but if it doesn’t and you are in a maintenance role while the project team is still developing the system, you have a window of time in which to negotiate documentation issues with the project manager. Such negotiations should be aimed at the project manager’s agreeing to ensure that documentation tasks are included in the project plan so that you will have the information you need.
But what if you are not in an ideal situation? Then it will be up to you to drive your team to fill in any document gaps. If the task is large enough, you may want to consider hiring a technical writer to assist you. The technical writer will not be able to replace your knowledgeable people, but a writer can relieve you of the wordsmithing tasks and deliver higher quality and more readable documentation.
If your team does the writing, you will have to set expectations and to plan time for your team members to do that work. It is challenging to schedule work that is less urgent when you must deal with high-priority production issues. Setting a reasonable schedule over many months to fill documentation gaps is the best approach. Communicate to your team your expectations of what is meant by effective documentation and what the documentation should include.
When you start having people document what they know, you very often will encounter some quiet resistance. Very few people actually like writing down what they know. The reasons can range from finding the task boring to not wanting to lose their expert status by having everyone know what they know. You need to make sure that team members who are responsible for documentation fully understand the necessity of providing clear information in their documents on how to support the system for future team members. Place requirements in an individual’s performance expectations to produce specific documents and to train less-experienced team members.
As with code development, documents should be tested for quality. This testing does not have to be a formal process. Testing can be performed by a less-experienced team member who walks through the steps listed in the document. If the instructions make sense to the tester, the document serves its purpose. If not, the tester can prepare specific questions that serve as an effective gap analysis that points out what needs to be improved.
Documentation Types
What constitutes appropriate system documents? At a high level, you want anything that will be useful. This chapter recommends the following five documents to help you effectively maintain the system:
• Operations Guide
• Maintenance Manual
• Design Document
• Business Process Document
• System Notebook
The first four documents have a specific focus. The fifth is a collection of many separate documents. The sections that follow present the specific content for these five documents.
If your company has different standards and terminology than those used in this book, or if you simply break down the documents into different categories, then use that approach in your documentation. You can still use the remainder of this chapter to verify that your own company’s approach is not missing any content.
Document Matrix
You need to have a clear idea of the types of documents that are needed for your team, whether it’s using the approach in this chapter or a different one. Having a clear idea of what is needed and why it is needed will make you more effective in reviewing a project team’s plan for producing documents.
Developing a Document Matrix is an effective way to track the documents you have, where they are, and what still needs to be developed. Figure 9-1 provides such a matrix. It shows an example of three applications, two interfaces, and one entry for reports. In a similar fashion, you should break down and enter the names of the applications your team supports in the most logical manner. You don’t need individual documents for each application, interface, or report. For example, one design document can cover two applications and all the interfaces as long as the content deals with all necessary items.
For a given document that is not developed yet, you can indicate the status in the matrix instead of the location and filename. Other types of documents can also be entered. When complete, this matrix becomes the central catalog for documentation that your team needs to perform its job.
The following sections contain outlines of the five documents that the maintenance team needs. You can combine these into fewer documents or further divide them so long as each subject is covered for each application. The important point is that each of the content items (such as shutdown and startup) identified under the five document types is addressed somewhere in your documentation, which should be readily available to the maintenance team.
Operations Guide
An IT Operations Guide is analogous to a car’s operating manual found in the car’s glove compartment. It tells the operator of the car how to do important things such as unlocking the door, starting the engine, and turning on the windshield wipers. But it does not tell the operator how to change the brakes.
The purpose of the Operations Guide is to address all the necessary operational procedures, requirements, and information needed to keep the system running soundly. The focus of this Guide is on the continued running of the application as it currently exists and on prevention or minimization of downtime caused by any kind of incident.
The Operations Guide should include the following sections:
1. Overview. Describe the application’s business functions and how the application operates.
2. References. Reference other documents created during the development and implementation of the application. Also incorporate references to any vendor manuals.
3. Assumptions. List any assumptions about the operating of the application.
4. External Agreements. Besides the agreement in the form of an SLA with your customer, your team may have agreements with other groups to provide some of the support functions. These agreements should be listed here and should include contact information and limitations of the service.
5. Production Diagram. Accurately depict the production environment. Use pictures and diagrams liberally. They can show all support systems, including interconnecting interfaces and technical architectures.
6. Directory Structure. This section will clearly define where production executable files reside for the applications.
7. Configuration Management. Define how the application pieces are maintained and fit together. Include application, database, and hardware configurations. Also reference any software licenses utilized for the application. All the versions of the components will be identified. This is extremely important when there are multiple versions of the application running in production for different customers.
8. Shutdown and Startup. Define the procedures used to properly shut down and restart the applications and databases.
9. Disaster Recovery. This section should provide the plan to reduce the impact of a major application problem.
10. Backup and Recovery. To address disk crashes and other unexpected data losses, clearly define the procedure for backing up the database and any data files. Include the frequency for full and partial backups. Also define the steps to recover the system using the backup media.
11. Data Archiving. The business rules specify how long data will be online in the application. These rules should also define any archiving needs for the data, including when the data should be removed from the production system and saved to conserve storage space for the online production system. This section will define the procedures to meet these business needs, including archiving the data and databases to backup media, purging data from the online application that is no longer needed, and restoring archived media when a business request is made.
12. Performance Tuning. This section will address procedures for monitoring and tuning the application as more data is entered and the use increases. Define the storage capacity and transaction volume of the application.
13. Security. This section will address security administration procedures for adding new users, changing rights, and auditing.
14. Metrics Monitoring. Set up the procedures to monitor and record the metrics defined in the SLA and any lower-level metrics not defined in the SLA but needed to provide an early warning if something is trending off track.
15. Help Desk. This section will address how users contact your team. The methods can be through using an independent help desk, a group pager that is rotated throughout your team, or a team member’s direct contact information. Whatever method is used, the steps that the maintenance team member must follow should also be documented in this section, including any help desk scripts required to obtain the necessary information for the customer.
16. Error Handling. List any known operation errors and the appropriate problem resolution, including restarts of batch jobs. This list could also be a separate, cumulative list in the System Notebook.
17. Training. Define the training of both users and maintenance team members. List the actual training materials used for training end users.
18. Appendix: Contacts. List the necessary business, maintenance team, and other IT team contacts for addressing operational issues and appropriate customer/user contacts that need to be informed of status. This section may not be necessary in this document if already defined elsewhere. This is listed in the appendix so it can be updated more readily.
Maintenance Manual
Continuing with the analogy of a car, the Maintenance Manual is not the manual you find in your car’s glove compartment that tells you how to use the windshield wipers. The Maintenance Manual is the manual you keep at home to tell you how to open the hood and work on the engine.
The purpose of the Maintenance Manual is to provide information to support enhancements and fixes of the application or interface. This document should address everything needed to find source code, compile the programs, and configure and deploy the application. The manual should include factors that were addressed when the application was first built.
The Maintenance Manual should include the following sections:
1. Overview. Describe how the application’s components interact with one another to deliver the required business function. It may be beneficial to specify what functions are not included.
2. References. Reference the other documents created during the development and implementation of the application. Some of these other documents might address the specific requirements of this document but may have to be updated to reflect implemented enhancements. There is no need to rewrite them; just reference them. Also reference any vendor manuals.
3. Assumptions. Start with the assumptions from the design phase and record new assumptions that answer questions as they arise.
4. Criteria and Standards. Reference any departmental software development standards used for maintaining the system. If no standards exist, list any other standards or criteria used by the project team that should also be used by the maintenance team.
5. System Diagram. The system diagram will depict the interconnections within the application and the interfaces to other applications. This depiction can include a high-level view and an appropriate lower-level view.
6. Directory Structure. This section will clearly define where the source code and other files reside.
7. Developer and Test Environment. Define how a team member can access the development environment; include the security rights the person needs. The test environment or any other environments should be defined in a similar way.
8. Instructions. List the necessary instructions specific to this application, such as compiling the application or migrating between environments.
9. Tools. List the tools, such as compilers and debuggers, used to build and maintain the application.
10. Migration Flowchart. Show the complete migration process, from starting to make a code change all the way through to deploying it into production. This flowchart should refer to the different environments used. This section can include any installation scripts that are used.
11. Testing Plans, Test Cases, and Test Scripts. The easiest way to deal with this section is to reference the original test plan along with any modifications to it. If doing this is not practical (maybe because such a plan never existed), then define the necessary steps to completely test and sign off on any changes.
12. Communication. List the business contacts who need to be informed of any new software deployment or defect fixes.
13. Source Code. List the actual location of all the source code versions.
Design Document
The Design Document details how the system was to be built. It can provide valuable information for someone trying to figure out the inner workings of the system code for the first time.
No document outline is included in this section because whatever the project team used as a Design Document is what you are going to receive. It does not make any sense to rewrite this document. However, even though the maintenance team doesn’t write the Design Document, the maintenance team should update the document as enhancements or other changes are made. If diagrams and flowcharts to make the design easier to comprehend are not already included in the Design Document, you should add them.
Business Process Document
Another important document to have, but less important than the others for the IT maintenance team, is a Business Process Document. This document describes the process that the business follows to do its work, including the steps that use the computer system that you maintain.
Your customer will probably have an easier time conducting its business if this document exists. A carefully mapped out business flow is an excellent tool for training the businesspeople on how to perform their jobs. Because of the nature of the information, members of the business will be the ones most capable of producing the Business Process Document. The IT team frequently plays a role in development of this document, because of team members’ skills in creating flowcharts and process diagrams.
The maintenance team will have an increased challenge if no Business Process Document exists for the system. You may receive trouble calls about the system that are due to the users not fully understanding their jobs and what the system is designed to do. If the document does exist, it will help your team understand how their efforts help meet business needs.
System Notebook
The System Notebook is the last document we will present in this chapter. Where the Operations Guide and the Maintenance Manual are formal documents, the System Notebook is more of a collection of notes and a catchall for important, short documents. Anything worth writing down about the system should be included or at least referenced in the System Notebook.
The System Notebook does not have to be a physical book; it can be in an electronic folder or on a web site so that every team member will have access to it. For new team members, the System Notebook should be the first place they look for information.
The information contained in the System Notebook will augment the other documents covered in this chapter by providing additional information that is necessary for carrying out maintenance tasks. The following outline presents suggested content for any System Notebook. There are no firm recommendations of what should be included for the System Notebook; the System Notebook is a free-form, evolving, and growing document when used properly.
The System Notebook could include the following sections:
1. References. List all documents that are not contained in the System Notebook but that are important to the effective maintenance of the system. This will serve as the one, all-encompassing master list of all system documents.
2. Team contact list with home phone numbers.
3. Other IT contact information.
4. Business system owners’ and stakeholders’ contact information.
5. Copy of the SLA.
6. Policies and procedures of the maintenance team.
7. System performance data from when the system was first implemented, from the present, and from periods in between implementation and the present. This data will provide good performance benchmarking information.
8. Any common issues with the system, such as batch processes that commonly fail, and the resolution to the problems.
9. System Administration Guide providing processes for activities such as adding and removing users.
10. Licenses for software.
11. Warranties on hardware.
Maintenance of the Documentation
Do not think that ongoing work on documentation isn’t needed after the documents are written. The documents themselves must be maintained. The system documents must be updated to reflect the system changes due to enhancements and defect fixes. If these documents are not updated, the maintenance team will encounter erroneous information when it uses the documents and will come to distrust the entire documentation. The documentation would thus become useless.
Make sure your team is disciplined in its updating of documents when system changes are made or when errors in the documentation are noticed. Documents should be updated at the same time that a system change is made. An annual review of the documentation should be conducted for the purpose of noting any documentation deficiencies and taking corrective action.
Just because your maintenance team is running well now doesn’t mean that trouble will not appear. At any time, your team can resolve current problems based on its determination, knowledge, and knowledgeable contacts. These are the easy situations that you may never even hear about. But for the hard cases—and particularly those hard cases that occur after knowledgeable team members leave the team and take their knowledge with them—if there isn’t complete and reliable documentation available to your team, you will soon hear of many problems. Capturing knowledge in your documentation is thus essential to running a healthy maintenance team.