The goals and structure of this appendix are slightly different from the rest of this book. Whereas Chapters 1 through 6 convey information to the CISA candidate, here, the focus shifts to the professional world of the information systems (IS) auditor. It addresses the nature of different professional engagements common to information systems auditors. I review the stages of and responsibilities involved in performing a risk-based information systems audit for both internal and external auditors. This appendix also serves to introduce and frame examples of professional situations that may challenge an auditor.

This appendix reviews the process of performing an information systems audit, and in doing so, identifies how sections of the study materials in this book can be applied in the “real world.” By bringing the subject of conducting an IS audit “up a level,” I provide associations between concepts found in the main chapters in this book so you have real-life examples of a number of these concepts. This should help solidify material you learned from the rest of the book, and hopefully assist in recalling information while sitting for the test.

In addition, this appendix can be employed as a guide (or adapted to create a checklist) for the reader who is executing or participating in an information systems audit. The material here is based on methods used in professional environments that have succeeded in achieving high client satisfaction ratings and delivering quality audits.

To employ an automotive metaphor, the study material in the main chapters of this book may teach the reader about how a vehicle functions and relates to the road, while this appendix teaches the reader about driving.

Understanding the Audit Cycle

The information systems audit cycle is central to the profession of an IS auditor. The cycle itself could be executed by a single auditor, or the responsibilities could be distributed to individuals making up the audit team. In some professional situations, one or more sections discussed here may be unnecessary. The IS audit cycle described here is not the only cycle an IS auditor may perform, as the needs of different situations may require alternate procedures or approaches.

Candidates for the CISA exam will have had some experience with information systems auditing, but not all candidates will have had visibility into the whole end-to-end business process. Here, the stages of the cycle are illustrated as they would be considered by someone managing a professional audit.

For persons early in their professional career, this appendix unveils some of the stages that their supervisors may perform. Understanding these phases will help individuals early in their career see the big picture, deliver meaningful work, and hopefully hasten their advancement.

How the Information Systems Audit Cycle Is Discussed

Some components of an information systems audit cycle are uniform, regardless of the size of the client and the scope of the audit. Each stage discussed is a valid consideration during the course of performing a moderately complex audit project. This appendix provides a relevant audit skeleton, regardless of whether the auditor serves as an internal auditor within an organization or is brought in from outside the organization being audited. This is relevant for working on a variety of audit services, including SOC1/SOC2, SOX, OMB Circular A-123 auditing, financial audits, internal audits, report writing, compliance audits, and other services.

Although the focus is on executing an audit involving controls testing, many project stages will apply when performing other projects where an information systems auditor’s skills are required. It is not meant to be a complete reference when a project’s needs go beyond the scope of this appendix. Additional procedures will be required to deliver services supporting other functions, such as enterprise-wide risk assessments, project life cycle evaluations, and disaster recovery planning.

For the sake of “telling the story,” terms are introduced from outside of the CISA exam terminology.

“Client” and Other Terms in This Appendix

To ensure that the examples in this appendix are clear to the reader, this section explains how an experienced auditor’s vernacular employs the versatile term “client” contextually. In this appendix, the terms “client” and “client organization” refer to the business entity or departments within a project’s scope.

• Client   The organization, department(s), and individual persons being audited.

• Client organization   The broader legal entity being audited. In some cases, this can be defined as subdepartments within a larger organization.

To say “in front of the client” can refer to being outside the building, in a meeting, or sitting with a control owner.

More specific terms are employed for parties encountered within client organizations. This will assist in this discussion of the information systems auditing process. In this appendix, I use the following definitions to categorize client personnel as having the following roles:

•Audit sponsor   The person or committee within the client organization that has determined that the audit needs be performed. When audits are required by regulation, the lead executive—commonly the CFO (chief financial officer), CIO (chief information officer), CAE (chief audit executive), CRO (chief risk officer), or CCO (chief compliance officer)—over the group being audited is most often the audit sponsor.

• Audit audience   The party that will review and employ the information contained in a report. In the case of an internal audit, this is most likely the board of directors and/or audit committee of the organization, whereas for SOC1/SOC2 reporting purposes, this would more likely be the external auditors or customers of client organizations.

• Primary contact   The person who serves as the initial point of communication between the audit team and client organization’s control managers and owners. The primary contact has the authority to schedule meetings and address issues, and may be provided regular status reports.

• Control owners   The person(s) performing manual control activities or maintaining the successful performance of automated controls.

• Control managers   The members of management who oversee control owners. They are ultimately responsible for ensuring the successful execution of control activities, and have a role in remediating issues discovered during testing.

“Client” as a Term for Internal Auditors

The term “client” usually implies parties are not under the same roof. In this appendix, “client” means the auditee—whether an audit is external or internal. A department within a larger organization may be the audit “client.” (As an example, in an audit focused on reviewing procurement practices, the procurement department is viewed as the “client.”)

With internal auditing, while the audit cycle will lack a bidding process, contract negotiation, and engagement letters, an auditor within an organization is still an independent party. Within this appendix, if there are points in the auditing process where there is a recognizable difference between performing work internally as opposed to externally, the difference is noted.

Overview of the IS Audit Cycle

This section describes the IS audit cycle and covers background information that may be pertinent to an auditor’s engagement. Included is a discussion on where audits originate and some of the particularities of different engagement types. Different reasons are addressed regarding why a client organization may initiate a project requiring the assistance of an IS auditor.

The IS audit cycle is a fairly standardized process, in that established steps are agreed upon as providing the basic structure for performing an information systems audit. Common milestones have been established. IS audit projects will involve some, if not all, of these milestones. This appendix explores the details of these milestones and activities, but at a high level, they can be viewed as:

•  Project origination

•  Engagement letter or audit charter

•  Ethics and independence

•  Launching a new project

•  Developing the audit plan

•  Test plan development

•  Performing a pre-audit

•  Organize a testing plan

•  Resource planning

•  Performing control testing

•  Developing audit opinions

•  Developing audit recommendations

•  Managing supporting documentation

•  Delivering audit results

•  Management response to audit findings

•  Closing the audit

•  Following up on the audit

Each of these stages is covered in more detail within this section.

Project Origination

This section addresses the origination of information systems audit projects. This is the beginning of the information systems audit cycle. The following service areas are included in this discussion, although some of these service areas are not fully covered in this appendix:

•  External attestations

•  Internal audits

•  Incident response and disaster response

•  Life cycle reviews

•  Governance reviews

•  Staffing arrangements

This appendix surveys how the need for audit work in each service area is identified and originates as a project. This continues with a discussion of how an auditor’s help is solicited and their common roles in supporting certain projects.

External Attestations

An attestation is a statement made by an auditor that certifies or affirms the results of the audit. Often, an attestation takes the form of a letter or report that is signed by an owner or partner in an auditing firm (or by the leader of the audit department in the case of an internal audit).

Many organizations are required to have an audit based on government regulation or contractual obligations. External auditors provide results free from the pressures of a management reporting relationship. Strict ethical, and sometimes legal, guidelines prohibit relationships that can impair independence, whether in fact or in appearance. These independence measures seek to remove obstacles to an auditor’s objectivity when confirming the existence of practices and assessing the operation of controls within their organization—think of this as similar to the idea of testing for conflicts related to segregation of duties.

Examples of these services include

•  Financial audits

•  Bank system controls testing

•  Lending or equity arrangements

•  SOC1/SOC2 and other attest audits

•  Certifications such as ISO and PCI-DSS

For external attestations services, a bid solicitation process is most commonly followed. The client organization issues a request for proposals (RFP) from external parties. The RFP will identify, at a high level, the scope of the work and some of the technologies involved. Proposals are collected and reviewed by the client organization, often including the audit sponsor and/or the primary contact. Proposals are vetted for approach, skills, terms, fees, expenses, and other considerations. The party selected by this process is then brought in to negotiate a contract (discussed further in the section, “Engagement Letters and Audit Charters”).

Management’s Need for an Independent Third Party

In addition to externally required attestations, the executive management of an organization can initiate projects for outside auditors. It is not uncommon for management to decide that a task should be handled by an independent third party. Management may have many reasons for hiring independent third parties to perform reviews, some of which include

•  Unbiased

•  Fresh perspective (“a new set of eyes”)

•  Professional perspective, if a certified or accredited auditor is employed

•  Not employing the necessary skills in-house

•  Answering inquiries by external parties (i.e., performing agreed-upon procedures)

•  Support management decision-making (such as “buy versus build” and system selection decisions)

•  Gain access to advice from outside professionals with deep industry experience

Projects at the request of management are likely to report to a CFO, CIO, CAE, CRO, or CCO. Such projects may involve testing that supports goals that are not standard audit goals. It is important for auditors to be clear with a client regarding objectives and scope, and how to address requests for additional work.

Internal Audits

Internal audit (IA) departments usually report to the organization’s audit committee or board of directors (or a similar “governing entity”). The IA department usually has close ties with and a “dotted line” reporting relationship to finance leadership in order to manage day-to-day activities. This department will launch projects at the request and/or approval of the governing entity and, to a degree, members of executive management.

Regulation plays a large role in internal audit work. For example, public companies, banks, and government organizations are all subject to a great deal of regulation, much of which requires regular information systems controls testing. Management, as part of their risk management strategy, also requires this testing. External reporting of the results of internal auditing is sometimes necessary.

A common internal audit cycle consists of several categories of projects:

•  Risk assessments and audit planning

•  Cyclical controls testing (SOX and A-123, for example)

•  Review of existing control structures

•  Operational and IS audits

The central function of an internal audit department is the entity-wide risk assessment process. Annually, an attempt is made to identify and weigh all risks to an organization. This process results in ranking the organization’s “areas of greatest risk.” This is provided to the governing entity (commonly the audit committee) to review and determine the scope of the organization’s internal audit function. Areas of greater risk command more attention by the governing entity. They may choose to have the scope of the IA department’s work address these areas.

It is common for the IA department to maintain a multiyear plan (as discussed in Chapter 3), in which it maintains a schedule or rotation of audits. The audit plan is shared with the governing entity, along with the risk assessment document, and the governing entity is asked to review and approve the IA department’s plan. The governing entity may seek to include specific reviews in the IA department’s audit plan at this point. When an audit plan is approved, the IA department’s tasks for the year (and tentative tasks for future years) are now determined.

Even if the risk assessment is carried out by other personnel, IS auditors are often included in a formal risk assessment process. Specific skills are needed to communicate with an organization’s IT personnel regarding technology risks. IS auditors will use information from management to identify, evaluate, and rank an organization’s main technology risks. The outcome of this process may result in IT-related specific audits within the IA department’s audit plan. The governing entity may select areas that are financial or operational and that are heavily supported by information systems.

Internal audits may be launched using a project charter, which formalizes the project to audit sponsors, the auditors, and the managers of the department(s) subject to the audit.

Cyclical Controls Testing   A great deal of effort has recently been expended getting organizations to execute a controls testing cycle. Most frequently, these practices are supporting the integrity of controls in financially relevant processes. Public corporations have needed to comply with Sarbanes-Oxley Section 404 requirements, and U.S. government organizations have been subject to OMB Circular A-123, compliance with the Federal Information Security Management Act (FISMA), and other similar requirements. Countries outside of the United States have instituted similar controls testing requirements for publicly traded companies and governmental organizations. Many industries, such as banking, insurance, and health care, are likewise required to perform control testing due to industry-specific regulations.

Financial leadership is required to affirm that controls testing cycles are operating successfully and that controls surrounding financial reporting are operating effectively. This includes control testing by qualified and independent auditors (common regulations require that a portion of the control testing require using an external IS auditor).

Organizations employ software tools to assist with tracking controls testing. These systems track the execution and success of control tests performed as part of a testing cycle, and can frequently manage archiving supporting evidence. Organizations may employ IS auditors to implement a system for tracking controls testing.

Many organizations have functioning internal audit departments. Most internal auditors come from a financial background and have limited knowledge of the practice of information systems auditing. Organizations that lack an existing internal audit department may outsource their whole internal audit function via the RFP process. IA departments may seek to augment their staff with IS auditors to cover internal shortcomings.

Establishing Controls Testing Cycles   Young or growing organizations may not have established or documented internal controls testing cycles. IS auditors, working in conjunction with individuals focused on manual controls, will participate in the establishment of controls testing. The auditor produces documentation of controls through a series of meetings with management. During the process, auditors will develop process and controls documentation and confirm their accuracy with control owners through the performance of control walkthroughs.

These engagements are likely to occur when companies prepare to go public. Such companies need to comply with Sarbanes-Oxley Section 404 requirements, which involve documenting controls and performing a test of existence, also known as a “test set of 1” or a “walkthrough,” for each identified key control.

Private companies will maintain SOX-equivalent documentation to retain the option of seeking public financing, or when lenders or private investors require it. Many organizations will find external resources to assist in the documentation and testing of applicable internal controls.

When auditors are bidding on engagements for establishing controls testing cycles, there are uncertainties regarding the amount of time this process will take. Factors that may add unexpected amounts of time include

•  Functions prove complex or disorganized.

•  Documentation provided by management could be out of date.

•  The need for auditors to produce control procedures where none exist.

•  Control weaknesses and failures may be uncovered, requiring unplanned remediation.

•  Fraud may be uncovered, resulting in project interruptions and turnover of control owners.

If management has documented procedures or experience with controls testing, this process will take less effort on the part of auditors.

Reviewing Existing Controls   Control structures change as an organization and the regulatory landscape change. Outside the organization, a change in regulations may change the focus of certain processes and controls testing. Guidance covering SOX and A-123 auditing has changed over time. For example, changes over time by the American Institute of Certified Public Accountants (AICPA) and other organizations have allowed a greater degree of reliance upon monitoring controls. Within an organization, objectives may change, and with them the technologies and procedures. New systems or new lines of business may change which controls are most material. IS auditors are often asked to review and update the individual controls within a controls testing cycle. They will update wording and may advise management on how to bring the controls testing closer in line with external guidance.

Early SOX compliance efforts often led to long lists of control activities that would be considered excessive by today’s standards. Older control structures predate more recent directives, causing an excessive testing burden for the organization. Business, process, and technology changes may have left parts of a control structure obsolete or not yet included. Organizations often seek third-party assistance performing controls rationalization—updating and “streamlining” their list of key control activities to realign their controls with both current regulatory directives and recent risk assessments. IS auditors may be tasked with updating controls documentation as an additional service while performing control tests.

Operational Audits   Operational audits are typically internal audits. These audits involve internal reporting to management and/or an organization’s governing committees, and can cover any area of the business. These reports are done at the request of a member of executive management or the governing entity, and may originate from the discovery of an issue in the business, the annual risk assessment, or the established audit planning cycle. These audits often will be performed over a limited period, have a clearly specified scope, and result in issuing an internal audit report.

Operational audits require an auditor with experience in the function being audited. Many organizations will want to grow out their internal audit function from within their ranks so their auditors have a functional knowledge of the departments they audit. If an organization lacks an internal audit function, they often will seek audit professionals with experience in their industry.

Audits focused on operational elements will employ whatever methods of testing or analysis support the audit’s objectives, and may not include controls testing. Some examples of the objectives for operational and IS audits could include the following:

•  Evaluate procedures within a department.

•  Process reengineering within a complex function.

•  Test compliance with policies.

•  Prepare for a system implementation.

•  Uncover internal inefficiencies.

•  Perform data analysis for management decision-making.

•  Identify improvement opportunities to existing systems.

•  Detect fraud and the risk of fraud.

When an operational area relies heavily on supporting systems, it is common for an IS auditor to own part or all of the cycle of performing operational audits. For certain operational audits, an IS auditor’s background with the operation is important. Familiarity with the business of the department and the procedures performed is often required. A project could require an IS auditor with a financial background or experience with certain financial business processes.

Auditing Incident Response and Disaster Response

Incidents can take many forms, and can include but are not limited to:

•  Internal and external hacking

•  Network traffic failures

•  Post-implementation problems

•  Misuse of information systems

•  Failure of key equipment

•  Natural disasters

Many occurrences might lead to the formation of a “response team,” sometimes referred to as a computer incident response team (CIRT). This team is assembled to involve the appropriate parties from within an organization to handle the issue. Internal auditors are often included in a CIRT.

An organization that has experienced an “incident” is unlikely to have a clear approach to how IS auditors can assist, but auditors often have a role to play in the aftermath of an incident. An unplanned damaging event often leads an organization to gather special teams. These teams will seek to assess, contain, and recover from the incident, as well as design and implement changes that will prevent future occurrences.

An IS auditor has skills that are useful in these situations, but it is important to recognize that some of the skills needed are beyond the scope of the CISA study materials. A CISA certification does not by itself provide the preparation needed for gathering forensic evidence or performing technological remediation. A person with a CISA certification may, with appropriate oversight, prove a useful assistant to a forensic auditor or other expert. An IS auditor does have the skills to assess where control structures broke down or were insufficient, and can assist in or advise while developing controls as part of remediation. Independence, however, must always be considered, even during an investigation, and IS auditors should not be responsible for the actual implementation of controls. Once remediation efforts have been completed and new controls have been implemented, the auditor can confirm controls are operating effectively.

Incidents can be embarrassing for an organization, and can affect their reputation and relationship with a customer base, potential investors, and other organizations. During the course of remediation from an incident, it is not uncommon for executive management to hire external IS auditors to provide independent verification of the progress of remediation plans. Executive management can see this service as providing assurance to their customers and/or partners that they should be protected from the incident recurring.

Some “incidents” may be understood by executive management, but the necessary skills to perform remediation are in short supply—for example, a system implementation with faults requires an assessment independent of management responsible for the project, or an acquiring company has problems related to data migrated onto their systems from companies they acquired. It is not uncommon for an IS auditor to be contacted for these jobs. Some of these tasks can be handled by information systems auditing techniques; however, the more the task involves a consulting solution, the more it is likely to stray from the standard information systems auditing process. When an IS auditor is brought into these situations, they should be very clear about the scope of the work to be performed and ensure they have the appropriate skills at their disposal.

Project Life Cycle Reviews

Project life cycles are a central function of an IT department, and project management practices can come under review as part of risk assessments and other projects. Areas frequently addressed in project life cycle reviews include

•  Tasks supporting implementing or upgrading existing software

•  System development life cycle (SDLC) methodology

•  Asset management

•  Change management

•  Patch management of critical systems

•  Configuration management

IT projects often involve a great investment on the part of an organization. The success of these projects can be critical to an organization’s future well-being and may have a bearing on members of management’s future with an organization. Management has a strong interest in ensuring that their investment in the project goes well.

An IS auditor can assess life cycle reviews to cover an IT department from a number of different angles. SDLC reviews can cover segregation of duties of project personnel, quality assurance measures, or a review of issues tracking tools and associated controls. Reviews may be designed to assess whether certain project management “best practices” are followed, such as maintaining project plans and meeting minutes, performing and documenting appropriate user-level testing, and capturing approvals on customization design documents. These audits may assess compliance with management’s policies in addition to looking at common industry practices.

Examples of possible life cycle projects include the following:

•  An organization has just implemented new financial accounting software. Financial auditors determine the change to systems is material and seek to gain an understanding of controls in the process of implementation. An IS auditor is called in to review project documentation and speak with key project personnel. The auditor will review scope documents, approvals for customizations, segregation of duties/responsibilities, test plan records, issues tracking, and other key records from the process. The IS auditor reports to the financial audit team whether the process is well controlled, and the audit team incorporates this information into their test plan development.

•  Internal audit may perform an IS review that addresses the controls in an organization’s SDLC to ensure that proper review, approvals, version retention, and segregation of duties are performed in accordance with controls documentation.

•  An organization is experiencing delays on an implementation project. Management is not sure whether this is due to the performance of the project manager or because of an underestimation of the project requirements. An independent reviewer is asked to speak with persons involved in the project and review project documents to provide feedback.

•  A government agency is preparing to comply with new legislation and is hoping to clarify the scope of compliance projects. The new legislation will result in increased traffic through their agency. Agency management seeks to learn whether procedures and technology are prepared for the increased traffic. They don’t have the bandwidth to task their own people with the review, so they hire outside reviewers to report on what changes are necessary and what changes may be desired.

•  An organization’s network security has been successfully compromised by ethical hackers (“ethical hackers” is a term signifying a professional services firm hired to test an organization’s security controls). Management has committed to performing remediation activities to prevent future intrusions. IT management is strengthening existing controls and pursuing projects that aim to institute new control measures. IS auditors are brought in at the request of executive management to validate IT management’s claims regarding the successful implementation of controls measures.

Life cycle reviews may be covered in part by methods discussed in this appendix, but some reviews may require procedures not addressed here.

Implementation Problems   Problems from system implementations and data conversions are common, and management may be strapped to remediate the issues. Client management may seek the help of IS auditors as additional “bandwidth.”

Here is an example: A company is acquiring smaller competitors and migrating their information systems into one consolidated system. Attempts to report using migrated data show problems in the underlying data. Auditors are tasked with learning about the process and controls related to correctly loading data sets onto the system. The auditor will then test report data and attempt to identify the specific problems.

These tasks can be handled by information systems auditing techniques. The more the task involves a consulting-style solution, the more the project is likely to stray from the standard information systems audit process. When an IS auditor is brought into these situations, the auditor should be very clear about the scope of the work to be performed and ensure they have the appropriate skills to deliver on the task at hand.

IT and IS Governance Reviews

Often, IT and IS governance reviews are required by external regulations. Governance reviews are usually focused on management’s risk management and performance measurement responsibilities. Financial auditing procedures require governance evaluations. An auditor’s risk analysis could identify information systems as an area material to an organization’s control structure, such as within an e-commerce company.

Management’s risk analysis could also identify areas of IT governance to review. Management could request that an internal audit department or external reviewers be requested to assess whether an IT department is aligned with a company’s strategies or is delivering appropriate value for an organization’s investment in IT.

A few examples of IT governance projects include the following:

•  Management is facing some long-term budgeting decisions, possibly including eliminating positions. Rather than determining which positions to eliminate, management finds an independent party to provide an impartial perspective on the value each of the groups within the IT department provides. Management wants IS auditors to provide feedback on whether each group within IT is efficiently delivering value to the organization and is appropriately sized.

•  A manufacturing company is preparing for growth. The IT department has not changed much since the company was small. Management wants an outside reviewer to recommend ways to “tune up” the IT department ahead of the expansion. Management hopes IS auditors will identify key IT risks facing expansion and work on developing an IT governance structure appropriate for a larger organization.

•  Auditors are asked to review management’s security policies and offer recommendations for improvement.

Organizations may seek the help of IS auditors to provide recommendations to strengthen their governance function.

Staff Augmentation

When an organization has the ability to oversee the work of an IS auditor but needs additional resources to get work accomplished, they may opt for outsourced or co-sourced staffing arrangements. It is not uncommon to temporarily requisition the help of a skilled IS auditor to support controls testing or to serve on special teams (such as a CIRT). In this situation, the IS auditor reports directly to management in a client organization. In these situations, the auditor may perform a limited part of the information systems auditing cycle.

Engagement Letters and Audit Charters

Engagement letters govern the terms of the audit engagement when external auditors are used. This section lists a few of the general terms and goes into more detail on a few subjects of interest to auditors. General subject areas addressed within the engagement letter include

•  Scope of work to be performed

•  Distribution of the report

•  Rates, time estimates, and fees

•  Ownership of workpapers

•  Terms for addendums

•  Nondisclosure agreements

•  Audit charters

Audit organizations and client organizations will both review a contract, and each party may require specific wording. Some contract negotiations can prove lengthy.

When audits are externally required, the party serving as audit sponsor may not be supportive of the audit. In these situations, the audit may not be welcomed by the primary contact or the control managers. Thus, it can be beneficial to give extra attention to:

•  Audit terms on turn-around time for requests

•  Availability of control owners and other key personnel

•  The relationship with the primary contact

•  Frequency and type of status reporting

Distribution of the Audit Report

The outcome of an audit is an audit report, a written explanation of the audit project, including the project’s objectives, the controls tested, and the auditor’s opinion of the effectiveness of each control. Most audit reports are solely for use by an organization’s management and governing entities. Reports will contain language reflecting the limited distribution and use of the report. The audit organization will only reply to inquiries regarding the report with members of management and no other parties. For example, the cover sheet and the footer on each page of the report can include the following phrase:

“This report is restricted to business use by management of XYZ Corporation and is not to be relied upon by any other party for any purpose.”

Certain reports, such as an SSAE 16 or SOC1 (formerly SAS 70) report, are for distribution to third parties. The engagement letter will state clearly the terms under which parties are permitted to receive these reports, and will provide a process for getting permission from the audit organization if they seek to provide the report to another party.

As an example of contract terms surrounding a report, SSAE 16 clients are forbidden from distributing a report to parties other than those using the control information in the SSAE 16 report for management’s review and to provide to their financial auditors. This means client organizations are not permitted to share this report with a potential client as a marketing tool without express written permission from the audit organization. If management does distribute a report beyond the terms of the engagement letter, the audit organization is no longer responsible for the content of that report if relied upon by a nonpermitted third party.

Rates, Time Estimates, and Fees

Part of an engagement letter will address the billing for services. Many attestation engagements are fixed-fee engagements, although additional billings may be agreed upon with time budget overruns or changes in scope. Many nonattestation (in other words, professional services instead of audit) engagements will bill based on hourly rates. Rates could be blended across teams, or could identify individual rates for specific resources. The contract could identify the degree of detail the client will receive on their statement.

Expenses are usually addressed in this section. Any conditions on expenses will be spelled out here. Clients may only permit certain kinds of expenses, or may ask for invoices with itemizations. Clients often select nearby audit firms that can provide resources without incurring travel or lodging expenses. Larger audits may draw resources from all over the country, and may incur sizeable expense bills.

Ownership of Workpapers

For external auditors, ownership of workpapers is different in attestation and internal audit engagements.

When a certified professional is signing off on an audit, they retain ownership of workpapers because they may be asked to defend their opinion with evidence. In some engagements, auditors may not have a problem providing a copy of part or all of the workpapers to management. Audit documentation of procedures is sometimes shared with management, and the documentation of test results beyond the report may be shared with management to support remediation efforts. In some external audits, ownership of audit workpapers is retained by the auditors, such as a bank requiring compliance testing from service providers.

Internal audit engagements are services done on behalf of management, and management retains ownership of audit documentation. IA departments commonly document their work in a combination of paper and electronic workpapers. Document retention requirements vary by industry and individual organization, and should be clearly understood by internal auditors to ensure internal and/or external retention compliance.

The ownership and sharing of workpapers will be addressed in the engagement letter.

Terms for Addendums

Most contracts prepare for the possibility that they can be extended with an addendum or change order. The addendum can increase or remove scope, extend deadlines, and add audit cycles based on the terms of the original engagement letter.

Nondisclosure Agreements

Because auditors gain access to proprietary information during the course of the audit, they are almost always bound by nondisclosure agreements (NDAs). These may be signed by the individual auditors, or for auditing firms, and they may be signed at an engagement level covering all team members. It is worth noting that NDAs usually do not cover disclosure to legal or regulatory authorities if fraud or other illegal activities are discovered during an audit. In some cases, an NDA is not signed but similar nondisclosure-type language is included in an engagement letter or contract.

Audit Charters

Audit charters are used for projects internal to an organization. Internal audit projects will often employ an audit charter. They prove useful to ensure management’s buy-in within an organization. Audit charters provide assistance to the project by communicating and formalizing the following:

•  Sponsorship by executive management

•  The goals of the project

•  The planned time frame for audit activities

•  Obligations of team members

•  Expectations of team members

Chartered projects will often be kicked off with a meeting. This will help the team by allowing introductions to be made between team members in different departments. The event also serves to promote teamwork and reinforce the goals, obligations, and expectations of the project.

When an outsourced IS audit resource is brought in to a chartered audit project, it is important that the auditor become familiar with the audit charter. This will allow the auditor to understand what has been communicated to client management and control owners.

Ethics and Independence

It is important that the auditor maintain independence from the client organization in both fact and appearance. This appendix will provide a few examples, but to comprehensively address the subject, refer to the discussion on ethics, independence, and the ISACA Code of Professional Ethics in Chapter 3.

Independence in Fact

Avoiding issues of independence in “fact” is rather straightforward. An auditor may not audit their own work, and may not report on testing if the subject of testing is a function owned or managed within their reporting relationship. The auditor may not design or be part of implementing controls and be called on to test those controls. Some examples of this include the following:

•  An auditor has had the responsibility of implementing the new AR (accounts receivable) module as part of the ERP (enterprise resource planning) implementation team; the auditor should not be performing control reviews on this system.

•  An auditor has been tasked with the daily monitoring of the firewall log. They may not perform testing as to whether the firewall log is regularly monitored.

•  An auditor has a reporting relationship with a control manager. The auditor may not test controls managed by that control manager.

In addition, one should avoid testing the work of control owners or control managers when the auditor has family, intimate personal relationships, or external business relationships involved.

Independence in Appearance

Avoiding issues of independence in “appearance” is where an auditor faces more challenges. Gifts from the client are one area where judgment can be required. Fortunately, an auditor often is able to lean upon workplace policies in such situations. Common workplace policies include forbidding gifts over certain values and getting permission to accept certain gifts (such as tickets to a baseball game). Regardless of policies, an auditor must exhibit care when accepting gifts. A few examples are discussed here.

•  The client organization’s CFO offers the team coffee mugs with the company’s new logo. Since this item has a limited cost and has marginal value, and is also a promotional tool for the company, there should not be an issue of independence in appearance.

•  A control manager at the company offering to pay for a coffee may be a limited and acceptable gift.

•  The CFO met the audit team for dinner and covered the bill. This can be acceptable as team building. The CFO then seeks to fund a night on the town with drinks and entertainment; this may be perceived as crossing the line.

•  Small talk with a control owner about their office decorations leads to them offering you a gift from their collection of sports memorabilia. This could be perceived by the client as impairing independence.

Launching a New Project: Planning an Audit

A new project is on the table. The client wants auditors to start work soon, and so the process begins. Often in external audits, clients will limit the information provided until engagement letters and nondisclosure agreements have been signed. Most external audit organizations severely limit the amount of work auditors are permitted to perform on a project before the client has signed the engagement letter. To ensure that valuable time is not wasted, management waits until both the audit firm and the client have a clear and formal understanding on the scope and purpose of the audit.

Understanding the Client’s Needs

When a client organization decides that an audit is needed, they will usually describe their needs in writing and use a formal or informal selection process to choose an auditor. This selection process is centered on communication between the audit organization and the client about the client’s needs. A client needing a signed attestation still looks for the best audit firm for their “needs.” A client has reasons in addition to price to consider when selecting an audit firm. If an audit firm does not have experience in the client’s industry or technologies, the auditors may not work effectively with management. If a client is the smallest in an auditor’s book of business, they might be concerned about the level of service they will receive. A firm may be selected because of experience with an area of a client’s needs that are peripheral to the audit scope, and management’s decision to perform such an audit could relate to these needs. Understanding the reasons behind an audit can be important to successful planning and meeting client expectations.

A client organization may open up with more detailed reasons once auditors are selected and nondisclosure agreements are signed. Having such conversations with the primary contact or the audit sponsor early in the audit can provide valuable information to the audit team.

Some examples of a client organization’s needs that may factor into an audit include

•  Augment documentation of new or changed procedures

•  Get an internal audit function operating

•  Update an outdated controls infrastructure

•  Assist in the education of a new executive

•  Support a financing relationship

•  Repair relationships damaged by a previous control failure

•  Meet contract conditions by providing an audit report by a certain date

Knowing the reason behind management’s decision to perform an audit will allow audit personnel to:

•  Better understand the client’s risk environment

•  Provide more useful feedback on their controls structure

•  More accurately plan for the audit

•  Focus extra testing on the most critical control objectives

•  Meet client expectations and deadlines

•  Provide meaningful reporting based on the results of the audit

IT managers frequently ask their auditor about how they compare with their peers.

Preliminary Discussions   Preliminary discussions between audit management and client management will set the stage for how well the parties will work together. It is important at this phase to anticipate challenges that may be faced during the audit. Common things to address in these initial discussions are

•  Clarifying scope by confirming an understanding of client needs and their risk environment

•  Acquiring more detailed information on employed technology and a deeper understanding of how it supports the organization’s objectives

•  Establishing engagement procedures, such as scheduling control owner time and requesting documentation

•  Setting expectations, such as frequency and depth of status reports and review of testing exceptions

Both the client and the audit manager have an important investment in the success of this phase. The client organization is hoping to maximize the benefit of the service, so they may identify areas where they would seek professional advice. The client representative may aim to minimize internal disruptions and ask the auditor to observe certain practices.

Understand the Technologies Employed   The audit manager uses information on employed technologies when developing the audit plan and when assigning resources to test controls.

When involving a third-party audit resource, a client’s selection process will limit the amount of information they share publicly about their systems. If there is a bidding process, it may permit formal or informal Q&A, where answers will be provided in response to vendor inquiries for the purposes of estimating effort. When the audit is launched and NDAs are signed, the client should be willing to share relevant documentation and information freely.

An audit manager also will be gathering information on the nature of testing that will be performed. Some questions an audit manager may consider include

•  What kind of security testing is required

•  What kind of process evaluation is required

•  What kind of application testing is required

•  What relevant customizations exist

•  Are any in-house-developed technologies employed

In these preliminary discussions, the audit manager needs to gather more specific information on the technologies involved and the testing to be performed. The version numbers, implementation dates, and an idea of the transaction volumes are useful information. The audit manager will use such information during audit planning.

Performing a Risk Assessment

A risk assessment process takes into account the innate risks of a certain operation and considers information from within the organization. Auditors will weigh information from several sources when performing a risk assessment, as illustrated in Figure A-1.

Figure A-1   Different considerations in a risk assessment

It is common to have financial information available for the purpose of assessing the materiality of certain activities. Here are some examples:

•  An organization may have extensive automated transactions in revenue and have very few assets tracked within their asset management system. Therefore, there is less innate risk surrounding asset tracking, but thorough attention needs to be given to the systems supporting the revenue cycle. There could be a high risk of data redundancy or incomplete capture of information in the event of system failure.

•  A debt collection service outsources the software maintenance for their core collections-processing software to a software vendor. Therefore, risks relating to change management controls surrounding their core systems are reduced. However, because it is a high-transaction environment, data backup and restoration controls are elevated in criticality.

A risk assessment is arguably the most important aspect of an audit. Without a risk assessment, high-risk situations may not be addressed sufficiently during an audit. Management may not realize the opportunity to reduce serious risks.

Audit Methodology

Audit methodologies are designed by audit management and standardize how parts of audits are performed. Methodologies are the procedures used by the audit team to perform an audit. They can be as simple as requiring scope to be documented and approved, to employing audit software and detailed procedures that govern the entire audit process.

ISACA considers the following items so central to the audit process that all audits will at least generate documented statements addressing:

•  Audit charter

•  Audit scope

•  Audit objectives

•  Audit testing program

Audit organizations that regularly provide certain services will standardize methodologies for performing their audits. These methodologies can assist an organization in ensuring completeness, maintaining standards, and streamlining the process of management’s review of audit work that is done.

Methodologies can include policies, procedures, software tools, templates, checklists, and other means of providing uniformity across the audit process. These methodologies are documented and taught to new members of the organization.

Documented methodologies can serve to govern many stages of the process, such as:

•  Bidding on RFPs

•  Risk assessments

•  Scope

•  Objectives

•  Resource allocations

•  Comprehensiveness of testing

•  Sample size guidelines for testing

•  Report templates

•  Completion checklists

Methodologies will provide a structure for achieving milestones within the audit process. For example:

•  Risk assessment   An audit firm’s risk assessment approach involves employing a spreadsheet predesigned to compute an aggregate score from several different risk measurements. Form letters may be used to communicate with management and collect their feedback on the client organization’s risks. Management’s feedback can be populated into the spreadsheet along with auditors’ assessments, and the risks are ranked.

•  Budget   A budget tracks time and rates incurred by auditors. This will be developed initially in the RFP process and updated in the planning process. Actual time incurred will be compared with the budget so audit management can better understand how to plan for future engagements.

•  Lead sheets   Lead sheets are intake forms used by auditors to capture and organize test information. They provide a uniform method to represent test results and enable audit management to perform a formulaic review of test results.

•  Testing standards   In order to maintain a rigorous standard of testing to support their reputation, an audit organization institutes testing standards. These standards require different methods of testing in order to pass a control test. Testing methods are identified as follows: collaborative inquiry, observation, inspection, and reperformance. In addition, each control objective must be supported by one form of substantive testing.

•  Auditing software   Large audit organizations frequently enforce their audit methodology with software that attempts to accommodate as much of the audit process as possible. These programs may accommodate most audit possibilities and enforce that certain procedures be executed by the audit team. They may even manage images of workpapers so that the software captures all audit documentation.

Methodologies used by audit firms may be designed to meet requirements published by regulatory organizations, such as the AICPA or the U.S. government’s Office of Management and Budget.

Developing the Audit Plan

An audit plan is a project plan designed for performing an IS audit. The audit plan, like a project plan, is a tool for tracking tasks and forecasting the time and resource needs of the audit process. It will describe the audit methodology to be used and lay out milestones and the sequential dependencies for the different tasks within the audit. The plan is updated with progress milestones, and may be adjusted with certain audit changes.

In addition to serving as a high-level audit plan, in the beginning, the audit plan serves to organize the stages of risk assessment, audit objectives, and the initial assessment of client procedures.

The audit plan does not track the detail of audit testing—this is tracked in the audit testing matrix and the lead sheets.

Gathering Information: “PBC” Lists

A “Provided by Client” (PBC) list is a common tool used by auditors for managing information requested from the client. It provides a consolidated list that the auditor can use as a record of request for process documents and records, and provides an effective checklist for tracking receipt of information. It will also help a primary contact manage fulfillment of auditor information requests. Several PBC lists may be needed during an audit. PBC lists should be dated when they are delivered, and, if possible, they should document an agreed-upon delivery date.

Initial Information Requests   At the beginning of an audit, the auditor will require information about the organization. Common requests include

•  Organizational charts

•  Company directory

•  Controls documentation

•  System documentation

•  Relevant reports or other information

This information will be used to prepare for and execute the audit. The list may also identify documents that a client has indicated exist, such as an information security policy document.

A Client’s Preparedness for an Audit

When an organization is facing their first audit, auditors frequently include in their plans an evaluation of the client’s preparedness. A client may have a need for an audit, but may not be prepared. In attestation situations, the client organization may not yet be ready for an audit. A few possible examples of this include the following:

•  A company hopes to undergo an initial SSAE 16 audit, but the control infrastructure is not yet documented or in place.

•  A company has experienced significant growth. New ways have been devised to perform key processes. Procedures are inconsistent, and documentation is incomplete.

•  Changes to business products, processes, and supporting technologies have left controls documentation out of date.

•  New control procedures have been only partly implemented.

•  Logging in key systems hasn’t been configured correctly, so there is inadequate capture or retention of audit information.

•  There has been turnover of key control owners or control managers.

If a client organization’s support for an audit is below par, the first order of business is housekeeping. If the engagement letter does not account for providing services to help the client prepare, it may be necessary to delay the launch of an audit. Some challenges may be addressed by expanding the scope of the engagement letter to include the auditors providing assistance (such as with updating procedures) ahead of an audit. This is, however, a tricky issue: in an attestation or external audit, this is most frequently not possible or desirable due to independence or regulatory issues—auditors can’t audit the structures they help to develop.

Developing Audit Objectives

An audit’s objectives clarify the goals of the audit. Audit objectives also ensure the audit complies with applicable standards, laws, or regulations. Objectives are clarified in a formal document and retained in project workpapers. The objectives provide a basis for measuring the success of testing, and are central to an audit report’s opinion.

Objectives are developed by giving consideration to several different sources:

•  The engagement letter or audit charter addresses the nature of the subject of testing and the expectations of reporting, and provides a central pillar to an audit’s objectives. This may focus on external security, operating effectiveness, or the correctness of transactions and processing.

•  At this point, auditors will understand the nature of the client organization’s business and have discussed the key processes at a high level.

•  An overall understanding of the organization’s risks from the risk assessment process will be incorporated.

•  An understanding of a client’s needs for launching the audit will also be considered. This may reveal the goals of management in conducting the audit or clarify the nature of a third party’s interests in the outcome of the report.

Statements of audit objectives may incorporate additional perspectives. Figure A-2 illustrates how an audit objective is developed through the consideration of many information sources.

Figure A-2   Audit objectives are developed using information from several sources.

For example, an audit engagement letter identifies a client as needing controls documentation and limited financial controls testing. Auditors are aware of new financial systems. The risk assessment shows that financial auditors annually perform test procedures on manual and automated controls within the financial software of an organization. Conversations with management reveal that they hope to update documentation, confirm the success of their system implementation, and provide financial auditors with a report that shows system controls are operating effectively so that they can reduce the scope and cost of the financial audit. The objectives of the audit will be focused on updating procedure and controls documentation for the new system and testing new software controls.

Developing the Scope of an Audit

An audit’s scope is documented in a series of statements addressing the processes and/or systems to be reviewed and to what depth. It will address how to enact the audit’s objectives. The stages leading to the development of an audit’s scope are illustrated in Figure A-3.

Figure A-3   Audit objective and risk assessment help to determine audit scope.

A risk assessment has identified the areas with the greatest risk, so an audit scope will be constructed to address areas of greater risk. For example, testing of low-risk areas may focus on a small number of key controls when more robust testing is called for in areas with greater risk.

Similar to casting a net, the scope statement will identify what is included under the net and set the boundaries of what is outside it. A well-defined scope will assist in the development of focused test plans.

Examples of project scope statements include

•  Testing addresses internal and external access to key systems, including procedures to set up accounts within Active Directory, password controls, VPN administration, and reviewing the network configuration and the firewall rule base. Not included in the scope are inquiries into firewall rule-base areas beyond those controlling external access, application access beyond network connectivity, or network penetration tests.

•  Controls surrounding files feeding financial information into the financial system are the focus of testing. This includes testing controls and validating report values from systems in the revenue cycle, billing, and AP, which are managed outside of the financial system. Excluded will be any testing of data once it is within the financial system.

If, for some reason, testing issues reveal a need to go beyond the scope of an audit, auditors and the client will need to discuss the expansion of scope. This can happen when controls fail and compensating controls need to be tested. When auditors are externally sourced, any augmentation of scope will need to be formalized through a signed addendum to the engagement letter.

Prior Period Issues   When an audit fails a test, succeeding audits will retest the failed test. When developing scope, audit reports from a prior period must be reviewed so auditors understand which issues require revisiting. An exception from a prior period may have been fixed, in which case audit documentation and testing plans may need to adjust to changes. Projects remediating issues to primary controls may be in-process, so only secondary controls are available for testing.

Expanding Scope   In certain audits, such as internal audit reports, management may have some leeway in changing the scope during the reporting period. Understanding procedures or testing exceptions may reveal an area where management needs to dig deeper. It may be more economical to augment the current audit than to perform a procedure as part of a subsequent, currently unscheduled audit. Auditors are most likely available and are currently immersed in the procedures. Management may wish to push deeper to get to the root of an issue immediately.

Developing a Test Plan

When an audit’s scope has been approved, it’s time to develop a test plan. This section covers stages that go into developing the test plan. Audits that are performed on a previously established cycle may base much of their test plan work on plans used in prior audits. However, auditors must avoid any temptation to simply reuse an audit plan. It is very important that an auditor revisit audit objectives and reevaluate the scope of an audit for each audit cycle. Failure to do so can lead to serious audit problems when testing performed fails to satisfy audit objectives. This is especially important in situations where new systems have been implemented, changes have occurred in the business, or control issues have been identified in the past.

Understanding the Controls Environment

When an auditor is preparing the test plan, information is drawn from several sources. If the audit is not in its initial year, documentation will be available from previous audits. Auditors who have performed the same audit in prior periods will have historical knowledge. Client management is consulted to update procedure and controls documentation as well as to identify the control owners.

When auditors collect information on the controls environment, they often use PBC lists. Upon review, auditors may find provided information falls short of an auditor’s needs, requiring additional information requests. Shortfalls might be due to procedures that are new to the control structure or that have not been previously tested. It is common for auditors to meet with client management during this process.

Understanding the Client’s Procedures   An auditor must understand a procedure before she can effectively plan and perform testing. Auditors begin by reviewing information provided by the client. Procedure documentation may be provided in a number of different forms:

•  Financial audit write-ups   CFOs usually keep copies of the process documentation generated by financial auditors. It is common for a CFO to make these available to audit teams when areas are in scope. These could also include written procedures and systems-level documentation.

•  Internal audit documentation   If the internal audit department tests controls on a cycle, the department keeps procedure documentation regarding the controls they test. Other auditors can benefit greatly from this documentation when it is within an audit’s scope.

•  Management procedure documentation   Management may have procedure documentation, such as instructional or reference material for employees (e.g., desktop procedures). A department’s policies may contain procedure documentation. Management may also retain procedure documentation from previous auditors.

•  Instruction manuals   When management trains many people to perform the same procedures, a training department may provide instruction to employees. Training material may provide instruction on control procedures that are within the scope of the audit.

•  Checklists   Management may oversee a process with a checklist. If a checklist is provided, it will offer a high-level understanding to auditors (as well as provide evidence of a controlled environment), but follow-up is likely to be required.

•  Walkthroughs   Auditors may determine that they need to perform a walkthrough of a process or transaction in order to document their own understanding of procedures used. The result of these walkthroughs can be a narrative of the process, flowcharts, or a control matrix that documents the control points in place.

Other sources may provide information on procedures as well.

An auditor must understand a procedure in the context of the audit. The impact of a procedure in relation to the audit objectives is important when understanding controls within the procedure. This will affect the degree or depth of testing that should be employed.

Many auditors bring to the table experience with procedures performed (or technologies employed) at other organizations. An auditor who has experience elsewhere can often more quickly understand a similar procedure (or technology) in a new environment. Such experience can also assist by being available to a junior auditor who is responsible for a subject for the first time.

Understanding the Technology Environment   With an understanding of a business procedure, an auditor can then understand how technology is employed to support it. Procedure documentation will often identify existing systems at a high level. To develop an audit program, the audit team needs to “look under the hood.” A PBC list may have been sent to the IT department. Conversations with IT may be required to identify what kind of information they keep on their systems. Possible sources of information include

•  Audit documentation   Previous audits of different kinds may have documented certain processes within IT. Because of the speed at which many IT environments change, information from an audit a few years back may no longer be relevant.

•  Network and system diagrams   Information about network and data security can be provided in network diagrams. All network documentation should be dated, and auditors should be sure to inquire about its accuracy and completeness. Diagrams can help an auditor quickly identify areas where they will need a greater depth of information.

•  System inventories   An IT department may keep a consolidated list of the systems they support. System inventories are performed to different depths and may not contain exactly the information an auditor seeks, but they are often useful tools for drilling into that information. One can review items on the list and inquire as to their relevance to procedures, or identify where a system’s data resides, or whether it is on a standard backup schedule.

•  Management’s procedure documentation   Management may have formalized certain procedures as part of developing a controlled environment. This information, when available, is often quite useful.

•  Disaster recovery plans   IT departments may have formalized within disaster recovery plans certain procedures to be performed in the event of an incident. At times these reflect common procedures performed by management. An auditor may find relevant procedure, technology, and controls information from reviewing a disaster recovery plan.

It is important for an auditor to validate understandings based on documentation received from IT personnel.

In addition to being outdated, it is not uncommon for IT documentation to come up short of auditors’ needs. A few examples include the following:

•  Documentation may reveal that data entry and processing controls are employed within PeopleSoft, but it might not identify the Linux-hosted Oracle database on the back end, which should be included when testing data security.

•  A written description of network security controls might identify the Cisco PIX firewall, but might omit it being used in series with an intrusion prevention system.

These clarifications are important for a test plan to be well designed. It is not uncommon for auditors to sit down with members of IT leadership at this phase to confirm the understanding of key systems and how they are managed.

Changes to IT Environments   Technologies and technological procedures change with some regularity. If the auditors learn of changes, they should be sure to address them with client IT personnel. Conversations with IT should address the landscape of current IT projects and review whether they will have an impact on testing controls.

New system implementations must be considered carefully when designing test procedures. When new systems are employed, there are many questions regarding the success of the deployment. Auditors may seek to review life cycle controls employed during a development process to determine if the system’s implementation method introduces control risks. With new systems, there is a risk that documentation was not updated to reflect the use of a new system, or that management shared documentation produced ahead of the system going live. “To-be” documentation could contain claims that certain controls exist that were actually omitted from the production launch.

In addition, it is important to know of systems that are due to be implemented before or during a testing period. These can prove problematic to testing plans, as there could be an interruption or a change in control structures with a new system. For example: An SSAE 16 engagement is testing controls over a long period, typically 12 months. Control objectives are signed off by management ahead of the testing period, and they include performing weekly backups. Four months into the testing period, the IT department upgrades their backup software. In doing so, they change the cycle of backups. This introduces a number of problems:

•  Lack of testing evidence   The old backup system may have housed records on success and failure of backup jobs. If the system has been retired, records of backup success and failure might no longer be available when auditors request evidence.

•  Outdated control objectives   Control objectives and control activities might be outdated as documented. The control objective reads that full backups are performed weekly, but the newly implemented practice performs daily incremental backups and full backups only every two weeks. This new practice could fail the stated control objective.

•  Outdated controls and control failures   The IT department may have encountered problems with the new software performing backups on certain technologies, and may, for a period, fail to perform backups of systems hosted on certain critical servers.

Because of instances such as these, it is important for auditors to be included in communications about potential changes/updates. Without prior knowledge of changes like these, an auditor likely would report failures of control objectives, and client management would argue that failures should not be reported because they have a reasonable controls structure in place. With a full understanding of IT plans, auditors are able to work with the client to accommodate the changes. Appropriate controls language can be used, and coordination with management can be done to ensure the transition does not interfere with the audit.

Controls and Control Objectives   Controls are selected for testing because they support a control objective. The control objective is achieved when testing shows tested controls are operating effectively. When building a test plan, auditors organize controls they will test by their support of control objectives.

Controls are implemented to mitigate risks within an organization. Multiple controls often work together to mitigate risks within a process or procedure. The control objective statement summarizes the risk-mitigation goals of controls within a procedure. The control objectives will collectively support the audit’s objectives.

For use in the test plan, control objectives are listed with their supporting controls. This is depicted in the example in Table A-1.

Table A-1   Control Objectives and Their Supporting Controls

Though not all engagements will involve auditors developing lists of control objectives and control activities, many will involve auditors reviewing and providing the client organization feedback on existing controls. Lists like the example in Table A-1 can provide a great deal of assistance to individuals who are new to an audit or new to auditing in general as these lists provide a connection between the individual control activities and the true purpose (or objective) for performing and testing those activities. This can be beneficial in helping to understand the “big picture” of why audits are being performed.

Developing Control Objectives and Supporting Controls   When an auditor is tasked with developing the control objectives and the list of controls to test, they must keep the audit’s objectives and scope in mind. It is important that control objectives be properly phrased to both reflect the actual control activities performed by management and support the audit objectives.

When examining existing control objectives and control activities, the auditor should determine if each control activity actually supports the control objective. Control activities can exist within procedures that do not support, or poorly support, the control objective. Any such control activities should be removed from the list. A replacement control may need to be identified so an objective is effectively supported. If an auditor experiences trouble in this area, they should consult upward within their organization.

With the list of supporting controls, the auditor should determine if any of these controls ultimately perform the same function. If two control activities protect against the same problem, the auditor should determine which one should be selected as the key control and remove the other one from the list. An auditor may wish to learn which of these controls can be more efficiently tested before selecting equal controls as key. Management may agree that one of the controls is redundant and elect to cease performing it.

Key Controls and Compensating Controls   Compensating controls are valuable to identify during this phase of audit planning. When a control failure occurs, the organization relies on a compensating control, which is a secondary measure that mitigates the same risks addressed by the key control. A good test to determine if a control is a compensating control is whether a failure of the key control is caught by the compensating control.

An example of a key control is a formal request and approval process for provisioning new user access. A request made by an individual’s supervisor with approval by department management is key to ensure an individual’s access levels match their job responsibilities. In the event that access was provisioned outside of the normal process, a periodic (quarterly/annual) review of all user access may provide comfort that user access levels overall are appropriate.

In the event of a control failure of a key control, a compensating control can be considered for testing to determine the materiality of the failure of the key control.

Reviewing Control Objectives and Supporting Controls   Over time an organization will change elements of its controls infrastructure. Control structures may evolve due to changes in the organization’s business model, changes in management, or possibly in response to guidance from governing entities, such as how guidance on SOX controls testing now emphasizes a greater focus on governance and monitoring.

When an auditor is tasked with reviewing a control structure, her goal is to make three key determinations:

•  Do control activities correctly support management’s activities?

•  Do control activities support the control objectives?

•  Do the control objectives effectively mitigate risks?

Auditors usually provide feedback to management on how the client can improve the wording of certain controls. Problems with the control structure could be identified, and management may need to institute additional control measures. Occasionally, auditors will determine that two controls mitigate the same risk and one of them can be relegated to the level of compensating control, be omitted from testing, or even be eliminated from management’s control structure altogether.

Helping a Client Understand and Identify Controls   Client management often has the obligation of writing and maintaining their control objectives and control activities. In these situations, auditors often still need to evaluate whether the provided control structure is sound and pertinent to the goals of the audit.

An IS auditor coming in to audit a new organization must keep in mind that not all organizations employ the skills to understand, identify, and document controls. Client organizations may be seeking the assistance of outside auditors to update or even write their controls documentation. If this is the case, auditors should make sure the engagement letter identifies this service.

Certain engagements, such as SSAE 16, agreed-upon procedures (AUPs), and outsourced or co-sourced SOX testing, presume management owns and maintains controls documentation. If a client is seeking these services, the burden is upon client management to have controls identified and documented. These engagements involve testing “management’s controls.” If management does not have a control structure (which happens), these engagements can run into problems. Consider the following examples:

•  A company is undergoing its first SSAE 16 audit. They are responsible for providing the controls structure for testing, as well as writing “Section 3,” which is management’s description of controls. This is a smaller, growing company that doesn’t have controls experience in-house. They have agreed with a partner, acquiring company, or a potential client that they will perform an SSAE 16 audit. They figure the auditors they have hired will help them get through a process they have yet to understand.

•  A bank has had trouble with an information system conversion. They have data problems, and outside parties have concerns. The bank hires a third-party auditor to perform agreed-upon procedures on converted data, but the auditor is not sure what they need to test.

•  An internal audit department has relied upon external IS auditors. Since their last testing, a new ERP system has recently been implemented, but controls documentation has not been updated to reflect current controls. Deadlines require testing be completed soon, so they line up auditors to perform the testing, though their controls are not current.

In these situations, the client may presume that the auditors will write the controls that will help them through the process. In the internal audit example, the burden of working with outdated controls may introduce problems meeting the deadline and may introduce problems when attesting to controls over a defined period. Moreover, in the SSAE 16 and agreed-upon procedures examples, there is a line an auditor must not cross. Auditors are not permitted to test controls they have designed. Feedback that does not involve writing controls language can be provided.

Documenting Procedures   Certain projects call for the IS auditor to generate systems and procedure documentation. When an auditor must draft these documents, it may take several meetings with client personnel before documentation goals can be achieved. Providing draft documentation allows an auditor to more effectively communicate with control owners and control managers to confirm an understanding of the subject matter.

The following are examples of documentation formats an auditor may use:

•  Written text and lists

•  Flowcharts

•  Network diagrams

•  Spreadsheets

•  Data structure and data flow diagrams

•  Screenshots

•  Text files of command output

The task of documentation faces several challenges. When dealing with information systems, an auditor is often attempting to abstract complex concepts in an organized manner to the correct audience. It is common to rely on multiple visual tools to fully communicate processes and the systems that support them (see Figure A-4).

Figure A-4   Different methods of diagramming can support information systems auditing.

When developing documentation, it is useful to share documentation with control owners and managers. Control owners are often helpful when their area of expertise is being represented in new ways. It is also important for the success of testing that the auditor and control owner agree closely on what is being tested. Draft documentation proves a convenient tool for capturing accurate feedback, as notes and corrections can be written on the draft documentation.

Documentation frequently reflects a version number, the date of latest update, and the name of the person who created the document.

Mapping Controls to Documentation   For an auditor’s purposes, individual control activities are often included in the procedure documentation. These may be cited within text and repeated at the end of a section of writing, or they could be identified on a process or data flow diagram. This technique of mapping controls to process diagrams is shown in Figure A-5.

Figure A-5   Diagrammatic process mappings can visually overlay controls and tie them to controls listings.

One way to confirm that controls documentation is complete is to examine a list of controls and verify that each control is reflected in supporting documentation.

Performing a Pre-Audit (or “Readiness Assessment”)

When audit management is unsure as to whether an audit program will be successful, they will often perform a preliminary review, which may be called a “pre-audit” or a “readiness assessment.” This is frequently employed when an organization is facing its initial audit, but may be performed when employing new auditors or after significant changes to business processes or information systems.

The goals of a readiness assessment are to confirm that the control structure is correctly documented and that control activities are correctly represented. The readiness assessment should serve to avoid the embarrassment and disappointment of failed testing because of misunderstandings. It will determine if control procedures are implemented as documented and hopes to confirm that documentary evidence needed for testing is available. The process involves auditors reviewing controls documentation and conducting meetings with control owners. Auditors may perform some testing to the level of compliance, such as observing controls in walkthroughs, to confirm the existence of key controls.

If the engagement plans to test controls over a defined period, the pre-audit must be performed before the beginning of the test period. Adequate time after a pre-audit permits the client time to remediate issues discovered in the pre-audit. It may take time to remediate a control issue.

An example of this would be a company’s first SSAE 16 audit, when the client owns the audit program and external auditors are to perform testing on management’s stated controls. The auditor will confirm that they have learned management’s procedures and controls, and will determine if the client is prepared for testing to begin.

This phase is in addition to procedures performed to generate an audit report, and no testing burden or required documentation is generated. Auditors will outline their observations and recommendations to management. Documentation gathered during this phase may contribute to procedure documentation in the audit workpapers, but may never be used as documentation in support of testing. It will not serve to reduce any testing to be performed during the testing period because it is collected outside of the testing period.

Presenting Pre-Audit Results

Any deficiencies identified are often presented in letter form to management. Any controls not in place also will be reported to management. The letter can address concerns at a number of different levels. Some examples of feedback that might be delivered on a readiness assessment include

•  Control language requires updating or is inaccurate.

•  Evidence of control activities is not captured.

•  Certain transactional records are moved off-site monthly, but testing will require this information to be retained on-site for testing purposes.

•  Control practices are not uniform in certain situations.

•  Planned system changes will affect the performance or relevance of a control activity during a given period.

Some methodologies involve management agreeing in writing that they understand and will address any deficiencies in controls descriptions or performance before the attestation period begins. If management lacks time for remediation, findings are likely. Management may accept an audit despite problems, perhaps citing that they accept the findings in a report, and intend to have the issue fixed by the succeeding audit cycle.

Correcting Control Language

Most IS auditors are skilled at writing controls language. When an auditor stumbles upon control wording that is incorrect, there are specific situations where an auditor should exhibit caution. Control language is the responsibility of management, and an auditor testing controls is forbidden from writing controls language. This would amount to auditors performing management’s function and testing their own work. An auditor can report why a control statement is not accurate and can suggest that management reword the control to correctly reflect the control activity.

Client personnel may be unsure what the audit team is seeking, especially when management currently has limited experience with proper control language. This scenario has the potential to become frustrating for the client. The client is likely looking to expedite the audit progress. Audit management should clearly express to management why this separation of duties is important, and can perhaps provide limited advice on what makes effective control language.

Organizing a Testing Plan

The audit team now has a sense of the control landscape. They have reviewed system and procedure documentation, and have spoken with control owners about the control environment. Once control wording is in its final form, the team is ready to assemble the audit’s test program, sometimes referred to as the “test plan,” “testing matrix,” or abbreviated as the “matrix.”

The test program is often in the form of a spreadsheet. Control activities are listed in the document, and auditors design a set of tests for each control. Depending on the control, there could be one to several test actions to perform, which could range from performing inquiries with control owners to complex substantive testing. Certain methodologies may require multiple types of testing to support the passing of each control. The degree of testing required for each control may consider the risk assessment, the audit objectives, and the project’s budget.

A test plan outlines the testing down to the individual task level and provides a structure for capturing test results in a central document. The document serves to ensure that the auditors perform testing completely. A test plan is an audit team’s internal document. It helps auditors manage the process of testing and serves to track their progress.

Contents of a Test Plan

A test plan organizes a set of control objectives and controls within a spreadsheet. The test plan is designed to assist auditors in performing complete testing and tracking their progress, and will contain many fields that will be populated later during the course of testing. In order to prepare for testing, certain fields will need to be populated before an auditor is ready to start testing. Fields required to be populated include

•  Control number and name

•  Control description

•  Control objective supported

•  Method of testing

•  Date that testing will take place

•  Test description

•  Control owner

•  Auditor resource assigned to test the control

Fields that are prepared by an auditor when developing a test plan include test results in narrative form. This part should provide an answer to a reviewer’s question, “Was testing adequately performed?” and may include

•  Dates and names of people interviewed

•  Short responses to inquiries when a full memo is not required

•  Short description of the testing process

•  Discussion of determinations made during testing

•  References to supporting documentation

•  Summary of test results

•  Testing status

•  Test results

•  Residual risk

•  Recommendations to management

An example test plan, shown in Figure A-6, illustrates how a control may be documented as an entry in a testing plan.

Figure A-6   A testing plan helps to organize the details of an IS audit.

The testing section of the test plan in Figure A-6 is developed to capture test results even though testing has not yet occurred. Auditors could choose to expand this with sections for capturing residual risk and recommendations.

Review of Test Plans

Before testing begins, audit management will approve the test plan. The review will consider the following:

•  Do planned test procedures support a valid test of the control?

•  Is the degree of testing appropriate to support each control objective? The audit’s objective?

•  Are all scope areas covered by testing?

•  Do planned test procedures appear to fit in the planned time frame?

•  Are auditors appropriately skilled to test their assigned controls?

The approved test plan document is placed in the audit’s workpapers. Audit management may share this plan (or parts of it) with client personnel, or it may be kept internal to the audit team to protect the integrity of testing. If management knew precisely how each control was going to be tested, control owners could manipulate their controls to pass the audit even if they were not effective controls.

Estimating Effort

Evaluating the time required for testing is important. In the course of designing testing, the time it would take for an auditor to perform the task should be estimated and tracked. Estimates should include the amount of time to schedule meetings, perform interviews, review test materials, document testing results, update the test plan, and file documents. Experience is the best guide to estimating the amount of effort necessary to perform testing.

When reviewing a draft test plan, it can be helpful to review how many hours are estimated for each test, each control objective, and the plan as a whole. Plans should avoid spending too much time on less important controls or control objectives, and ensure that testing for the most material control objectives is thorough.

It is important to compare test plan time estimates with the planned time budgets in the audit plan. If the testing plan is much larger than the time allotted, several issues could arise, such as:

•  Testing is too ambitious, and there may need to be a reduction in test activities, control tests, or even control objectives.

•  The scope of testing is appropriate and individual controls tests are appropriately sized; however, testing time estimates were inaccurate. The client may agree to pay for the testing or may wish to work with auditors to reduce the scope.

Resource Planning for the Audit Team

Depending on how many audit professionals are available, resource planning will be more or less complex. For external engagements, the selection and bid solicitation process is the first place where an audit team assesses whether they have the skills available to perform an audit. Once more detailed information is gathered from the primary contact, it is time to determine who will perform what work. The most efficient work will be performed by persons who have done testing on that technology (and the supported procedure) in the past. A project will be managed more efficiently with fewer persons involved; however, more hands on deck (with skilled supervision) can accelerate completion—to a point.

An auditor may find it useful to list the technologies in-scope and assign audit resources to the technologies on the list.

If the audit organization has resources with “deep skills” in specific areas, those resources might not need to be included on the team in the field. A skilled auditor can meet with management to gather test materials and then “push down” specific testing tasks to persons with deeper skills. Without bringing them into the field, they can still contribute their skills, perhaps without requiring them to travel or meet with auditors. Here is an example situation:

You have Auditor A (“Anne”), who is reasonably familiar with Unix and available to the audit team, and Auditor X (“Xavier”), who is a Unix expert with limited availability. Anne can own several responsibilities on the audit team, including working with the Unix control owners. When information is gathered for the purposes of testing, Anne can pass the test materials, with detailed testing instructions, to Xavier. Xavier returns test results to Anne, identifying the nature of any exceptions found. Anne is then able to address testing exceptions with the control owner and control manager. Anne has been freed up for other meetings, the testing has been performed by the most capable resource, and the audit has been expedited. Anne also gains valuable Unix testing experience.

In a situation where younger auditors are mentored, developing the technology skills and expanding the experience base of the audit team is a forward-thinking consideration. Exposure to new processes or technologies is best done when a more senior person is guiding them and able to serve as backup. By doing so, an organization grows its skills from the inside.

Preparing Staff

Staff members are usually informed of the project when an engagement letter is signed. Details on a project’s requirements may be scarce at the time, but management may share with staff a high-level summary of the engagement, the intended roles, and the planned hours per their proposal.

Once resource planning has been completed, it will be clear which personnel will be performing which tasks within an audit. When this occurs, it is important to meet with audit personnel to explain the project and define the roles of auditors during the project. During this meeting, audit management may provide resources or web links for auditors to use when researching the company, and perhaps update their knowledge on technologies they will review. If audit team members will be tasked with technologies with which they have limited or outdated experience, this is the time to consider providing training.

Once schedules are set for resources visiting the client site, guidelines should be provided to audit staff on the logistics of traveling to their location and, if possible, finding acceptable lodging in the area. This is also a chance for audit personnel to raise questions about the audit. Frequently, dress code and other auditee office norms are communicated at this time.

Performing Control Testing

Thorough preparation for testing will reap benefits when the audit team enters the testing phase. It may appear that effort spent preparing is excessive compared to the task still at hand. Preparation’s greatest benefit will be avoiding problems during testing, review, and developing reports. Failure to include a few needed tests may set a team back several days. An error in objectives, scope, or an understanding of the test environment can cause greater problems.

A test plan developed with proper care will provide the audit team with a structure that facilitates effective and efficient testing. Testing is busy enough when it all goes well.

Control testing involves execution of the test plan, consisting of interviews and evidence collection.

Project Planning with the Client

The readiness assessment is now complete, the client has addressed any preparedness issues, and the attestation period is approaching. The primary contact will help the audit team plan and schedule testing. Table A-2 is a quick summary of topics to address at this phase.

Gathering Testing Evidence

The process of gathering evidence should be orderly. Several common ways of requesting evidence are

•  In person

•  E-mail

•  PBC lists

•  Collaboration sites

It is common when interviewing personnel to make in-person requests while discussing the control being tested. Direct requests may involve an auditor sitting with a control owner observing software controls and gathering screen-prints and other electronic information as evidence. An auditor may confirm the availability of policies, procedures, and disaster recovery plans by asking a person to show them where they are located. Direct requests may be made for information to be supplied later. An auditor might follow up a meeting with an e-mail summarizing requested information and what is to be expected.

Table A-2   Project Planning to Audit Project Planning

PBC lists are a tool that brings order to the process of gathering test information, but are best employed when an auditor does not have a need to observe an activity directly. Concise and easy to understand, PBC lists are a tool for client management to track information they are providing to auditors, and for auditors to track what information they have requested and received. PBC lists are ideal for requesting transaction records, data, and process documents.

Depending on the testing being performed, there can be several phases of requests for information supporting testing. The audit team should be orderly regarding requests. If requests are made directly of control owners, the requests should be tracked.

Testing can demand a great deal of time from client personnel, and a client may seek to minimize the impact. When gathering test evidence, the client contact may request that the team limit the frequency of information requests to their staff. Some requests may need to be made through the primary contact or their designee. Clients also may request liberal turn-around times so as not to overburden their personnel.

Sample Testing   Sample testing usually requires multiple rounds of requests. Initial requests are made by auditors for sampling populations, from which auditors will select their test populations. Assuming the sample population information is correctly delivered and understandable, auditors then submit a request for test evidence for the selected sample. Unless test results are extremely successful, there could be requests for follow-up information. Unclear sample test results may require further investigation, resulting in requests of additional records or a live discussion with the control owner.

Security and HR Procedure Testing   Testing of information security often covers setup and removal of user account permissions. To determine if access granting and removal procedures are operating, auditors will involve two key lists:

•  Current employee listing

•  List of employees terminated within a given period

These lists must be dated and should be provided at the same time. Lists should include (at a minimum) employee name; manager’s name, department, or title; hire date; and termination date. Testing surrounding HR procedures (such as approvals) may require these lists as well. They may be requested early during testing or, in some cases (such as for an SSAE 16), after some time has passed within a test period. These lists may also help auditors determine if user account permissions are granted appropriately for their role.

Requests for Follow-up Information

Often, testing doesn’t go precisely as planned. Certain information may be lacking from provided evidence. This may occur when the course of business requires exceptions to standard procedures. An auditor may not learn of all of these possibilities when meeting with a control owner, but instead the auditor may find out later while examining evidence. Auditors usually need to make follow-up requests or have follow-up discussions to determine if a notable occurrence during testing is an exception or is acceptable.

Launching the Testing Phase

The audit team and client personnel are ready for testing, so it is time for auditors to begin. The availability of client personnel may determine the schedule of certain tests. Planning meetings will set expectations as to when auditors are expected to complete their work. Once schedules are set, logistics should be set for getting auditors into the field (such as securing transportation and lodging).

A testing period often starts with a series of kickoff meetings before interviews are scheduled with individual control owners. These meetings serve the purpose of introducing auditors to the control managers and their team of control owners. This forum allows for:

•  Auditors to meet personnel in person before testing launches

•  Auditors to explain the scope of their audit and clarify testing expectations

•  Control managers to frame the event for their personnel and to set rules and expectations

•  Control owners to raise questions they may have about the process

Control owners may be experiencing some degree of nervousness, especially if they have not spoken with auditors before. Auditors will often find testing interviews work more smoothly when control owners have already met the auditor and have a deeper understanding of the scope and objectives of the audit. Gathering evidence will work more smoothly when control owners have had expectations set by management as to what they can expect during interviews and what they are permitted to provide auditors. In addition, explaining to control owners that results of testing will be reviewed with them before they are reported will often help control owners relax when communicating with auditors.

Performing Tests of Control Existence

Tests of control existence determine if a control is in place and operating effectively. They also determine if the control activity occurs. Many controls will pass if a document simply exists or a software configuration setting is confirmed.

Control existence does not imply that a control is operating effectively over time, but whether or not it works when successfully operated. Existence tests are often performed using the following testing methods:

•  Inquiry and corroboration   Auditor questions control owners about controls.

•  Observation   Auditor views a control being performed.

•  Inspection   Auditor reviews material evidencing compliance with stated control.

•  Reperformance   Auditor performs control activity himself.

Many existence tests are performed when accompanying the control owner. Existence testing is a leading method of testing automated controls enforced by software.

Certain controls testing engagements, such as SOX 302 testing and an SSAE 16 Type I, test for a control’s existence. An SSAE 16 reports that at one moment in time these controls are in place. SOX 302 requirements also verify that the controls environment is documented and that the controls have been tested once. In contrast, when the burden of controls testing involves confirming operation over a given period, such as SOX 404 or an SSAE 16 Type II report, control tests for existence may be performed multiple times during the test period.

Automated Controls   Automated controls are tested for existence. Many tests involve confirming that software configurations are set to specific values or observing whether systems enforce a rule.

Automated controls are most often observed in person by an auditor, and the auditor documents the occurrence for testing workpapers. Screen-prints or other captured images provide evidence of software configurations. These images can be accompanied with an auditor’s descriptive text for use as documentary evidence of the control test.

Governance Controls   Governance controls are often tested for existence—for example, policies, procedure documentation, committee meeting minutes, approvals, and other documents can be tested for existence. The provided documents may be subject to additional review per the test plan. If the evidence does not exist, additional test procedures will not be performed and tests will likely fail.

Testing Existence via Observation   Tests by observation are performed by an auditor witnessing whether a control activity is performed. This can involve the auditor looking over the shoulder of the control owner and requesting they perform certain tasks as they demonstrate a procedure, or by viewing software settings within applications. Examples of observation testing include

•  Confirming that software requires and tracks approvals

•  Observing physical and environmental controls for a server room

•  Noting the existence of signed policies and a background report in an HR file

•  Viewing system-generated alerts reaching a pager

•  Reviewing a firewall rule base to confirm settings

Control tests by observation are recorded in writing in testing workpapers, or through retention of screenshots showing configurations. Recording observations should be sure to include the date and time, as well as the full name and title of the control owner. If observations can be supported by documentation beyond an auditor’s recorded observations, this is preferable. Not all client or technology situations will provide such evidence.

Testing by Inquiry and Corroborative Inquiry   Testing by inquiry involves asking questions of control owners. Inquiry is most commonly performed in person, over the phone, or via e-mail. Some standards of testing require corroborative inquiry, meaning two persons with knowledge of a control must make agreeing statements. Discussions are documented, often in memos, and placed in the workpapers.

When documenting inquiry in workpapers or reports, the auditor must take care to phrase any statements by management as “representations” made by the client rather than facts. Wording might read: “Per the Unix system administrator, security logs are reviewed on a weekly basis,” or “The CIO represented that spending is reviewed monthly.”

The audit workpapers will include the record of the conversation and a memo addressing the result of the test.

Testing Existence by Inspection   Inspection is performed by reviewing the content of client-provided evidence. Testing may seek to analyze the nature of information discovered within the material being tested, which could be a report, policies, meeting minutes, or other document. Testing by inspection might seek to determine whether:

•  Forms reflect the proper approvals signatures

•  Data backup software schedules match documented schedules

•  Committee meeting minutes reflect management’s discussion of the log file review

•  Network and data flow diagrams are dated and current

The audit workpapers should include a memo addressing the results of testing, which identifies the inspected documents and copies of the documents. Any exceptions should be marked in the document and addressed in the memo.

Testing Existence Through Reperformance   Auditors may test security controls and automated controls via reperformance. In this type of test, auditors are taking information that is input to a control and performing tasks or calculations on their own to see if they achieve the same results that the control does. Examples of reperformance testing include

•  Attempting to set a password that is noncompliant with policies

•  Confirming VPN authorization is required and no guest account is enabled

•  Checking whether specific employee key cards permit access to a restricted area

•  Reproducing report values from business rules and raw data

The audit workpapers should include a description of the reperformance test. Workpapers for a reperformance test may then resemble other forms of testing, such as observation (VPN example) or sample testing (transaction records example).

Control Existence Failures   One possible result of testing is that a control being tested is not implemented. This is often discovered during initial discussions with control owners or during a procedure walkthrough. This can occur for several reasons:

•  Documentation of controls has not been updated for new procedures. Effective controls may exist, but they are not included in testing programs.

•  Changes in personnel resulted in a lack of ownership of the control process.

•  Controls were documented as “to be implemented,” or implementation was not successful.

When controls are found not to exist, they should be brought quickly to the client’s attention. If it is possible (or prudent), validate the absence with a control manager before elevating the issue to the primary contact. A control manager may be able to clear up the confusion and prevent embarrassment, such as identifying a compensating control that could be tested to support audit objectives.

Performing Testing of Control Operating Effectiveness

Tests of control effectiveness confirm that the performance of a control activity has been successful. Audits most often test control effectiveness over a defined period. In order to test for successful operation, evidence needs to indicate that a control repeatedly occurs correctly or appears to occur without interruption.

There are several methods of testing operating effectiveness, including inspection, reperformance, sample testing, continuous auditing/monitoring, and automated testing methods (not to be confused with testing of automated controls).

Testing Effectiveness by Inspection   Inspection can be used as a test of effectiveness as well as existence. Inspection can reveal that evidence indicates continued operation of a control activity. To test for control effectiveness, inspection is often used to review system log files or similar reports. Some examples of how inspection methods are employed effectively for testing of reports or log files include

•  Confirming log entries were captured without interruption

•  Reviewing changes to administrator passwords, confirming they are periodically changed in accordance with policy

•  Reviewing changes to key settings, confirming that logs were not turned off and that settings were not changed at times during the testing period

Inspection may be employed as a method within sample testing as well. Testing may produce documents that are then tested by inspection.

Testing Effectiveness Through Reperformance   Effectiveness testing using reperformance can involve an auditor reproducing control activities performed by clients. This method can allow an auditor to confirm that controls have been operating correctly because they can re-create the control activity themselves. Reperformance can involve recomputing figures or confirming the reported values from source data. Some examples of testing by reperformance include

•  Comparing a list of active system users against a list of terminated employees to see whether terminated employees’ accounts were removed.

•  The control owner performs a reconciliation on reports provided by two different systems and initials the documents before processing. To test that the reconciliation was performed correctly, an auditor reviews whether the figures match on a sample of initialed documents.

•  A billing application generates reports on hours worked by querying a database of hours worked on each project. An auditor seeks to validate that report totals are accurate. This data is imported into a database, and the auditor runs queries to attempt to reproduce the values generated by the billing application.

If the auditor is using a database or spreadsheet application to manipulate the data, workpapers may include evidence in electronic form. A standard, noneditable format (such as a CD/DVD-ROM) is preferable to rewriteable media. Workpapers should contain a memo discussing the testing methodology and identifying how and where (in the workpapers) the information is stored digitally.

Sample Testing   Sample testing is conducted when a control is performed on a regular basis and a record of the control activity exists. An auditor will select a population of control activity records to confirm that the control is being performed correctly. An auditor must review the population of control tests and then request the evidence supporting elements of that population. This often involves two rounds of requests from a control owner to gather a sample population and test evidence.

Auditors may select sample populations by random test sets, or they can judgmentally select a population from the test set. Judgmental selection is often helpful when the materiality or criticality of transactions varies. Auditors might select transactions that relate to a company’s largest clients or most critical systems. Judgmental selections may also seek to test a variety of transactions. Auditors might include different kinds of clients, or from applications hosted on different systems.

When an auditor performs sample testing, the audit workpapers should include

•  A written description of the testing process, including a discussion of how sample selection was performed and what sampling criteria were used

•  A record of the sampling population

•  List of the test set population

•  Test results

•  At least one example of a successful test and each exception (if all test materials are not retained)

An auditor’s workplace may have policies covering workpaper documentation requirements. Testing sometimes includes lengthy printed reports, and some places permit audit management to use their judgment regarding when retaining a sample of the report is sufficient evidence.

Continuous Auditing/Continuous Monitoring   In instances in which the volume of data is very large or the frequency of transactions is high, organizations may choose to utilize tools to perform continuous audits. This type of approach can be done with varying frequency (hourly, daily, weekly, monthly) based on the risk, volume, and type of data, and incorporates the use of technology to perform an analysis of data/information. This analysis is designed with a level of precision such that anomalies in data are identified and reported upon for further detailed follow-up. Continuous auditing can both provide a greater level of assurance (since high-powered computers can audit 100 percent of a population in much less time than an IT auditor can audit a sample) and make the audit program more efficient (as IT auditors are freed up to perform additional audit tasks, such as follow up on continuous audit findings, perform tests where the population or control type does not align with continuous audit techniques, etc.).

In instances related to testing IT general controls, technology may be used to continuously monitor items like security provisioning. In that case, while there may not be a significant amount of activity over a period of time, tools can be put in place that will send an alert to a responsible party immediately if an unauthorized or inappropriate change in access is made. As mentioned, the use of a tool like this can provide a high level of assurance and security, while at the same time allowing an organization to spend less time on periodic access reviews.

Continuous auditing should not be confused with automated testing or testing programs, described next. While those techniques involve the use of technology, that technology is generally utilized at a particular point in time to assist in the performance of audit procedures.

Automated Testing   Automated testing can quickly provide an auditor with large amounts of critical system information. It often involves the use of testing programs (sometimes off-the-shelf) or test scripts. Test scripts may be designed by the audit organization, the software’s manufacturer, or a third party. Running scripts should only be done in close cooperation with a control owner, such as a database administrator or operating system administrator, and approved by appropriate client personnel.

Testing programs and scripts may need to be run on a client’s system. Many organizations will require that a script or testing program be reviewed before auditors are permitted to run it on their system. A client’s review will usually confirm that a script is only executing inquiry commands. Special user accounts may be enabled temporarily for test programs to run. The use of automated testing should be brought up early in conversations with the client, as approval for their use may take some time.

Testing Programs   Testing programs will often be developed by a software vendor. To assist a client’s review of a software program, the client can be provided with the software make and version number, and, if possible, the auditor can provide software documentation.

Examples of automated testing programs include

•  Penetration testing

•  Segregation of duties analysis

•  A program that runs on a network to identify all workstations connected to the domain and to confirm that antivirus definitions are current on all workstations

•  Network analysis programs that review network components and the protocols enabled on network devices

Testing programs often provide organized reporting of the results of testing. An auditor’s test consists of reviewing output reports from the test and recording the observations in a document. In addition to the output report and a record of an auditor’s testing, the workpapers should include details on the testing program used, including name, manufacturer, version number, and relevant configuration settings used. The results of analysis are then recorded in the testing matrix.

Test Scripts   Scripts are programs that are run on systems and that usually generate a file showing the commands executed and the results. Scripts are written in scripting languages (such as SQL), which execute commands sequentially. Scripts can run a query and write query results into an output file. Test procedures will involve reviewing scripts’ output files.

When a script is employed, an auditor should observe management running the script and collect the evidence without delay. The audit workpapers should include

•  Information on the system being tested, including version, current patches, and relevant configuration settings

•  Text of the script itself, plus any information on the publisher, name of the product, and version number

•  Output report from running the script

•  Results of the auditor’s review of the output report

Scripts designed for certain technologies have the issue of expiring as the technology becomes obsolete, so infrastructure is required to keep them current. Unless an auditor is an expert in a given scripting language, he should avoid writing or editing scripts. Automated tools are more common with large audit organizations and cutting-edge audit shops.

Discovering Testing Exceptions

Auditors will face problematic test results, and will need to determine the nature of any exception. It may not be clear whether an unexpected result equates to a test failure. Despite preparation, an auditor may not immediately understand test results. A clear understanding of the procedure, and the role of the control within that procedure, is needed so that the auditor can determine if a nonstandard test result amounts to an exception, or simply a test that was run incorrectly. If an auditor reports prematurely on findings, it can lead to challenges with the client relationship and loss of professional integrity for the auditor. Auditors should attempt to confirm an understanding of how the test results amount to an exception.

An auditor should communicate an exception with audit management first. Audit managers will confirm their understanding as to why it appears to be an exception. Then the control manager and primary contact will be informed—perhaps at regular update meetings, or more immediately if the finding is highly material.

When an exception is confirmed, it is documented in test matrices and the workpapers. If possible, the description of the exception should accompany the source document, including where it was identified. If the test was discovered in data testing or images of a screen, the evidence should be captured electronically (and perhaps printed), and a written description should be included.

Reporting exceptions in audit reports usually includes reporting to the client the residual risk and making recommendations.

Issues Requiring Follow-up   There can be several reasons why an auditor must return to the control owner for a clear understanding of test results and exceptions. Most often, this will be because a procedure or a control is not completely understood, possibly because control documentation is insufficient or outdated. When following up with a control owner, if e-mail or a short conversation is not possible, it is often best to inform the primary contact and control manager that you will require more time with control owners.

After completing follow-up with control owners, any clarifications the auditor acquires should be documented in the workpapers as support for the results of testing. In addition, procedure and controls documentation could require updating. If follow-up confirms the exception is valid, it must be documented as such.

Confirming an exception with the control owner and control manager should prevent disagreements when an audit report is presented to a client organization.

An example requiring follow-up involves network security. Upon an initial review of the client’s firewall rule base, the firewall appears to permit traffic that includes less secure protocols. The auditor does not yet know if this is an exception. Further inquiry is required for the auditor to learn whether compensating controls address the residual risk:

•  Traffic could additionally be managed by a router, isolating it to specific servers.

•  An intrusion prevention system (IPS) may be employed to isolate traffic that introduces security issues into the network.

•  Valid business reasons may have led to management accepting the risk of permitting this protocol. Staff may regularly monitor the activity.

Auditors may be told that management is aware of and accepts the risk of these protocols. Auditors would document management’s communication in the workpapers, perhaps with an e-mail printout or a memo recounting the conversation.

Discovering Incidents Requiring Immediate Attention

During the course of testing, an auditor could discover information that requires immediate attention by the audit team, client management, or both. An auditor has a professional duty to be aware of possible indicators of illegal activities, fraud, hacking, or other improper actions. Situations that could require immediate attention include

•  Fraudulent evidence provided by client personnel

•  Discovery of critical vulnerabilities that compromise the computing or network infrastructure

•  Improper or fraudulent actions on the part of client personnel

•  Manipulation of financial figures reported to internal or external parties

•  Requests made to auditors that could compromise the integrity of the audit process

An auditor must make sure they correctly understand the nature of the discovery. In many instances, auditors will not have a problem consulting with control owners to confirm their understanding; however, situations such as fraud could require a high degree of confidentiality if a proper investigation is required. If the audit team together is still unclear on how to handle a situation, it is best to consult certified professionals on how best to proceed.

Discovery of Fraud   If an auditor uncovers evidence of possible fraudulent or criminal activity, it is urgent that they address this with audit management. They must begin thoroughly documenting all communications and should write a description of how they made their discovery. The auditor should confidentially consult with fellow auditors to confirm their interpretation of the evidence. If the team suspects a possible issue, they must then decide how to proceed.

If there is agreement among auditors that evidence appears to show fraudulent or criminal activity, the audit team will need to notify appropriate client personnel of the incident. The audit team should consider the nature of the incident and carefully consider which members of client management or the governing board are the most appropriate to inform.

In small to middle-sized organizations, the audit team may consider informing the CIO, CFO, legal counsel, internal audit director, or other members of executive management. In larger organizations, a level below executive management may prove more appropriate. In extreme situations, the chair of the audit committee or other governing board might need to be informed first. A meeting should be set up to discuss the evidence and see how the client would like to proceed.

Handling Evidence of Fraud or Criminal Activity   If the auditor possesses documentary evidence of a potentially fraudulent activity, they will need to isolate the document and establish a chain of custody for the potential evidence should criminal investigation proceedings occur. This appendix does not claim to be an authoritative resource on procedures to follow in the event of discovering fraud. If fraud is discovered, a professional investigator should be consulted immediately for guidance in these situations. This person will advise the audit team on how to act until an investigator is able to take over the chain of custody for evidence and continue the investigation.

An auditor may be required to handle evidence relating to the discovery of fraud or criminal activity. In these situations, the auditor must secure evidence and establish a chain of custody. It is preferable to establish the chain of custody of evidence following the instructions of a certified professional. A certified professional may not be able to assume the chain of custody of evidence the day it is identified, so an auditor may be required to perform certain actions. The following points are important to consider when developing a chain of custody of evidence:

•  Do not make any marks or additional marks on the evidence or disfigure it in any way (for example, don’t punch any holes in it to insert it into a binder).

•  Begin an evidence log spreadsheet that will track the location and possession of the evidence. The spreadsheet should be constructed to track the following information:

•  Evidence log number.

•  Date and time evidence was received from the source.

•  Information on the source of the evidence, including person, source system, report names, and other details.

•  Name of person submitting evidence to custody.

•  Date and time evidence is entered into the evidence log.

•  A discussion of the information located within the documentary evidence that may indicate fraud.

•  The method of storage of the documentary evidence. Evidence should be stored behind locked doors or in locked cabinets when not in the custodian’s direct possession.

•  The name of the person responsible for keeping documentary evidence secure and in their possession, and the time and date when they accepted possession of the evidence. If the security of the evidence is transferred between members of the audit team, be sure to record the date and time when this transfer of ownership occurs.

•  If the evidence is a piece of paper that has information on only one side, some parties advise that one may write the following on the back of the page: document number, date, and the source of the document, and sign and date it.

•  Place the evidence into a tamper-evident envelope.

•  Lock the evidence in a locking cabinet or safe.

If anyone on the audit team has experience with fraud investigations, they should oversee this process, and as soon as possible a certified fraud investigator should be consulted. When a certified fraud investigator arrives, auditors will formally hand over custody of any documentary evidence and the evidence log to the investigator, and record the transfer of custody in the evidence log.

Improper Actions by Management   Management may behave improperly during the course of the audit, and client personnel may fear for their reputation with client management when auditors are around. Client personnel may not act appropriately. It is important for an auditor to maintain professional composure when client personnel act inappropriately. Less serious issues of inappropriate behavior can be addressed between audit management and the primary contact. More serious issues may require a meeting between control managers and audit sponsors.

Certain improper actions may be severe enough to interfere with the execution of the audit, such as:

•  Refusing to provide test evidence

•  Providing fraudulent or “doctored” evidence

•  Requesting audit personnel act inappropriately

•  Threatening audit personnel

Violations at this level will require action by the audit team. Auditors should document and communicate the incident immediately to audit management. Audit management and client management should then meet and address the issue.

Certain improper actions by management could strongly affect the audit execution and the final report. In some situations, refusal to provide evidence or providing fraudulent evidence can be reason for the audit team to cease performing an audit.

Materiality of Exceptions

The auditor employs judgment to assess the materiality of exceptions. During testing, an auditor will determine if the exception is serious enough to warrant the immediate attention of management or is a discrepancy of limited consequence. The question of materiality is addressed partly in the testing matrix, when an auditor assesses residual risk for a control that did not pass.

The materiality of controls and control failures is discussed in more detail in Chapter 3.

Assessing Residual Risk   After the nature of a control failure is confirmed, the auditor can assess its residual risk. Certain failures will introduce a relatively limited amount of risk, such as identifying low-access accounts that are not compliant with a password policy. Other failures can be highly material and require immediate attention. An example of a material failure could be the following:

Auditors are performing tests of network security controls. When testing a requirement that employees use VPN for external access, auditors learn that RDP (Remote Desktop Protocol) traffic is permitted through the firewall and enabled on workstations. Inquiry reveals that a number of IT personnel work from home and that they access their workstations remotely using RDP. The residual risk is that unencrypted traffic, including authentications, is permitted through the firewall and is potentially visible to third parties. This is clearly a material breach of compliance with policies and compromises network security controls.

Examples of how different exceptions from the same test could result in different levels of residual risk are depicted in Table A-3.

Table A-3   Different Kinds of Exceptions and How Residual Risk Is Evaluated

In Table A-3, the high-risk example would definitely be brought to the attention of the report’s audience; however, it would be up to the auditor whether to bring medium- and low-risk exceptions to the attention of management.

Categorizing or Ranking Exceptions   In some reports, such as internal audit reports, the severity of exceptions may be weighed. Categorizations of materiality can be selected, such as:

•  Ranked most to least important

•  Linear, such as low, medium, and high, or on a scale of 1 through 10

•  Stoplight: Green (controls operating or compliant), yellow (requiring attention), red (controls not operating or noncompliant)

Several parties can use weighted exceptions:

•  Auditors may choose to present only the most material exceptions to the governing entity. These exceptions are again ranked by importance to the governing entity. Exceptions of lesser materiality are addressed with management.

•  The IA department can use the ranking when scheduling retesting of failed controls.

•  Management may use the ranking to prioritize their remediation plans.

Weighted results lend effectively to diagrammatic representations of the residual risk. This helps management better understand which residual risks are the most important.

Developing Audit Opinions

Auditors will develop opinions on individual control tests, control objectives, and at times, an audit as a whole. Reviewing test results and developing an opinion can be performed after all testing is complete, any follow-up with the control owner has been performed, and the test evidence is ready to be documented in the workpapers. In the event testing revealed exceptions, the control owner and the auditor will have already agreed to the facts relating to a performed test. An audit opinion is entered into the audit testing matrix.

An auditor will conclude his opinions on all controls supporting a control objective before developing an opinion on a control objective.

Management Representation Letter

When certain licensed professionals are performing an external attestation, it is sometimes the practice to have client management sign a “Management Representation Letter.” A Management Representation Letter states that client personnel have provided truthful information throughout the audit process. By signing, management takes responsibility for information provided to auditors. This provides an auditor with a degree of legal protection in the event the report’s contents are subject to litigation. When this practice is followed, audit checklists require this letter before delivery of a signed report to the audit client.

Control Activities

Developing opinions involves weighing the materiality of the exception, the residual risk, and the impact on control or audit objectives. Opinions on control activities will most often take the form of

•  Control passes

•  Control passes with observations

•  Control passes with notable but not material exceptions

•  Control fails due to exceptions

•  Control fails because the control activity is not performed

Tests that merely “pass” the planned test procedures are easy to handle. Passing controls are entered into the testing matrix and reflect simply that the test passed. There is no burden for the auditor to develop statements of residual risk or recommendations.

Testing observations are generated when a situation is uncovered during the course of testing that relates to the control environment. Inquiry, for example, may reveal control weaknesses outside of the test plan scope. An auditor might also identify possible improvements to the control structure, or note that certain procedures are not always performed consistently.

When an audit test does identify exceptions but the audit team considers the materiality of the exceptions to not constitute a control failure, the control may pass, but the exceptions may be brought to the attention of management. For example, an Active Directory control states that no shared accounts are permitted on the network. Inquiry confirms that setup of shared accounts is not permitted, but a review of accounts identifies several old shared accounts, though follow-up reveals the access granted to these accounts is appropriately limited and that the accounts were not used by personnel.

Testing a control activity could result in failure before all supporting tests are performed. In these situations, an auditor will need to decide whether to curtail any more testing of that control and conserve effort. An auditor decides whether additional tests of that control will provide any benefit to the client and whether the opinion of the control objective or the audit as a whole is influenced by these control activity test results.

Control Objectives

An auditor opines on control objectives once the opinions of all supporting controls activities have been documented. Auditors will determine how the results of controls testing support the control objective. Opinions on control objectives will state that controls pass with notable observations, or fail. An audit’s methodology may provide guidance in determining if a control objective passes or fails. Audit methodologies might provide guidance such as:

•  Each passed control objective must be supported by at least one substantive test confirming the effectiveness of supporting controls.

•  A single control failure shouldn’t fail a control objective unless the control is a key control or the auditor documents a clear justification. (Individual audit organizations each have their own tolerance for addressing control failures and any resultant additional testing that can be performed to determine whether a failure is an isolated incident.)

•  A failure of two or more control activities should fail a control objective, unless an auditor documents a clear justification of their determination.

If auditors are weighing a control objective, when testing failures exist, auditors must consider how the control objective supports the audit’s objectives. In the course of developing an opinion on a control objective, auditors may determine that evidence is inconclusive and that additional testing is needed before a determination can be finalized.

Final determinations on control objectives should be documented in the workpapers and approved by audit management.

Audit Opinions

Certain reports will deliver an opinion on the audit’s results. An SSAE 16 Type II report will attest to whether controls appear to be operating effectively. To develop an audit opinion based on control objectives, the opinions on control objectives will be weighed against the audit’s objectives. Audit opinions typically pass, pass with qualifying conditions, or fail. A statement supporting this determination should be approved and entered into the workpapers.

Developing Audit Recommendations

At the conclusion of an audit, it is common for an auditor to provide recommendations on improvements to a client’s control environment. This can be done formally or informally.

Formal Recommendations

Auditors can provide formal recommendations to the client. This could take the form of internal audit reports or a letter accompanying the audit report. Recommendations included in a report are frequently accompanied by management’s responses and possibly remediation plans.

Formal recommendations are carefully worded. Auditors must be careful to advise that management take action to remediate a weakness, avoiding statements that may appear that auditors are making decisions on the part of management. Wording is commonly along the lines of “management should consider” performing a certain action. Auditors may avoid highly specific recommendations, because there are often several options on how a control weakness can be mitigated and clients may argue specific methods rather than agree on a need to fix controls.

Informal Recommendations

Auditors often discuss informal recommendations with control managers during the course of testing. It is constructive for both the auditor and control managers to discuss improving an organization’s controls. Executive management, such as an audit sponsor, may have some pointed questions they would like to ask auditors at the close of an audit. Similar to formal recommendations, auditors should phrase responses with care and maintain impartiality and professionalism in their answer.

Managing Supporting Documentation

In the process of auditing, an auditor will handle a significant volume of audit documents, process documentation, and testing evidence. Having a complete set of documentation is necessary to ensure integrity of the audit process. In addition, having clear and organized documentation will benefit the execution of the audit. Some of these benefits include the following:

•  Auditors will be able to answer detailed questions about test results.

•  Audit managers will be able to confirm work is being done to expected standards.

•  For auditors in accounting firms, the audit may be subject to a peer review, where other firms review a project’s report and its workpapers and scrutinize the audit. An accounting firm may also be audited by the Public Company Accounting Oversight Board (PCAOB), in which case the auditor will be required to show all of the documentation that has been collected during the audit.

It is important that the format of testing documentation be clear when staff auditors begin testing. Auditors also need to make sure that documentation keeps up with the progress of testing. Memos recording conversations should be inserted promptly into workpapers.

During the course of the audit, it is often convenient to manage testing evidence in binders or electronic folders that are separate from audit documents.

Complete audit documentation will often have most or all of the following sections:

•  Table of contents

•  Engagement documentation, such as the signed engagement letter

•  Memos providing direction on understanding documentation

•  Meeting minutes and memos from meetings between auditors and client personnel

•  Procedure documentation and background information on key systems

•  Testing matrices and lead sheets

•  Supporting documentation generated during testing

•  Checklists for audit completion

•  Testing methods used

•  Sample guidance

•  Document review

Storing Electronic Documentation

It is increasingly common to retain as much as possible in the form of electronic versions of documentation. Electronic documentation can be stored as files within a file system, such as on a shared network, but it is sometimes managed by supporting software.

In some engagements, outside auditors provide a copy of audit workpapers to the client. Sharing electronic versions of documentation can make this easy to do. When delivering electronic documentation, the auditor must confirm the preferred media and schedule of delivery with the client.

Electronic storage can introduce challenges if it is to be archived for a specific period. Storage media preferences evolve and could render documentation difficult to access if it is stored on outdated media. Another consideration is that certain media has a limited shelf life. Data that is to be stored for extended periods (generally, for more than five to ten years) may need to be periodically rearchived onto newer media to ensure that the data can be retrieved if needed.

Storage of data used within audit management software runs the risk of becoming difficult to access in the distant future. Using software can be expensive, and an organization could face the risk of software compatibility issues once the systems that generated the archive files have been upgraded or replaced, or are no longer supported.

Lead Sheets

One common practice for organizing testing documentation is to begin a documentation section for each control with a “lead sheet.” Lead sheets often contain similar information to what information was tracked in the testing matrix. Lead sheets provide the reviewer with a tool for following the testing process and understanding the accompanying documentation. Lead sheets can also capture a reviewer’s sign-off.

Lead sheets typically end with the result of testing clearly stated, as in Figure A-7. Evaluations of residual risk, auditor opinions, and recommendations will often be tracked in separate tools, but this is not necessary.

Figure A-7   A testing lead sheet contains comprehensive information on the control and the testing performed.

Delivering Audit Results

An audit’s fieldwork has been completed, and control owners and managers have agreed to the facts of testing. Before delivering audit results, the auditor updates all matrices with test results, residual risks evaluated, and draft versions of audit opinions written. The next task is to compose results into presentable form.

Typical presentations of audit results can include

•  Delivery of a formal report for review by the audit sponsors and other management

•  Presentation of the report to the organization’s audit committee of the board

•  Publication for distribution to and review by third parties (such as SSAE 16 audits)

•  Memo or letter summarizing audit results and reporting issues to control managers and/or audit management

If IS audit procedures are a subset of a larger audit, IS audit results are communicated to audit management for inclusion in their report.

Discussions with audit sponsors and the primary contact will have clarified what a client expects for their deliverable at the beginning of the audit. The degree of formality will vary depending on client needs and the company culture.

Contributing to Larger Audit Reports

When an IS auditor is working for a team that includes non-IS auditors, it is frequently the case that audit management does not have an IS audit background. In this situation, the IS auditor may have a stronger understanding of the materiality of certain test results and the residual risks than does the audit manager responsible for writing and delivering the report. This will require close coordination between the IS and non-IS auditors, which can be a challenge toward the end of an audit when time schedules are tight and several tasks are being pushed to completion. Ensuring that audit results are communicated correctly in final reports and during presentations can be a challenge. If audit reporting over- or under-represents the materiality of information systems risks, it could have consequences to the audit relationship and the reputations of control managers or auditors.

IS auditors should make sure they participate in reviewing the report’s representations. The audit manager should be receptive and allow this participation. This situation may be delicate, because the audit team is balancing the success of its relationship with the audit manager with the needs of the audit organization and the client.

Audit Report Contents

Frequently, either the audit organization or the client organization will provide an audit report template. At a high level, a report communicates scope, procedures, and conclusions. In audits involving numerous auditors, several parties may contribute content. Coordination between the report writer (frequently the audit manager or senior) and auditors who performed testing is standard during this time. The report writer will confirm with the field auditors that their statements correctly represent audit activities.

An audit report often includes the following:

•  Standard language addressing the parties involved, audit scope, and auditing standards invoked during the audit

•  Rules regarding the distribution of the report

•  If applicable, the audit opinion

•  If applicable, statements by client management regarding company history, financial activities, processing activities, and related controls

•  Audit objectives

•  Controls to be tested and how they support control objectives

•  Test results and conclusions from testing, including exceptions, and evaluations of materiality and residual risk

•  Auditor recommendations

•  Client-provided feedback on exceptions, conclusions, or recommendations

During this process, audit staff and management review how to appropriately present the audit results. Younger auditors stand to experience mentoring, corrections, and rework during this phase.

Audit Management and Staff Disagreements   A staff auditor and an audit manager may draw different conclusions from test results. For an audit opinion to be defensible by audit management, the report and supporting detail must agree. During report writing and review, as audit management becomes familiar with test result details, the practices and judgment of a staff auditor may face criticism. Disagreements can arise surrounding the assessments of the nature of an exception and the residual risks.

When a staff auditor concludes differently about an exception’s residual risks than does audit management, a discussion must follow. The following are several scenarios where disagreements may occur:

•  A staff auditor may lack a thorough understanding of specific technologies or client procedures, and may draw incorrect conclusions from test results.

•  Staff auditors frequently have more familiarity with a procedure than does audit management, so their conclusions may consider as-of-yet-undocumented knowledge of the situation. If an audit staff person does not thoroughly articulate the reasons for their conclusion, they may omit information critical to audit management’s review. Audit management may require augmenting documentation when they come to understand the reasons behind their judgment.

•  A staff auditor may be younger and less experienced with a technology than the control owner, and may prove impressionable in the presence of an “expert.” A young staff auditor’s judgment may be affected by representations of the control owner that an exception is not serious. Staff auditors may require mentoring on the nature of the exception and related risks.

Audit management may need to engage in discussions with the client regarding the exception. Additional procedures may be required, such as increasing test set size or testing a compensating control. This may prove frustrating to the client and audit team, as both client and audit team personnel are not expecting additional burdens on their schedules. This situation should be preventable through clear communication during audit procedures and the regular oversight by management.

Writing the Report

Once test results are clear, writing the report should be a rather straightforward process. Most audit firms or IA departments have developed reporting templates, though in some cases clients will provide a report template. When no standard template is provided (or expected), it is common to select a report from a similar engagement and benchmark. When developing the shell of a report, it is common to identify within the shell the parties responsible for providing content for the different sections.

A report can be drafted in sections, even as testing progresses. As soon as a test has been captured fully in the workpapers, the section of a report relating to that test can be drafted for inclusion. It is common for the greatest amount of writing to be done to account for exceptions.

Attention to these few practices will deliver effective audit reports:

•  Write things in the clearest way possible. This can mean breaking up run-on sentences into single statements and saying things with the fewest words.

•  It is often preferable to refer to control owners by their title. The person can be listed with their title as a participant. A report will be meaningful to more people when individual names are avoided, and it will serve as a more effective tool for future management and reviewers.

•  If multiple writers are contributing report sections, employing uniform language throughout the report is essential, but can prove challenging. If dissimilar language is used, more time may be required during review. It is handy to provide writing examples upon which the auditors writing each section can benchmark.

The testing matrix plays an important role when writing the report. Some reports will present results from the full testing matrix, such as SSAE 16 reports. Internal audits may focus reporting on the most material exceptions.

The report is not complete until the workpapers and testing matrix have been reconciled to the report detail. This means evidence in the workpapers is organized and supports the content of the report.

Management Response

When the conclusions of the audit report are clear to the audit team, the team sits down with the primary auditee contact, and possibly control managers, and delivers the results of testing. Presentations should share draft language of exceptions wording and audit opinions from testing, which will be included in the final report.

Some report formats seek management’s responses to the results of testing. Client management will review the wording and the nature of the audit findings and compose responses. The auditor will include these responses in the final report.

Discussing the Auditor’s Wording

Client management may wish to discuss the wording of exceptions or opinions. There may be situations where the wording of the audit opinion could trigger confusion for the auditee organization. So long as a rewording doesn’t affect the message or the nature of the opinion, audit management might agree to adjust certain wording.

Examples of word changes include the following:

•  Data backups have been a persistent problem, and the control objective of “daily incremental, weekly full” backups is not being met. Configuration problems existed within the backup software. Because resources were busy on other critical projects, the configuration problems were not addressed. IT management argues the successful weekly full backup is a mitigating factor not mentioned in test results or the opinion, though they agree the control failed as worded.

•  Partway through the audit period, an organization replaced a system that had been problematic from a controls issue. Per testing guidance, testing at different points during the period reveals material exceptions. Management impresses the point that the situation is now controlled effectively and argues test wording does not fully reflect the effectiveness of their current and ongoing controls.

Auditors will have to consider these situations carefully. Client management may be seeking to avoid embarrassment for any exceptions, and an auditor must take special care not to give in to pressure or suggestions that result in misrepresenting exceptions or residual risks. If an auditor is unsure whether a change is appropriate, they should consult with higher levels of audit management regarding management’s requests.

Management’s Responses to Auditor Recommendations

Reponses from client management about audit recommendations usually involve one of three responses:

•  Management will perform auditor-suggested remediation.

•  Management will seek an alternate solution to their control weakness.

•  Management believes no remediation action is necessary and assumes the risk of control failure.

Management may address the audit opinion in their written response to defend their position.

The auditor will then review management’s responses and develop his own opinion on management’s action plans. The auditor may believe that the response is appropriate and agree with management about the plans, but this is not always the case. When management responds by saying no action is necessary, an auditor may be compelled to report this inaction to regulators or governing entities, or include in the report that management’s responses do not satisfy the auditor’s own concerns about the risks involved.

Report Audiences

The audience of an audit report will be defined at the beginning of the audit engagement. The report should be written with language and at a level of depth appropriate to that audience.

Reports to Audit Committees   Reports presented to audit committees are written to communicate at a high level. Audit committees consist of members of the board of directors, who are usually removed from the day-to-day operations of an organization. Their main concern is whether there are problems that they need to be aware of.

Frequently a report will start with a high-level “executive summary.” Language in the executive summary must be clear and direct without diving into details. For subjects warranting more thorough attention, additional pages provide supporting detail so interested members can “drill down” further into a subject. Auditors will make sure that the issues of greatest concern get the attention of the committee.

Audit committee reports often make a point of:

•  Clarifying the scope of the audit

•  Drawing attention to significant issues

•  Communicating management’s response to issues

Reports attempting to describe mid-to-low-level operations are generally too detailed for the needs of an audit committee and likely to lose their attention or, in the worst case, confuse members. Care must be taken not to introduce confusion, as this will waste valuable time and might complicate the auditor/client relationship.

Audit management will frequently attend the meeting to present results and field any questions the committee may have about the report.

The report typically will be provided to committee members several days in advance of an audit committee meeting. Auditors will need to be aware of the committee’s deadlines and procedures for submitting reports on time.

Reports to Client Management   Client management has a high level of interest in the detailed results of an audit. Reports written for management will contain operational-level information with which they are familiar. The report’s audience will include control managers and their directors. It is important that information in the report be accurate to ensure the report is deemed credible. Within the client organization, there may be parties with motives to discredit a report or the audit organization, so an auditor’s diligence in presenting undisputable facts will protect their reputation.

Reports to management will often contain information on the results of each test performed. Exceptions will be addressed with more detail, and auditors will present their assessment of residual risk and will include recommendations.

In audit situations where management is tasked with providing their own responses addressing exceptions and recommendations, a report’s completion can be held up by management delaying in providing their response to the audit. Auditors may need to engage their primary contact and perhaps their audit sponsor to ensure management delivers timely responses.

Frequently, when an audit is being presented to an audit committee, a second, more detailed report is issued to client management. Composing a report to management before drafting the report to the audit committee will ensure that all communications with management are complete.

Depending on the organization, a senior executive may ask to sit with auditors and control managers to discuss the detailed report’s findings.

Reports to Third Parties   Certain engagements involve presenting testing results to parties outside of the client organization. A few examples of such audits are the following:

•  Organizations that provide services to third parties will hire their own auditors to publish an audit report. A service organization thus avoids opening their doors to auditors hired by their customers. The final report is provided to customers and their auditors. An example of this is the transaction-based SSAE 16 reports that are governed by the American Institute of Certified Public Accountants (AICPA).

•  When two organizations agree to a partnering relationship or are considering a merger or acquisition, one party may request the results of an audit of the second party.

•  Banks and other parties extending credit to an organization may require a set of procedures be performed by auditors as terms of lending.

Reviewing the Draft Report

Once the report is a final draft form, it is subject to review by the head of audit management. They will proofread the report for correct language and will tie the report back to the testing matrix and supporting workpapers. When a certified professional is signing a report, they will go through this process with great care before putting their name on the report. Reviews often include reviewers initialing sections as complete during the review, such as initialing lead sheets when review of testing documentation has been completed.

Ideally, a review will go well; certain points of feedback will be delivered to improve the report, testing matrix, and workpapers; and the audit team will have a limited number of points of cleanup: opportunities for feedback and retesting may be required.

In lengthy audit projects, it may also be appropriate for auditors to conduct periodic reviews of audit results throughout the audit, instead of waiting until the end of the audit.

Signed Reports   Certain reports must be signed by a certified professional. For example, an SSAE 16 audit report must be signed by a CPA (certified public accountant). The certified professional will have taken care to make sure the audit process has complied with the standards set out by the certifying organization. The audit will have followed guidelines for documentation, review, sample size, internal review, and perhaps other areas. Checklists may be employed that must be completed before an audit may be signed.

Internal audits performed by outside parties may request that the audit report be signed by a certified professional or specialist. Client management can have reasons to seek a certified professional’s sign-off on specific testing or internal audit reports.

For example, a company has had challenges with their fledgling internal audit department. Another company is seeking to partner with the company, but has concerns, and places conditions on their partnership agreement that an outsourced certified internal audit firm provide a signed report.

Delivery of the Report

Audit management will clarify with the client how they would prefer to have the report delivered. Depending on the engagement, a report may be delivered during a closing meeting. Some clients expect a bound hardcopy report, while others will be happy with a printout or electronic file. (This information should be agreed upon with the auditee at the commencement of the audit.) Reports may be delivered at a formal occasion, such as a closing meeting or an audit committee meeting.

Reports delivered to management often will be provided to specific personnel. Audit management will get a distribution list from the primary contact and provide sufficient copies.

Service providers often need to make their audit reports available to their customers’ auditors. Usually this is achieved by mailing hardcopy reports, sending electronic copies via e-mail, or posting audit reports on a website.

Delivering Electronic Reports   If reports are delivered in electronic form, they should be prepared so that they cannot be altered. Additional controls, such as preventing text or figures from being copied out of the report or preventing the report from being printed, may also be appropriate in some circumstances. These controls will help to prevent the original audit report from being misrepresented, abused, or distributed outside of its intended audience.

Additional Engagement Deliverables

A client organization, or the department being tested, could request additional information from an audit process. Examples of this include

•  Feedback on policies

•  Suggestions for improving the control environment

•  Test results information in greater detail than in the report

•  The feasibility of mitigating controls

The client organization may have selected the contracted audit party because of the experience of team members. Certain audit projects give an auditor a significant understanding of a client organization’s environment. After testing, it is not uncommon for clients to ask the auditor how their systems compare to their peers’ systems.

Auditors can be helpful when asked questions, but ought to be cautious in their responses. There are several potential pitfalls to avoid:

•  Auditors should avoid making statements that go beyond their experience with the client and their personal areas of expertise.

•  It would be problematic if a manager justifies a decision based on the fact that “the auditor told me we should do it.” An audit is usually not a consulting arrangement, and engagement letters often are not written with disclaimers used by business advisors.

•  An auditor should exhibit prudence when performing assessments at clients that are business competitors. An auditor should be careful not to reveal details of a competitor’s systems. Even without identifying the party, providing certain information could be perceived as unethical and in violation of professional standards. If a client believes an auditor is overly liberal regarding sharing business information, they may fear that the auditor will treat their sensitive information similarly in dealing with other clients.

An auditor is in no way discouraged from providing advice to management, but must do so in accordance with an engagement’s requirements and in accordance with ethical and professional standards. Auditors may seek to avoid putting certain comments or advice in writing to protect them from exposure to liability or professional criticisms.

Audit Closing Procedures

The audit process comes to a conclusion when reporting is finalized and workpapers are ready for storage. Methodologies may require certain checklists and that approvals are followed when wrapping up an audit.

Audit Checklists

Audit management may follow checklists for certain milestones during the audit. These may include requirements during the bidding and launching cycle, as well as closing procedures that make sure the audit is complete. Audit closing procedures may include the following:

•  The report and workpapers have been reviewed.

•  The report has been delivered.

•  The signed management representation letter is in the workpapers.

•  Workpapers are signed off on and archived.

•  Final invoices have been sent.

Audit checklists are typically filed along with the engagement workpapers.

Closing Meetings

At some point after the delivery of reports, audit management and the client will discuss the performance of the audit. In situations where the audit organization plans to continue performing audits for the client in the future, these inquiries help to develop business relationships as well as refine the performance of the audit. It is common to identify points of friction in the audit and seek ways to improve interactions. Clients might prefer to perform testing within certain calendar periods, or want to centralize requests for information. This occasion is a good opportunity to address changes in personnel. A client may choose to have another person serve as the primary contact, or the audit team may announce a change in staffing or management.

When delivering reports to management, these discussions frequently append the meeting where reports are delivered. Auditors should be prepared to discuss such points at the end of the audit. It may prove best to address certain issues at a later time. For example, specific details of findings and recommendations for improvement may be best discussed with individual control owners, who may not be participants in the closing meeting.

Final Sign-off with the Client

Audit methodologies may involve formalizing the closing of an audit with a client. Audit organization management may seek closure to know that they can close the book on an audit. Auditors will know all work on a project is complete and no more hours will go against project budgets. Final billings can be processed, and the auditor can send a final invoice for audit services to the client.

One way of formalizing the end of an audit is to have management sign a letter accepting its completion. A final document is often signed by the client that states the auditor has provided all services as contracted under the engagement letter. Such letters are placed in the engagement files.

Client Feedback and Evaluations

After an engagement is completed, the audit organization frequently will ask for feedback from client management. Some audit organizations track a complex set of feedback metrics. It is common to use this information for internal performance evaluations and bonuses.

An audit firm may follow up with clients with surveys. These surveys will solicit feedback on different parts of the audit. The client will provide the audit team with different “grades,” which the audit firms may consider when seeking to improve service quality in the future. If a client is pleased with the service, this proves an opportunity for the audit firm to ask a client if they may serve as a referral when the audit firm is bidding on services to a new client.

Some audit firms will bring the audit team together to review the feedback from the client. In cases where audits are performed on a regular schedule, the results of “what went well” and “potential improvements” in a prior audit may be discussed during kickoff meetings to ensure focus is given to following up on feedback received.

Audit Follow-up

From a company’s perspective, an audit is a part of an ongoing cycle. After the initial growing pains of the “first time through,” an organization usually experiences audits on a regular cycle. Control managers and owners are often familiar with the audit process, though specific testing practices may evolve to fit changes in business, technology, and regulation.

A mature audit function will track issues over successive audit periods. The results of testing cycles or prior audit reports will have identified areas of improvement. Management will have agreed to certain remediation measures. The audit cycle provides an auditor with an opportunity to revisit controls that have failed in previous periods.

Follow-up on Management’s Action Plans to Remediate Control Failures

Internal audit cycles will suggest improvements to control activities, and management will reply to these recommendations. Sometimes, management implements the auditor’s recommendations; sometimes, they reply with alternate approaches to the issue. Often, a project is started by management as remediation and the planned completion date is provided with management’s reply. Internal audit departments will track management’s remediation plans on the calendar and will follow up with the project owner regarding completion. Testing will be reperformed on failed controls, and may be performed on controls newly introduced by a project as well.

Retesting Issues in Succeeding Periods

Controls with audit exceptions will be retested either until improvements are made and the control passes or the control is replaced with a new control activity. A test with repeated control failures will be shared with appropriate executive management and, depending on materiality, may be reported to a governing entity.

External Audit   External audit agreements often approve performing audits over several periods. The audit organization develops in-house knowledge of the client organization’s issues. Audit recommendations aim to work with management to improve the controls environment so testing failures in previous periods are avoided. It is standard to retest areas where previous tests have failed.

In the event that tests continue to fail over successive periods, auditors will draw attention to the issues in their reports; audit reports will draw attention to issues that persist. In high-risk areas, audit management may deem the issue material enough to issue a qualified or adverse audit opinion.

When an audit organization is performing their first audit of a client, they will review previous periods’ audit reports. These help the auditors evaluate the risk of different areas, and include in-scope failed tests and management’s remediation plans.

Internal Audit   Internal audit departments frequently track controls to be tested over cycles. Certain controls testing cycles, OMB A-123, for example, will permit less critical controls testing on a less frequent basis. If a control has passed in previous testing, a risk evaluation may deem the control need only be tested on a two- or three-year cycle. However, control tests that have experienced exceptions, have a high level of materiality/risk to an organization, or that include processes that have changed in previous periods probably will need to be retested on a more frequent basis.

Cyclical control testing is only a portion of internal audit work. An internal audit will, at the direction of the board of directors, pursue audits of different functions within an organization. Unlike internal audit testing cycles or external audits, audits of specific functions may be a one-time event. Issues discovered in these reports are tracked within the internal audit department. Internal audit management and the audit committee should allocate time each year to revisit the status of different issues uncovered in these audits.

Management Projects   Frequently, management will agree to remediate issues by initiating a project to solve the problem. Information systems projects end up in a priority queue, where they are allocated time based on available resources as well as complexity, relative importance, and other factors. Management is frequently cooperative in understanding the importance of a remediation project, but these are evaluated and prioritized among dozens, if not hundreds, of other IT projects. Auditors may find themselves pressing management on delayed projects relating to ongoing control issues.

Summary

This appendix has provided you with an in-depth study of the IS audit cycle, with the goal of providing a link between CISA study materials and the discipline of professional information systems auditing. By providing real-world associations for the subject materials addressed in this book, your memory will benefit from this material. After all, successfully passing the CISA audit exam requires more than being able to recite facts; it requires you to understand all aspects of IS auditing and many aspects of IT management and security.

An additional goal of this appendix has been to portray the “organic” challenges that auditors face in the field. Examples have illustrated scenarios that experience has shown are reasonable considerations during an audit.

Hopefully, you feel you have a better grasp of the process of information systems auditing and an awareness of the challenges encountered in the field. To the experienced auditor, it is frequently the case that as soon as one audit cycle ends, planning for the next period’s audit is just around the corner. There are always opportunities to improve a current process, and experience will unveil more perspectives and stories.