Measuring performance

Internal Auditor, Feb, 2000 by Douglas E. Ziegenfuss

What are the critical success factors for internal audit departments? A new study sheds light on the performance indicators that CAEs regard as the most important measures.

Since 1993 The IIA's Global Auditing Information Network (GAIN) has collected diverse benchmarking information for participating organizations. However, many GAIN participants--and others--have been interested in identifying the performance measures that are most useful in evaluating internal auditing department effectiveness.

New research, based on a study involving CAEs participating in the GAIN project, has pinpointed five top performance measures: staff experience; auditing viewed by the audit committee; management expectations of internal auditing; percentage of audit recommendations implemented; and auditor education levels. Further classification and analysis of the findings, along with supplementary studies, captures additional information that is likely to be intriguing, not only to practitioners, but to chief executive officers (CEOs), audit committee members, and others with keen interest in the success factors of internal audit departments.

DEVELOPING THE STUDY

A survey based on the 1997 GAIN Questionnaire and Sample Report was sent to 501 CAEs participating in GAIN. The pool of non-U.S. respondents--30 percent--was large enough to merit analyzing them separately from U.S. respondents, but statistically significant differences were not found.

PART ONE As depicted on page 40, researchers classified each of 84 GAIN performance measures into 16 performance areas, which were further categorized into four audit processes: audit environment, input, process, and output. Survey respondents, who received a list of 73 performance measures taken from the original GAIN list, were then asked to rank each of these indicators on a scale of one to five according to their perception of its appropriateness in measuring audit quality. The top 20 performance measures, as well as their correlating audit process and performance area, are listed in Exhibit 1.

Interestingly, 50 percent of the key performance measures were associated with the "audit environment." Five were associated with "input," three with "output," and only two with "process." In terms of performance areas, four of the 20 key measures were associated with auditor quality, three with management statistics, and three with management satisfaction.

PART TWO Respondents were asked to list the performance measures they would use if they were limited to only five. Exhibit 2 shows the 20 performance measures listed most often and identifies the audit process and performance area associated with each measure.

In general, the results of Parts One and Two were similar. Seven of the 10 highest ranked performance measures identified in Exhibit 1 were also among those indicated in Exhibit 2:

* Auditee satisfaction survey results.

* Percent of audit recommendations implemented.

* Audit committee satisfaction survey results.

* Audit committee importance of audit issue.

* Management expectation of internal auditing.

* Staff experience.

* Training hours per internal auditor. Three measures that ranked among the top 10 performance measures in Exhibit 1 were not among the 10 most frequently mentioned in Exhibit 2:

* Auditing viewed by the audit committee.

* Auditor education levels.

* CAE reporting relationships--functional.

Conversely, three performance measures were ranked among the 10 most frequently listed in Exhibit 2 but were not among the top to ranked performance measures in Exhibit 1:

* Number of management requests.

* Completed vs. planned audits.

* Number of process improvements.

Although these results might seem to indicate a strong correlation between the results of Exhibits 1 and 2, statistical tests indicate no significant relationship. In addition, significant differences were indicated in the distribution of audit process and performance areas, especially in the number of performance measures associated with the "output" audit process. Similar differences appeared in Exhibit 1 with regard to the distribution of performance areas.

PART THREE The third part of the questionnaire requested that CAEs rank the 16 performance areas, with No. 1 as the most important determinant of auditing department quality. The results of this ranking are shown in Exhibit 3, along with pair-wise rankings acquired through supplementary research.

CORROBORATING EVIDENCE

Rankings of the relative importance of performance areas were confirmed through a focus group of to CAEs from the IIA's Tidewater Chapter. In addition, 16 additional CAEs from the IIA's Tidewater and New York City Chapters completed an Internal Auditing Area Preference Questionnaire by mail. Pair-wise matching was used in the questionnaire to force respondents to choose between one of two performance areas in each set. The responses, which appear in Exhibit 3, were then tabulated.

In general, the pair-wise matching results confirm the ranking compiled in Exhibit 2. Eight measures were among the top 10 ranked performance areas in both the GAIN questionnaire and pair-wise matching: Auditor quality, quality of findings, and accuracy of reports were at the top of both lists.