The audit team’s decision-making process

Substantive decisions
Audit tasks. Before the audit, the team selects appropriate targets and formulates audit probes that comprise the initial audit tasks. Some audit tasks will be constructed by the TEAC staff before the audit in response to particular concerns of the Accreditation Panel director. Before the audit visit, by consensus, the team agrees to a set of about 65 initial audit tasks.

Audit report . The audit team uses its final on-site session to come to consensus on the three key issues that must be reported in the audit report:

The lead auditor, unless the team has agreed to another plan, writes the first and final drafts of the audit report, in consultation with the other member(s) of the team. The precise language of the report is negotiated by consensus, and, as above, both versions of any stalemate are noted in the final audit report.

Process decisions.
The team makes many decisions about how the audit will be conducted. These decisions are made by consensus and based on equity and special competence. The principal decisions center on the schedule and the assignment of responsibility for various audit tasks to the team members. The lead auditor, however, has the final say in these matters should consensus be elusive.

Guidelines for the audit strategy

The lead auditor, the director of the Accreditation Panel, and TEAC’s director of audits select the audit tasks beyond the standard audit tasks. The particular audit trail (the sequence and nature of the audit tasks) is a matter of the auditors’ professional judgments. The entire audit (the initial, standard, and follow-up probes and tasks) is guided by the following general principles.

• A target is appropriate for probing if it is related to one of the elements and components of the TEAC system (1.0–4.0).

Centrality. There must be an audit task for each element, component, and subcomponent of the TEAC quality principles and standards of capacity (elements 1.0–4.0 and the components and subcomponents). The centrality criterion for the selection of audit tasks provides a challenge for the auditors because the authors of the Brief are free to address the quality principles and capacity components in differing ways. The auditors are required to audit each element, component, and subcomponent of the TEAC system, but the Brief authors may write claims and provide evidence that spans more than one element of the system. Because the same claim and data source may serve more than one element, a single audit task may help verify more than one part (e.g., the verification of the student teaching evaluation data may be related to claims of teaching skill, pedagogical knowledge, student services, student feedback, and subject matter knowledge). Nevertheless, the auditors must verify the evidence in the Brief that is associated with each claim, and they must also verify and certify that the evidence cited in the Brief fully addresses each element, component, and subcomponent of the TEAC principles and standards.

Priority. Some parts of the TEAC system play a larger role in the accreditation decision than others and have a higher priority on that account. The evidence behind Quality Principle I, for example, is a determinative factor in the accreditation decision and for that reason it is important that the evidence be conclusively verified for components 1.1–1.3.

Quality Principle III also has a determinative role in the accreditation decision, which is why special attention is given to the verification of the i internal audit report. On the other hand, the evidence of institutional commitment has less influence on the accreditation decision than does the evidence behind the quality principles. Thus, the audit probes of some components have a higher priority in the strategy of the audit than other components.

Difficulty. Some components may be relatively easy to verify (such as the faculty’s acceptance of the program goal or the Brief) while others, such as teaching skill and knowledge of subject matter, may require more probes to verify that the evidence is as it was presented because there may be multiple measures or weak measures administered in inconsistent ways. Thus, a target for 4.2.1 (acceptance of the TEAC goal) may require only a single probe that amounts to little more than a question to a sample of faculty members, while the target for 4.2.2 (the faculty’s balanced and accurate understanding) may require several probes to verify the evidence of an accurate and balanced view.

Variability. If the outcome of a probe fails to confirm the target, the auditors should probe an additional target, or a related target, until they find a stable pattern of outcomes. If the auditors cannot find a stable pattern, then it is unlikely that the target can be verified because fewer than 75 percent of probes of the target will have succeeded in confirmation.

For example, if the sequence of courses on the transcript in a randomly selected student file is the target of a probe to verify whether graduates majored in their teaching subject (as may have been claimed in support of Quality Principle I), and the probe fails to confirm that fact, then other students’ transcripts need to become targets of probes until the pattern of majoring for the program’s students can be discovered, confirmed, or disconfirmed.

Internal consistency and corroboration. Auditors are encouraged to seek targets that could corroborate evidence that is in the Brief. If, for example, the Brief cites a grade point index of 3.25 in certain mathematics courses in support of the claim that students know their mathematics, the index could be the target of an audit probe that could calculate the index of a sample of students to confirm that the index indeed was 3.25. The scores of the sample of students on Praxis I (math), Praxis II, the scores on the SAT (math), the variability in math course grades, or the math lessons in student teaching could also be probed to see if they were consistent with, and corroborated, the math grade index.

If, to take another example, the program faculty supports a claim of institutional commitment by reporting that the median salaries of assistant professors in education are insignificantly different from the median salaries of all assistant professors, the auditors might probe other sources of salary information in the institution to check if they corroborate the evidence reported in the Brief.

Conclusive and persuasive. Some sources of evidence are more persuasive and compelling than others. The evidence for the claim that the program’s graduates possess teaching skill (1.3) might be attributed in the Brief to their student teaching course grades, their employers’ ratings of them, their cooperating teachers’ opinion, or the academic accomplishments of the graduates own pupils. The latter source of evidence, if it were available, would be more persuasive than the student teaching course grade, for example, and would be a preferred target of the probe over the target of the grades in the student teaching course.

Disconfirmation. Upon seeing the results of a survey of employers cited in support of a claim, an auditor could verify and confirm the results. The auditor might also wonder what would falsify or disconfirm the evidence. If the survey response rate were 10 percent, or if the employers were also employees of the program, or if the survey instrument had a bias for positive ratings, the evidence would be disconfirmed although the actual results of the survey were accurate as reported in the Brief. The auditors may construct probes to examine these areas of potential disconfirmation as a further way of verifying the survey evidence.

For example, if the Brief claims as evidence of teaching skill that 90 percent of the “teachers of the year” in their state are graduates of the program, the auditors could easily confirm the number and percentage. They could also probe how many teachers in the state are graduates of the program. If 90 percent of the teachers in the state are also graduates of the program, the teacher of the year data would probably not be persuasive to the Accreditation Panel. However, if only 10 percent of the state’s teachers are graduates of the program, but 90 percent of the teachers of the year are graduates, the panel might be more persuaded that the teacher of the year data indicated something about the program’s quality with regard to the acquisition of teaching skill.

Similarly, if the program cites as evidence of teaching skill that 90 percent of its graduates secure teaching positions within three months of graduation, the panel could direct the auditors to determine what the hiring rates are for the program’s region. If there is a severe teacher shortage in the region, and 100 percent of applicants are routinely hired by local districts, the program’s evidence for its claim would be less persuasive to the panel.

Primary sources. Whenever possible, it is better to trace the evidence back to its origin, the raw data. When the results of a survey of graduates, for example, are cited as evidence that the graduates care about their students, the verification of the results of the survey is on surer ground when the auditors inspect the survey instrument, inspect some completed forms from graduates, re-tally a random sample of returns, and perhaps interview one or two of the respondents. (Top)

Auditors’ heuristics

The audit report must include a judgment, or opinion, about the trustworthiness of the program’s evidence for each of the elements of the TEAC system, including the components of 4.0. The auditors use the following five guidelines, or heuristics, to determine their opinion of the evidence for the quality principles and standards of capacity (1.0–4.0) as they are presented in the Brief :

2. An element (1.0–4.0), or component of 4.0, receives a clean opinion if at least 90 percent of its targets are confirmed. If more than 10 percent of the targets are not confirmed, the element (or a component of 4.0) cannot receive a clean opinion and must receive some other opinion, depending on the circumstances described below.

3. An element or component is given a qualified opinion when at least 75 percent, but less than 90 percent, of its targets are confirmed. An element that would otherwise receive a clean opinion is also given a qualified opinion if more than 25 percent of the targets reveal misstatements of any kind, either trivial or consequential.

4. An element is given an adverse opinion if more than 25 percent of its targets cannot be confirmed. An element is also given an adverse opinion if more than 25 percent of the targets cannot be verified owing to the fact that the targets could not be confirmed for any reason (including disclaimer).

5. An element is given a disclaimer opinion if more than 25 percent of the targets associated with it can not be verified because of missing data, limited access to information and informants, or evidence that the findings reported in the Brief are not genuine.

These five guidelines are heuristics for formulating an audit opinion about each element and each component of element 4.0. They are not algorithms or rules: a simple counting of outcomes of probes could be misleading with regard to the trustworthiness of the Brief. Some audit tasks may be more revealing than others. For example, some may have targeted only minor points, and some may be merely following up on other audit tasks on a single point. The guidelines may prove unreliable in cases where the number of audit tasks is small. The audit team knows that they are not to treat the heuristic as an algorithm or rule that can be mechanically applied. If the findings suggest anomalies that make the heuristic unworkable, the auditors will rely on their good judgments, explaining in their audit report the difficulties they experienced and the reasons for their opinions.

Heuristics, by definition and design, only guide decision making. Because TEAC cannot predict or accommodate all possible outcomes and circumstances, the auditors make judgments when the findings are complex and lack a regular pattern. When there is doubt, the auditors will render a lower, more conservative audit opinion rather than a higher audit opinion to alert the Accreditation Panel and the Accreditation Committee to possible dangers in interpreting the Inquiry Brief or Inquiry Brief Proposal as trustworthy and reliable. Should a TEAC auditing team make errors in judgment in these matters, the lower and more conservative audit opinions always can be adjusted in the process that requires the mutual acceptance of the a audit report or through the TEAC appeals process.

Overall auditors’ opinion. If no element received an adverse or disclaimer opinion, the auditors give the Brief a clean audit opinion overall if 90 percent or more of the targets are verified, and they give it a qualified opinion if at least 75 percent of the targets, but less than 90 percent, are verified. The Brief can go forward to the Accreditation Panel only with a clean or qualified opinion (i.e., at least 75 percent of the targets are verified). Otherwise, it is returned to the program faculty for reworking and resubmission.

Auditors’ judgment of commitment. The auditors are charged not only with verifying the evidence for commitment, but with determining whether the evidence is sufficient to support the program’s claim that the institution is committed to the program. The program faculty members are free to provide any evidence they find convincing of their institution’s commitment to their program, but they must address the issue of parity between the program and the institution in the components of capacity (4.0) in making their case for commitment.

With respect to 4.0, the standards for capacity for quality, TEAC expects the program faculty to provide evidence that the program is fairly treated and not appreciably different from the institution overall. As measured by each of the seven components of 4.0, the program, in other words, should receive its fair share and be treated like most of the other programs at the institution.

Before the auditors can conclude that the institution is committed to the program, the evidence of parity in Appendix B must receive a clean or qualified opinion. In forming their conclusion, the auditors are guided by the same heuristic that guides the Accreditation Panel with regard to its judgments of how much evidence is sufficient to support a claim. This heuristic, when applied to the evidence of commitment, supports the conclusion that the institution is committed to the program when at least 75 percent of the points of comparison show parity or favor the program.

Parity between the program and the institution is taken as signifying the institution’s commitment to the program. Unless there is a credible rival hypothesis to the contrary, it is prima fascia evidence of commitment.

For example, the mean salaries of the teacher education faculty and the mean salary for the institution as a whole could be indistinguishable and show a parity that would seemingly signify commitment. One salary might be for 12 months of effort, however, and the other for nine months of effort, or one might include overload teaching assignments while the other does not, etc. Thus, the salary parity, as reported in the Brief, between the program and the institution may not always indicate institutional commitment, but may indicate the institution’s exploitation of the education program faculty. Or the allocations of resources to the program faculty and the institution’s faculty in general may be the same, but the allocations to the education faculty may include unique costs not shared by the others (e.g., payments to cooperating teachers, a curriculum resource center, mileage for student teaching supervision, and so forth). The auditors must consider the possibility that parity in resource allocation may have come about for reasons that might signify that the institution is really not committed to the program.

While parity usually signifies commitment, the lack of parity may not be prima fascia evidence of a lack of commitment. For example, the faculty may claim that a discrepancy between program and institutional salaries is in fact evidence of commitment: the institution has added a disproportionately large number of new, junior-level positions to the program, positions that were not available to other programs. The auditors would have targeted this salary claim, and if they had verified the evidence for the claim, they could easily have concluded that the salary discrepancy, as explained, indicated the institution was in fact committed to the program with regard to compensation. (Top)

Return to the audit