| Before
the audit
On-site audit
activities
Post-audit
activities
Before the audit
TEAC’s process of formative evaluation includes reviewing
the Brief in draft form to ensure that it (1) is complete; (2) addresses
the appropriate elements and components of the TEAC system; and
(3) is written clearly and precisely. When TEAC is satisfied that
the draft Brief meets all requirements, then it declares the Brief
“auditable.” At that point, TEAC instructs the program
to submit a final version of the Brief (and the checklist),
in multiple copies.
Once the program’s Brief has been accepted
as auditable, TEAC staff and the program faculty schedule and plan
the audit. TEAC will assign a team of auditors and schedule the
audit. (See the TEAC audit schedule .)
Once selected, the audit team members insure that
there are no undeclared conflicts of interest surrounding their
participation in the audit. In this initial period of planning for
the auditors’ visit, program faculty members have an opportunity
to review the resumes of the members of the audit team to identify
any potential conflicts of interest that may exist. TEAC officers
and program faculty will negotiate claims of conflicts of interests.
After their study of the Brief, the auditors will
propose audit tasks (see constructing audit
tasks, below) and review the audit tasks proposed by others.
Audit tasks can be conducted without an on-site analysis; in those
cases, the auditor may conduct the analysis before arriving on site.
Then, TEAC and the program faculty will determine
what evidence the program must make available to the auditors, the
interviews that need to be scheduled, and the observations that
are required. In some instances, TEAC and program faculty may agree
that some data records can be sent to the auditors prior to the
visit to make more efficient use of their on-site time.
Next, the auditors prepare a summary of the case.
The summary of the case explicates the case the program has made
to support its claims; it tells the program’s story. The purpose
of the summary is at least threefold: (1) to convey to the authors
(and to others) the auditors’ interest in fully understanding
the Brief’s meanings and contexts; (2) to facilitate the construction
of the final audit strategy; and (3) to provide the members of the
Accreditation Panel and Accreditation Committee with an accurate
summary of the case the Brief makes.
Following these five guidelines, the auditors prepare
the summary of the case:
1. The auditors restate all the claims advanced
in the Brief related to TEAC’s Quality
Principle I, the evidence related to the reliability and
validity of the measures used to assess the claims (Quality
Principle II), and the standards
of capacity for program quality.
2. The auditors summarize the results, linking
particular findings to claims, including any summative judgments
the faculty members make about their claims (e.g., our students
have mastered their teaching subjects because they meet a 3.0 grade
index in their major and meet the state’s Praxis II examination
standard).
3. In a separate section of the summary, the auditors
write out the principal results of the program’s internal
audit and the findings reported in Appendix
B (e.g. all students in the sample who were admitted to student
teaching met the committee’s GPI standard).
4. The auditors quote from the Brief itself in
composing the narrative and cite page numbers of the quote.
5. Finally, because the auditors are telling the
program’s story, they do not comment about aspects of the
case for accreditation that they may think are weak or problematic.
Nor do they make the case stronger than the program faculty made
it. The summary is about the program’s case, not the case
the auditors would have made.
The lead auditor prepares the initial draft of
the summary of the case; other members of the team review it, and
once the team accepts the summary, TEAC sends it to the program
head.
Constructing
audit tasks.
The audit is a series of tasks, each assigned to an aspect of the
Brief that is also associated with one of the principles or standards
of the TEAC system.
There are three kinds of audit tasks: (1) those
the auditors create for the verification of a particular Brief;
(2) those the panel director requests or creates for the Brief;
and (3) tasks that are common to all audits, which the TEAC staff
has created. These latter audit tasks, called standard audit tasks,
typically target the subcomponents of the program’s capacity
for quality (4.0).
Before the auditors arrive at a campus, the TEAC
staff and the auditors will have created a set of initial audit
tasks that are directed at the parts of the Brief that are relevant
for one or another of the TEAC principles and standards. They are
called initial tasks because the auditors may also employ follow-up
tasks and new tasks that they create during the audit or that they
draw from the larger set of potential audit tasks TEAC has created.
The standard audit tasks are constructed without reference to a
particular Brief.
An audit task is composed of a target and a probe.
The audit task is constructed by selecting some aspect of the Brief
text (the target) and probing it.
A target is what the auditors are seeking to verify
in the Brief. A target can be a particular sentence, claim, statistic,
or piece of evidence. Each target is linked to an element, component,
or subcomponent of the TEAC system. For example, if the target is
the sentence all faculty teach in their areas of competence, the
purpose of choosing the target is its link to subcomponent 4.2.3,
the faculty’s qualifications.
A probe is a specific action
taken by the auditor to establish whether the evidence for a target
(in the first sense) is accurate. If the result of the probe of
a target is ambiguous or in cases where the outcomes of a probe
are variable or uncertain with regard to the accuracy of the evidence
for a target in the Brief, the auditors probe further until a stable
pattern is uncovered or until a probe’s result is unambiguous.
A probe is said to be
confirming if the auditor determines that the evidence, statistic,
or claim, representing the target is accurate. This judgment can
be made even if there are slight and inconsequential inaccuracies
in the targeted text of the Brief.
The following are examples of possible audit tasks
(targets and probes):
| 1.
Checking records, such as minutes of meetings or memos on
file of faculty actions in making program decisions in approving
the Brief, and similar group decisions.
2. Reviewing notes taken of interviews
with focus groups and with students at their exit from the
program from which summaries are prepared or generalizations
included in the Brief are induced.
3. Inspecting the responses received from
stakeholders who are surveyed in the self-study process
when the responses are summarized in tables or in narrative
and included in the Brief.
4. Re-computing percentages, means, and
standard deviations reported in the Brief from the original
raw data taken from institutional files, state reports,
or other sources.
5. Using institutional records to re-compute
the means and standard deviations of grade point averages
reported in the Brief.
6. Tracing the sources of claims having
to do with budget allocations, space allocations, and similar
matters with institutional officials (e.g., provosts, bursars,
or vice presidents).
7. Re-applying the coding schemes used
to draw inferences from qualitative data to see if the results
can be reproduced.
8. Interviewing faculty or staff who applied
the coding schemes to qualitative data to learn how well
they were trained, how their reliability was checked, and
the depth of their understanding of the process.
9. Re-computing correlations and other
statistics that were used to support claims of reliability
and validity using, when appropriate, the faculty’s
statistical packages.
10. Checking reports concerning the reliability
of multiple observers by inspecting the raw reports and
re-computing the appropriate coefficients.
|
11. Touring
the campus to verify evidence about claims concerning services
available to program candidates, the availability of computers,
faculty office space, and other capacity-related issues.
12. Checking brochures, catalogs, and local Web sites to
make sure the information found in the Brief is consistent
with the information found in these sources.
13. Examining both the data (video tapes,
transcripts, field notes) and the procedures for coding
the data if the faculty’s claims are supported by
qualitative analyses of interviews with informants.
14. Examining data sets (also institutional
reports where those same data are provided) to verify evidence
present in comparisons of funds, space, full-time faculty
equivalent per student enrollment of the target program
with other programs on campus.
15. Interviewing faculty who participated
in the deliberations leading to program change, examining
minutes of meetings, and inspecting the copies of proposals
that were taken to the faculty or administration for action
to determine if the Brief claims that changes were made
in the program after considering data generated by the quality
control system.
16. Targeting text to see if there is in
fact a referent for the language, particularly if the language
is educational jargon or is unqualified.
17. Interviewing faculty who conducted
the internal audit probes and asking that their efforts
be described to discern how familiar faculty members were
with the internal audit and its purposes, findings, and
recommendations.
18. Interviewing students who were the
focus of the internal audit probes to ascertain that the
characterizations found in the internal audit report in
Appendix A about these students are accurate.
19. Interviewing faculty who were the focus
of an internal audit probe to ascertain that the characterizations
of those faculty members found in the internal audit are
accurate.
20. Examining files and archives describing
actions taken by the faculty to improve the program to document
the accuracy of the characterizations of these actions in
the Brief.
|
Of course, situations will vary from site to site.
Claims and the sources of data for claims that have not been anticipated
may arise, and auditors may need to consider additional kinds of
probes to use in their efforts to determine if the statements and
evidence found in the Brief are accurate.
TEAC prescribes the following additional features
for some of the audit tasks. At least one audit task must meet each
of the following conditions:
- The auditors must observe a session of at least two regularly
scheduled courses that the program offers.
- The auditors must interview the students in the program’s
sample for its internal audit or for the evidence for Quality
Principle I.
- The auditors must interview a sample of cooperating teachers.
- The auditors must select four facilities cited in the Brief
and tour each to verify their existence and similarity to their
description in the Brief.
- The auditors must verify a plan to investigate, or an investigation,
of a link between student learning and any program factor.
- The auditors must interview members of the administration
to verify their commitment to the program, their allocation
of resources to the program, and their qualifications for their
positions.
- The auditors must verify that the call for comment from third
parties was distributed to the parties required by TEAC policy.
- The auditors must verify that the raters were trained and
the rating forms and instruments exist.
The auditors must note any discrepancies between
characterizations of the institution described in the Brief and
the experience of the site visit, particularly facts at variance
with what is reported in Appendix
E.
On-site
audit activities
The auditors’ verification process entails the review of relevant
documents and interviews with representatives of the institution,
faculty, staff, and students associated with the program. The sorts
of activities the auditors might undertake, and the data to which
the auditors need to have access, are described below.
Review of the
pre-audit tasks
The auditors tell the story of the program seeking accreditation
so that the program faculty can be assured that the auditors have
understood the Brief in the manner intended by the program faculty.
The story (summary of the case) will have been sent to the program
faculty before the audit so the faculty members can respond with
corrections and amendments. The point is to ensure that the auditors
and program faculty can conduct the rest of the audit from a common
basis of understanding of the Brief.
Understanding the local context.
During this first meeting with program faculty,
after the introductions are complete, the discussion turns
to the summary of the case prepared by the TEAC staff and
auditors and sent to the program faculty before the audit
visit.
The auditors seek the program faculty’s
reaction to the summary of the case: Does it hit the mark?
Is it complete? Has it distorted any elements of the Brief?
The auditors should receive feedback from the faculty without
argument or debate. When the auditors write their report,
they will also amend the summary, based on these comments
from the program faculty.
Having determined that the auditors understood
the Brief at a level acceptable to the program faculty,
the auditors move to clarifying their own understanding,
or misunderstanding, of the Brief.
Clarification. Before
the audit visit, the auditors ask the authors and endorsers
of the Brief to clarify any language used in the Brief that
may be unclear to the auditors. This effort is critical
because it is essential that the program faculty believe
that the auditors understand the Brief. TEAC believes that
this feature of the audit process helps to build the rapport
between the audit team members and the program faculty that
comes when one party feels the other party understands its
positions.
Before the audit visit, the auditors sample
from a pool of statements in the text that are unclear to
them and ask the program faculty to put in writing their
explanation and clarification of the text. The auditors
need to probe assertions made in the Brief to determine
if the referents exist and mean exactly what they seem to
mean. The purpose of these probes is to verify that the
match between the referent and the language in the Brief
is accurate and precise. The auditors can verify the program’s
assertions only if the language is clear and precise.
The audit tasks focused on language are
designed to clarify text that is ambiguous or that, when
explained, may be particularly revealing of the program
faculty members’ thinking about matters related to
the quality principles and capacity standards. Through this
process, the auditors provide the Accreditation Panel members
with a basis for determining the degree to which the language
and evidence in the Brief mean exactly what they seem to.
|
On-site audit tasks
The main purpose of the audit is to verify the evidence the program
faculty has cited in support of its claims with respect to the quality
principles and capacity standards. From a pool of audit targets,
the auditors select a sample that is particularly revealing and
representative of the totality of the evidence the program faculty
has presented in the Brief. The auditors divide some tasks among
themselves, and others, such as interviewing students, administrators,
and faculty or observing classes, together as a full team. Throughout
the entire visit, the auditors are alert and sensitive to unobtrusive
information that may have a bearing on the targets of the audit.
While the auditors are on site, they use the evenings
and team meals as opportunities for de-briefing. The auditors make
mid-course corrections in the audit tasks, modify the agenda and
schedule as needed, develop new audit trails, and review preliminary
impressions and observations.
Verifying
the evidence related to specific claims. The Brief
includes the evidence the faculty uses to support its claims
related to the program’s goal of preparing competent,
qualified, and caring professionals as well as to support
the claim that the institution has the capacity to offer
a quality program. The auditors do not judge whether the
claims are true or even credible. The auditors do not judge,
for example, whether or not the program’s graduates
understand pedagogy. They judge only whether or not the
evidence is what is reported in the Brief. For example,
if the program faculty relies on a mean score on a standardized
test to advance and support its claim that the program’s
graduates understand pedagogy, the auditors will check to
see if, in fact the score the program’s graduates
earned on the test is as the program faculty reports in
the Brief.
Corroborating evidence.
Throughout the audit the auditors are alert to the discovery
of evidence that was not cited in the Brief but has a direct
bearing (positive or negative) on the verification of the
evidence and the precision of the language in the Brief. The
auditors are charged with assuring the Accreditation Panel
that there is evidence behind the claims made in the Brief.
There are two kinds of errors the auditors need to avoid:
(1) false positive errors (concluding the evidence is present
and accurate when it is not); and (2) false negative errors
(concluding there is no evidence for a claim when in fact
there is).
Errors. The auditors must
determine whether any errors they find in the Brief are trivial
or are of some consequence to the meaning of the text. When
a misstatement is trivial and of no consequence, the targeted
text is not misleading in spite of the error and the statement
means more or less the same thing with the error as without
the error.
For example, if the auditors had recalculated
a mean and found it was 3.16 instead of the 3.06 reported
in a table or in some text, it is probably the case that
the targeted text would have the same meaning whether the
mean is one or the other value. If the faculty claimed they
are constructivists and it turns out in response to the
auditors’ probe that they meant only that they are
Piagetians, the statement is still acceptably accurate.
The errors, or misstatements, that are of
consequence are those that alter the meaning of a targeted
statement in the Brief in such a way that the statement is
not verified. If the Brief asserts, for example, that the
faculty endorsed the Brief at a particular faculty meeting,
but the minutes of the meeting do not report the action, or
a sample of faculty do not recall the endorsement or is unfamiliar
with the contents of the Brief, the misstatement is of consequence
and may signify to the panel that one part of the standard
for faculty qualification in the TEAC system could not be
confirmed (e.g., 4.2.1).
The evidence claimed for the endorsement, in other words,
cannot be relied upon. If the recalculated mean (to take the
example above) differed by more than 25 percent of the standard
deviation from the reported mean, the misstatement of the
mean is of consequence and the auditors would conclude that
the reported and misstated mean was not confirmed and verified.
|
Final on-site session
In its final on-site session, the audit team considers the findings
from each audit task and formulates its audit opinions. The team
also analyzes the evidence about institutional commit-met and determines
whether or not the evidence is sufficient to support the conclusion
that the institution is committed to the program. The team uses
the session to start planning the audit report.
Judging.
The auditors must come to a conclusion about whether or not the
evidence advanced by the faculty in support of the TEAC quality
principles, the capacity components, and internal audit was in
fact verified. The auditors also must make a separate determination
of whether the evidence of institutional commitment is sufficient
to support the claim that the institution is committed to the
program.
In their audit report, the auditors give one
of the following four judgments (audit opinions) about the overall
trustworthiness of the Brief and about each element (1.0–4.0)
and the seven components of 4.0:
· Clean opinion:
An element is assigned a clean opinion when the evidence in
the Brief that bears on it is found to be trustworthy.
· Qualified opinion:
An element is assigned a qualified opinion when some of the
evidence in the Brief that bears on it has significant errors,
but overall the evidence for the element is found to be trustworthy.
· Adverse opinion:
An element is assigned an adverse opinion when a significant
portion of evidence in the Brief that bears on it cannot be
confirmed and verified.
· Disclaimer opinion:
An element is assigned a disclaimer opinion when it is not possible
to verify a significant portion of the evidence in the Brief
that pertains to an element owing to missing data, limited access
to information and informants, or evidence that the findings
in the Brief are not genuine.
Post-audit activities
After the visit, the team drafts the audit report. TEAC then sends
the draft to the program for review. With TEAC staff, the auditors
respond to any comments from the program faculty, negotiate points
raised by the faculty, and finalize the audit report. The auditors
might meet in person, if convenient, or communicate by phone or
electronically. Finally, the lead auditor, as a non-voting member
of the Accreditation Panel, participates in the panel meeting devoted
to the program’s Brief.
Audit report
Immediately after their campus visit, the auditors prepare the audit
report, which is submitted to TEAC and the program faculty within
two weeks of the visit, first in draft form inviting comment, and
subsequently in final, official, form.
In the audit report, the auditors give their opinion
about the accuracy of the evidence in the Brief. The auditors do
not comment on the implication the evidence holds for the accreditation
decision.
Within two weeks of receiving the audit report,
the program faculty must correct any factual errors made by the
auditors (the program faculty simply points out the errors). At
this time, the program may formally respond in writing to the findings
of the audit; however, at this time the program faculty cannot make
or offer any corrections or changes to the Brief or the facts of
the program.
After correcting factual errors in the auditors’
findings and considering any responses by the program faculty, the
auditors submit a final audit report to the TEAC staff, program
faculty, and Accreditation Panel.
Once accepted by the program faculty and the TEAC
staff, the audit report becomes part of the record submitted first
to the TEAC Accreditation Panel and then to the Accreditation Committee.
Each body considers the report in its respective deliberations and
in support of the recommendations and decisions concerning the Brief
and the appropriate accreditation decision
The audit report includes four major sections:
Section 1: Abstract. The first
part of this section contains the final version of the summary of
the case. The second part gives the auditors’ overall opinion
about the trustworthiness of the Brief coupled with a summary of
the principal findings of the audit. The auditors’ judgment
about the level of institutional commitment to the program is also
included in the abstract.
Section 2: Method. This section
briefly describes the character and method of the audit: what the
auditors did, with whom they met, what they examined, and the schedule
of the audit.
Section 3: Findings. The third
part is a full report of the findings from the auditors’ probes
into the evidence included in the Brief related to each of the TEAC
quality principles and standards of capacity.
Section 4: Judgments. The fourth
section contains the auditors’ judgments, given as audit opinions,
about whether or not the evidence advanced by the faculty in support
of each element was verified. If a sufficient number of the probes
confirm, or fail to confirm or verify the evidence, the report explains
the findings and reasoning behind the auditors’ opinions.
Finally, the auditors make a determination of whether the evidence
of institutional commitment is sufficient to support the claim that
the institution is committed to the program.
Return to The TEAC audit
|
