What
is an internal audit? The internal audit of the quality
control system is similar to an accountant’s audit of a financial
system.
In a financial audit, the accountant randomly selects
a payment and follows it through the institution’s financial
policies and regulations (the audit trail) to see if the payment
was duly authorized, was in the correct amount, was recorded and
reported properly, was for a proper purchase and purpose, was backed
by deposited funds, and so forth. These many probes are the basis
for the accountant’s professional opinion about whether or
not the corporation’s financial system can be trusted or depended
upon to provide the corporation’s directors, stockholders,
and federal agencies with sound results regarding the corporation’s
financial integrity and quality.
Figure 2 gives a visual
representation of an internal academic audit. It indicates the starting
point of the audit (in this case, a student folder) and shows the
pathways and the elements that the faculty members checked from
that point forward. These pathways are a sequence of audit tasks:
targets and probes to understand how the quality control system
works. (Return)
Getting started: To carry
out the internal audit of its quality control system, the program
faculty should follow these steps:
|
Conducting the internal audit:
1. Understand the program’s quality control system.
In whatever way is effective and efficient for the program,
assemble a picture of the quality control system. List all
the elements, group them, and note their relationship to each
other.
2. Describe the quality control system visually.
Create a schematic (Figure 1 is
an example of one way to do this).
3. Describe the quality control system in
words. Write a narrative of the system.
4. Determine an overarching set of questions
about the quality control system.
5. Develop an audit plan. Determine the focus
and the point of entry. Determine targets and probes. Assign
roles and responsibilities for carrying out the audit. Be
clear and sensitive to potential weaknesses in the system.
(See Entering the quality control system)
6. Ask for formal approval of the audit plan.
Everyone involved should know and understand why and how
the audit will be conducted, and should approve the plan in
some formal way.
7. Carry out the audit. Keep track of the
process and findings.
8. Create an audit map or audit trail. (e.g.,
Figure 2)
9. Write the audit report (see Writing
the internal audit report) (Return)
|
Entering the quality
control system
The internal audit can begin at any point in the system. The entry
point has no particular significance and merely provides a manageable
way to begin the audit probes and constrain the amount of information
that must be considered. The faculty may enter the system in several
ways.
For example: As in Figure 2, the faculty identifies
a student folder through an unbiased method (e.g., randomly, or
by a student whose birthday is closest to a randomly selected date).
To see if the quality control system works as it is designed to
work in that instance, the faculty probes each element in the system
that is linked to the folder (follow Figure
2).
The internal audit probe would determine, for example,
whether or not
-The particular faculty member who gave the grade in a randomly
selected course in the student folder was appointed, reviewed, and
assigned properly;
-The student was admitted and enrolled properly;
-The work on which the grade was given was evaluated properly;
-The course became required properly;
-The course was evaluated and reviewed properly;
-The course was properly funded;
-The course was given in an appropriate facility,
-The program in which the course was required was properly
evaluated.
The faculty examines all links to the course grade
that are implicated in the system to see whether or not the system
functioned properly in the particular instance. If the program faculty
members feel their system is probed better by starting with a particular
faculty member, the initial audit probe can begin there and move
through the system from that point. The questions of the system
are the same:
-Was the faculty member’s appointment, assignment
proper?
-Was the tenure or promotion decision conducted properly?
-Were the faculty member’s students selected and admitted
properly?
-Does the faculty member evaluate student learning properly?
-Was the faculty member’s course properly approved
and evaluated?
-Was the course properly funded?
-Was the course given in an appropriately equipped classroom?
The probe continues until each element in the system
that bears on the quality of a particularly selected case has been
examined. (Return)
Entering the
audit at points of suspected weakness
If the faculty members are concerned about some aspects of its system
(for example, the process of hiring adjuncts or the trustworthiness
of grades), it would be appropriate to audit the system from this
perspective and with these elements perhaps receiving an initial
focus of the audit procedures. Thus, the faculty would specifically
address these areas of suspected weakness and begin the audit trail
at these points and enter the system, for example, with the agent
within the system that was supposed to produce quality adjunct appointments.
(Return)
Audit probes
The number of probes necessary to the internal audit depends, as
in all sampling, on the degree of variability that is revealed.
The number should be of a magnitude that would convince the faculty
and others that a reasonably accurate reading of the system had
been taken. A sample of 10 percent is usually sufficient.
Sampling just one or two students, for example,
or one or two courses, will probably not provide sufficient confidence
to the interpretations gleaned from the audit. When the faculty
determines how many elements should be sampled, a rationale for
the number should be provided so that the reader can be assured
that the audit findings are truly representative of the system.
(Return)
Understanding the audit findings
The purpose of the internal audit is for the faculty to make some
judgments about how well its quality control system is working.
It is unlikely that a faculty could come to an unqualified conclusion
that the system works as it was designed. Almost nothing works just
as it was designed. The faculty’s conclusions will be credible
only if the audit uncovers some exceptions and problems.
The faculty might advance any one (or more) of
the following conclusions:
-Our quality control system is working well, overall, except
we have learned that we cannot put a great deal of faith in the
course grades our students receive because they are not predictors
of the subsequent performance in student teaching.
-Our quality control system has several significant breakdowns:
violations of our appointment policies in the hiring of adjunct
faculty, inconsistencies in content and practices within various
sections of the same course, and inconsistencies in the way clearances
into student teaching are administered.
These conclusions would give direction to faculty
for strengthening their quality control system and their program.
However, a fair-minded reader might still ask, What is your evidence
that the system is working well, or that breakdowns exist?
The faculty finds the basis for its judgments in
the evidence that it collects during the internal audit. So, it
makes sense to see judgments as flowing from evidence. For this
reason it is important to keep the reporting of the evidence separate
from the reporting of the judgments or conclusions about the quality
control system. (Return)
Writing the internal audit report
TEAC suggests that program faculty organize the internal audit report
in the following way:
| Introduction:
The introduction to Appendix A explains who conducted the
audit, how the plan for the audit was approved by faculty,
and how the internal audit complemented the evidence presented
for Quality Principle III in the Inquiry Brief.
Description of the quality control
system (QCS): The program faculty provides both
a narrative and a schematic of the quality control system.
Audit procedures: In this
section, the faculty members describe how they conducted
the audit, what evidence they collected, what trail they
followed, how many elements (students, courses, and faculty
members) they audited, and who participated in organizing
and interpreting the findings. Figure
2 represents such a procedure. The faculty should provide
a similar visual representation.
Findings: What did the
faculty discover about each part of its QCS? The faculty
may organize the findings by either the various elements
of the program’s quality control system or the TEAC
components 4.1–4.7.
Conclusions: What are
the internal auditors’ summative judgments? Here the
faculty addresses two key questions:
-How well is the quality control system working
for our program?
-Is there evidence of student learning that is attributable
to any capacity dimension?
Discussion: In this section,
the faculty addresses several questions:
-What are the implications of the evidence?
-What are the faculty’s conclusions for further
action?
-What modifications, for example, will the faculty
make in its QCS as a result of the findings and conclusions
of the internal audit?
-What investigations will the faculty undertake
to test whether the system is enhancing the quality of the
program and the quality of student learning in particular?
In the discussion section, the faculty
will also recommend ways to conduct the internal audit in
subsequent years. (Return)
|
The quality control system
and Quality Principle III
Whereas Quality Principle I represents the core outcome
of the TEAC accreditation system, student learning, Quality
Principle III represents the core activity of the TEAC accreditation
system, institutional learning. Quality Principle III requires
the program faculty, in monitoring the program’s quality,
to engage in a systematic program of inquiry about the factors that
support the program’s success and failure with respect to
Quality Principles I and II.
TEAC believes that programs must continuously investigate
the factors that contribute to their success and failures, so they
can better understand them and improve upon them. TEAC’s Quality
Principle III and the QCS are precisely about the need for
programs to investigate and understand the sources of their success
and failure, to search for ways to improve the program, and to discard
unproductive practices. (Return)
Linking the quality control system
and student learning
In truth, almost no one has done the inquiry needed to establish
the suspected links between the components of quality control and
the quality of student learning.
For example, through the quality control system,
faculty can monitor whether or not the students admitted to the
program in fact meet the entrance requirements. Yet few programs
are in the habit of asking whether or not the entrance requirements
accomplish the program’s mission or how they might be modified
if they fail to support the mission.
TEAC requires the internal audit so that the faculty
has the occasion to inquire about how the program yields student
learning. TEAC requires the program to include in the internal audit
report its plans to research the link between its capacity for quality
and student learning. (Return)
Return to Appendix A.
report of the internal audit of the quality control system
|
