Validity issues
There are validity issues for each category of evidence.
Rates. Hiring rates, for example, based upon
the hiring district’s own evaluation of the subject matter
knowledge, pedagogical knowledge, and caring teaching skill (Quality
Principle I components), may not be as much an indicator of
student accomplishment in times of teacher shortages, such as are
expected in the decade ahead, as they would be in times of teacher
oversupply. In times of shortage, hiring rates may indicate very
little about quality because virtually everyone is hired. The rate
of first choice hires, for example, may prove to be a more persuasive
indicator of student accomplishment.
Similarly, some categories of evidence may be relatively
meaningless if the rates are low or less than the normative rates.
The rates may indicate something important about the program’s
quality, however, if the rates are significantly higher than the
norm.
Passing rates on the currently available teaching
license tests, for example, are surprisingly high*,
but some passing-scores are set as low as the 25th percentile of
actual cohort performance and with fewer than half the test’s
items answered correctly in some cases. Retention, program completion,
and graduation rates average 50 percent in most cases. Rates have
meaning in the TEAC framework only if they are based upon an evaluation
by a third-party of some aspect of Quality Principle I.
Survey data particularly that
derived from survey forms created by amateurs are known to be affected
by a number of extraneous factors. For example:
· the order in which questions
were presented
· the context in which questions
appeared
· whether the questions weed out those
with no opinion (filtering)
· the range and order
of choices· whether middle categories were provided
· whether the format was open or closed |
Survey results need to be examined for their reliability
and validity, as do course grades.
Course grades are meant to be
a measure of subject matter understanding, but their validity is
threatened by the fact that they are frequently measures of other
matters that may have only a tangential or no relationship with
the student’s mastery of the subject matter of the course.
Some of the common
threats to the validity of course grades occur when they become
influenced by other factors and become as a result measures of these
other factors. In contemporary higher education, it is fair to say
that grades may be, in varying degrees, measures of any, or all,
of the following:
The inference that grades, or any other measures
of learning, are valid can be based on a number of considerations
and investigations:
| • Are the
grades the faculty members give consistent and correlated
with other known measures of student learning (e.g., standardized
tests of the same content)?
• Are they based on the appropriate content so that
they measure only what they are supposed to measure?
• Are they correlated with and predict later accomplishment
that depends on student learning?
• Are they related to other factors that one would
expect, in theory, to be related to what the grade measures
(e.g., intelligence, prior grades, aptitudes, specialty
training, beginning or end of the program accomplishment,
motivation)? |
In general, the correlations about .50 provide
confidence that the measure is valid for the purposes to which it
is put.
* Pass rates of 100 percent are becoming common, but many programs
achieve them by using the state’s license test as a program
admission test or screening test for latter stages of a program.
High pass rates in this instance are of little use as indicators
of program quality.
Return to Inquiry Brief Methods
of assessment or Results or Inquiry
Brief Proposal
Format |
