The faculty also describes the research design it has employed to secure the evidence. Was the evidence based on all the students and graduates of the program? Some representative sample? If it was based on a sample, how was the sample drawn and determined? The faculty members also describe how the research design addresses rival explanations for the results and how they will address potential aggregation errors and other threats to the validity of their findings.

The methods section describes any assessments and measures that will provide corroborating evidence for the faculty’s main findings and any other evidence that has a bearing on any rival or alternative explanations of their findings.

The methods section speaks to TEAC’s Quality Principle II: the program faculty documents its claim that the inferences it draws from the evidence of student learning are sound and valid. This section is about whether the methods will prove to be credible, trustworthy, and reliable.

The design of the faculty’s investigation must support the faculty’s interpretations of the results of its assessment system and the appropriateness of the uses to which it puts them. The faculty members must consider several factors: evidence about the content of the assessments, the assessment criterion relationships, the theoretical and scholarly basis of the construct they assessed, and the uses to which they put the assessments.

In the Inquiry Brief, a program faculty will typically provide evidence of the quality of student learning in the program. Typically, programs use some combination of the categories of evidence presented in the chart following this page. However, each program is encouraged to present novel and tailored evidence of student learning, in place of or in addition to, these categories.

Qualitative assessments and measures. When a program faculty uses qualitative assessments and measures, those writing the Inquiry Brief describe the methods of procuring the evidence and give a rationale for them, just as with any quantitative assessment. The program faculty would present precisely the procedures it employs: for example, team-recorded observations; interview protocols with students, alumni, faculty, administrators, employers; representations of student products or artifacts; interpretations of student journals, lessons, field notes, audio/video presentations.

Linking to Quality Principle I. Whether qualitative or quantitative, each source of evidence must have a clear link to a component of Quality Principle I. Without such links, the measures may still have value, but only in documenting the context of the program or providing corroboration for subsidiary claims in the Inquiry Brief.

Categories of evidence. The program faculty actually has a fairly limited number of sources of evidence with which to make its case for its claims about Quality Principle I.

Multiple measures. Because each kind of evidence (grades, surveys, portfolios, standardized tests, etc.) can be misleading, it is important that the faculty commit to include several measures that converge, triangulate, and indicate true student learning. The faculty should also take steps to reduce factors that which affect the validity of the faculty’s interpretations. (See Comment on issues of reliability and validity.)

The methods section of the Inquiry Brief gives a complete account of the measures and the faculty’s case for the reliability and validity of the measures. Table 1 offers one way to present the information about measures that is required in the methods section. Table 1 summarizes the kinds of information that the faculty would include in the methods section to address Quality Principle I. The table makes clear that the program faculty must provide at least two measures for each component of Quality Principle I and must describe and report its methods of investigating the reliability and validity of the measures.

In the case of qualitative measures, the faculty should present the triangulation methods used to reduce error and increase the trustworthiness, dependability, and authenticity of the measures.

COMMENT on validity issues and reliability issues