QRSR allowed for mapping of individual test items to specific program objectives. With the support of an NSF grant, an interdisciplinary team mapped the existing items and developed ten additional items to assess curricular outcomes not covered by the original QRSR.
In academic year 07-08, all students with junior status were randomly assigned to either CAAP or the modified version of QRSR. In total 655 took QRSR and 580 took CAAP. Student information systems were used to collate data about participants major, GPA, ACT, number of math and science courses taken, and grades in those courses. In fall 2008, 135 first-year students were also tested on QRSR.
Results/Conclusions
The tests had comparable indices of reliability: CAAP manuals report KR20 between .84 and .86 and alpha for QRSR was calculated between .80 and .86. Each test also showed the expected relationships with existing measures of math and science ability. For example, math and science majors scored significantly higher than nonmajors. Each test correlated moderately well with ACT Math and Science scores (r values ranged from .517 to .685) . QRSR had small but significant correlations with the number of courses taken in Biology, Chemistry, Math & Physics and with grades in introductory Biology, Math & Physics. CAAP correlated with classes in Agricultural Sciences, Biology, Chemistry, Math, Physics & Statistics and with grades in Biology, Chemistry, Math & Physics. However, the correlation with number of Agricultural Sciences courses was negative.
QRSR showed promise for measuring “value-added” effects. The mean for first-year students was approximately 11 percentage points lower than the average score for juniors (t (730) = 14.12, p < .001).
QRSR showed a clear advantage in transparency. Each item mapped to at least one curricular objective, and subscales could be computed. Information about how the process of test selection and the results are being used as part of curriculum revision will also be discussed.
