This NSF funded project enjoyed the participation of five partner institutions with highly diverse institutional missions and student populations. A primary goal was to assess the generalizability of a locally developed (at James Madison University) measure of quantitative (QR) and scientific reasoning (SR) to other institutions with the goal of enhancing assessment practice at home institutions. Our primary focus was direct assessment of student learning and development of collegiate quantitative and scientific reasoning.
The instruments were developed over a 10-year period with all items written by faculty teaching in the University's general education program. Four partner institutions conducted content alignment exercises to map the instrument items to their respective home institution goals and objectives. This alignment process is useful for developing and testing any type of assessment instrument. Finally, conference participants will be shown how to conduct such exercises to select instruments and how to fill any identified content gaps.
The review of the instrument development shows continued positive growth in reliability and validity estimates. In relation to the content alignment exercises, our results indicate very strong content alignment of the items comprising the JMU QR and SR tests with the goals and objectives of the four partner institutions' home goals and objectives. Every institution mapped well over 90% of the JMU developed items to their home student learning objectives. When content gaps were identified, new items were developed to establish better content validity and coverage.
Results/Conclusions
The use of a test blueprint and content alignment exercises will be demonstrated. Our research clearly indicates that use of the item by item methodology results in judges assigning individual items to only one objective, whereas use of the objective by objective methodology results in assignments of individual items to multiple student learning objectives. Further, faculty involvement in content alignment exercises results in much greater engagement and confidence in assessment results.
During the same content alignment exercise, it is an extremely cost effective strategy to ask faculty to make judgments about the percentage of students who have completed the appropriate coursework that should respond correctly to each item. Aggregation of these judgments across faculty members produces a criterion referenced interpretation of student performances. We are going to provide our students for the first time with feedback on their performances in QR and SR in March 2009 via JMU’s eCampus application.
