Evaluating reform outcomes: Is there evidence for efficacy?

Background/Question/Methods

Momentum continues for restructuring introductory STEM courses to better reflect the skills and habits of mind of practicing scientists. Numerous studies indicate performance and affective gains in the context of reformed courses based on data from course exams, concept inventories, and surveys. Yet, there are few data about longer-term impacts beyond the reformed classroom. Evidence of reform efficacy is critical as universities are pressured to demonstrate programmatic outcomes and warrant their resource distributions.

We designed and implemented a reformed, large-enrollment, introductory biology course for life science majors.  Course design was grounded in existing theory about how people learn and consistent with objectives articulated in the 2011 report, Vision and Change in Undergraduate Biology Education. From Spring 2008 through Fall 2011, 9 instructors representing 11 sections (n=2240 students) adopted reformed approaches in their classrooms; 6 instructors representing 13 sections (n=2938 students) maintained a traditional, instructor-centered approach. 

We are evaluating longer-term impacts of reform by analyzing differences between Reformed and Traditional student cohorts in terms of: (1) achievement in upper-division courses in the life science curriculum (e.g., grades and test scores in Genetics, Ecology, Evolution), and (2) patterns of attrition/retention from STEM majors. Students from instructors using mixed (reformed and traditional) instructional methods were not included in our analyses.

Results/Conclusions

Reformed and Traditional cohorts were determined equivalent in terms of demographics (e.g., race, sex) and prior achievement (e.g., GPA, SAT scores, etc.).  Of the variables tested, only AP English and SAT Critical Reading scores differed significantly, with students in Traditional sections having higher scores in each case (p < .05).

Our preliminary analyses indicate: (1) students from reformed and traditional sections do not differ in their subsequent performance in upper-division life science courses.  This suggests that reformed instruction does not hinder students’ preparation for their major on account of “lower content coverage” as is often claimed anecdotally.  However, low alignment between the assessment strategies used in reformed and upper-division biology courses does illuminate the need for clarifying expected outcomes for biology majors at all points in the curriculum.  (2) Reformed and Traditionally prepared students differ statistically in their patterns of retention/attrition from STEM majors (c² = 10.531, df = 2, p = 0.0052). More students from the reform cohort were retained and fewer lost from STEM majors compared to traditional. We are currently exploring drivers of this trend, including whether patterns can be explained by differences within certain demographic or achievement categories.