Complexity versus novelty: A factorial approach to assess student learning in an introductory biology course

Background/Question/Methods

A major challenge of creating summative assessments in the biological sciences is crafting questions that directly address objectives for student learning. We focused on this issue in an introductory biology laboratory course by examining one learning objective (that students will be able to apply conceptual and analytical knowledge to increasingly more novel and more complex situations). To assess student performance regarding this learning objective, we created sets of multiple true-false questions that followed a factorial design, with questions in four categories based on course content: (A) familiar content and moderate complexity level (Control); (B) unfamiliar content and moderate complexity level; (C) familiar content and higher complexity level; and (D) unfamiliar content and higher complexity level. Questions were administered as part of a final practical exam for students over three semesters (n=1150 total students), and a discrimination index was used to test the ability of all questions to discriminate between high and low performing students on the basis of final practical exam scores. We hypothesized that students would perform the best on questions in category A, the worst on questions category D, and at equal intermediate levels on questions in categories B and C.

Results/Conclusions

On average, students got 79% of the questions in category A correct, 73% of the questions in category B correct, and 71% of the questions in categories C and D correct. Compared to encountering questions in category A, encountering questions in all other categories reduced the odds of scoring correctly, but to different extents. Discrimination index values for the questions did not differ significantly across the four categories, indicating that the ability of the questions to discriminate between high and low performing students was equal across categories, and that no one category contained questions of lesser quality than the others. Surprisingly, category C questions reduced the odds of scoring correctly more than category B questions, and students performed equally poorly on questions in categories C and D, even though category D questions were hypothesized to be the most challenging for students. These results suggest that, in an introductory biology course based heavily in ecology and evolution, students have more difficulty with questions that increase the complexity of a situation (even if content is familiar) compared to questions that ask students to apply concepts they have learned to a novel situation.