Cause and goal of the project
When your doctor is unable to tell whether you are suffering from disease A or disease B, would you rather have that they flip a coin to decide which disease to treat you for, or would you prefer that they defer their judgment until more information is available, for instance through blood works or the consultation of an expert? Most people prefer the second option, yet when we examine students in higher education through multiple-choice exams, we are fine with them venturing a guess when they do not know the correct answer to a question. Is this the signal we want to provide to future professionals? In this project, we investigated whether there is an alternative to the common way in which multiple-choice exams are administered at institutes of higher education.
Plan and method
We administered two versions of the same multiple-choice exam to students. One version employed standard setting, the most commonly used procedure at Dutch universities. The other employed elimination marking. Standard setting requires students to answer all questions, even if they do not know the answer. Because students will get questions correct by mere chance, the standard for passing the exam is raised to compensate the guessing rate. It has the downside that it makes the exam result less reliable and that it signals to students that it is okay to guess when their knowledge fails them. Elimination marking, on the other hand, sees students eliminate the answer alternative they know to be incorrect. Students are rewarded for partial knowledge: the more faulty alternatives they can eliminate, the better their result will be. While the procedure aligns with the way students tend to answer multiple-choice exams, signals professional behaviour, and makes for more reliable results, it comes with disadvantages too. The procedure is more complex, making it cognitively taxing and potentially stress inducing. The procedure also does not necessarily eliminate strategic responding and guessing, leaving room for extraneous influences such as gender differences in risk aversion and text anxiety to influence the results. To assess whether the measurement and pedagogical benefits of elimination marking outweigh its potential practical disadvantages compared to standard setting, we compared the students’ results on both versions of the exam, taking into account their gender, risk taking propensity, and text anxiety.
Evaluation and results
Students taking the elimination version of the multiple-choice exam used the procedure to display partial knowledge as intended. They ruled out fewer alternatives on the more difficult questions, for example. We did not find significant differences in the mean grade or passing rate of the two exam versions, and students rated the versions similarly in terms of clarity, fairness, and stress inducing character. The elimination marking results were not systematically affected by gender and did not demonstrate a more pronounced relationship with risk taking propensity and text anxiety than the standard setting version did. In their qualitative feedback, students indicated that they felt the elimination marking version of the exam represented their knowledge well, but they also signalled stress and difficulties adjusting to the new examination format. From a practical point of view, the administration and grading of the elimination marking version was perceived to be more complex by teaching staff because it is not yet incorporated in current examination software. Overall, these results are encouraging, considering that standard setting is considered the default reference for students, and suggests that teachers can consider elimination marking for future multiple-choice exams without risk of systematic bias.
Literature
De Bond, A. E., Bodger, O., Skibinski, D. O., Jones, D. H., Restall, C. J., Dudley, E., & Van Keulen, G. (2013). Negatively-marked MCQ assessments that reward partial knowledge do not introduce gender bias yet increase student performance and satisfaction and reduce anxiety. PLoS ONE, 8(2), e55956. https://doi.org/10.1371/journal.pone.0055956
Vanderoost, J., Janssen, R., Eggermont, J., Callens, R., & De Laet, T. (2018). Elimination testing with adapted scoring reduces guessing and anxiety in multiple-choice assessments, but does not increase grade average in comparison with negative marking. PLoS ONE, 13(10), e0203931. https://doi.org/10.1371/journal.pone.0203931
Wu, Q., De Laet, T., & Janssen, R. (2019). Modeling partial knowledge on multiple‐choice items using elimination testing. Journal of Educational Measurement, 56(2), 391-414. https://doi.org/10.1111/jedm.12213
