Assessing the academic capabilities of students should play a key role in both stimulating their learning process (formative assessment - FA) and in the accurate evaluation of their knowledge and capabilities in relation to a topic (summative assessment - SA). Therefore, based on the principle of constructive alignment, any form of assessment needs to be carefully designed to match the learning outcomes of a course, needs to be delivered in an appropriate format (paper-based versus computer-based), and needs to be scored in a suitable manner.
However, this is a challenging task, due to the substantial amount of time teachers need to spend on grading open questions. On the one hand teachers use less appropriate assessment methods (Multiple Choice questions) or on the other hand they spend less time on innovating their courses (implementation of FA).
In this project the application of Machine Learning (ML) technology for supporting grading of open questions, for both SA and FA, is investigated. Expected results include the design of a method supporting grading of open questions with ML, an investigation into the most suitable ML algorithms for small samples of tests, and insight into how ML can be applied in FA to make its usage more appealing to teachers. Additionally, as the most innovative aspect, the feasibility of a state-of-the-art version of FA (assessment as learning) is investigated, focused on student self-assessment with the help of short-form open question quizzes assessed with ML techniques.
Implementation of the project: web application Remindo developed
The majority of the tasks that were described in the initial proposal have been executed according to the plan. This includes the literature reviews, the interviews with teachers, the re-design of the open question structure, the gathering of the requirements, and the design of the interface for the web application.
The development of the web application took a bit longer than expected. However, with the help of an external software development company the prototype has been completed according to the designed specifications. Additionally, several improvements for usability and user experience were implemented.
Due to the COVID-19 pandemic not all teachers were able to test the prototype or provide feedback on the design.
Further development
The prototype has been fully developed according to the specifications of the mockup and is available on the servers of the University of Twente (UT). Any teacher who uses the Remindo system for digital examination and has exams with open questions can test and use it. The aim is to continue the development of the prototype into a more robust tool with additional functionality based on the feedback from the teachers who tested it. In the near future, variations of the tool will be investigated for other types of assessment purposes, such as essay assignments, standardised testing for secondary education, and formative assessment.
The prototype will be hosted on one of the servers of the UT where every teacher who wants to test and use the prototype are able to do so when they want to. Additionally, maintenance and the implementation of additional improvements can be done with the help of the BMS lab.
The adoption of the web application is also supported by the close participation of the Centre for Educational Support (CES) and the Technology Enhanced Learning and Teaching (TELT) staff. These two groups are the driving force behind the adoption and usage of new technology at the University of Twente.
| Ontwikkelperiode: | 2019-2021 |
|---|
Producten of presentaties over dit project
- Interview and article for the UToday magazine - 6 januari 2020
- Aldea, A. I., Haller, S. M., & Luttikhuis, M. G. (2020). Towards grading automation of open questions using machine learning. In J. van der Veen, N. van Hattum-Janssen, H.-M. Järvinen, T. de Laet, & I. ten Dam (Eds.), Engaging, Engineering, Education: Book of Abstracts, SEFI 48th Annual Conference University of Twente (online), 20-24 September, 2020 (pp. 584-593). University of Twente. https://ris.utwente.nl/ws/portalfiles/portal/247489928/Kleiner2020towards.pdf
Comenius Teaching Fellowship 2019. Raadpleeg hiervoor nro.nl.
