The paper was accepted at the prestigious SIGCSE conference, which is supported amongst others by Google, Microsoft, and the U.S. National Science Foundation, and focuses on topics related to computing education. On December 5, the Data Science students present their preliminary work, which aims to facilitate the comparison of automatic grading systems. In it, they combined seven commonly used English short-answer grading datasets, which included a total of 19,000 datasets with questions, answers, and grades. This allowed them to evaluate a range of current SAG methods. The results showed that AI-based approaches, while promising, are still a long way from human performance. The insights gained by analyising these results on more detail open up new avenues for future research on human-machine SAG systems.

Other findings from the study are that Large Language Models (LLMs) are transferable to different situations and can evaluate new questions. Specialised grading systems have more difficulty doing this – in these cases, the AI needs to be "fine-tuned", i.e., specifically trained. Moreover, the ability to generalise improves as the size of an LLM increases. In a next step, the students plan to address the problems they encountered during their work. They want to extend the benchmark in the future with more diverse datasets to reduce biases, include multilingualism, and conduct a more thorough evaluation of automatic grading solutions.

So it will be some time before university exams can be created and marked fully automatically. Ultimately, work like this gives us solid insights into the possibilities and limitations of artificial intelligence in concrete applications - how we use it is up to us humans.