Strengthening literature evaluation in the age of generative AI

Check & double check

For Tamas, the starting point was a concern about how students engage with literature: “Engagement with literature is often sloppy, both when it comes to content as well as with references. The rise of genAI adds a problem to this, being that in the worst-case scenario, students are no longer engaged in literature at all: they accept the genAI summary, which means that students never learn good scientific practice.”

Kamminga notes that when using genAI tools to search for information, it is very tempting to copy-paste the AI-generated answer. She warns against this: “When a statement is made in a source with a reference to where the statement was derived from, it is commonly assumed that this is correct. Unfortunately, this is not always the case. Even when the model gets the quote right, that is not enough. Authors can interpret the text in the original source differently. When working with scientific information, it is essential to spend time reading the primary source, the first source that makes the claim you want to work with.”

Students are often asked to check their sources. In practice, this is time-consuming to do and is rarely checked. The assignment used in the study forces students to check & double check to ensure the information they use in their work is correct.

Looking beyond fluent prose

The assignment was also developed in response to a broader challenge created by genAI. Tamas notes that “as teachers, we always relied on fluency of the report to measure some combination of competence and care. With genAI, that proxy is in serious trouble: we often can’t tell the difference between careful, competent crafting by a student and fluent continuation produced by a model.” He puts it rather bluntly:

“We have long treated the quality of a student’s prose as an adequate proxy for their care and competence. This proxy is no longer adequate. Prose is dead.”

Kamminga stresses that proper engagement with literature takes time and effort: “It takes real time and effort, particularly the first few times, both to engage in literature properly and to learn why this is important. Courses must provide sufficient time and structure for these tasks to prevent students from coming to think that it is normal to trust first impressions. This has never been adequate, and genAI has made the risk of such naive trust far worse.”

"Students, particularly early in their education, must discover for themselves why it is worth the time to read & check sources, and they must be provided the opportunity to learn how to do it", she adds. 

A simple citation log

For the assignment described in the article, students were required to submit an additional file documenting their use of sources, along with the sources themselves. The log was an Excel file with three columns:

• Quote from report
• Relevant material from the source
• Filename of source

The file was submitted along with all sources in a zip file, which was checked using a simple script. The log is a method to keep track of all the different steps that are taken when building an argument. This helped to keep an overview of work and make it transparent. For Tamas, the purpose goes beyond administration: “It forces students to document the dependencies of their argument on the literature in a manner that is consistent with work on active citation that is now, given genAI, necessary.”

Following claims back to their source

The article also describes a protocol for testing citations that has been used successfully in undergraduate classes. Students were asked to analyse statements in a scientific source and rank them by importance to the most important conclusion in the article. The question was: “If this statement were wrong, would the conclusions the article makes hold water?”

Students ranked statements by their potential impact on the conclusion and, for those supported by a citation, traced that citation back to the source. Tamas points out that the exercise consistently produced two outcomes: “students discover that articles are not as sound as they seem at first glance, and citations are often bad.”

How students responded

The exercise was very well received by undergraduate students. “In overview, the students accepted the assignment as a natural part of their course programme. It was not remarked as odd until I mentioned that it was odd. Most of the time, it was seen as a reasonable reaction or adaptation to our genAI-infused world,” Tamas reflects. The most encouraging result was the students’ willingness to participate. Tamas is enthusiastic about the outcome:

“Students went for it. No, really, that is the cool bit. Students just accepted this work as a natural part of responsible and transparent science. This means that the normative roots required to grow responsible scientific practice are present.”

Tamas believes all that is needed is to offer ways in which students’ care can appropriately be expressed: “Part of that is keeping the weeds back (blocking casual abuse of genAI and part of that is showing them how to demonstrate their care and their competence in ways that fit today’s realities.”

One piece of advice

If Kamminga could give students one piece of advice, it would be simple: “genAI tools are just that: tools; they can’t take over your work. Always make sure to engage with literature: read it, understand it, challenge it, make connections.”

For Tamas, the lesson is equally clear: “We should teach students from the start that checking sources is a normal part of our academic culture and responsible science.”

In an era where genAI can produce fluent text instantly, the challenge is not simply preventing misuse. It is about helping students learn how knowledge is built, assessed and justified by engaging directly with the literature itself. Tamas argues that these activities should be part of students’ primary introduction to science and socialisation: “students should learn that this is a standard part of academic culture, so it should be in the very first course where any undergraduate is asked to engage with the literature.”

The value of libraries

WUR Library teaches information literacy in many bachelor's and master's programmes, although the level of integration varies. Some programmes offer a comprehensive learning trajectory, with information literacy embedded throughout the curriculum. For Kamminga, responsible use of information is a vital skill to teach students, now more than ever.

“In the current era, where scientific information is provided via different channels, it becomes increasingly important to be aware of the information landscape, how scientific information is generated, and how it is being processed.”

Information literacy teachers have always stressed that readers should critically assess sources. In practice, however, publication source and authority have often been used as proxies for adequacy. “This has never been completely sound, but it has, perhaps, been tolerable. Today, these proxies are no longer tolerable. We must train our students to check all sources, no matter how fluent,” Kamminga argues. She believes that a critical attitude towards scientific literature should not be something added later in a student’s education:

“Students should never be allowed just to ‘trust’. If we start with that stance, then getting critical later is recognised as an expensive exception. It is far more prudent for trust to be the exception condition.”

Over the last few years, WUR Library has integrated genAI into its training activities. “During our training sessions, we make clear at which moment during the cycle of literature retrieval genAI tools can be used and at what stages it is better to refrain from it", Kamminga explains.

This is reflected, among other things, in the Library’s recently revised and publicly available e-learning modules. WUR Library is also currently running several projects on genAI and information literacy. One focuses on testing genAI tools for finding and analysing literature. Another focuses on updating the information literacy learning trajectory to include the responsible use of genAI tools for literature searching.