The problem with AI citing retracted research

“Bad information can lead to bad decisions,” says Yuval Harari in an interview about the dangers of AI and information. There is now a rising concern that Generative AI like ChatGPT will use retracted research articles to provide answers to prompts. The implications of this issue are problematic to say the least.

It has been known that ChatGPT hallucinates from time to time and you can often mitigate this issue by asking for a source. There is a possibility that even with a real source, ChatGPT will still be wrong because the article was retracted as errors or inconsistencies were found in that article.

Having ChatGPT use retracted articles is problematic for several reasons. First, the user will get incorrect information, like a doctor getting incorrect information on cancer imaging, as was found in the research mentioned above. Second, the retracted article will become part of the dataset of a model and remain there, ultimately reshaping the output. Third, if the retracted article was found to be AI generated (which is becoming more and more a thing), we have AI citing AI content which can eventually cause data degeneration.

The research was using ChatGPT and not a Deep Research tool. It would interesting to see whether Deep Research AI can identify retracted articles and avoid them when conducting a search. Otherwise, Deep Research output will become a lot less trustworthy.

Yuval Harari states that “in a free market of information, the truth is bound to sink to the bottom and not rise to the top. We need to give it help.” Academia should be vigilant in keeping the truth afloat lest it will drown in a sea of information.