AI is writing convincing fake research papers. How one scientist is fighting back

BINGHAMTON, N.Y. — “Publish or perish” has long been the mantra of academia. But what happens when the publications are penned not by perishing professors but by perpetually productive AIs? As artificial intelligence muscles its way into scientific writing, one researcher is fighting back with a tool that could change the game.

Large language models like ChatGPT continue to become increasingly sophisticated, and there’s growing concern about their potential misuse in academic and scientific circles. These models can produce text that mimics human writing, raising fears about the integrity of scientific literature. Now, Ahmed Abdeen Hamed, a visiting research fellow at Binghamton University, has developed a groundbreaking algorithm that might just be the silver bullet in this high-stakes game of academic authenticity.

Hamed’s creation, aptly named xFakeSci, is not just another run-of-the-mill detection tool. It’s a sophisticated machine-learning algorithm that can sniff out AI-generated papers with an astonishing accuracy of up to 94%. This isn’t just a marginal improvement; it’s a quantum leap, nearly doubling the success rate of conventional data-mining techniques.

“My main research is biomedical informatics, but because I work with medical publications, clinical trials, online resources and mining social media, I’m always concerned about the authenticity of the knowledge somebody is propagating,” Hamed explains in a statement.

His concern isn’t unfounded. The recent global pandemic saw a surge in false research, particularly in biomedical articles, highlighting the urgent need for robust verification methods.

In a study published in Scientific Reports, Hamed and his collaborator, Professor Xindong Wu from Hefei University of Technology in China, put xFakeSci through its paces. They created a testbed of 150 fake articles using ChatGPT, evenly distributed across three hot medical topics: Alzheimer’s, cancer, and depression. These AI-generated papers were then pitted against an equal number of genuine articles on the same subjects.

The algorithm uncovered distinctive patterns that set apart the AI-generated content from human-authored papers. One key difference lies in the use of bigrams – pairs of words that frequently appear together, such as “clinical trials” or “biomedical literature.” Surprisingly, the AI-generated papers contained fewer unique bigrams but used them more pervasively throughout the text.

“The first striking thing was that the number of bigrams were very few in the fake world, but in the real world, the bigrams were much more rich,” Hamed notes. “Also, in the fake world, despite the fact that were very few bigrams, they were so connected to everything else.”

This pattern, the researchers theorize, stems from the fundamental difference in the objectives of AI models and human scientists. While ChatGPT aims to produce convincing text on a given topic, real scientists focus on accurately reporting their experimental methods and results.

“Because ChatGPT is still limited in its knowledge, it tries to convince you by using the most significant words,” Hamed explains. “It is not the job of a scientist to make a convincing argument to you. A real research paper reports honestly about what happened during an experiment and the method used. ChatGPT is about depth on a single point, while real science is about breadth.”

Study authors warn that as AI language models grow more sophisticated, the line between genuine and fake scientific literature could blur further. Tools like xFakeSci could become crucial gatekeepers, helping maintain the integrity of scientific publications in an age of ubiquitous AI-generated content.

However, Hamed remains cautiously optimistic. While proud of xFakeSci’s impressive 94% detection rate, he’s quick to point out that this still leaves room for improvement.

“We need to be humble about what we’ve accomplished. We’ve done something very important by raising awareness,” the researcher notes, acknowledging that six out of 100 fake papers still slip through the net.

Looking ahead, Hamed plans to expand xFakeSci’s capabilities beyond medicine, venturing into other scientific domains and even the humanities. The ultimate goal? A universal algorithm capable of detecting AI-generated content across all fields — regardless of the AI model used to create it.

Meanwhile, one thing is clear: the battle against AI-generated fake science is just beginning. With tools like xFakeSci, however, the scientific community is better equipped to face this challenge head-on, ensuring that the pursuit of knowledge remains firmly in human hands.