AI in the publication business

Open and replicable science
1

I am sure many of you have had experiences with journals using artificial intelligence (AI) for multiple purposes. We had recently one experience with a journal (one of the Frontiers journals), which shows to me some journals are losing control of the situation.

As it happened, after receiving two reviews to our submitted paper from two referees, it did not take us long to perceive striking similarities in their criticisms. Although referees claimed not using AI, the reviews raised our suspicions.We thus suspected foul play and ran the reviews through an AI detection system that returned an assessment that the texts were “100% AI-generated”.

While we initially thought about informing the journal, we decided to try to demonstrate the hazards of allowing AI to perform most of the chores humans should do. We thus prompted an AI system to suggest responses to the presumptive AI-generated criticisms in a manner that would present us with the least amount of work in revising the paper, and to then re-write the manuscript guided by its responses. Note this was a hypothesis-driven experiment: that the AI-guided content “corrections” to the manuscript would go undetected and likely be accepted. The hypothesis was confirmed: our bot minimised the work needed, the paper was revised guided by its responses, the AI contribution went undetected, and the referees endorsed the publication.

At this point we disclosed our experiment to the journal. It had shown that if we had continued on we may have ended up publishing a paper which contains conceptual misunderstandings, which is expected considering the features of LLMs: the bot tried to create a coherent story based on its responses to the referees’ AI-generated critiques, but the story in the end was not really our story.

We see then that with the increased automation in the submission and review process, things can get out of control ( Scientists Invented a Disease to Test Whether A.I. Knew It Was Fake. Then, Chatbots Started Saying It Was Real ) with the consequent results in quality of research.

It goes without saying we quit that journal and submitted the paper to another (still under review).

But the fun never ends. To make matters worse, last week we received an (undoubtedly AI generated) email from that Frontiers journal saying that our paper “has been read and cited well past what most work in the field see”… But we didn’t publish it yet and surely won’t be published in that journal!

So unless some sense is used with the AI processes in the publication business and science in general, I am afraid things will become weird indeed.

1 Like
2

This reminds me of an interesting item from the NYT last week: This Is a Rare Arena in Which People Think A.I. Is Helping. It’s Making Things Worse.

I like the point about the discretionary difference between AI solving a mathematical problem inside a black box, which doesn’t actually count as solving the problem, and AI as tool in biology, where it can solve a protein folding problem or whatever, which either works or doesn’t work. The scientific community is extremely concerned with the mission creep aspect of AI in the lab, but I think that this Navier-Stokes debacle actually proves that there is a pretty bright line between AI and human jobs in the research workplace.

The people who look at AI as an easy way to make money are not going to care about the quality of the work or its output. The people whose income depend on quality and accuracy can use AI as a kind of benchmarking tool, but for now, it seems like the speed of development is not quite relevant to the question of whether AI can replace human researchers.

I think there could be a similar trend with publishers and peer review that could widen the gap between the already well-resourced high quality journals and all those competing in the space–whether they are predatory or not. Sadly, OA outlets could greatly benefit from the right kind of AI intervention to manage quality control and possibly even do preliminary review.

Would you object to an AI gatekeeper designed to weed out fraudulent submissions? Or tooling that could cross-check citations, and possibly even validate the use of cited materials as forms of evidence?