NeurIPS is an annual machine learning conference. Here in the AI bubble, NeurIPS is huge and flooded with chatbot cash. [Atlantic, archive]

The AI vendors at NeurIPS want to talk about imaginary problems, like the super-AI destroying humanity. The researchers want to get on with doing science.

You want to present your work at NeurIPS. You submit your conference paper. It goes to a reviewer — usually someone else who submitted a paper. They check your paper’s up to standard.

Unfortunately, a lot of reviewers lately are passing the review job to the chatbot. This misses the entire point of peer review. But it sure saves time!

NeurIPS has a clear rule: it flatly forbids reviewers from using the chatbot: [NeurIPS]

Our policy is strict: reviewers may not use any LLMs or AI agents in the review process.

But a few reviewers didn’t listen. And now they’re yelling in august scientific venues such as LinkedIn that they got caught fobbing their work off to a chatbot.

How did they get caught? NeurIPS put chatbot prompts hidden in the PDFs they sent out for review. These told the chatbot to use a particular phrase in the review.

The bros were deeply aggrieved: [Transmitter]

You do not build a healthy reviewing culture by treating your reviewers as suspects.

You don’t build it by fobbing your reviews off to a chatbot, either.

NeurIPS wasn’t the first machine learning conference to do this.The International Conference on Machine Learning did the same thing earlier this year.

ICML actually had two policies for chatbot reviewing. “Policy A” was no chatbots allowed. “Policy B” was: “LLMs allowed to help understand the paper and related works, and polish reviews.” [ICML]

That’s easy, right? But AI bros refuse to understand consent.

So a pile of reviewers signed up for the no chatbots track — and they used chatbots anyway! And they got busted:

795 reviews (~1% of all reviews) written by 506 unique reviewers who were assigned Policy A (no LLMs) were detected to have used LLMs in their review. Again, recall that these are reviewers who explicitly agreed to not use LLMs in their reviews.

… If the designated Reciprocal Reviewer for a submission produced such a review, their submission was rejected.

So the bros who cheated got their own paper kicked out.

How did this go down? Was there a flood of deep concern?

It turns out the AI bros whining on LinkedIn are not the majority. The scientists were largely delighted with the ICML approach:

Researchers expressed “overwhelming support” for the strategy, says Shah, who adds that he shared the methodology with the NeurIPS team. “I have been working on conference peer review for several years, and I have hardly seen such strong support for anything,” he says. “People were really tired of reviewers copy-pasting AI-generated reviews without putting any effort.”

Academic reviews are work — but nobody likes a cheat.

• Video — Podcast