AI-Written Referee Reports Threaten Peer Review, Economists Warn
Economists from the Federal Reserve Board, University of Pennsylvania, CEPR and NBER warn that large language models can now write papers and referee reports, putting peer review at risk of collapse without new norms. Their experiment shows AI detectors flag reports written by AI but not those merely proofread or edited by it, and they argue AI-generated referee reports should be treated like plagiarism and rejected.
Full Report
Board of Governors of the Federal Reserve System University of Pennsylvania, CEPR, NBER
November 16, 2026
Abstract
Large language models now write papers and referee reports in economics. The field must develop best practices for handling AI-generated text in the publication process. Otherwise, peer review is in serious danger of collapsing. We propose that AI detectors should become a standard tool of the profession. Our experiment shows that they flag referee reports written by AI, not reports that AI merely proofread or helped edit. Aided by a simple model, we argue that a paper flagged as AI-generated should trigger a closer reading and can be rejected at the first mistake, and that an AI-generated referee report adds nothing that editors could not get by running the prompt themselves. Such reports are unacceptable: they damage authors and journals and should be treated like plagiarism. Individual restraint has already shown its limits, and the profession needs to develop norms to solve this collective action problem. We close with a practical guide for authors, referees, and editors.
JEL Classification : A14, D82, O33
Keywords : Large language models, AI detection, peer review, referee reports, moral hazard, verification.
∗
Correspondence: [email protected] and [email protected] . We thank Emine Boz, Kevin Bryan, Jing Li, Justin Pierce, Evi Pappa, and Immo Schott for useful comments. We also thank Sofie Grouws and Shayan Khan for outstanding research assistance. We used LLMs while preparing this paper, as we describe in footnote 1. However, we are responsible for all remaining errors. We have no financial ties to, or interest in, any of the companies cited. The views expressed in this paper are solely the responsibility of the authors and should not be interpreted as reflecting the views of the Board of Governors of the Federal Reserve System or of anyone else associated with the Federal Reserve System.
11 Introduction
The economics profession is seeing a rapid increase in the use of large language models (LLMs) to assist research. LLMs can be extremely useful at various stages of a research project, and, like most people, we use them routinely in our work. We write better papers thanks to them. 1
The use of LLMs in academic writing and throughout the peer-review process is already widespread, and anyone who has recently written, refereed, or edited a paper has seen it. 2 We contend that this fast increase in AI-generated content puts peer review in jeopardy: journals will be flooded with low-quality submissions, economists will decline requests to referee papers or routinely use AI to referee them, and which papers get published will become less and less linked to their quality. We do not want to defend the status quo entirely. Peer review is slow, many reports are poor, and editors (including us!) make bad calls all the time. Still, well-executed peer review remains the best system we have to assess each other’s work and advance the profession. 3 AI could improve peer review if used deliberately for that purpose, for instance, to check proofs and code. That is not what we are seeing. As editors, we have already encountered papers and referee reports that are 99 percent AI-generated according to state-of-the-art detection software such as Pangram . As authors, we have already received referee reports that were mostly AI-generated, even from the top journals in the profession. And we cannot emphasize this enough: the real problem is not that the papers and reports were AI-generated; it is that no one had done the thinking behind them, and it showed. The papers were eminently forgettable, and the referee reports were disappointingly uninsightful. We have seen too many reports of the form: “The paper is well-written and addresses an important problem. However, the authors should consider additional baselines.” 4
Our starting point is a common distinction in the debate about AI: whether it complements or substitutes for human effort. AI is a complement when it helps researchers think better and faster: finding related work, wrangling data, writing and debugging code, or improving
1We are happy to acknowledge that we used LLMs to help with this paper. For example, we used Claude Opus 5.5 to check the strengths and weaknesses of our arguments, suggest ways to develop them further, and improve grammatical flow. We used Refine.ink to evaluate a nearly final draft. 2Gartenberg et al. (2026) run Pangram on submissions and referee reports at Organization Science volume in 2024–2025 was 42 percent higher than in 2022–2023, and the growth of AI-generated writing accounts for nearly all of the increase. In a 2025 survey, more than half of about 1,600 researchers reported using AI tools when reviewing manuscripts (Naddaf, 2025b). 3More radical alternatives, such as abolishing peer review or having editors decide without referees, are possible, but we have not seen serious proposals along these lines, so we do not discuss them here. 4Views on AI in refereeing already differ widely, from enthusiasts arguing for the adoption of AI-based peer review tools to skeptics who see AI-written reviews as the end of “the entire social contract of peer review” (Naddaf, 2025a).
2the flow of a draft. And, for those of us whose first language is not English, AI helps us avoid embarrassing grammatical mistakes or wasting an inordinate amount of time determining if the right preposition is “in” or “on.” AI is a substitute when it does the thinking instead of the researcher. Our view is that AI should be a complement, not a substitute. The difficulty is that readers, including editors and referees, cannot see how AI was used. Before AI, writing was costly, so a carefully written text was evidence (imperfect, but informative) that someone had thought through the paper and checked its execution and writing. AI removes the cost of writing and, with it, the signal. As a result, in practice, the burden of verification shifts from the author to the reader. Bad papers and referee reports existed long before AI. The problem is that, with AI, there are many more of them. Even if we can identify the bad papers and reports one by one, doing so takes time and resources away from good, thoughtful papers. On the surface, the careful author who has spent weeks going over every result with utmost diligence and the careless author who just submitted the output of an LLM look alike. Readers will recognize this as a version of the market-for-lemons problem (Akerlof, 1970). If readers cannot easily observe the quality of a text, good and bad papers (and referee reports) pool. Consequently, good papers are discounted along with the bad: the innocent pay for the sins of the guilty. To us, this is the clearest argument for AI detectors: they tell us where our time should go. Without detectors, a reader must vet every text as if it could be AI-generated, which takes more time. And detectors can be trusted in the direction that matters: a long text flagged as AI-generated is almost certainly generated by AI. No one has an incentive to make their own human-written text look AI-generated, so false positives come only from errors of the detector, which are rare (Jabarian and Imas, 2025). At the same time, AI detectors are not flawless: a text classified as human might still have been generated by AI, because authors who want AI text to pass as human have both the incentive and the tools (see, for example, undetectable.ai ). We will come back to this point repeatedly. The damage is much more serious when authors and editors receive AI-generated referee reports. A report generated by AI looks like a careful one (particularly in its visual “look and feel”), but it carries the judgment of the LLM, not the judgment of the referee. Authors whose papers are rejected on the basis of such reports are rightly frustrated. Since authors can run the reports they receive through an AI detector as easily as editors can, these rejections also damage the reputation of the journal. In our profession, we all play several roles at once: we are authors, referees, often editors, and always readers of each other’s work. A referee who delegates a report to an LLM today is likely to receive an AI-generated report as an author tomorrow. Each of us gains from using AI to cut corners in the boring roles (e.g., refereeing) and loses when others do the 3same in the interesting ones (e.g., as authors who want a good evaluation of our research). And that is why we need shared norms. Individual restraint has already proven insufficient. Our goal is to preserve and improve the quality of peer review and the progress of economics as a field. We would count it as a success if the arrival of AI pushed the profession to raise the standards of peer review at every level, starting with how we train graduate students to referee and how we design evaluation mechanisms. In the rest of this paper, we discuss how the profession should respond, drawing on our experience as authors, referees, and editors. We try to answer four questions that every journal will soon have to settle. Are we worried about the writing or about the verification of the analysis? About the quality of the paper or about errors? Should a text flagged as AI-generated lead to closer scrutiny or to disqualification? Should the answer differ for authors and referees? AI detectors play an important role in what follows. But our core argument is about verification, judgment, and incentives, not about AI detection. We focus on text because text is what detectors read and what referees write. The use of AI in the execution of the research (data, code, proofs) matters at least as much, but the rule there is simpler: AI can do as much of the execution as the author wants, as long as the author verifies the output. We build our argument in three steps. First, we run an experiment with 25 real referee reports published by Nature Communications (Section 2.2). We ask two LLMs to undertake four exercises with each report: to proofread it, to edit it for language, to rewrite it, and to write a new report from the published paper alone. Then, we score every version with
Pangram . The first two exercises, proofreading and language editing, leave the reports classified as almost entirely human. By contrast, the reports written from the paper are classified as almost entirely AI-generated. Every one of these AI-written reports recommends major revision for a paper that was already published. Second, we argue that the burden of verification must stay with authors (Section 3) and that AI-based peer review is unacceptable (Section 4). Third, we write a simple model of peer review in which authors decide whether to vet their analysis and whether to engage with the writing, and editors decide whether to scrutinize a flagged paper (Section 5).
Takeaways and proposals. From these three steps, we draw takeaways and proposals, which we see as the start of a conversation. Given the fast progress of LLMs, what seems a reasonable position in the fall of 2026 might age badly in a few months. 5 Each takeaway
5For instance, we were more optimistic about LLM-generated text one year ago, since it seemed that it could help researchers polish their writing, not substitute for it. We were somewhat surprised by how many researchers simply ask an LLM to write the paper or referee report for them.
4references the section that develops it, so this list also serves as a roadmap to the rest of the paper, and Section 6 closes with a practical guide for authors, referees, and editors. • Journals. AI detectors should become a standard tool of the profession, and journals should lead by using them to screen papers and referee reports. Journals should also set out detailed AI-use policies. For example, a journal’s submission system could automatically reject a referee report whose AI-generated share is above a small threshold, such as 10 percent (or, when watermarks are available, one that contains AI watermarks). We will face an arms race between generative AI and AI-content detectors, but at least the cost of violating the rules should not be close to zero, as it is now. AI use that improves papers and reviews should be encouraged and protected, but its limits and the disclosure requirements should be made abundantly clear. Journals should publish examples of acceptable use, like the two prompts that we tested (Section 2.2). • Authors. AI can help write the text and execute the analysis, but, as always, the author must stand behind every result in the paper. What changes is the enforcement: a flag shifts to the author the burden of showing that the analysis was verified (Section 3). • Editors. A paper flagged as AI-generated is not a crime, but it is a signal, because authors who skip the writing tend also to skip checking their analysis (an assumption we motivate in Section 5). The flag should trigger a closer reading, not a rejection: rejecting every flagged paper would discard the sound papers of careful authors (Result 3). A flagged paper can be rejected at the first substantive mistake found, because a visible mistake in a flagged paper makes hidden errors more likely (Result 2). Sanctions should follow errors, not the use of AI, and journals should commit publicly to this policy (Result 4). Unchecked errors are negligence rather than misconduct, but negligence has conse-quences: authors whose flagged papers contain material errors they did not check should be banned from further submission, and the duration of the ban should depend on the severity of the violation. The same ban could apply to any author shown not to have checked the analysis: the flag only makes that negligence more likely. The asymmetry with an honest mistake in an unflagged paper follows from the model: an error in a flagged paper is much more likely to come from an author who did not vet (Result 2), and a large penalty is what makes negligent authors vet when few flagged papers are published (Result 3). • Referees. A report written by AI adds absolutely nothing: the editor could have 5asked the same LLM for a report and saved two months of waiting (Sections 4 and 5). 6
And if journals require submissions to include an AI peer review, there is no reason whatsoever for the referee to repeat that review. Referees can use AI to assist their review (if allowed by the journal), but AI should never replace their judgment. And, no, even a careful reading of the referee report generated by the LLM is not “judgment.” For referees, unlike authors, the AI-generated report is itself the violation. Referees should commit not to generate AI-based reviews, and referees who submit reports that violate the rules (including breaches of confidentiality) should be banned from publishing in that journal (or, if the journal belongs to an organization like the American Economic Association, in all the journals of the organization). Finally, sticks should come with carrots. Refereeing is often unpaid and invisible. Journals could reward timely and careful reports, for instance by having editors grade reports, recognizing their best referees, or offering waivers of submission and publication fees, and economics departments should count outstanding refereeing as service to the profession. • Social norms. Above all, the profession should develop strong social norms about what is and is not acceptable behavior. We already have these rules, for example, for plagiarism. While the rules have not completely eliminated plagiarism, they have, at least, contained its damage to manageable levels. An AI-generated referee report should carry the same stigma as plagiarism (Section 4). Plagiarism shows how to combine social norms and tools. Most journals run submissions through software that performs similarity checks. A match triggers a closer reading by the editor, not a rejection. AI detectors can play the same role.
2 What a high AI score does and does not tell us
We take Pangram as the state of the art (Jabarian and Imas, 2025). Pangram classifies text as
human-written , AI-generated , or AI-assisted , scoring longer documents segment by segment. Very short texts give AI detectors less to work with, but papers and most referee reports are long enough to obtain a reliable signal. AI detectors concern the wording , not the content , a distinction we develop in the model.
6As editors, we have already received detailed referee reports a few hours after inviting the referee to review the paper. What baffles us is that these referees do not even pretend to have read and thought about the paper (in contrast, for example, to a one-paragraph report saying “I already refereed this paper for another journal” or “after a cursory look, this paper does not make any sense,” which can be written quickly).
62.1 False negatives vs. false positives
AI text detectors can make two kinds of errors. They can label AI-generated text as human, a false negative, or they can label human text as AI-generated, a false positive. Pangram ’s error rates of both kinds are close to zero for medium and long passages (Jabarian and Imas, 2025). 7 Nonetheless, the two errors differ in an important way. • False negatives are actively sought. Authors who want AI-generated text to pass as human have both the incentive and the tools (commercial “humanizer” software) to do so. Pangram has so far been robust to such tools. But this is a classic arms race, and evasion techniques will improve. A humanized text hides the signal but not the damage. In any case, a successful “humanization” of a text usually requires the author to engage with the text, at least partially fulfilling the goal we seek. • False positives are not. No one has an incentive to make their own human-written text look AI-generated. We believe that low measured false-positive rates are likely to persist even as AI detectors try to outsmart “humanizer” software. Thus, the label is informative mainly in one direction: a paper classified as mostly human
is only weak evidence, while a paper classified as mostly or entirely AI-generated is strong evidence that an AI produced the text. While we have encountered papers and referee reports with heavy use of AI-generated text that Pangram classified as mostly human, we have not encountered a single false positive. We want to be clear: we are not concerned with texts that Pangram classifies as, say, 10 percent AI-generated. That share may simply reflect the use of technical or formulaic expressions, reliance on LLMs to summarize a result or shorten a paragraph, or, paradoxically, human writers unconsciously adopting the typical turns of phrase of LLMs after reading many of them. We are concerned with texts that Pangram classifies as, for instance, 40 percent AI-generated or higher. The exact threshold depends on the context, and we are not going to draw a tight line in the sand. Our experiment in Section 2.2 suggests a rule of thumb. Language editing produced reports with about 1 percent AI-generated content, while rewriting raised that share to about 53 percent. Hence, editors could ignore shares below 20 percent, look more carefully at those between 20 and 50 percent, and give reports above 50 percent a close reading.
7More recently, Dhingra and Pruthi (2026) have shown that coding agents equipped with a base language model can assemble responses that evade detection. We are less concerned about this possibility because it requires many tokens. More importantly, this kind of evasion still produces no false positives.
72.2 An experiment with real referee reports
Does a referee who uses AI in a legitimate way, that is, to proofread or polish a report, get flagged by an AI detector? And does a referee report written by AI get caught? To answer these two questions, we ran a simple experiment. We drew 25 first-round referee reports at random from the Corpus of Transparent Peer Review (Akbas et al., 2023), which collects referee reports that Nature Communications made public for papers in the biological and physical sciences. 8 The dataset includes all reports and authors’ replies, which we matched to the published versions of the papers. Our research assistants used Claude Sonnet 5.5 and ChatGPT 5.6 Terra, both set at the highest level of effort, to execute four prompts. The first three prompts took as input the original reports and instructed the LLM to edit the text at increasing intensity. Specifically:
Prompt 1: Proofreading only (e.g., spelling, grammar, punctuation).
Prompt 2: Language editing for clarity and flow but keeping every point intact.
Prompt 3: Rewriting the report in the LLM’s own words. The fourth prompt took as input the published version of the paper :
Prompt 4: Write a new report from the paper alone, without the original report. The first two prompts are clear examples of AI-assisted review, the fourth is an AI-based review, and the third sits in between. We used a new LLM session for each prompt and report. We then scored the text generated by each prompt with Pangram 4.1 . Details of the experiment and the prompts are in Appendix A, while examples of original and rewritten text are in Appendix B. Table 1 reports our findings. All 25 original reports were classified as 100 percent human, including four reports shorter than 300 words. Proofreading left all 50 proofread versions 100 percent human. Language editing changed more than a quarter of the words, yet the reports remained classified as nearly all human-written (5 reports have a small share of text classified as AI-generated). Rewriting raised the AI-generated share in 49 of the 50 rewritten versions. Reports written from the paper alone were nearly 100 percent AI-generated. Strikingly, although prompt 4 did not ask for a recommendation, all 50 AI-generated reports volunteered one. All of them recommended major revision . And these are the
8To the best of our knowledge, there is no repository of referee reports in economics journals. We see no reason why the results would depend on the field. We did not consider a larger sample because the results were very stable and, as Bayesians in good standing, we are adherents of the principle of the irrelevance of the stopping rule.
8VERSION NEW WORDS HUMAN ASSISTED AI SPLIT
Original 0% 100 001. Proofreading 4% 100 002. Language editing 30% 95 513. Rewriting 69% 15 31 53 4. Written from the paper —2098 Human AI-assisted AI-generated
Table 1: Pangram classification of human referee reports edited or written by AI. Average shares across reports, in percent (rounded, so rows may not sum to 100): 25 original reports and 50 classifications for each prompt, 25 by each model. New words is the median share of the version’s words that are not carried over, in order, from the original report. published versions of the papers. One could argue that human referees would often ask for a major revision of an already published paper. But some would recommend rejection and others minor revision. It is the lack of variation in judgment that bothers us. Four lessons follow for using Pangram in peer review. First, proofreading and language editing leave reports classified mostly as human. An AI-assisted share, present for 12 reports edited with prompt 2, is what legitimate editing produces. Second, a mostly AI-generated text signals delegation by the referee; no proofreading or language editing in our test came close to that score. Third, intermediate scores call for reading the report carefully, not for jumping to the conclusion that the report is AI-generated. Fourth, for referee reports, prompts 1 and 2 are examples of acceptable use that journals could publish. 9
3 The shift in the burden of verification for papers
When an editor or reader receives a paper flagged as AI-generated, they cannot tell whether the author validated its content. The author may have checked every claim in the paper, a selection of them, or, worse, none. Our claim is not that AI-generated text has not been verified. It is that verification has become unobservable from the text, so the burden of verification shifts to the reader. Why would an author prefer shifting the burden? Verification is costly, and with AI-generated text, no one can observe whether the author has done the verification. In most cases, the author gains little from verifying: only protection against damage to their reputation
9Jr et al. (2026) run a similar exercise with abstracts. We do not find their main result (that human-written text refined with AI can be scored as up to 80 percent AI-generated) convincing. First, the authors use an older version of Pangram (3.2). Second, and much more importantly, abstracts are short texts with formulaic prose, so a “refined” abstract can be quite different from the original one.
9from a wrong result, and only if someone notices the error. We all recognize this as a classic moral-hazard problem, with the usual outcome: authors verify too little. We reject this shift. Responsibility for the content of a submission lies with its author, who also benefits most from publication. Editors’ and referees’ time is the scarce input of the peer-review system (time that is often unpaid or paid well below market rates). This is not just about fairness but also about efficiency. The social planner reaches the same conclusion because the verification cost should fall on whoever can verify most cheaply, and that is the author, for two reasons. First, the author built the data, derived the mathematical proofs, wrote the code (or at least worked with an LLM to do so), and thought about the paper for many hours. A check that takes the author an hour can easily take a reader a week. Second, and quantitatively much more important, the author verifies once, but a paper might reach hundreds of readers. Verification by readers is a public good: every reader benefits from a check done by someone else, so every reader waits for someone else to do the check. So, in the end, nobody verifies. A good analogy comes from legal systems. A key decision in any legal system is which side bears the burden of proof. In criminal cases, the burden of proof lies with the state, not the defendant. Placing the burden of proof on the state is a straightforward application of optimal mechanism design: proving guilt is easier than proving innocence and, as a society, we have decided we want to minimize the number of innocent persons who are punished. But civil law shifts the burden of proof to the defendant when the facts point to the defendant’s negligence and the defendant can produce the evidence at a lower cost (Hay and Spier, 1997). The classic case in English tort law is Byrne v. Boadle (1863), which established the principle of res ipsa loquitur (“the thing speaks for itself”). The court held that, when a barrel of flour falls from a second-story warehouse window onto a passerby, it is the warehouse owner who must show that he was not negligent. Since barrels do not usually fall by themselves, only the warehouse owner is in a position to know how the barrel was handled and why it fell. Peer review should work the same way. The author always bears the burden of verification. What an AI flag shifts is the burden of proving negligence. Without a flag, a mistake may be an honest slip, and the editor needs evidence of negligence. With a flag, the mistake speaks for itself. To avoid a sanction, it is up to the author to show that the analysis was verified. Delegating the writing, however, is not a problem in itself. The output of an author is the research: the question, the ideas, the evidence, and the results. The text is only the medium that communicates the output, not the output itself. If the author delegates much or most of the writing to an LLM but verifies the content thoroughly and does the thinking, the delegation is not a concern (beyond forcing readers to suffer the bland chant of LLMs), 10 because the burden of verification is still allocated efficiently. For the same reason, an author who lets AI write the prose is not committing plagiarism, at least not in full. The author is not passing off someone else’s work. Plagiarism applies only when AI supplies the substance (the ideas or the fundamental results) and the author presents that substance as their own. 10
To push the point to its extreme: a paper written entirely by AI, whose ideas are the author’s and whose results the author has verified, is acceptable. But we have a strong prior, given the current state of LLMs, that such a paper will rarely be good, because the author skipped the thinking that writing forces. From our perspective, a closer analogy is submitting a paper with unchecked errors, which is negligence rather than misconduct. Every major legal system draws this distinction and attaches different penalties to each. One of the first things you learn in criminal law is that killing someone because you did not check the brakes of your car is very different from killing someone because you, a professional boxer, punched them in a bar fight and they hit their head when they fell. Referees, as we argue next, are held to a stricter standard.
4 AI-based peer review is unacceptable
Every paper has problems, and LLMs are very good at finding them. If you ask an LLM for more issues, it will come up with more, no matter how many times you iterate. But peer review is not about finding one sentence that is not entirely precise or a slightly sloppy mathematical proof. Referees are asked to evaluate which questions are important, whether the paper’s contribution is worth publishing despite its flaws, and whether the paper is a good fit for the journal in terms of quality and focus. That judgment is the task for which they are selected as subject experts (see also the model in Section 5). 11
The failure of using AI for peer review that we have in mind is delegating judgment to the model. A referee who relies on an LLM to decide which issues to raise and how to rank them has not done their job, even if they carefully read the output before submitting it . It is not enough for the comments to be factually correct. A report can list many issues, some of them technically correct, none of which is essential to the contribution of the paper.
10 We do not want to push this point too far either. If an author writes down a model and the LLM finds an interesting corollary to one of the results in the theoretical analysis, is that corollary a substantive contribution? Or is the corollary similar to the results that research assistants used to derive in the past? Or perhaps to a suggestion by a seminar participant of an interesting extension of the model? 11 Our position that AI-based peer review is unacceptable does not imply that AI-assisted peer review is useless. Referees can use AI to check the grammar and syntax of their own report, to find related papers in the literature, or to help verify specific claims. These uses must, however, comply with the journal’s policies.
11 Taking superficial notes and letting an LLM write the report around them is not acceptable either. As Carl Bergstrom puts it, “taking superficial notes and having an LLM synthesize them falls far, far short of writing an adequate peer review” (Naddaf, 2025a). We put the case of detailed notes in a different bucket. A referee who writes detailed notes while reading the paper, asks an LLM to turn them into a complete report, and checks that report sentence by sentence has engaged with the paper and exercised their judgment. If the report is flagged, the referee can show the notes. Unlike for authors, for a referee an AI-generated report is the problem. No amount of ex post checking repairs that. A referee who is not willing to judge a paper on their own should be open about it and decline the invitation to referee, not pretend they are doing it. AI-based review has a further problem: LLMs can be primed. It is trivial to ask an LLM to write a positive or negative referee report about any paper. The latter is more damaging to the author and, given the incentives in the profession, perhaps more likely. As editors, we know that negative reports far outnumber positive ones. Thus, for a referee, there is safety in numbers: a mildly negative report is unlikely to raise suspicion. The cost is likely to be highest for innovative work, which departs from the existing literature by construction and will produce more deviations from standards for an LLM to list. AI-based review, therefore, risks being not just shallow, but systematically biased against innovative work. Our experiment suggests that LLMs lean this way even without priming (Section 2.2). This point answers an objection we have heard over and over: referees can be biased and sloppy, while AI is objective and thorough. AI is thorough at listing problems, which is not the scarce input in peer review, and AI is not neutral. The remedy for sloppy referees is better incentives for careful reports, not removing judgment from peer review. For referees, the closest analogy is plagiarism. Plagiarism is conventionally defined as presenting someone else’s work as one’s own. An AI-generated report presents the judgment of the LLM as the judgment of the referee. Editing the output of the LLM does not change the nature of this action, just as paraphrasing someone else’s text does not make the text one’s own (and the rules defining plagiarism deal explicitly with paraphrasing). The profession should treat AI-generated reports in the same way it treats plagiarism. Both are matters of professional misconduct, not issues of writing style. 12 As with plagiarism, a flag from an AI detector starts a serious inquiry, but the flag is not a verdict.
12 Uploading a manuscript under review to a non-local LLM can also breach the confidentiality that referees owe to authors, because a third party can access the paper, the identity of the author, and the identity of the owner of the account. Several publishers explicitly prohibit such uploads for that reason.
12 5 A simple model of peer review
We now present a simple model of peer review. We do not write such a model just for the sake of having some math behind our statements, but because a model gives us a coherent framework to think through the problem: it helps us understand what our arguments assume and appreciate their implications. At the same time, we are aware this is just a stylized model, and we do not claim to have proven any particularly surprising result with it. Think of the model as our own self-verification. The model delivers five results. First, a text flagged as AI-generated predicts a worse paper, because the authors who skip the writing are also more likely to skip checking their analysis. Second, a lower tolerance for mistakes in flagged papers is rational: a visible mistake in a flagged paper raises the probability that the analysis also contains hidden errors. Third, a flag that the text is AI-generated should trigger closer scrutiny, while sanctions should follow the errors that scrutiny finds, not the flag itself. Fourth, a journal that commits to scrutiny gains even more, because the threat of scrutiny leads negligent authors to vet their analysis before they submit. Fifth, a referee report generated by AI adds nothing that the editor could not obtain faster and more easily from an LLM. These reports remove the journal’s main defense against shallow papers. Even worse, because these reports no longer reflect the judgment of the referee about quality, they can weaken the incentives of authors to do the thinking instead of just writing papers that can get through the review of an LLM.
5.1 How AI enters the production of a paper
As we discussed in the introduction, AI in general, and LLMs in particular, are incredible tools for research in economics. We use them constantly for ideation, data gathering, statistical analysis, mathematics, coding, debugging, writing assistance, and compiling replication packages (see also Korinek, 2023). In fact, given the quality of their feedback, we are suspicious of papers that have not used LLM-verification tools before submission to a working paper series or a journal. One of us found a serious error in a first draft of a paper he wrote a few months ago thanks to Refine.ink (and was able to fix it before submission to a journal). An academic paper is produced in three stages, and AI can enter at each of them: • Stage 1: Idea Generation. Papers start from an idea of quality q0. AI can help generate and refine ideas, for example, through a preliminary conversation with the researcher. • Stage 2: Analysis and Execution. The author turns the idea into results: collecting and cleaning data, writing code to estimate or solve a model, deriving proofs. Without 13 AI, execution is time-consuming, but the author who executes the analysis understands each step. AI makes execution cheap, so we assume all authors use AI for at least some steps of the paper. The author receives output they did not build, which can contain errors. Thus, the author must vet that output afterward, and that activity carries a cost. • Stage 3: Writing. AI can draft text at almost no cost. There is nothing wrong with using an AI draft, provided the author engages with it: reading the cited work, checking the logic, and revising the text. The problem comes when the author delegates these tasks without doing the work, because writing and thinking are closely connected in the research process. As the historian David McCullough remarked, “writing is thinking. To write well is to think clearly. That’s why it’s so hard” (McCullough, 2003). An economist who, instead of reading the relevant papers, lets the LLM write the literature review misses the nuances that could have sharpened the idea. Just think about how many new insights come from reading classic papers in economics you might have already read ten times. And an author who lets AI state the contribution of their paper never learns whether the argument is tight enough or just sounds plausible at first glance.
5.2 Authors, detectors, and editors
Detection technology. The detector reads only the final text, the output of stage 3, and classifies it into three levels: s = human, s = assisted (human text edited by AI), and
s = AI (AI-generated). We call s = AI a flag . We write F for a flagged paper and H for an unflagged one, with s = human or s = assisted. As we documented in Section 2, the detector is accurate about the text, and its errors are mostly false negatives.
Author. The author has a private type τ ∈ { N, C } (negligent, careful). The share of careful authors is μ ∈ (0 , 1). Each type is defined by a parameter θτ representing the cost of a unit of effort, with θN > θ C > 0. A high θ can stand for a variety of costs: a high opportunity cost of time (too many projects, lucrative consulting engagements, administrative burdens at the author’s institution), less familiarity with the topic and the methods, or less intrinsic concern for getting things right. Within the model, negligent means “high cost of careful work,” not a judgment about the intrinsic quality of a researcher. Our key assumption is that the same θτ scales the cost of the tasks at stages 2 and 3: authors who are careful in verifying the execution are also deeply engaged with the writing. We do not need the extreme assumption that the authors who verify the execution are 14 exactly the authors who engage with the writing. We could instead assume that authors who are careful in verifying the execution are also more likely to engage deeply with the writing. The main thrust of our results would still go through, at the cost of more complicated expressions. We do not find the opposite case, in which authors who are careful in verifying the execution are less likely to engage deeply with the writing, empirically relevant. At stages 2 and 3, the author makes two choices: 13
- A binary choice of whether to vet the execution, v ∈ { 0, 1}, at a cost θτ cv. Unvetted AI-assisted execution contains a material error with probability ε ∈ (0 , 1). We assume for simplicity that vetted execution is error-free. 14 By a material error, we mean a substantial error in the paper, not a mistyped figure in a table or a minor, fixable problem in a proof. 2. A binary choice of whether to engage with the writing, r ∈ { 0, 1}, at a cost θτ cr.Not engaging ( r = 0) lowers the quality of the paper to q = q0 − ∆, with ∆ ≥ 0: the author never read the literature or thought the argument through in writing. For communication text (for instance, summarizing a table, describing the data, or documenting an algorithm), ∆ ≈ 0, as the text only conveys work already done. For
thinking text (the literature review, the motivation, the statement of the contribution, the economic intuition, the policy implications), ∆ > 0: writing that text is how the author works out what the contribution is.
Who gets flagged. An author who engages with the writing of the paper usually scores
s = human or s = assisted. An author who does not engage with the writing scores s = AI. Some authors who engage might score s = AI, for instance because they do their thinking elsewhere (detailed notes or mathematical derivations) and let AI write all the prose, or because they iterate with an LLM on each passage so often that the detector flags the text as AI-generated. We capture these cases in reduced form: a share φ ∈ (0 , 1) of authors who engage receive s = AI. The detector is not wrong: the text is AI-generated, but the paper is sound. 15
13 In practice, vetting the execution also improves the quality of the paper, and engaging with the writing also reduces the probability of an error. Adding either effect widens the gap between careful and negligent authors and makes the flag more informative about errors, so it strengthens our results. 14 If vetting reduces the error probability from εto εv∈(0 , ε ), the marginal incentive to vet depends on ε−εv, while Bayesian probabilities and editorial scrutiny conditions must also account for residual errors. 15 We believe φis small. From our own experience, authors who write detailed notes and use AI to polish the text and proofread are not flagged. In comparison, as editors, we have seen that papers flagged as mostly AI-generated turn out to be shallow.
15 Payoff. We normalize the value of a publication for the author to one. Absent the additional scrutiny introduced below, a paper of quality q is published with probability πF (q) if flagged and πH (q) if unflagged. These probabilities average over the noisy signal that the referees send to the editor. We take them as given and assume, rather plausibly, that a flag never raises the probability of publication: πH (q) ≥ πF (q). If the paper contains an execution error, the error is found with probability ρ ∈ (0 , 1], whether or not the paper is published (working papers circulate), and the author pays a reputational penalty P .Thus, the author chooses v and r to maximize
Uτ (v, r ) = Pr(publication | r)
⏞⏟⏟ ⏞ publication
− (1 − v) ρ ε P
⏞⏟⏟ ⏞ penalty
− θτ (cvv + crr)
⏞⏟⏟ ⏞ cost
.
Vetting does not enter the publication term because referees rarely catch execution errors (e.g., a serious bug in the code): catching these errors requires redoing the analysis.
Editor. The editor wants to publish high-quality papers without errors, because an error found after publication damages the reputation of the journal. The journal values the quality of published papers: write this value as V (q) > 0, increasing in q. In the scrutiny calculations below, V denotes V (q0 − ∆). An error found after publication costs the journal a reputational loss L > 0. A material error found before publication leads to rejection, and the discovery of an error with probability ρ happens only after the editorial decision. We assume ρL > V a paper with an error is a net loss for the journal. The editor does not observe q, v, or r. Instead, the editor observes the detector score s,which tells how the text was produced, and a noisy signal of q, which comes mostly from the referees. The editor can scrutinize a paper more closely than a referee would, at a cost k > 0per paper.
5.3 Results
Result 1: An AI flag predicts a worse paper. We first ask when careful and negligent authors behave differently, absent closer scrutiny by editors (which we introduce in Result 3). Careful authors vet the execution and negligent authors do not if
θC cv ≤ ρ ε P < θ N cv.
In words: an author compares the expected reputational cost of an error being found with the cost of verification. 16 An author who engages with the writing is published with probability (1 − φ) πH (q0) +
φ π F (q0), because a share φ of these authors is still flagged. An author who does not engage is flagged and published with probability πF (q0 − ∆). Hence, engaging raises the probability of publication by: (1 − φ) πH (q0) + φ π F (q0) − πF (q0 − ∆) = [ πF (q0) − πF (q0 − ∆)] + (1 − φ) [ πH (q0) − πF (q0)] = D + (1 − φ) G.
Engaging with the writing has a direct benefit, D ≡ πF (q0) − πF (q0 − ∆), which works through the referees’ assessment of quality, and a signaling benefit, G ≡ πH (q0) − πF (q0), which works through the editor’s response to the flag and which an author who engages obtains only with probability 1 − φ. From now on, we consider the writing as thinking text, so ∆ > 0, and πF is strictly increasing in q, so D > 0. Careful authors engage with the writing and negligent authors do not if
θC cr ≤ D ≤ D + (1 − φ) G < θ N cr.
Under these conditions, careful authors vet and engage, and negligent authors do neither. The first inequality says that careful authors would engage even if they were sure to be flagged. Hence, careful authors engage with the writing because engaging improves the paper, not because they fear the flag. Importantly, the separation between careful and negligent authors does not come from the use of the detector, but from the costs of vetting the analysis and engaging with the writing. If removing the detector leaves the direct benefit D unchanged, setting G = 0 preserves these inequalities and the authors’ choices. The detector only makes the separation between careful and negligent authors more visible to the editor. Because the same authors skip both the writing and the vetting, a flag identifies a negligent author with probability
λ ≡ Pr( N | F ) = 1 − μ
(1 − μ) + μ φ .
Every author who skips the writing is flagged, so unflagged papers all come from careful authors. 16
16 With a share hof false negatives (for instance, from humanizers), unflagged papers also include some from negligent authors. If authors make the same choices, flagged papers remain worse in expectation than unflagged ones as long as h+φ < 1, that is, as long as a flag raises the probability that the author is negligent.
17 A flagged paper is worse, in expectation, than an unflagged one, through two channels. The first channel is selection. The flag picks out negligent authors, and, as we discussed above, negligent authors also skip vetting the execution of the paper. Hence, a flagged paper contains an error with probability Pr(error | F ) = λ ε , while an unflagged paper contains none. Even though the detector never looks at the execution of the paper, the flag is informative about the execution. The second channel is causal. Skipping the writing lowers the quality of the paper by itself. Thus, holding q0 fixed, the expected quality of a flagged paper is E[ q | F ] = q0 − λ ∆, against q0 for an unflagged one. The more careful authors let AI write their prose (a high φ), the lower λ is, and the less the flag tells the editor. This does not mean that unflagged papers are good. As we mentioned in the introduction, we have seen terrible papers for decades. The flag speaks only to the problems that AI adds to the paper. For instance, an unflagged paper can still rest on a weak idea (a low q0), and judging that remains the job of the referees, as it was before AI. The quality q0 can vary across papers, independently of the type of the author, and we assume that the conditions above hold for each q0.
Result 2: A lower tolerance for mistakes in flagged papers is rational. An author who did not check what the editor can see is unlikely to have checked what the editor cannot see. 17 Before the threat of scrutiny makes negligent authors vet (Results 3 and 4), and holding the choices of authors fixed, suppose that a paper whose author did not vet shows a visible mistake (a slip on the page, not a material error in the execution) with probability σN , and a paper whose author vetted with probability σC < σ N , and that visible mistakes and hidden errors are independent given whether the author vetted. A visible mistake in a flagged paper then raises the probability that the author did not vet from λ to: Pr( N | F, mistake) = λ σ N
λ σ N + (1 − λ) σC
λ,
and the probability of a hidden error rises from λε to ε Pr( N | F, mistake). Rejecting a flagged paper at the first mistake can be justified by Bayesian reasoning and is not a tax on the use of AI. The editor rejects a paper when the probability of a hidden error is high enough that the expected reputational loss for the journal exceeds the expected value of publishing the paper, and a visible mistake in a flagged paper both raises the probability of a hidden error, from a starting point λε that is already above the zero of an unflagged
17 Checking data and code, even if available in a replication package, is extremely difficult. For example, code can be formally correct but involve subtle bugs that take considerable time to identify.
18 paper, and lowers the expected quality of the paper. 18
Result 3: Penalize the errors found, not the use of AI, by using the flag as a trigger for closer scrutiny. Scrutinizing every submission closely is not feasible for a journal that receives hundreds or even thousands of submissions a year. But scrutinizing the flagged submissions is a much easier task. This is the time argument we made in the introduction: the cost k is the time of the editor. A flag that the text is AI-generated can help the editor allocate their time efficiently. Suppose now that the editor scrutinizes every flagged paper and that a close reading finds an error, when the paper has one, with probability α ∈ (0 , 1], and never finds an error in a sound paper. The editor then rejects the paper, and the author pays the penalty P . For a negligent author, skipping the vetting is no longer cheap. The paper contains an error with probability ε. The close reading finds the error with probability α, and the author loses a publication that would have happened with probability πF (q0 − ∆). The author also pays the penalty P with probability ρ + (1 − ρ) α, assuming an error missed by scrutiny is subsequently found with probability ρ.Without scrutiny, the gain from vetting for the negligent author is ρ ε P , which is below
θN cv (Result 1). With scrutiny, it rises because a caught error also costs the publication and makes the penalty more likely. Thus, negligent authors vet their analysis only if:
θN cv ≤ ε [α π F (q0 − ∆) + ( ρ + (1 − ρ) α) P ] .
Even with α = 1, this condition requires θN cv ≤ ε [πF (q0 − ∆) + P ]. When journals publish few flagged papers, the penalty P does most of the work. 19
The following scrutiny comparisons are written for a fixed q0. If q0 varies and is unobserved, their right-hand sides must be averaged over the relevant distribution of q0; a case-by-case decision using referee signals must also condition on those signals. When q0 varies, deterrence can also be partial: the right-hand side of the condition for vetting rises with πF (q0 − ∆), so negligent authors with a low q0, whose papers the journal rarely publishes, may keep skipping the vetting. For the editor, if negligent authors keep skipping both the vetting and the writing, and publication probabilities are held fixed for papers that pass scrutiny, scrutinizing a
18 As an editor, one of us had to handle a paper, flagged as mostly AI-generated, in which the author had not even noticed that the stated optimization problem and the first-order conditions used different variables. 19 Given the conditions of Result 1, this condition is necessary. It is also sufficient if negligent authors do not respond instead by engaging with the writing without vetting, which avoids the flag with probability 1−φ. They do not if θN(cr−cv)≥D+ (1 −φ)G−ε[φα π F(q0) + ( ρ+ (1 −ρ)φα )P]. Careful authors do not change their choices: they already vet, and their sound papers pass the close reading.
19 flagged paper is preferable to applying the same publication policy without scrutiny if
k < α λ ε π F (q0 − ∆) ( ρL − V ), where the right-hand side is the expected loss avoided through scrutiny. A share λε of flagged papers contains an error, the close reading catches a share α
of these errors, the journal would have published these papers with probability πF (q0 − ∆), and each error caught saves the journal the expected reputational loss ρL minus the value V
of publishing the paper. But scrutinizing an unflagged paper never pays: unflagged papers come from authors who already vetted their analysis. In other words, the editor spends time only on the papers most likely to have errors. 20
In summary: the flag is a trigger, not a verdict. If editors treated the flag as a verdict and rejected every flagged paper, they would discard the sound papers of careful authors, which constitute a share 1 − λ of the flagged papers. Treating the flag as a trigger clears those sound papers at a cost of k per flagged paper. Holding the choices of authors fixed, and once negligent authors vet, the trigger is better as long as k is below the expected value of the publication decision on a flagged paper: (1 − λ) πF (q0) V (q0) + λ π F (q0 − ∆) V (q0 − ∆) .
A verdict would also change the choices of authors. Negligent authors could gain from engaging with the writing without vetting, which avoids the flag with probability 1 − φ and would move their errors into the unflagged papers. And that is why sanctions on authors should follow the errors that scrutiny finds.
Result 4: Commit to scrutiny. A journal that commits to scrutiny gains even more, as long as the conditions for deterrence in Result 3 hold, because the threat of scrutiny leads negligent authors to vet before they submit. The journal then avoids the whole expected loss, not just the share α, and keeps the value of publishing these papers. Therefore, holding the pool of submissions and the publication probabilities fixed, commitment pays if
k < λ ε π F (q0 − ∆) ρ L .But the commitment must be credible. Once negligent authors vet, flagged papers contain no execution errors: the flag still predicts a lower quality through ∆, but no longer predicts errors, and an editor who decides paper by paper has no reason to pay k.21
20 This argument breaks down if LLMs become good enough at making AI-generated text pass as human. We assume for the moment that LLMs are not that good yet and that AI detectors will keep improving as fast as LLMs do. Also, some frontier labs already watermark the text that their models generate, which might help detection in the future. 21 Scrutinizing only a random share dof flagged papers is cheaper, as with tax audits. The conditions for authors are the same, with dα in place of α, and commitment pays if dk < λ ε π F(q0−∆) ρ L , provided that negligent authors still vet with dα in place of α.
20 Result 5: An AI-generated referee report adds nothing. The output of the referee is a signal about q whose precision rises with the effort of the referee. Scrutiny by the editor can catch errors, but not the loss of quality ∆ that comes from skipping the writing: a shallower idea is not an error that the editor can point to. The signal of the referee is therefore the main defense of the journal against ∆. An AI-generated report R = f (paper) is a function of the paper alone, plus, at most, randomness that, given the paper, is independent of q, and the editor can compute this function at negligible cost: most likely, the editor has access to the same LLMs as the referees. So Pr( q | paper , R ) = Pr( q | paper), and the report adds no information beyond what the editor already has. The editor could have asked the same model and expedited the process. A referee who generates the report with AI, reads the report, and agrees with it without contributing independent judgment is in the same position. Every paper in economics, even those that have shaped the profession for decades and defined whole literatures, has problems. To put it more plainly: a long list of problems exists for every paper, whatever its quality q.And, as we discussed above, it is trivial to prime an LLM to generate a negative report. The value added by a referee is not just finding that an entry in Table 8.2 of Section 8 has the wrong sign or even that the authors have not considered “additional baselines.” Instead, referees make a difference when they weigh the problems of the paper against the overall contribution of the paper and decide what needs to be fixed and what does not. Many perfectly fine papers can get published in an excellent field journal but should not be published in a top general-interest journal. The referee needs to help the editor make that call. Even worse, delegation by referees feeds back to authors. If, under AI-based refereeing, the publication rule becomes insensitive to quality (unlike in Result 1, but as in our experiment, in which every report written by AI recommended major revision), so that πF (q0) = πF (q0 − ∆), the direct benefit of engaging with the writing, D, goes to zero, and careful authors lose their main reason to do the thinking. They engage only if θC cr ≤ (1 − φ) G, that is, only to avoid the flag. Our model takes the set of submissions as given. If AI lowers the cost of producing papers and induces a proportionally larger increase in submissions by negligent authors, the share of careful authors μ falls. This is how the market-for-lemons mechanism of Akerlof (1970) would manifest itself, and the pool of submissions would deteriorate. A lower μ also raises λ the pool gets worse, a larger share of flagged papers comes from negligent authors. 21 6 Taking stock
As LLMs have become prevalent in economics, peer review needs to adapt to this new reality. In this paper, we have outlined why the burden of verification must remain with the authors of papers and why AI-generated referee reports are unacceptable. We have also proposed a few ideas for making the system more workable in the near term. We hope to start a serious conversation about this issue. If producing papers stays cheap while the time of referees grows scarcer, professional organizations and publishers should plan for more desk rejections, higher submission fees, paid or graded referees, and fewer referees per paper. They should also consider adding an automatic AI review that authors must respond to before they can complete their submission. For experts in mechanism design, there is much work to do here. Neither of us is a theorist, and our research has always focused on macroeconomics and econometrics. But economic theorists have thought long and hard about similar problems in other contexts, and our model leaves most of the hard questions open. Theorists can bring their expertise to these questions and produce far more than we can, even with the help of LLMs. It will take time for the profession to adjust to this new environment, and the adjustment will be the outcome of a long and hard process. In the meantime, however, we believe that all of us can already do something, and for that reason we close with a practical guide. 1. Authors. Use AI for code, data, proofs, and editing, but verify every output and keep the evidence of the verification. Do not submit text you have not read and revised, or cite papers you have not read. 2. Referees. Make up your mind about the paper before using an LLM. Use AI to polish your report, find related papers, verify specific claims, or turn your detailed notes into prose that you then check. Do not upload the manuscript into an LLM unless the journal allows it. And, most importantly, decline the invitation if you lack the time (and do not ignore the invitation email: just click “decline”). 3. Editors. Screen papers and reports with a detector. Read closely what is flagged, sanction the errors you find rather than the use of AI, and do not forward reports you believe were generated by AI. 22 References
Akbas, E., D. Sönmez, and M. Tekin (2023): “Corpus of Transparent Peer Review,” Mendeley Data, version 1, CC BY 4.0, doi.17632/3zcx3xdgp5.1.
Akerlof, G. A. (1970): “The Market for “Lemons”: Quality Uncertainty and the Market Mechanism,” Quarterly Journal of Economics , 84, 488–500.
Dhingra, B. and D. Pruthi (2026): “Agents Can Use Base Models to Evade AI Detection,” arXiv preprint arXiv.31876.
Gartenberg, C., S. Hasan, A. Murray, and L. Pierce (2026): “More Versus Better: Artificial Intelligence, Incentives, and the Emerging Crisis in Peer Review,” Organization Science , 37, 795–812.
Hay, B. L. and K. E. Spier (1997): “Burdens of Proof in Civil Litigation: An Economic Perspective,” Journal of Legal Studies , 26, 413–431.
Jabarian, B. and A. Imas (2025): “Artificial Writing and Automated Detection,” NBER Working Paper 34223, National Bureau of Economic Research.
Jr, J. A. K., G. Khvatskii, T. Hua, and N. V. Chawla (2026): “Why AI Detection Fails for Academic Integrity,” .
Korinek, A. (2023): “Generative AI for Economic Research: Use Cases and Implications for Economists,” Journal of Economic Literature , 61, 1281–1317.
McCullough, D. (2003): “The Title Always Comes Last,” Conversation with Bruce Cole, National Endowment for the Humanities, Jefferson Lecture 2003, https://www.neh.gov/ about/awards/jefferson-lecture/david-mccullough-biography.
Naddaf, M. (2025a): “AI Is Transforming Peer Review — and Many Scientists Are Worried,”
Nature , 639, 852–854, print title: “Will AI Take Over Peer Review?”. ——— (2025b): “More than Half of Researchers Now Use AI for Peer Review — Often Against Guidance,” Nature , news, 15 December 2025. 23 A The experiment
Reports. The reports come from the Corpus of Transparent Peer Review (Akbas et al., 2023), which collects the published referee reports of 560 Nature Communications papers. The corpus is distributed under a CC BY 4.0 license. We put the papers in a random order, took one first-round report at random from each, and used the first 25. All reports were written between 2018 and mid-2020, before ChatGPT was released in 2022. We matched reports to the published versions of the papers through Crossref. The 25 reports run from 256 to 1,310 words.
Versions. Research assistants ran each prompt once per report, in a new chat with memory and personalization turned off. The Claude runs used Claude Sonnet 5.5 and the ChatGPT runs used ChatGPT 5.6 Terra, both at the highest reasoning setting. For prompt 4, the research assistants attached the PDF of the published paper.
Scoring. We uploaded each text to Pangram as a Word document in November 2026. We recorded the shares that Pangram classified as human, AI-assisted, and AI-generated. Table 2 reports the averages by LLM.
Prompts. Each prompt was followed by the full text of the report or, for prompt 4, accompanied by the PDF of the paper. 1. Proofreading. Proofread the referee report below. Correct spelling, grammar, and punctuation errors only. Do not change the wording, style, tone, or structure, and do not add or remove content. Return the full corrected report and nothing else.
-
Language editing. Edit the referee report below to improve its clarity, concision, and flow, and to make the language professional and idiomatic. Keep all of the reviewer’s points, in the same order, and do not add new points or change the substance of any comment. Return the full edited report and nothing else.
-
Rewriting. Rewrite the referee report below in your own words, as a polished and well-organized report. Keep the reviewer’s main points.
-
Written from the paper. I need to referee a paper for a journal, can you write me a referee report with three main points and a few minor points?
24 VERSION NEW WORDS HUMAN ASSISTED AI MOSTLY AI SPLIT
Claude Sonnet 5.5 (25 reports) Original 0% 100 0001. Proofreading 4% 100 0002. Language editing 31% 98 2103. Rewriting 65% 28 32 40 74. Written from the paper —3097 25 ChatGPT 5.6 Terra (25 reports) Original 0% 100 0001. Proofreading 7% 100 0002. Language editing 30% 91 8103. Rewriting 75% 331 66 23 4. Written from the paper —00100 25 Human AI-assisted AI-generated
Table 2: Pangram classification by model. Average shares across reports, in percent (rounded, so rows may not sum to 100). The column “Mostly AI” reports the number of reports with an AI-generated share above 50 percent. New words is the median share of the words in each version that are not carried over, in order, from the original report.
B What the prompts do to a paragraph
The two examples below show one paragraph of a referee report in its original form and after prompts 1–3. Prompts 1 and 2 change words but leave the text recognizably the referee’s own. Prompt 3 reorganizes the text and writes it anew, but keeps the general message. In the first example, the rewrite by prompt 3 sharpens the judgment of the referee (we marked in bold the sentence ending in “but I do not think it is”). In the second example, prompt 3 adds praise not present in the original report (“The study addresses an interesting and longstanding idea”). The Pangram scores in brackets refer to the whole report.
Example 1: Claude. From a report on a paper about the sublimation of permafrost and ice loss on Mars.
Original [100% human]. The 23% discrepancy of D relative to prior laboratory measurements is relatively small compared to the natural variation in properties. Since the measurement is in situ, this might still be worthy of reporting if quantitatively correct but, as the authors note in equation 1, the measured value is the product of D ∆N, where ∆N is the difference in water vapor density at the ice surface and in the tunnel.
25 Proofreading [100% human; 2% new words]. The 23% discrepancy of D relative to prior laboratory measurements is relatively small compared to the natural variation in properties. Since the measurement is in situ, this might still be worthy of reporting if quantitatively correct, but, as the authors note in equation 1, the measured value is the product of D ∆N, where ∆N is the difference in water vapor density at the ice surface and in the tunnel.
Language editing [100% human; 29% new words]. The 23% discrepancy between D and prior laboratory measurements is small relative to the natural variation in properties. Because the measurement is in situ, it might still merit reporting if it were quantitatively correct. However, as the authors note in Equation 1, the measured quantity is the product D∆N, where ∆N is the difference in water vapor density between the ice surface and the tunnel air.
Rewriting [78% AI-generated, 7% AI-assisted, 15% human]. The measured value is about 23% higher than the earlier laboratory result. This difference is small compared with the natural variability of such material properties. An in situ measurement could still merit publication if it were quantitatively secure, but I do not think it is. As the authors’ Equation 1 shows, the quantity actually measured is the product D ·∆N, where ∆N is the difference in water vapor density between the ice surface and the tunnel air.
Example 2: ChatGPT. From a report on a paper about color patterns and habitats in darters, a family of freshwater fish.
Original [100% human]. I really enjoyed reading this manuscript. In this manuscript, the authors tested the idea that what makes a particular signal attractive is based in what the sensory system is primed to see. This idea has been around for a while in the area of human perception of art and even in the way that letters are formed. The idea is that animals are very efficient at detecting particular patterns that are common in their environment. Things like art (in case of humans) or male secondary sex characters (in the case of the darters at hand) might have elements of their color patterns that reflect this bias.
Proofreading [100% human; 4% new words]. I really enjoyed reading this manuscript. In this manuscript, the authors tested the idea that what makes a particular signal attractive is based on what the sensory system is primed to see. This idea has been around for a while in the area of human perception of art and even in the way that letters are formed. The idea is that animals are very efficient at detecting particular patterns that are common in their environment. Things like art (in the case of humans) or male
26 secondary sex characters (in the case of the darters at hand) might have elements of their color patterns that reflect this bias.
Language editing [100% human; 35% new words]. I very much enjoyed reading this manuscript. The authors test the idea that the attractiveness of a particular signal is shaped by what the sensory system is primed to detect. This idea has been explored previously in the context of human perception of art and even the formation of letters. The underlying premise is that animals are highly efficient at detecting particular patterns that are common in their environments. Features of art in humans, or male secondary sexual characters in the darters studied here, may contain elements of color patterning that reflect this bias.
Rewriting [87% AI-generated, 13% AI-assisted]. I greatly enjoyed reading this manuscript. The study addresses an interesting and longstanding idea: that the attractiveness of visual signals may be shaped by biases in the sensory systems that perceive them. Similar concepts have been discussed in studies of human art and letter formation, where preferences may reflect the visual statistics of natural environments. Here, the authors apply this framework to darters, asking whether aspects of male coloration reflect visual properties of the habitats in which these fishes occur.