Pangram’s 99% AI finding puts prize-winning novel under review

Pangram, an AI-text detector, classified 99% of David Silva Rojas’s prize-winning novel “Where the Water Works” as AI-generated, prompting its Mexican publisher Almadía to suspend promotional events and open a detailed review. The author has denied using generative AI at any stage of the book’s creation. The result does not mean Pangram assigned a 99% probability that the novel was made with AI. It represents the share of the more than 25,000-word text that the tool classified as AI-generated. Pangram labels passages as human, AI-assisted or AI-generated, but it cannot determine precisely what part a human may have played in producing a text. Silva Rojas said on Oct. 27 that he conceived, wrote and revised “Donde el agua trabaja,” the book’s Spanish title, without generative-AI tools. He said it emerged from research into the conditions of agricultural workers in Ica, beginning with first-person fragments written while he transcribed interviews. He later developed scenes, characters and situations, revising the manuscript through different Word versions. The novel’s notably spare style has become central to the dispute. Silva Rojas said it was influenced by Japanese haiku, and the book averages about five words a sentence, against roughly 16 words in two other texts by the author examined by Pangram. The detector also classified 97% of his written answers to an Ulrica magazine interview as AI-generated, and gave an 80% AI-generated assessment to a first-person account of the novel’s origins. “Where the Water Works” won the inaugural Almadía Ventosa-Arrufat International Short Novel Prize in July, receiving a €20,000 advance against royalties. The book was selected from 2,204 manuscripts by a jury of Cristina Rivera Garza, Selva Almada and Mónica Ojeda. The jury praised its silences and short, subtle sentences, comparing their rhythm to irrigation dripping through vineyards. The prize rules barred work written wholly or partly with AI tools and required disqualification for any indication or suspicion of a breach. Almadía said Silva Rojas had contractually declared that he had not used generative AI and had personally reaffirmed that declaration. The publisher said neither its editors nor the jury perceived AI use, but said automatic detectors needed interpretation and cross-checking. Almadía and the Ventosa-Arrufat Foundation said they would examine the matter seriously without rushing to judgment, in order to protect their commitments and readers’ trust. The publisher halted planned presentations in Madrid and other Spanish cities after the controversy. Silva Rojas argued that even a high detector score should not be treated as definitive proof without other evidence. Pangram says it is calibrated to avoid falsely accusing human authors, accepting more false negatives as a consequence. Its own testing says it mislabels about one in every 38,000 human-written texts as AI-generated. EL PAÍS cited at least seven independent evaluations with false-positive rates of 0% to 0.2%, including tests of older books and pre-ChatGPT scientific writing. But accuracy claims for automated detectors remain difficult to independently verify because their data and methods are proprietary, according to the University of Chicago’s Becker Friedman Institute. Its research evaluated Pangram alongside OriginalityAI, GPTZero and RoBERTa across a corpus of 1,992 passages, including novels, news, blogs and reviews. More generally, a detector’s threshold involves a trade-off: reducing false accusations increases the chance that AI-generated writing goes undetected. The case therefore turns both on the author’s account of his writing process and on how much evidentiary weight publishers and prize bodies should give a detector’s classification. Almadía’s review remains under way.