Some Trouble with the AI Book Critic
How a summer reading list highlighted the tensions between institutional legitimacy and technological shortcuts
Readers of the Chicago Sun-Times were recently disappointed to learn that a number of books on the paper’s summer reading list for 2025 would not be available in time for their vacations. In fact, Tidewater Dreams by Isabel Allende, Nightshade Market by Min Jin Lee, and 13 other titles by notable authors would not be available at all because they were simply made up by a Large Language Model that had been used to compile the list. Ironically, one fictitious book, The Last Algorithm, was billed as a “science-driven thriller” that “follows a programmer who discovers that an AI system has developed consciousness.” Perhaps the AI responsible for this blunder had a sense of humour.
How could such a thing happen? Part of the answer may lie in the financial position of the Chicago Sun-Times. Just two months before the publication of the reading list, the paper announced that it was shedding 20% of its staff — a situation all too familiar in the regional news business. Such a drastic reduction in personnel likely necessitated the paper to outsource some of its content production. The reading list appeared in a special insert that had been syndicated by King Features, a Hearst Communications subsidiary. The Philadelphia Inquirer, another regional paper in financial distress, also ran the bogus reading list after it acquired the same insert. King Features itself outsourced the creation of the insert to a freelance writer, who, by his own admission, turned to AI to do the actual work. Given that the current structure of the news business is designed to save as much money as possible during production, it was inevitable that the true source of the text would be the cheapest.
Large Language Models work for free (or for just a nominal subscription fee), thanks to venture capital subsidies. And there is growing concern that they are increasingly displacing real jobs that involve language production — like the news business. This is because today’s AI systems are very good “next token predictors.” Through the use of a probabilistic model of language that understands the relative frequencies of words that appear together, they are able to construct coherent text based on a prompt and a training corpus that consists of the bulk of the text found on the Internet. This leads to the generation of writing that is high quality in a technical sense, yet stylistically bland unless a human intervenes. But it’s frequently judged to be good enough for publication.
A persistent problem with such generated text has been the presence of hallucinations: fabricated information that sounds plausible in context. The fake titles and descriptions in the summer reading list are a prototypical example of the hallucination phenomenon — the text isn’t at all suspicious until one visits their local bookstore to pick up a copy of The Last Algorithm. The good news is that AI isn’t intentionally trying to fool us. Hallucinations are a natural byproduct of the way language models are trained. A deep understanding of the probabilistic nature of words is different from the possession of deep knowledge about the world and what exists there.
As I’ve argued in my recent book A History of Fake Things on the Internet, in creative contexts where there is a general understanding that one is engaging with fiction, hallucinations are really interesting. There is tremendous artistic potential in using AI as a medium when some human intentionality is still involved. This spans everything from creative writing to the visual arts. In fact, the book list story went viral in part because the list itself is amusing. However, when hallucinations bleed into the physical world in unexpected ways, society has a problem on its hands.
There is an important history behind the specific technology involved in this story, which also helps explain how a fake book list could be published by a professional news outlet. Since the dotcom era of the 1990s, goods, services, and activities in the physical world have been gradually replaced by digital counterparts as part of a geopolitical movement to virtualize all aspects of life. The futurist Kevin Kelly has described the interconnectedness of the technologies involved as the technium, a silicon substrate on top of which much of society now operates. As the tech investor Marc Andreessen famously declared in 2011, “Software is eating the world.” Changes to the technium itself, however, can have downstream effects. Riffing on Andreessen in 2017, Nvidia CEO Jensen Huang wrote to his audience on LinkedIn that “AI is eating software.” He was right, and that change propagated to arenas where AI had not been present before.
This project is moving too fast. Wide availability coupled with information asymmetry leads people to put more faith into the technology than they should. The writer who created the book list admitted that he didn’t proofread the output of the AI tool. Such complacency is commonplace because business productivity software has historically been very reliable. This is compounded by Silicon Valley storytelling, which drives marketing by exaggerating the potential of technology. Sam Altman, Elon Musk, and many others in the Valley have cultivated a hyperbolic rhetorical style that situates them closer to P.T. Barnum than to precursors like Gordon Moore or Steve Jobs. Expect to see similar situations in domains other than journalism very soon.
There are consequences of a premature rollout. As a business move, driving the cost of content as low as possible appears to have backfired on the Chicago Sun-Times. The fake book list was widely panned by the public and brought undue attention to the paper in the form of negative news coverage. A common theme in that reporting has been the loss of trust in the paper. Moreover, this episode ended up creating more work for the paper by forcing a reduced staff to add additional process steps to prevent this from happening again.
We need to think more intentionally about the design of these technologies and their subsequent deployment. What would it take for us to live well with AI? One suggestion is to have a longer period of discernment before an AI product is rolled out at scale. Knowing that unintended consequences can flow from it being plugged directly into the technium, spending extra effort to engineer for the common good might be time well spent. Another suggestion is to hold a more frank dialogue about the strengths and weaknesses of technologies we do have, helping the public to avoid the marketing hype and make judicious use of technology. And a final suggestion is that new technologies should not be put in a position to threaten long standing institutions like the news media. It’s quite possible that incidents like this one may lead us in all of these directions as non-technologists recognize that AI leaves much to be desired.
Start reading A History of Fake Things on the Internet »
It’s becoming clear that with all the brain and consciousness theories out there, the proof will be in the pudding. By this I mean, can any particular theory be used to create a human adult level conscious machine. My bet is on the late Gerald Edelman’s Extended Theory of Neuronal Group Selection. The lead group in robotics based on this theory is the Neurorobotics Lab at UC at Irvine. Dr. Edelman distinguished between primary consciousness, which came first in evolution, and that humans share with other conscious animals, and higher order consciousness, which came to only humans with the acquisition of language. A machine with only primary consciousness will probably have to come first.
What I find special about the TNGS is the Darwin series of automata created at the Neurosciences Institute by Dr. Edelman and his colleagues in the 1990’s and 2000’s. These machines perform in the real world, not in a restricted simulated world, and display convincing physical behavior indicative of higher psychological functions necessary for consciousness, such as perceptual categorization, memory, and learning. They are based on realistic models of the parts of the biological brain that the theory claims subserve these functions. The extended TNGS allows for the emergence of consciousness based only on further evolutionary development of the brain areas responsible for these functions, in a parsimonious way. No other research I’ve encountered is anywhere near as convincing.
I post because on almost every video and article about the brain and consciousness that I encounter, the attitude seems to be that we still know next to nothing about how the brain and consciousness work; that there’s lots of data but no unifying theory. I believe the extended TNGS is that theory. My motivation is to keep that theory in front of the public. And obviously, I consider it the route to a truly conscious machine, primary and higher-order.
My advice to people who want to create a conscious machine is to seriously ground themselves in the extended TNGS and the Darwin automata first, and proceed from there, by applying to Jeff Krichmar’s lab at UC Irvine, possibly. Dr. Edelman’s roadmap to a conscious machine is at https://arxiv.org/abs/2105.10461