Generative AI Does Not Hallucinate

One of the more interesting cultural products of generative AI is the idea of 'hallucination.' I find it noteworthy how quickly and totally this term has taken over the discussion of false or otherwise untrue generative AI outputs. I get why, too. It sounds kind of cool, which is an affect one wants when they think the underlying technology is also cool. It fits with the mentalist themes of intelligence, too. And, from a branding perspective, it's quite useful to be able to say AI is 'hallucinating' rather than saying what is actually happening, which is it is not working.

Generative AI does not think about you (or anything)
I think the term 'hallucination' is a bad one. Certainly, I think it is wrong.

Let's start with a boring reason why it is wrong: AI does not think. All the discussions around consciousness, or AI being some equivalent to human intelligence, are just bogus. This is not to say that AI is not capable of human-level performance or emulating human-level actions, or even exceeding them. But this should not be regarded as unique, either. Cars and trains move faster than humans. Calculators compute sums faster than we do. A plough undertakes farming actions more efficiently than could be done manually (I know nothing about farming). Technology emerges in response to demands for greater efficiencies from activities currently undertaken by humans. Generative AI can produce images faster than people can at a consistent technical quality in excess of most people, just as it can write faster and with a higher average technical quality. None of this should be surprising. Neither should it be considered intelligence.

Intelligence, as Matteo Pasquinelli argues in The Eye of the Master is essentially a social phenomenon. Those dorks who think they can measure intelligence are typically the stupidest people in the room. Intelligence is whatever people think is intelligent. It is why I have previously argued that the most fruitful use of AI in the intelligence discussion is not whether it functions as a model of the brain or cognition, but rather, as a prompt for us to ask what encourages people to label anything as intelligent? This question could be asked of AI as much as it could be of a dog, a dolphin, a cephalopod, and so on.

Generative AI does not think, and thus it does not hallucinate. It takes in an input, runs the input through a network of weights and biases, perhaps applies some randomisation to the output to create the illusion of natural language, and that is it. There is no intent to lie, and certainly nothing typically associated with a human hallucination, such as a chemical imbalance (accidental or otherwise) prompting the sporadic firing of neurones in the brain. This, as above, is the boring reason why AI does not hallucinate. But from here, we build towards something more interesting.

The Gell-Mann Paradox
One thing that is more interesting is that generative AI follows the same generative process for every output it produces, regardless of whether that output turns out to be 'right' or 'wrong', 'true' or 'false'. That is interesting because it suggests that whatever 'causes the AI to hallucinate' is not actually contained within the AI system itself, but somewhere else. Again, intelligence is a social phenomenon, and it is something we impose on other things.

Something to consider in this discussion is what I have always known as the 'Gell-Mann Paradox', though looking online, it seems to be better known as 'Gell-Mann Amnesia' and was not, apparently, coined by physicist Murray Gell-Mann at all (I would like to write a popular science book about all of the paradoxes that effect our everyday lives, and seeing as I wish to include Gell-Mann Amnesia in this book, I will continue to call it a paradox. For those who are interested, the others (so far) are Jevons' paradox, and Simon's paradox). The paradox goes as follows. You are Nobel Prize winning physicist Murray Gell-Mann. You're reading a popular newspaper, and pleasantly stumble across a news story about a recent physics discovery. Being Murray Gell-Mann, the story catches your attention, and you eagerly begin reading. Unfortunately, the story is shit. It's sensationalist, wrong in places and misrepresentative in others. It's nothing like the deeper research that you know so well, and care so much about. Alas, you turn over to the next page. It's a story about the President, with some political commentary thrown in to keep you informed. You care about current events, but politics isn't your speciality, so you eagerly see what the newspaper's expert has to say...

The paradox is that there is no reason to trust the political commentary, which you lack the skills to assess, any more than the scientific commentary, which you have the skills to possess, and know to be shit. So, why do you treat the political commentary different to the scientific commentary?

Obviously, there is a parable here to draw about hallucinations. When an AI 'hallucinates', it is not doing anything different to when it is giving expert, 'super-human-level' intelligence answers, and so on. It's just that in the instance of a hallucination, you notice the bullshit. Of course, sometimes generative AI gets things right. My point is not that everything that ChatGPT and co. spew out is bullshit (or, as one fun paper dubs it, botshit). My point is that there is nothing separates what the AI gets 'right' from what it gets 'wrong', besides the person interacting with the AI system integrating their own perspective into the overall experience. Generative AI does not hallucinate. But sometimes you notice the errors, and anthropomorphise the system in response. And, equally, sometimes you are impressed by the system, possibly to the extent that you are one of those people who think it is the dawn of a new age of super-intelligence, or whatever.

From the perspective of the Gell-Mann paradox, none of this really makes a whole lot of sense. Someone who was resistant to the paradox would treat all outputs with the same level of scrutiny as they would when an output was obviously wrong. When I ask ChatGPT to explain nudge theory to me, I spot the errors. Why, then, should I assume its explanation of baroque music or nuclear fusion or farming practices are any more reliable? I shouldn't. The term 'hallucination' is just an easy way out for those of us susceptible to the Gell-Mann paradox to make our lazy lack of scrutiny feel consistent with our beliefs that we are, in fact, sceptical, critical thinkers. The hallucination allows us to deny the instance where we see that the emperor has no clothes (or more, is procedurally generated using a series of weights and biases), and has lacked them all along. It allows us to chalk the error up to an aberration--a bug in the machine, a glitch in the silicon neuronal structure of the super-intelligence.

Of course, it is not. In keeping with the Gell-Mann paradox, we know AI does not hallucinate. But it is easier to just turn the page, and keep on reading.

Automating Everyday Life
As above, I could make this argument because I am a pedant. I could essentially say that the word 'hallucination' is an inaccurate, overtly-anthropomorphising phrase that obscures what is exactly going on when ChatGPT says the capital of Mongolia is Mongolia City (it is actually Ulaanbaatar--I'm really fun at parties). But I think there is more to it than pedantry. I think the whole idea of the 'hallucination' masks something more much important about how these models are designed, and the political economy which goes into them.

If I were to ask ChatGPT how to make a bomb, it would not tell me. It would say something like my request is potentially dangerous to myself and others, illegal in various countries, so on and so forth. If I ask it a more innocuous question, say, should I buy shares in Tesla (no), I will get a more innocuous, but still automated, response. Something like 'as a large language model I cannot give financial advice.' Everyone by now should know the response limits of ChatGPT, and systems like it.

Clearly for these 'AI Safety' requests, some engineering intervention took place. There is some explicit protocol in action, or some specific training was used, such that the system can identify instances where it should not give a natural language response, and instead defer to the corporate line. With all this in mind, here's my question: why does AI 'hallucinate'? More specifically, why do the creators of generative AI systems allow their systems to produce outputs which cannot be verified against the inputs provided to them, when in other instances, there is clearly some intervention behind the scenes to moderate outputs? Why do AI companies allow these systems to fill in the gaps, to elaborate on background details when asked to produce a picture of a cat or to write the opening chapter to a fantasy adventure?

I think the most sensible answer is that generative AI is designed with an economic imperative to automate everyday activities. Most of the time, when AI 'hallucinates,' it does so by filling in contextual details that it may not have been given. For instance, if I were to ask it to write me a cover letter for a job, it might fill in the details of my employment history, and how long I'd held each position. This would all be bullshit, of course, because the AI would lack knowledge of my employment history. But the alternative--a rough outline with clear indications of where I should fill in the relevant contextual details--would still require me to do something. In this instance, that's OK. It is my cover letter, and I would be using the AI as a convenience tool. But in a commercial setting, this is value destroying, because it means keeping someone employed to manually fill in the context that the AI system lacks. If generative AI didn't fill in the gaps automatically, if it didn't 'hallucinate', then it's prospects as an automating technology would be substantially reduced. Bullshit is profitable, as long as nobody smells it.

And this is why I think it's most important to say that AI does not hallucinate. There is not an aberration in the system; there is no rogue neurone firing. There is simply an engineering choice in response to an economic imperative to build a technology capable of automating human activities in everyday life. To this end, I don't think AI 'hallucinations' will ever disappear. Instead, the inevitable transformation, if AI sticks around, is that AI integrated processes, and human interactions with said processes, will be twisted to fit into the mechanical limits of the AI system. Through re-engineering human activities, and human behaviours, the degree of context which the AI system must automate will be reduced to the minimum viable amount, so as to minimise the occurance of 'hallucination.' Of course, this will, from a human perspective, just result in a homogenisation of experiences which, when colliding with the heterogeneity of everyday life, will result in frustration and anger for all non-AI parties involved. For scholars of technology and society, this is nothing new, for it is people--and so far only people--who can demonstrate superb and sudden adaption to all manner of social, economic, and technological changes.