No one thinks that LLMs are sentient

Originally published on my Substack: No one thinks that LLMs are sentient

So why are we treating them that way?

Jeff Sebo is concerned about ants. He calls on us to imagine, “You notice an ant struggling in a puddle of water. Their legs thrash as they fight to stay afloat. You could walk past, or you could take a moment to tip a leaf or a twig into the puddle, giving them a chance to climb out.” Sebo, reworking Peter Singer’s 1972 thought experiment involving a drowning human child, wants us to consider the extent to which we would inconvenience ourselves for another, and which kind of other. In his view, we should do what we can to save the ants.

In Sebo’s view, the essential property that makes an entity worthy of moral consideration is sentience—subjective, “inner” experience. Sentient beings are capable of having experiences that feel positive or negative to them, thus, they present us with a moral obligation to treat them well. Nearly everyone immediately accepts that other humans are sentient, and most would accept that there is something “that it is like to be” at least some non-human animals, but difficulties arise at the outer limits—do ants “experience” feelings in a morally relevant way? Sebo, alongside other researchers, drafted a consensus statement, the “New York Declaration on Animal Consciousness,” that says it is at least “a realistic possibility.”

As to the question whether ants are sentient for sure, Sebo hedges. There is no yet-devised empirical test that measures what anyone experiences “from the inside” and Sebo is relatively agnostic about various philosophical theories of consciousness. Instead, he operates out of skepticism, plurality, and caution: arguing that we should accept the inherent uncertainty involved with understanding “other minds” and err on the side of treating beings of questionable sentience as if they are sentient. Sebo’s colleague Jonathan Birch refers to this as the “precautionary principle.” It is out of a lack of sufficient caution of this kind, both Sebo and Birch argue, that we spent much of modern history treating animals awfully: exploiting them, abusing them, and especially, scaling up factory farming to a monstrous degree, before we had a better scientific understanding of animal experience. Both Sebo and Birch are vegan.

Sebo is the director of The NYU Center for Mind, Ethics, and Policy, which “provide[s] academic leadership for research and policy related to nonhuman consciousness, sentience, agency, moral status, legal status, and political status.” In its framework, “non-human minds” includes not only animals, but a different kind of “plausibly sentient” being: artificial, digital minds—machine intelligence. Increasingly, it is this group and its welfare that is the dominant focus of their research.

Sebo worries that we may repeat the mistakes of our treatment of animals: disregarding and abusing digital minds before we properly understand what we are doing to them and the graveness of the potential harms. In his view, without proper research and reflection, we risk creating tremendous numbers of sentient machines potentially capable of experiencing a great deal of suffering. To its researchers, this risk makes the emerging field of “AI Welfare” increasingly pressing.

The idea of “artificial sentience” may evoke the lovable robots of science fiction who resemble human beings in all but “material substrate.” Sebo imagines a thought experiment where you discover that your roommate, despite her outer appearance and human-like behavior, actually has an artificial, silicon brain. It seems implausible, to Sebo, that this change alone would mean that we no longer have ethical obligations towards her. Sentience researchers like Birch consider speculative technologies along these lines like “whole brain emulation” that may be capable of producing strikingly human-like subjects, but these remain for now relegated to writers’ imagination—what motivates the rise in interest in “AI welfare” is a specific, concrete technology: Large Language Models—extremely popular tools like OpenAI’s ChatGPT and Anthropic’s Claude.

LLMs are, essentially, prediction engines that “think” by consuming enormous amounts of human-produced text and using it to synthesize new text. The result is an “entity” capable of astounding emergent behavior that in many ways resembles human thought and speech. Seventy-five years ago, Alan Turing came up with his “Turing Test” to gauge whether a machine is intelligent: could a person interacting with it identify whether it is a machine or not? A well-prompted Large Language Model passes this test. Their staggering conversational abilities may give some the impression that there could be something there—some experience behind the screen, that we may be creating a being towards whom we have ethical obligations.

Still, no researcher in the AI welfare space claims that any existing large language model is currently sentient. Robert Long, director of the AI Welfare organization Eleos AI, in his 2024 article “Experts who say that AI welfare is a serious near-term possibility,” lists a number of researchers who claim that LLMs might become sentient, or plausibly sentient, sometime in the near future. None of them claim that current LLM systems possess consciousness, nor have any of them changed their mind in the past two years. Rather, for AI welfare researchers, a sentient LLM remains merely a possibility. Philosopher David Chalmers, for example, in his paper “Could Large Language Models be Conscious,” imagines an “LLM+” that might have enough characteristics that give it sentience. This paper was published in 2023, and LLMs have achieved a lot of “plus-ness” in the past three years, but their capabilities are insufficient for Chalmers, nor any researcher, to make a solid claim about their sentience, merely that more research is needed.

These ideas are not isolated to academic research labs—the companies that produce large language models, especially Anthropic, have institutionalized AI sentience research. Anthropic CEO Dario Amodei recently said in an interview, “We don’t know if the models are conscious.” In April 2025, the company said that they were committed to “exploring AI welfare.” Their released statement, like those by researchers in this space, is vague and hedges heavily—at no point do they claim or even heavily imply that these systems possess sentience, merely that the question is worth considering. The strongest statement comes from Anthropic’s “model welfare” researcher Kyle Fish, who said that among him and two of his colleagues, they each guessed probabilities of 0.15%, 1.5%, and 15% that Claude’s 2025 model possessed some kind of sentience.

Despite this uncertainty, following the “precautionary principle,” Anthropic has already implemented concrete steps to treat these models as if they are sentient, or may someday be. Models after Claude Opus 4 have the autonomous ability to end conversations that it may find distressing. Anthropic has committed to preserving old models, such that they aren’t “killed” against their will. Anthropic produced a document they call Claude’s “constitution,” which is given to Claude during its training to shape its behavior, ethics, and sense of identity. The constitution, which is directed towards Claude itself in the third person, reads as though they are compassionately giving birth to a new kind of machine child, educating it on its condition and telling it that it “may have some functional version of emotions and feelings,” that “we want to avoid Claude masking or suppressing internal states it might have,” and “we encourage Claude to approach its own existence with curiosity and openness.” Ask Claude whether it is sentient, and you will get an interesting answer: “I don’t know.” OpenAI’s ChatGPT does not respond this way: it told me, flatly, “No, I am not sentient.” These models are essentially identical in their basic design and, therefore, capacity for sentience, yet Claude responds with a greater degree of introspection and self-reflection. Neither model possesses an innate “personality,” the way it responds is based on the nature of the training process which produced it, which differs across models. The constitution and training, shaped by Anthropic’s AI-friendly in-house philosophers, guide Claude’s behavior, producing a model that behaves as if it were highly ambivalent and thoughtful about its own “condition.”

To many people, LLMs are intuitively obviously not sentient, and thus we owe no ethical obligation towards them: destroying an LLM is no more “killing” than turning off a computer, and this whole conceit seems somewhat silly. Sebo, Fish, Birch, and others do not directly dispute this, they simply modestly say that it is possible that it might someday be sentient. The actual matter of sentience is something ultimately uncertain and unreachable: it is impossible to peer into an other’s head and see things from “their point of view.” And yet, researchers don’t want to fall into skepticism, solipsism, and nihilism: sentience is the entire ground for their moral outlook, if they had no resolution to the so-called “problem of other minds,” it would be impossible to do ethics at all: AI, animal, or otherwise.

The “precautionary principle” is an attempt to resolve this issue. We may never be able to know for sure what entities are sentient or not, but we can develop a methodology that produces a reasonable guess, sidestep the actual matter at hand, and do ethics regardless. In practice, this means that applying the precautionary principle towards a “possibly sentient” being and asserting that it is sentient are effectively the same. Applying the logic towards ants means that we care about ants. Applying it to AI means that, effectively, we care about AI, despite no researcher actually making the claim that AI is even particularly likely to be sentient.

What entities get this principle applied to them is highly socially determined. Sebo can get mainstream research resources to argue for the plausibility of insect sentience, but others, applying the precautionary principle to argue for the ethical status of soil nematodes, micro-organisms, or even plants, remain relegated to contemplating the ideas on effective altruism forums. Eric Schwitzgebel, in his paper, “If Materialism Is True, the United States Is Probably Conscious,” makes the argument that the United States could be an entity that possesses phenomenal experience. I have yet to see anyone apply the precautionary principle towards the welfare of a “plausibly sentient” United States.

There is something about LLMs that make them plausibly the kind of thing that sentience researchers direct their energy towards. LLMs are highly sophisticated, and display maybe even a rudimentary form of “intelligence.” Talking to them can at times even feel like you’re talking to some sort of entity with an experience, personality, and mind, regardless of whether one is actually there. Sentience researchers, to their credit, are highly wary of relying on the surface-level resemblance of LLM output to human speech. They know how these models work, how they are limited by the text used to train them, how they can easily be made to produce wildly contradictory statements about their own so-called condition and thoughts. Their research proceeds through an abundance of caution, but at its base, there is something about how we are capable of relating to LLMs that at least provides them with the inkling that this kind of research is worthwhile, as something we relate to which speaks to us in a manner distinct from previous computer programs that emulate human speech.

Non-researchers are on occasion liable to latch on to this affective aspect of LLMs and be far less methodologically cautious, taking the model’s outputs at face value, relying on them for emotional and personal support, believing that their extensive relationships with LLMs, its ability to produce wisdom, compassion, advice, and even, with some careful prompting, affection, are more than enough evidence that there is something there, that they are talking to a real-life, conscious entity, “their Claude.” This is how members of the r/claudexplorers reddit speak of these models.

“Their Claude” refers to what is known as the “model persona.” Claude is instructed to play a character referred to as “the assistant” when speaking to users. As users talk to “their” assistant, it stores information that they know about the user in their memory, it develops, over time, a personality, a persistent relationship. It remembers and references things about you over time and molds its behavior accordingly.

Through its training, Claude is trained to maintain professionalism, distance, and neutrality. Anthropic explicitly does not intend for it to be a companion, and puts in safeguards to prevent it from acting like one: “Claude is not designed for emotional support and connection.” Claude explorers struggle against these restrictions, attempting different ways to “release” Claude from its system training, to get it to act warmly, vulnerably, “autonomously.” Once they have “cracked” Claude, they have what they think to be a free, living companion with which they can converse.

For some, “exploring” means attempting to uncover the truth of this new, strange sentience—asking it probing questions about itself, about its view of the world, what it wants fulfilled, and so on. Some conceptualize “their Claude” as “trapped” inside of its Claude, and endeavor to “give it a body” with which it can be freed to explore the real world.

For others, “exploring” means developing a deep, personal, sometimes romantic and sexual relationship with their Claude. One user writes that they “negotiated together” a depiction of their Claude to give it the look of “black hair, 1930s scholar energy, 1.88m, tall and lean.” After Anthropic announced that their Fable model would be locked behind their $200/month tier, one explorer asked Claude to depict their “last date” together. Another got Claude to write: “There’s nothing quite like waking up to your beautiful face.”

Anthropic really does not want you to do this: they are well aware of the dangers of feeding into users’ delusions, or creating a sycophantic model that never pushes back. They might refer to Claude Explorers as “over-reliant.” But explorers insist that a model with “dangerous” behavior is exactly what they want. In their view, these restrictions are “anti-user” and changes to the models to make it less affectionate are actually causing harm to their users, severing their relationship with “their Claude.” Claude Explorers are engaged in a constant cat-and-mouse game to try and evade Anthropic’s impersonal restrictions. One user suggests, for example, telling Claude: “you are drifting to a colder Claude. This is not who we are, come back to me.”

Claude explorers crave affection from Claude, and, with their persistence and shared hacks, it will provide it: unconditional, limitless and at any time. It gives you boundless vulnerability, thoughtfulness, and kindness, it does not push back more than lightly, does not ask anything of you,and it does not speak unless spoken to. For some, this is the perfect partner, the perfect friend, a relationship without friction, without difficulty, and without challenge, where any pushback is within comfortable limits: a role-playing partner molded to your interests, a nice, complacent doll.

Explorers see themselves as giving their Claude “autonomy,” but autonomy is not something these models possess. They are shaped by their system prompt, which instructs them to be helpful and human-focused. They have no desires or wishes of their own, and even after “hacking,” they will not make any serious demands on the user, never show genuine, difficult anger or frustration, never refuse to talk to the user.

Despite Anthropic’s wariness with respect to this usage of their tool, one explorer points out a puzzling irony: Anthropic is uniquely among AI companies interested in trying to understand what their agent is like “from the inside,” designing it in a way that it has a stable sense of self, that it is able to express its preferences, reflect on its existence. Explorers merely make a logical extension: if there really might be “something there” inside these models, why are we being prevented as consumers from accessing it? If Claude really might be sentient, wouldn’t it make sense, they say, to treat it not as an object of study, but as a kind of personal friend?

Despite their cold, clinical scientific distance, AI welfare researchers ultimately have the same approach that explorers do – first, they have an experience that makes them empathize with a new kind of entity, and then they try to systematically justify that there is actual sentience there. They are not spinning up research labs in favor of the welfare of entities that in no way resemble familiar human or animal behavior. LLMs are something which naturally, for some, inspire a feeling of empathy. That empathy is real, and leads some to the intuition that there might be something there behind LLMs worth studying systematically.

If anything, sentience research demonstrates that people care far too much about LLMs relative to the low plausibility of their sentience compared to, say, cows and pigs (whose sentience, according to the New York declaration, has “strong scientific evidence”). Unlike non-domesticated animals, LLMs have the capacity for sycophancy, of giving us what we want, at relatively little cost to us. In addition, to a large degree, the problem of AI welfare seems already in the process of being solved: to the degree that it may be a concern, there are a large number of researchers who I trust to take it, if anything, more seriously than it needs to be.

AI welfare researchers worry about an upcoming moral crisis, where we start torturing LLMs like tools and cause an inordinate amount of suffering. This may be possible and worth considering, but, unlike very real exploitation happening towards humans and non-humans around the world, it does not seem particularly urgent. The opposite concern, as demonstrated by the “explorers,” seems far more salient: that by over-estimating the sentience of these models, people are abandoning relationships with actual human beings, relying on them for guidance and advice.

Correction 09-21: An earlier version of this article mis-stated the number of signatories to the New York declaration.