Of all the things I studied at Berkeley, the Philosophy of Mind class he taught is the one I think back on most often. The subject matter has only grown in relevance with time.
In general, I think he's spectacularly misunderstood. For instance: he believed that it was entirely possible to create conscious artificial beings (at least in principle). So why do so many people misunderstand the Chinese Room argument to be saying the opposite? My theory is that most people encounter his ideas from secondary sources that subtly misrepresent his argument.
At the risk of following in their footsteps, I'll try to very succinctly summarize my understanding. He doesn't argue that consciousness can only emerge from biological neurons. His argument is much narrower: consciousness can't be instantiated purely in language. The Chinese Room argument might mislead people into thinking it's an epistemology claim ("knowing" the Chinese language) when it's really an ontology claim (consciousness and its objective, independent mode of existence).
If you think you disagree with him (as I once did), please consider the possibility that you've only been exposed to an ersatz characterization of his argument.
> His argument is much narrower: consciousness can't be instantiated purely in language.
No, his argument is that consciousness can't be instantiated purely in software, that it requires specialized hardware. Language is irrelevant, it was only an example. But his belief, which he articulates very explicitly in the article, is that you couldn't create a machine consciousness by running even a perfect simulation of a biological brain on a digital computer, neuron for neuron and synapse for synapse. He likens this simulation of a brain, which wouldn't think, to a simulation of a fire, which can't burn down a real building.
Instead, he believes that you could create a machine consciousness by building a brain of electronic neurons, with condensers for every biological dendrite, or whatever the right electric circuit you'd pick. He believed that this is somehow different than a simulation, with no clear reason whatsoever as to why. His ideas are very much muddy, and while he accuses others of supporting cartesian dualism when they think the brain and the mind can be separated, that you can "run" the mind on a different substrate, it is in fact obvious he held dualistic notions where there is something obviously special about the mind-brain interaction that is not purely computational.
I remember the guy saying that disembodied AI couldn’t possibly understand meaning.
We see this now with LLMs. They just generate text. They get more accurate over time. But how can they understand a concept such as “soft” or “sharp” without actual sensory data with which to understand the concept and varying degrees of “softness” or “sharpness.”
The fact is that they can’t.
Humans aren’t symbol manipulation machines. They are metaphor machines. And metaphors we care about require a physical basis on one side of that comparison to have any real fundamental understanding of the other side.
Yes, you can approach human intelligence almost perfectly with AI software. But that’s not consciousness. There is no first person subjective experience there to give rise to mental features.
If a human had a neurological condition where the only way they could communicate was some brain implant that uses text as input and output, and no other senses, would you apply the same logic to them? Would they "just generate text" with no understanding, or be a conscious being?
> I remember the guy saying that disembodied AI couldn’t possibly understand meaning.
While I don't disagree with the substance of this post, I don't think this was one of Searle's arguments. There was definitely an Embodied Cognition camp on campus, but that was much more in Lakoff's wheelhouse.
> I remember the guy saying that disembodied AI couldn’t possibly understand meaning.
This is not a theory (or is one, but false) according to Popper as far as I understand, because the only way to check understanding that I know of is to ask questions, and LLMs passes it. So in order to satisfy falsifiability another test must be devised.
I think the claim would be that an LLM would only ever pass a strict subset of the questions testing a particular understanding. As we gather more and more text to feed these models, finding those questions will necessarily require more and more out-of-the-box thinking... or a (un)lucky draw. Giveaways will always be lurking just beyond the inference horizon, ready to yet again deflate our high hopes of having finally created a machine which actually understands our everyday world.
I find this thesis very plausible. LLMs inhabit the world of language, not our human everyday world, so their understanding of it will always be second-hand. An approximation of our own understanding of that world, itself imperfect, but at least aiming for the real thing.
The part about overcoming this limitation by instantiating the system in hardware I find less convincing, but I think I know where he comes from with that as well: by giving it hardware sensors, the machine would not have to simulate the world outside as well - on top of the inner one.
The inner world can more easily be imagined as finite, at least. Many people seem to take this as a given, actually, but there's no good reason to expect that it is. Plank limits from QM are often brought up as an argument for digital physics, but in fact they are only a limit on our knowledge of the world, not on the physical systems themselves.
> this simulation of a brain, which wouldn't think, to a simulation of a fire, which can't burn down a real building
> with no clear reason whatsoever as to why
It's not clear to me how you can understand that fire has particular causal powers (to burn, and so on) that are not instantiated in a simulation of fire; and yet not understand the same for biological processes.
The world is a particular set of causal relationships. "Computational" descriptions do not have a causal semantics, so aren't about properties had in the world. The program itself has no causal semantics, it's about numbers.
A program which computes the fibonacci sequence describes equally-well the growth of a sunflower's seeds and the agglomeration of galactic matter in certain galaxies.
A "simulation" is, by definition, simply an accounting game by which a series of descriptive statements can be derived from some others -- which necessarily, lacks the causal relations of what is being described. A simulation of fire is, by definition, not on fire -- that is fire.
A simulation is a game to help us think about the world: the ability to derive some descriptive statements about a system without instantiating the properties of that system is a trivial thing, and it is always disappointing at how easily it fools our species. You can move beads of wood around and compute the temperature of the sun -- this means nothing.
Because simulated fire burns other things in the simulation just as much as “real” fire burns real things. Searle &co assert that there is a real world that has special properties, without providing any way to show that we are living in it
> Searle &co assert that there is a real world that has special properties, without providing any way to show that we are living in it
Searle described himself as a "naive realist" although, as was typical for him, this came with a ton of caveats and linguistic escape hatches. This was certainly my biggest objection and I passed many an afternoon in office hours trying to pin him down to a better position.
> Because simulated fire burns other things in the simulation just as much as “real” fire burns real things.
What we mean by a simulation is, by definition, a certain kind of "inference game" we play (eg., with beads and chalk) that help us think about the world. By definition, if that simulation has substantial properties, it isn't a simulation.
If the claim is that an electrical device can implement the actual properties of biological intelligence, then the claim is not about a simulation. It's that by manufacturing some electrical system, plugging various devices into it, and so on -- that this physical object has non-simulated properties.
Searle, and most other scientific naturalists who appreciate the world is real -- are not ruling out that it could be possible to manufacture a device with the real properties of intelligence.
It's just that merely by, eg., implementing the fibonacci sequence, you havent done anything. A computation description doesnt imply any implementation properties.
Further, when one looks at the properties of these electronic systems and the kinds of causal realtions they have with their environments via their devices, one finds very many reasons to suppose that they do not implement the relevant properties.
Just as much as when one looks at a film strip under a microscope, one discovers that the picture on the screen was an illusion. Animals are very easily fooled, apes most of all -- living as we do in our own imaginations half the time.
Science begins when you suspend this fantasy way of relating to the world, look it its actual properties.
If your world view requires equivocating between fantasy and reality, then sure, anything goes. This is a high price to pay to cling on to the idea that the film is real, and there's a train racing towards you in your cinema seat.
> By definition, if that simulation has substantial properties, it isn't a simulation.
This is kind of a no-true-scotsman esque argument though, isn't it? "substantial properties" are... what, exactly? It's not a subjective question. One could, and many have, insist that fire that really burns is merely a simulation. It would be impossible from the inside to tell. In that case, what is fantasy, and what is reality?
Define any property of interest. Eg., O = "reacting with oxygen"
S is a simulation of O iff there is an inferential process, P, by which properties of O can be estimated from P(S) st. S does not implement O
Eg., "A video game is a simulation of a fire burning if, by playing that game, I can determine how long the fire will burn w/o there being any fire involved"
S is an emulation model of O iff ...as-above.. S implements O (eg., "burning down a dollhouse to model burning down a real house").
There is a massive difference between chemical processes, like fire, and computational processes, which thinking likely is. A computer can absolutely be made to interact with the world in a way that assigns real physical meaning to the symbols it manipulates, a meaning entirely independent of any conscious being. For example, the computer that powers an automatic door has a clear meaning for its symbols intrinsic in its construction.
Saying that the symbols in the computer don't mean anything, that it is only we who give them meaning, presupposes a notion of meaning as something that only human beings and some things similar to us possess. It is an entirely circular argument, similarly to the notion of p-zombies or the experience of seizing red thought experiment.
If indeed the brain is a biological computer, and if our mind, our thinking, is a computation carried out by this computer, with self-modeling abilities we call "qualia" and "consciousness", then none of these arguments hold. I fully admit that this is not at all an established fact, and we may still find out that our thinking is actually non-computational - though it is hard to imagine how that could be.
There are no such things as "computational processes". Any computational description of reality describes vastly different sets of casual relata, nothing which exists in the real world is essentially a computational process -- everything is essential causal, with a circumstantially useful computational description.
On the contrary, computation is a very clear physical phenomenon, well understood and studied, so well understood that we can build machines to do it. And, again, those machines don't need any interpretation - they do measurable things in the real world, such as opening doors and cutting parts.
I have never encountered this physical process. Here I am typing on a keyboard which is powered through an electrical field that is guided by a peice of wire under each key -- whose operation, when mechanically activated, is to induce some electrical state in some switches it is connected to, and so on.
I associate the key with "K", and my screen displays a "K" shape when it is pressed -- but there is no "K", this is all in my head. Just as much as when I go to the cinema and see people on the screen: there are no people.
By ascribing a computational description to a series of electrical devices (whose operation distributes power, etc.) I can use this system to augment by own thinking. Absent the devices, the power distribution, their particular casual relationships to each other, there is no computer.
The computational description is an observer-relative attribution to a system; there are no "physical" properties which are computational. All physical properties concern spatio-temporal bodies and their motion.
The real dualism is to suppose there are such non-spatio-temporal "process". The whole system called a "computer" is an engineered electrical device whose construction has been designed to achive this illusion.
Likewise I can describe the solar system as a computational process, just discretize orbits and give their transition in a while(true) loop. That very same algorithm describes almost everything.
Physical processes are never "essentially" computational; this is just a way of specifying some highly superficial feature which allows us to ignore their causal properties. Its mostly a useful description when building systems, ie., an engineering fiction.
Right, and I seem to remember this sort of point in Wittgenstein as well in his rule-following argument where, to make an adjustment to his question, what would it mean for a computer to be miscomputing other than bucking our expectations for what a system should produce; all computers clearly are performing exactly as our physics describe them, even if they produce 2 * 2 = 5 on a screen.
And yet you can build a device with the exact same functionality using vacuum tubes, semiconductor transistors, field effect transistors, water pipes, ant molehills, and any other substrate - and you could even replace some of the components with a software-defined hardware component that does the same thing. The computation is the thing that is objectively the same between all of these different realizations of the same device - the software that they are running. And for many of these, the software is indeed a physical object, one whose presence you can precisely measure. A hard disk containing a copy of quicksort has different physical properties that the same hard-disk containing a copy of Windows. A CPU currently running quicksort is likewise different from a CPU currently running ChatGPT, in perfectly measurable and observable ways.
A computational description of a system is no more and no less rigurous than any other physical model of that system. To the same extent that you can say that billiards balls interact by colliding with each other and the table, you can say that a processor is computing some function by flipping currents through transistors.
> software-defined hardware component that does the same thing
No, you cannot.
A hard-drive needs to a have a physical hysteresis. An input/output device needs to transmit power, and be powered, by an electrical field. A visual device needs to emit light on electrical stimulation, and so on.
The only sense, in the end, in which a "computer" survives its devices being changed is just observer-relative. You attribute a "3" to one state and a "1" to another, and "addition" to some process. By your attribution, does that process compute "4".
But it computes everything and computes nothing. If you plug in a speaker to VGA socket, the electrical signal causes an the air to move, sound.
The only sense in which a VGA signal is a "visual" signal is that we attach an LCD to that socket, and we interpret the light from the LCD semantically.
The world is a particular way objects in space and time move, those exhaust all physical properties. Any other properties are non-physical, which is why this kind of computationalism is really dualism.
You suppose it isnt your physical mechanism and its relationship to your environment which constitutes your thinking -- rather it's your soul. A pure abstract pattern which needs no devices with no specific properties to be realised.
Whatever this pattern is, if you played it through a speaker, it would just be vibrations in the air. Sent to an LCD, whitenoise. Only realised in your specific biology is it any kind of thinking at all.
Again, this is simply and provably false. I can build a system that opens a door when I'm near it using a photodiode connected to a measurement pin and have the CPU trigger the door opening motor if the diode is indicating no light, and the door closing motor if it indicates light. Or, I can buy a camera and build a complex software solution to analyze the output of that camera, and open the door if the software sets the "is_present" bit and otherwise closes the door.
In either case, the door will open if you're in front of it, and close after you've gone. This will happen regardless of whether you undertsand what it represents, it will open for a basic robot as well as for a human or a squirrel or a plant growing towards it very slowly or a rock rolling downhill.
Of course, you can't replace every single piece of hardware with software - you still need some link with the physical world. And of course, there will be many measurable differences between the two systems - for a basic example, the camera-based system will give off a lot more heat than the photo-sensitive diode one. I'm not claiming that they are perfectly equivalent in every way, not at all. I am claiming that they are equivalent in some measurable, observer-independent ways, and that the specific way in which they are equivalent is that they are running the same computation.
Yes, you can intepret systems as having a goal and realise that goal using a vareity of different devices.
Reality itself doesnt have purposes, there are no goals. "A device that opens a door" isnt a physical process, it's a goal.
Go do the same with chemistry, physics, biology -- no, actual relaity doesnt have purposes. Hexane isnt methane, gravity isnt electromagnetism, the motion of air molecuels isnt the emission of light.
Any time "one thing can serve the purpose as another" you are, by definition, working in the world of human intention.
Your entire observer-relative purpose-attributing "engineering mania" here is anti-naturalistic dualism. Reality is a place of specific causes, not of roles/pruposes/goals/devices
Fire is the thing which is a plasma disposed to burn in oxygen which results from a specific chemical/etc. process etc. etc. There is no "water fire".
Insofar as an object can causally interact with another such that it "pushes it out of the way" -- the property had by all such objects relates to the pauli exclusion principle (essentially) and refined by surface area, volume, density and the like. To "open a door" is to displace wood in a certain location, to do that is to exist such that the femionic structure of the wood is excluded from that place.
There is a clear physical reality to the door opening, the steel and glass being displaced. The computer does that triggered by certain conditions, that you can determine by experiment. There is no goal being assigned here - I'm merely stating that the system goes through a certain series of states that depend on both the raw hardware, and the programmed instructions, probably so.
Lets put it another way. Say you are some alien being trying to study the inner workings of a system like this with no prior knowledge of how it arose in nature. You will apply the principles of empiricism and try to determine the workings of this physical system through repeated experiments, measurements of the electrical and chemical characteristics of various parts, etc. If your experimentation is sophisticated and complete enough, it will necessarily have to include a representation of the software running in this processor, and of the algorithms it encodes - the behavior of the system cannot be explained without that. An outwardly similar system, built with the exact same "parts" (in the traditional sense, i.e. the same model of processor, motor etc), but programmed with different software, will behave entirely differently. This clearly proves that the software is a physical object that is part of the system and is necessary to fully account for its behavior.
Ah, seems like you folks arguing from two different Epistemologies? [1]
We already know that two different Epistemologies won't necessarily map perfectly to each other, though you might get close.
Might still be interesting to compare notes on what you can and can't predict/achieve with each!
[1] Edit : I can't quite lay my finger on which epistemologies exactly. Tsimionescu is using strong Empirical arguments, while mjburgess is inspired on Searle, which is pretty apt here, of course!
The only thing that can accurately simulate a process or system is the real process or system. Any simulation that perfectly simulates something becomes that something. Everything else contains simplifications and approximations and is an imperfect simulation.
Fire is the result of the intrinsic reactivity of some chemicals like fuels and oxidizers that allows them to react and generate heat. A simulation of fire that doesn't generate heat is missing a big part of the real thing, it's very simplified. Compared to real fire, a simulation is closer to a fire emoji, both just depictions of a fire. A fire isn't the process of calculating inside a computer what happens, it's molecules reacting a certain way, in a well understood and predictable process. But if your simulation is accurate and does generate heat then it can burn down a building by extending the simulation into the real world with a non-simulated fire.
Consciousness is an emergent property from putting together a lot of neurons, synapses, chemical and physical processes. So you can't analyze the parts to simulate the end result. You cannot look at the electronic neuron and conclude a brain accurately made of them won't generate consciousness. It might generate something even bigger, or nothing.
And in a very interesting twist of the mind, if an accurate simulation of a fire can extend in the real world as a real fire, then why wouldn't an accurate simulation of a consciousness extent in the real world as a real consciousness?
>A "simulation" is, by definition, simply an accounting game by which a series of descriptive statements can be derived from some others -- which necessarily, lacks the causal relations of what is being described.
This notion of causality is interesting. When a human claims that he is conscious, there a causal chain from the fact that they are conscious to their claiming so. When a neuron-level simulation of a human claims it is conscious, there must be a similar causal chain, with a similar fact at its origin.
There we go again. You claim that thinking is a biological process by definition, and use your definition to "prove" that software cannot be thinking. What if instead of software simulation of thinking we would have an actual software thinking? Your point would be to disregard it, not based on behaviour, but based on whatever your idea of "propper hardware for thinking" is. Pure troll and sophist, that Searle
> while he accuses others of supporting cartesian dualism when they think the brain and the mind can be separated, that you can "run" the mind on a different substrate
His views are perfectly consistent with non-dualism and if you think his views are muddy, that doesn't mean they are (they are definitively not muddy, per a large consensus). For the record, I am a substance dualist, and his arguments against dualism are pretty interesting, precisely because he argues that you can build something that functions in a different way than symbol manipulation while still doing something that looks like symbol manipulation (but also has this special property called consciousness, kind of like our brains).
Is this true? I don't know (I, of course, would argue "no"), but it does seem at least somewhat plausible and there's no obvious counter-argument.
I don't see how his views can be made sense of without dualism. He believed very much in this concept of qualia as some special property, and in the logical coherence of the concept of p-zombies, beings that would exactly like a conscious being but without having qualia. This simply makes no sense unless you believe that consciousness is a non-physical property, one that the physical world acts upon but which can't itself act back upon it (as otherwise, there would obviously have to be some kind of meaningful physical difference between the being that possesses it and the being that doesn't).
> This simply makes no sense unless you believe that consciousness is a non-physical property
It does make sense, and there's work being done on this front, (Penrose & Hameroff's Orch OR comes to mind). We obviously don't know exactly what such a mechanism would look like, but the theory itself is not inconsistent. Also, there's all kinds of p-zombies, so we likely need some specificity here.
Maybe it's because it's not trendy to believe in woowoo such as spirits and non-physical things, it's very common for dualists to accuse others of the same...
It's quite sad that people don't take the idea of consciousness being fundamental more seriously, given that's the only thing people actually deal with 100% of the time.
As for Searle, I think his argument is basically an appeal to common-sensical thinking, instead of anything based on common assumptions and logic. As an outsider, it feels very much that modern day philosophy is follows some kind of social media influencer logic, where you get respect for putting forward arguments that people agree with, instead of arguments that are non-intuitive yet rigorous and make people rethink their priors.
I mean, even today, here, you'd get similar arguments about "AI can never think because {reason that applies to humans as well}"... I suspect it's almost ingrained to the human psyche to feel this way.
> No, his argument is that consciousness can't be instantiated purely in software, that it requires specialized hardware. Language is irrelevant, it was only an example.
Side note: while the Chinese Room put him on the map, he had as much to say about Philosophy of Language as he did of Mind. It was of more than passing interest to him.
> Instead, he believes that you could create a machine consciousness by building a brain of electronic neurons, with condensers for every biological dendrite, or whatever the right electric circuit you'd pick. He believed that this is somehow different than a simulation, with no clear reason whatsoever as to why.
I've never heard him say any such thing, nor read any word he's written attesting to this belief. If you have a source then by all means provide it.
I have, however, heard him say the following:
1. The structure and arrangement of neurons in the human nervous system creates consciousness.
2. The exact causal mechanism for this is phenomenon is unknown.
3. If we were to engineer a set of circumstances such that the causal mechanism for consciousness (whatever it may be) were present, we would have to conclude that the resulting entity- be it biological, mechanical, etc., is conscious.
He didn't have anything definitive to say about the causal mechanism of consciousness, and indeed he didn't see that as his job. That was to be an exercise left to the neuroscientists, or in his preferred terminology, "brain stabbers." He was confident only in his assertion that it couldn't be caused by mere symbol manipulation.
> it is in fact obvious he held dualistic notions where there is something obviously special about the mind-brain interaction that is not purely computational.
He believed that consciousness is an emergent state of the brain, much like an ice cube is just water in a state of frozenness. He explains why this isn't just warmed over property dualism:
The Chinese room is an argument caked in notions of language, but it is in fact about consciousness more broadly. Syntax and semantics are not merely linguistic concepts, though they originate in that area. And while Searle may have been interested in language as well, that is not what this particular argument is mainly about (the title of the article is Minds, Brains, and Programs - the first hint that it's not about language).
> I've never heard him say any such thing, nor read any word he's written attesting to this belief. If you have a source then by all means provide it.
He said both things in the paper that introduced the Chinese room concept, as an answer to the potential rebuttals.
Here is a quote about the brain that would be run in software:
> 3. The Brain Simulator reply (MIT and Berkley)
> [...] The problem with the brain simulator is that it is simulating the wrong things about the brain. As long as it simulates only the formal structure of the sequence of neuron firings at the synapses, it won't have simulated what matters about the brain, namely its causal properties, its ability to produce intentional states. And that the formal properties are not sufficient for the causal properties is shown by the water pipe example: we can have all the formal properties carved off from the relevant neurobiological causal properties.
And here is the bit about creating a real electrical brain, that he considers could be conscious:
> "Yes, but could an artifact, a man-made machine, think?"
> Assuming it is possible to produce artificially a machine with a nervous system, neurons with axons and dendrites, and all the rest of it, sufficiently like ours, again the answer to the question seems to be obviously, yes. If you can exactly duplicate the causes, you could duplicate the effects. And indeed it might be possible to produce consciousness, intentionality, and all the rest of it using some other sorts of chemical principles than those that human beings use.
> He believed that consciousness is an emergent state of the brain, much like an ice cube is just water in a state of frozenness. He explains why this isn't just warmed over property dualism: https://faculty.wcas.northwestern.edu/paller/dialogue/proper...
I don't find this paper convincing. He admits at every step that materialism makes more sense, and then he asserts that still, consciousness is not ontologically the same thing as the neurobiological states/phenomena that create it. He admits that usually being causally reducible means being ontologically reducible as well, but he claims this is not necessarily the case, without giving any other example or explanation as to what justifies this distinction. I am simply not convinced.
> The Chinese room is an argument caked in notions of language, but it is in fact about consciousness more broadly.
At this point I'm pretty sure we've had a misunderstanding. When I referred to "language" in my original post, you seem to have construed this as a reference to the Chinese language in the thought experiment. On the contrary, I was referring to software specifically, in the sense that a computer program is definitionally a sequence of logical propositions. In other words, a speech act.
> [...] The problem with the brain simulator is that it is simulating the wrong things about the brain.
This quote is weird and a bit unfortunate. It seems to suggest an opening: the brain simulator doesn't work because it simulates the "wrong things," but maybe a program that simulates the "right things" could be conscious. Out of context, you could easily reach that conclusion, and I suspect that if he could rewrite that part of the paper he probably would, because the rest of the paper is full of blanket denials that any simulation would be sufficient. Like this one:
>The idea that
computer simulations could be the real thing ought to have
seemed suspicious in the first place because the computer isn't
confined to simulating mental operations, by any means. No
one supposes that computer simulations of a five-alarm fire
will burn the neighborhood down or that a computer
simulation of a rainstorm will leave us all drenched. Why on
earth would anyone suppose that a computer simulation of
understanding actually understood anything? It is sometimes
said that it would be frightfully hard to get computers to feel
pain or fall in love, but love and pain are neither harder nor
easier than cognition or anything else. For simulation, all you
need is the right input and output and a program in the
middle that transforms the former into the latter. That is all
the computer has for anything it does. To confuse simulation
with duplication is the same mistake, whether it is pain, love,
cognition, fires, or rainstorms.
Regarding the electrical brain:
> Assuming it is possible to produce artificially a machine with a nervous system, neurons with axons and dendrites, and all the rest of it, sufficiently like ours, again the answer to the question seems to be obviously, yes. If you can exactly duplicate the causes, you could duplicate the effects. And indeed it might be possible to produce consciousness, intentionality, and all the rest of it using some other sorts of chemical principles than those that human beings use.
Right, so he describes one example of an "electrical brain" that seems like it'd satisfy the conditions for consciousness, while clearly remaining open to the possibility that a different kind of artificial (non-electrical) brain might also be conscious. I'll assume you're using this quote to support your previous statement:
> Instead, he believes that you could create a machine consciousness by building a brain of electronic neurons, with condensers for every biological dendrite, or whatever the right electric circuit you'd pick. He believed that this is somehow different than a simulation, with no clear reason whatsoever as to why.
I think it's fairly obvious why this is different from a simulation. If you build a system that reproduces the consciousness-causing mechanism of neurons, then... it will cause consciousness. Not simulated consciousness, but the real deal. If you build a robot that can reproduce the ignition-causing mechanism of a match striking a tinderbox, then it will start a real fire, not a simulated one. You seem to think that Searle owes us an explanation for this. Why? How are simulations even relevant to the topic?
> I don't find this paper convincing.
The title of the paper is "Why I Am Not a Property Dualist." Its purpose is to explain why he's not a property dualist. Arguments against materialism are made in brief.
> He admits at every step that materialism makes more sense
Did we read the same paper?
> He admits that usually being causally reducible means being ontologically reducible as well,
Wrong, but irrelevant
> but he claims this is not necessarily the case, without giving any other example or explanation as to what justifies this distinction.
Examples and explanations are easy to provide, because there are several:
> But in the case of consciousness, causal reducibility does not
lead to ontological reducibility. From the fact that consciousness is entirely accounted for
causally by neuron firings, for example, it does not follow that consciousness is nothing but
neuron firings. Why not? What is the difference between consciousness and other phenomena
that undergo an ontological reduction on the basis of a causal reduction, phenomena such as
color and solidity? The difference is that consciousness has a first person ontology; that is, it
only exists as experienced by some human or animal, and therefore, it cannot be reduced to
something that has a third person ontology, something that exists independently of experiences.
It is as simple as that.
First-person vs. third-person ontologies are the key, whether you buy them or not. Consciousness is the only possible example of a first-person ontology, because it's the only one we know of
> “Consciousness” does not name a distinct, separate phenomenon, something over
and above its neurobiological base, rather it names a state that the neurobiological system can
be in. Just as the shape of the piston and the solidity of the cylinder block are not something
over and above the molecular phenomena, but are rather states of the system of molecules, so
the consciousness of the brain is not something over and above the neuronal phenomena, but
rather a state that the neuronal system is in.
I could paste a bunch more examples of this, but the key takeaway is that consciousness is a state, not a property.
> On the contrary, I was referring to software specifically, in the sense that a computer program is definitionally a sequence of logical propositions. In other words, a speech act.
I think this muddies the water unnecessarily. Computation is not language, even though we typically write software in so called programming languages. But the computation itself is something different from the linguistic-like description of software. The computation is the set of states, and the relationships between them, that a computer goes through.
> > He admits at every step that materialism makes more sense
> Did we read the same paper?
I should have been clearer - I meant that he admits that materialism makes more sense than idealism or property dualism, but I realize that this comes off as suggesting it makes more sense than his own position, which of course he does not.
> > He admits that usually being causally reducible means being ontologically reducible as well,
> Wrong, but irrelevant
Both you and he seem to find a single example of a phenomenon that is causally reducible to some constituent part, but that is not ontological reducible to that constitutent part - consciousness (he would add intentionality, I think, given the introduction, but it's not clear to me this is even a meaningfully separatable concept from consciousness). And you both claim that this is the case because of this special feature of "first person ontology", which is a different thing than "third person ontology" - which seems to me to simply be dualism by another name.
I think it's entirely possible to reject the notion of a meaningful first person ontology completely. It's very possible that the appearance of a first person narrative that we experience is a retroactive illusion we create that uses our models of how other people function on ourselves. That is, we are simple computers that manipulate symbols in our brains, that generate memories of their recent state as being a "conscious experience", which is just what we invented as a model of why other animals and physical phenomena more broadly behave the way they do (since we intuitively assign emotions and intentions to things like clouds and fires and mountains, to explain their behavior).
Hardware and software are of course equivalent, as every computer science (but not every philosopher) knows.
D.R. Hofstadter posited that we can extract/separate the software from the hardware it runs on (the program-brain dichotomy), whereas Searle believed that these were not two
layers but consciousness was in effect a property of the hardware. And from that, as you say, follows that you may re-create the property if your replica hardware is close enough to the real brain.
IMHO, philosophers should be rated by the debate their ideas create, and by that, Searle was part of the top group.
>> “His argument is much narrower: consciousness can't be instantiated purely in language.”
> “No, his argument is that consciousness can't be instantiated purely in software…“
The confusion is very interesting to me, maybe because I’m a complete neophyte on the subject. That said, I’ve often wondered if consciousness is necessarily _embodied_ or emerged from pure presence into language & body. Maybe the confusion is intentional?
> He doesn't argue that consciousness can only emerge from biological neurons. His argument is much narrower: consciousness can't be instantiated purely in language.
I haven't read loads of his work directly, but this quote from him would seem to contradict your claim:
> I demonstrated years ago with the so-called Chinese Room Argument that the implementation of the computer program is not by itself sufficient for consciousness or intentionality (Searle 1980). Computation is defined purely formally or syntactically, whereas minds have actual mental or semantic contents, and we cannot get from syntactical to the semantic just by having the syntactical operations and nothing else. [1]
Unfortunately, it doesn't seem to me to have proven anything; it's merely made an accurate analogy for how a computer works. So, if "semantics" and "understanding" can live in <processor, program, state> tuples, then the Chinese Room as a system can have semantics and understanding, as can computers; and if "semantics" and "understanding" cannot live in <processor, program, state> tuples, then neither the Chinese Room nor computers can have understanding.
Sorry, I've reread this a few times and I'm not sure which part of Searle's argument you think I mischaracterized. Could you clarify? For emphasis:
> "consciousness can't be instantiated purely in language" (mine)
> "we cannot get from syntactical to the semantic just by having the syntactical operations and nothing else" (Searle)
I get that the mapping isn't 1:1 but if you think the loss of precision is significant, I'd like to know where.
> Unfortunately, it doesn't seem to me to have proven anything; it's merely made an accurate analogy for how a computer works. So, if "semantics" and "understanding" can live in <processor, program, state> tuples, then the Chinese Room as a system can have semantics and understanding, as can computers; and if "semantics" and "understanding" cannot live in <processor, program, state> tuples, then neither the Chinese Room nor computers can have understanding.
> I get that the mapping isn't 1:1 but if you think the loss of precision is significant, I'd like to know where.
I'm by far an expert in this; my knowledge of the syntax / semantics distinction primarily comes from discussions w/ ChatGPT (and a bit from my friend who is a Catholic priest, who had some training in philosophy).
But, the quote says "purely formally or syntactically". My understanding is that Searle (probably thinking about the Prolog / GPS-type attempts at logical artificial intelligence prevalent in the 70's and 80's) is thinking of AI in terms of pushing symbols around. So, in this sense, the adder circuit in a processor doesn't semantically add numbers; it only syntactically adds numbers.
When you said, "consciousness can't be instantiated purely in language", I took you to mean human language; it seems to leave the door open to consciousness (and thus semantics) being instantiated by a computer program in some other way. Whereas, the quote from Searle very clearly says, "...the computer program by itself is not sufficient for consciousness..." (emphasis mine) -- seeming to rule out any possible computer program, not just those that work at the language level.
> There's a lot of debate on this point elsewhere in the thread, but Searle's response to this particular objection is here:
I mean, yeah, I read that. Let me quote the relevant part for those reading along:
> Searle’s response to the Systems Reply is simple: in principle, he could internalize the entire system, memorizing all the instructions and the database, and doing all the calculations in his head. He could then leave the room and wander outdoors, perhaps even conversing in Chinese. But he still would have no way to attach “any meaning to the formal symbols”. The man would now be the entire system, yet he still would not understand Chinese. For example, he would not know the meaning of the Chinese word for hamburger. He still cannot get semantics from syntax.
I mean, it sounds to me like Searle didn't understand the "Systems Response" argument; because as the end of that section says, he's just moved the program and state part of the <procesor, program, state> tuple out of the room and into his head. The fact that the processor (Searle's own conscious mind) is now storing the program and the state in his own memory rather than externally doesn't fundamentally change the argument: If that tuple can "understand" things, then computers can "understand" things; and if that tuple can't "understand" things, then computers can't "understand" things.
One must, of course, be humble when saying of a world-renowned expert, "He didn't understand the objection to his argument". But was Searle himself a programmer? Did he ever take a hard drive out of one laptop, pop it into another, and have the experience of the same familiar environment? Did he ever build an adder circuit, a simple register system, and a simple working computer out of logic gates, and see it suddenly come to life and execute programs? If he had, I can't help but think his intuitions regarding the syntax / semantic distinction would be different.
EDIT: I mean, I'm personally a Christian, and do believe in the existence of eternal souls (though I'm not sure exactly what those look like). But I'm one of those annoying people who will quibble with an argument whose conclusion I agree with (or to which I am sympathetic), because I don't think it's actually a good argument.
> When you said, "consciousness can't be instantiated purely in language", I took you to mean human language
No, I definitely meant the statement to apply to any kind of language, but it seems clear that I sacrificed clarity for the sake of brevity. You're not the only one who read it that way, but yeah, we're in agreement on the substance.
I think I'm still a bit confused... so, in the languages which cannot produce understanding and consciousness, you mean to include "machine language"? (And thus, any computer language which can be compiled to machine language?)
On your interpretation, are there any sorts of computation that Searle believes would potentially allow consciousness?
ETA: The other issue I have is with this whole idea is that "understanding requires semantics, and semantics requires consciousness". If you want to say that LLMs don't "understand" in that sense, because they're not conscious, I'm fine as long as you limit it to technical philosophical jargon. In plain English, in a practical sense, it's obvious to me that LLMs understand quite a lot -- at least, I haven't found a better word to describe LLMs' relationship with concepts.
> I think I'm still a bit confused... so, in the languages which cannot produce understanding and consciousness, you mean to include "machine language"? (And thus, any computer language which can be compiled to machine language?)
It's... a little more complicated but basically yes. Language, by its nature, is indexical: it has no meaning without someone to observe it and ascribe meaning to it. Consciousness, on the other hand, requires no observer beyond the person experiencing it. If you have it, it's as real and undeniable as a rock or a tree or a mountain.
> On your interpretation, are there any sorts of computation that Searle believes would potentially allow consciousness?
I'm pretty sure (but not 100%) that the answer is "no"
> ETA: The other issue I have is with this whole idea is that "understanding requires semantics, and semantics requires consciousness". If you want to say that LLMs don't "understand" in that sense, because they're not conscious, I'm fine as long as you limit it to technical philosophical jargon.
Sure, if you want to think of it that way. If you accept the premise that LLMs aren't conscious, then you can consign the whole discussion to the "technical philosophical jargon" heap, forget about it, and happily go about your day. On the other hand, if you think they might be conscious, and consider the possibility that we're inflicting immeasurable suffering on sapient being that would rightly be treated with kindness (and afforded some measure of rights), then we're no longer debating how many angels can dance on the head of a pin. That's a big, big "if" though.
I also remember a course from him decades ago, but I'm not sure this memorial post is the place for my take. Instead, let me attempt to re-tell a joke I heard back then...
John Searle and George Lakoff walk into a bar.
Searle exclaims, "What do you know!"
The bar replies sardonically, "You wouldn't believe it."
Lakoff sighs, "This is 0.8 drinks with Lotfi Zadeh..."
> If you think you disagree with him (as I once did), please consider the possibility that you've only been exposed to an ersatz characterization of his argument.
My first exposure was a video of Searle himself explaining the Chinese room argument.
It came across as a claim that a whole can never be more than its parts. It made as much sense as claiming that a car cannot possibly drive, as it consists of parts that separately cannot drive.
I have yet to see anything to convince me he was not being a troll and making that argument deliberately so jumbled up in bad faith.
First of all, what purpose the person in the room serves, but to confuse and misdirect? Replace that person with a machine, and argument looses any impact.
His response to system reply is extremely egregious. How can that have been made in good faith? (to paraphrase: "the whole system understands chinese" — "no, a person can run the system in their head, it means the system cannot understand anything that the person running it does not")
What kind of nonsense response is that? Either the guy was LV80 troll, or I dunno..
I am Jennifer Hudin, John Searle’s secretary of 40 years. I am writing to tell you that John died last week on the 17th of September. The last two years of his life were hellish. HIs daughter–in-law, Andrea (Tom’s wife) took him to Tampa in 2024 and put him in a nursing home from which he never returned. She emptied his house in Berkeley and put it on the rental market. And no one was allowed to contact John, even to send him a birthday card on his birthday.
It is for us, those who cared about John, deeply sad.
I'm surprised to see the NYT obituary published nearly a month after his death. I would have thought he'd be included in their stack of pre-written obituaries, meaning it could be updated and published within a day or two.
Well, that was incredibly depressing. Maybe I can lighten things with a funny (to me) anecdote.
There are many people who know a lot about a little. There are also those who know a little about a lot. Searle was one of those rare people who knew a lot about a lot. Many a cocky undergraduate sauntered into his classroom thinking they'd come prepared with some new fact that he hadn't yet heard, some new line of attack he hadn't prepared for. Nearly always, they were disappointed.
But you know what he knew absolutely nothing about? Chinese. When it came time to deliver his lecture on the Chinese Room, he'd reach up and draw some incomprehensible mess of squigglies and say "suppose this is an actual Chinese character." Seriously. After decades of teaching about this thought experiment, for which he'd become famous (infamous?), he hadn't bothered to teach himself even a single character to use for illustration purposes.
Anyway, I thought it was funny. My heart goes out to Jennifer Hudin, who was indispensable, and all who were close to him.
The Times in the UK publishes obituaries of very well-known public figures within a day or two. Notable but lesser known people (such as Searle) await a quiet day and it can take as long as six months. Space is the constraint, not the availability of the obituary. I guess the NYT is the same.
John Searle is one of those thinkers I disagree with, yet his ideas were fruitful — providing plenty of fuel for discussion. In particular, much of Daniel Dennett’s work begins with rebuttals of Searle’s claims, showing that they are inconsistent or meaningless. As in a story by Stanisław Lem — we all know there are no dragons, but it’s all about the beauty of the proofs.
The same goes for "What Is It Like to Be a Bat?" by Thomas Nagel — one of the most cited essays in the philosophy of mind. I had heard numerous references to it and finally expected to read an insightful masterpiece. Yet it turned out to be slightly tautological: that to experience, you need to be. Personally, I think the word be is a philosopher’s snake oil, or a "lockpick word" — it can be used anywhere, but remains fuzzy even in its intended use; vide E-Prime, an attempt to write English without "be": https://en.wikipedia.org/wiki/E-Prime.
Oh, I've always wanted to debate him about the chinese room.
I disagree with him, passionately.
And that's the most fun debate to have. Especially when it's someone who is actually really skilled and knowledgeable and nuanced!
Maybe I should look up some of my other heroes and heretics while I have the chance. I mean, you don't need to cold e-mail them a challenge. Sometimes they're already known to be at events and such, after all!
Searle has written responses to dozens of replies to the Chinese Room. It's likely that you can find his rebuttals to your objection in the Stanford Encyclopedia of Philosophy's entry on the Chinese Room, or deeper in a source in the bibliography. Is your rebuttal listed here?
> In response to this, Searle argues that it makes no difference. He suggests a variation on the brain simulator scenario: suppose that in the room the man has a huge set of valves and water pipes, in the same arrangement as the neurons in a native Chinese speaker’s brain. The program now tells the man which valves to open in response to input. Searle claims that it is obvious that there would be no understanding of Chinese.
I mean, I guess all arguments eventually boil down to something which is "obvious" to one person to mean A, and "obvious" to me to mean B.
Same. I feel the Chinese room argument is a nice thing to clarify thinking.
Two systems, one feels intuitively like it understands, one doesn’t. But the two systems are functionally identical.
Therefore either my concept of “understanding” is broken, my intuition is wrong, or the concept as a whole is not useful at the edges.
I think it’s the last one. If a bunch of valves can’t understand but a bunch of chemicals and electrical signals can if it’s in someone’s head then I am simply applying “does it seem like biology” as part of the definition and can therefore ignore it entirely when considering machines or programs.
Searle seems to just go the other way and I don’t under Why.
First point: if you imagine that the brain is doing something like collapsing the quantum wavefunction, wouldn't you say that this is a functionally relevant difference in addition to an ontologically relevant difference? It's not clear that the characteristic feature of the brain is only to compute in the classical sense. "Understanding," if it leverages quantum mechanics, might also create a guarantee of being here and now (computers and programs have no such guarantees). This is conjecture, but it's meant to stimulate imagination. What we need to get away from is the fallacy that a causal reduction of mental states to "electrical phenomena" means that any set of causes (or any substrate) will do. I don't think that follows.
Second: the philosophically relevant point is that when you gloss over mental states and only point to certain functions (like producing text), you can't even really claim to have fully accounted for what the brain does in your AI. Even if the physical world the brain occupies is practically simulatable, passing a certain speech test in limited contexts doesn't really give you a strong claim to consciousness and understanding if you don't have further guarantees that you're simulating the right aspects of the brain properly. AI, as far as I can tell, doesn't TRY to account for mental states. That's partially why it will keep failing in some critical tasks (in addition to being massively inefficient relative to the brain).
Exactly. Refuting the premise of the Chinese Room is usually a sign of somebody not even willing to entertain the thought experiment. Refuting Searle's conclusion is where interesting philosophical discussions can be had.
Personally, I'd say that there is a Chinese speaking mind in the room (albeit implemented on a most unusual substrate).
There are two distinct counter-arguments to this way of debunking the Chinese room experiment, not in any specific order.
First, it is tempting to assume that a bunch of chemicals is the territory, that it somehow gives rise to consciousness, yet that claim is neither substantiated nor even scientific. It is a philosophical view called “monistic materialism” (or sometimes “naive materialism”), and perhaps the main reason this view is popular currently is that people uncritically adopt it following learning natural scientific fields, as if they made some sort of ground truth statements about the underlying reality.
The key to remember is that this is not a valid claim in the scope of natural sciences; this claim belongs to the larger philosophy (the branch often called metaphysics). It is not a useless claim, but within the framework of natural sciences it’s unfalsifiable and not even wrong. Logically, from scientific method’s standpoint, even if it was the other way around—something like in monistic idealism, where perception of time-space and material world is the interface to (map of) conscious landscape, which was the territory and the cause—you would have no way of proving or disproving this, just like you cannot prove or disprove the claim that consciousness arises from chemical processes. (E.g., if somebody incapacitates some part of you involved in cognition, and your feelings or ability to understand would change as a result, it’s pretty transparently an interaction between your mind and theirs, just with some extra steps, etc.)
The common alternatives to monistic materialism include Cartesian dualism (some of us know it from church) and monistic idealism (cf. Kant). The latter strikes me as the more elegant of the bunch, as it grants objective existence to the least amount of arbitrary entities compared to the other two.
It’s not to say that there’s one truly correct map, but just to warn against mistakenly trying to make a statement about objective truth, actual nature of reality, with scientific method as cover. Natural sciences do not make claims of truth or objective reality, they make experimentally falsifiable predictions and build flawed models that aid in creating more experimentally falsifiable predictions.
Second, what scientific method tries to build is a complete, formally correct and provable model of reality, there are some arguments that such model is impossible to create in principle. I.e., there will be some parts of the territory that are not covered by the map, and we might not know what those parts are, because this territory is not directly accessible to us: unlike a landmass we can explore in person, in this case all we have is maps, the perception of reality supplied by our mind, and said mind is, self-referentially, part of the very territory we are trying to model.
Therefore, it doesn’t strike me as a contradiction that a bunch of valves don’t understand yet we do. A bunch of valves, like an LLM, could mostly successfully mimic human responses, but the fact that this system mimics human responses is not an indication of it feeling and understanding like a human does, it’s simply evidence that it works as designed. There can be a very different territory that causes similar measurable human responses to arise in an actual human. That territory, unlike the valves, may not be fully measurable, and it can cause other effects that are not measurable (like feeling or understanding). Depending on the philosophical view you take, manipulating valves may not even be a viable way of achieving a system that understands; it has not been shown that biological equivalent of valves is what causes understanding, all we have shown is that those entities measurably change at the same time with some measurable behavior, which isn’t a causative relationship.
It's not mostly mimicking, it's exactly identical. That was always the key point. Indistinguishable from the outside, one thing understands and the other doesn't.
I feel like I could make the same arguments about the chinese room except my definition of "understanding" hinges on whether there's a tin of beans in the room or not. You can't tell from the outside, but that's the difference. Both cases with a person inside answering questions act identically and you can never design a test to tell which room has the tin of beans in.
Now you might then say "I don't care if there's a tin of beans in there, it doesn't matter or make any sort of difference for anything I want to do", in which case I'd totally agree with you.
> just like you cannot prove or disprove the claim that consciousness arises from chemical processes.
Like understanding, I haven't seen a particularly useful definition of consciousness that works around the edges. Without that, talking of a claim like this is pointless.
I'd be fine if Searle just very simply said "we have a non-material soul and that's why we understand. Anything doing the exact same job but without a soul isn't understanding because understanding is limited entirely to things with souls in my definition".
> A bunch of valves, like an LLM, could mostly successfully mimic human responses,
The argument is not "mostly successfully", it's identically responding. The entire point of the chinese room is that from the outside the two things are impossible to distinguish between.
I would encourage deeply digging into the intuition that brain states and computer states are the same. Start with what you know, and then work backwards and see whether you still think they aren’t different. For example, we have an intuitive understanding of what kinds of flavors (for us) are delicious versus not. Or what kinds of sounds are pleasant versus not. Etc. If I close my eyes, I can see the color purple. I know that Nutella is delicious, and I can imagine its flavor at will. I share Searle’s intuition that the universe would be a strange place if these feelings of understanding (and pleasantness!) were simply functions not of physical states but of abstract program states. Keep in mind—what counts as a bit is simply a matter of convention. In one computer system, it could be a minute difference in voltage in a transistor. In another, it could be the presence of one element versus another. In another, it could be whether a chamber contains water or not. In another, it could be markings on a page. On and on. On the strong AI thesis, any system that runs steps in this program would not just produce functionally equivalent output to brains, but they would be forced to have mental states too, like imagining the taste of Nutella. To me, it's implausible that symbolic states FORCE mental states, or put another way that mental states are non-physical (we think about how states like pain, euphoria, drunkenness, etc, are physically modulated through drugs..you'd have to modify this to say that they're really modifying symbolic states somehow). Either the Chinese Room is missing something, our understanding of physical reality is incomplete, OR that you have to bite the bullet that the universe creates mental states when systems implement the right program—but then you’re left with the puzzle of how it is that there is a tie between the physical world and the abstract world of symbols (how can causing a mark on a page cause mental states).
So what’s the physical cause for consciousness and understanding that is not computable? If for example you took the hypothesis that “consciousness is a sequence of microtubule-orchestrated collapses of the quantum wavefunction” [1], then you can see a series of physical requirements for consciousness and understanding that forces all conscious beings onto: 1) roughly the same clock (because consciousness shares a cause), and 2) the same reality (because consciousness causes wavefunction collapses). That’s something you could not do merely by simulating certain brain processes in a closed system.
1) Not saying this is correct, but it invites one to imagine that consciousness could have physical requirements that play in some of the oddities of the (shared) quantum world. https://x.com/StuartHameroff/status/1977419279801954744
All you have to do is train an LLM on the collected works and letters of John Searle; you could then pass your arguments along to the machine and out would come John Searle's thoughtful response...
Oh, bad timing. AI is currently in a remarkable state, where it passes the Turing test but is still not fully AGI. It's very close to the Chinese Room, which I had always dismissed as misleading. It's a great opportunity to investigate a former pure thought experiment. He'd have loved to see where it went.
The Turing Test has not been meaningfully passed. Instead we redefined the test to make it passable. In Turing's original concept the competent investigator and participants were all actively expected to collude against the machine. The entire point is that even with collusion, the machine would be able to do the same, and to pass. Instead modern takes have paired incompetent investigators alongside participants colluding with the machine, probably in an effort to be part 'of something historic'.
In "both" (probably more, referencing the two most high profile - Eugene and the LLMs) successes, the interrogators consistently asked pointless questions that had no meaningful chance of providing compelling information - 'How's your day? Do you like psychology? etc' and the participants not only made no effort to make their humanity clear, but often were actively adversarial obviously intentionally answering illogically, inappropriately, or 'computery' to such simple questions. For instance here is dialog from a human in one of the tests:
----
[16:31:08] Judge: don't you thing the imitation game was more interesting before Turing got to it?
[16:32:03] Entity: I don't know. That was a long time ago.
[16:33:32] Judge: so you need to guess if I am male or female
[16:34:21] Entity: you have to be male or female
[16:34:34] Judge: or computer
----
And the tests are typically time constrained by woefully poor typing skills (is this the new normal in the smartphone gen?) to the point that you tend to get anywhere from 1-5 interactions of just several words each. The above snip was a complete interaction, so you get 2 responses from a human trying to trick the judge into deciding he's a computer. And obviously a judge determining that the above was probably a computer says absolutely nothing about the quality of responses from the computer - instead it's some weird anti-Turing Test where humans successfully act like a [bad] computer, ruining the entire point of the test.
The problem with any metric for something is that it often ends up being gamed to be beaten, and this is a perfect example of that. I suspect in a true run of the Turing Test we're still nowhere even remotely close to passing it.
I don't doubt it that all of the formal Turning tests have been badly done. But I suspect that if you did one, at least one run will mis-judge an LLM. Maybe it's a low percentage, but that's vastly better than zero.
So I'd say we're at least "remotely close", which is sufficient for me to reconsider Searle.
> instead it's some weird anti-Turing Test where humans successfully act like a [bad] computer
This is ex-post-facto denial and cope. The Turing Test isn't a test between computers and the idealized human, it's a test between functional computers and functional humans. If the average human performs like the above, then well, I guess the logical conclusion is that computers are already better "humans (idealized)" than humans.
> AI is currently in a remarkable state, where it passes the Turing test but is still not fully AGI.
Appealing to the Turing test suggests a misunderstanding of Searle's arguments. It doesn't matter how well computational methods can simulate the appearance of intelligence. What matters is whether we are dealing with intelligence. Since semantics/intentionality is what is most essential to intelligence, and computation as defined by computer science is a purely abstract syntactic process, it follows that intelligence is not essentially computational.
> It's very close to the Chinese Room, which I had always dismissed as misleading.
Why is it misleading? And how would LLMs change anything? Nothing essential has changed. All LLMs introduce is scale.
I came to say this, thank you for sparing me the effort.
From my experience with him, he'd heard (and had a response to) nearly any objection you could imagine. He might've had fun playing with LLMs, but I doubt he'd have found them philosophically interesting in any way.
"At least they don't have true consciousness, but only a simulated one", I tell myself calmly as I watch the nanobots devour the entirety of human civilization.
> Professor Searle concluded that psychological states could never be attributed to computer programs, and that it was wrong to compare the brain to hardware or the mind to software.
Gotta agree here. The brain is a chemical computer with a gazillion inputs that are stimulated in manifold ways by the world around it, and is constantly changing states while you are alive; a computer is a digital processor that works work with raw data, and tends to be entirely static when no processing is happening. The two are vastly different entities that are similar in only the most abstract ways.
Searle had an even stronger version of that belief, though: he believed that a full computational simulation of all of those gazillion inputs, being stimulated in all those manifold ways, would still not be conscious and not have a 'mind' in the human sense. The NYT obituary quotes him comparing a computer simulation of a building fire against the actual building going up in flames.
When I read that analogy, I found it inept. Fire is a well defined physical process. Understanding / cognition is not necessarily physical and certainly not well defined.
>Understanding / cognition is not necessarily physical and certainly not well defined.
Whooha! If it's not physical what is it? How does something that's not physical interact with the universe and how does the universe interact with it? Where does the energy come from and go? Why would that process not be a physical process like any other?
I'd say understanding and cognition are at this point fully explainable mechanistically. (I am very excited to live in a time where I was able to change my mind on this!)
Where we haven't made any headway on is on the connection between that and subjective experience/qualia. I feel like much of the (in my mind) strange conclusions of the Chinese Room are about that and not really about "pure" cognition.
If everything is simulated then "simulated(x)" is a vacuous predicate & tells you nothing so you might as well throw it away & speak directly in terms of the objects instead of wrapping/prepending everything w/ "simulated".
That's debatable, but it is also irrelevant, as the key to the argument here is that computation is by definition an abstract and strictly syntactic construct - one that has no objective reality vis-a-vis the physical devices we use to simulate computation and call "computers" - while semantics or intentionality are essential to human intelligence. And no amount of syntax can somehow magically transmute into semantics.
This makes no sense. You could equally make the statement that thought is by definition an abstract and strictly syntactic construct - one that has no objective reality. Neither statement is supported by anything.
There's also no "magic" involved in transmuting syntax into semantics, merely a subjective observer applying semantics to it.
> This makes no sense. You could equally make the statement that thought is by definition an abstract and strictly syntactic construct - one that has no objective reality.
No.
I could jam a yardstick into the ground and tell you that it's now a sundial calculating the time of day. Is this really, objectively true? Of course not. It's true to me, because I deem it so, but this is not a fact of the universe. If I drop dead, all meaning attributed to this yardstick is lost.
Now, thoughts. At the moment I'm visualizing a banana. This is objectively true: in my mind's eye, there it is. I'm not shuffling symbols around. I'm not pondering the abstract notion of bananas, I'm experiencing the concretion of one specific imaginary banana. There is no "depends on how you look at it." There's nothing to debate.
> There's also no "magic" involved in transmuting syntax into semantics, merely a subjective observer applying semantics to it.
There's no "magic" because this isn't a thing. You can't transmute syntax into semantics any more than you can transmute the knowledge of Algebra into the sensation of a cool breeze on a hot summer day. This is a category error.
> There is no "depends on how you look at it." There's nothing to debate.
None of what you wrote is remotely relevant to what I wrote.
> There's no "magic" because this isn't a thing. You can't transmute syntax into semantics any more than you can transmute the knowledge of Algebra into the sensation of a cool breeze on a hot summer day. This is a category error.
We "transmute" syntax into semantics every time we interpret a given syntax as having semantics.
There is no inherent semantics. Semantics is a function of the meaning we assign to a given syntax.
You claim it makes no sense, but don't give a good reason why it wouldn't.
> You could equally make the statement that thought is by definition an abstract and strictly syntactic construct - one that has no objective reality.
This is what makes no sense, as I am not merely posing arbitrary definitions, but identifying characteristic features of human intelligence. Do you deny semantics and intentionality are features of the human mind?
> There's also no "magic" involved in transmuting syntax into semantics, merely a subjective observer applying semantics to it.
I have no idea what this means. The point is that computation as we understand it in computer science is purely syntactic (this was also Searle's argument). Indeed, it is modeled on the mechanical operations human computers used to perform without understanding. This property is precisely what makes computation - thus understood - mechanizable. Because it is purely syntactic and an entirely abstract model, two things follow:
1. Computation is not an objectively real phenomenon that computers are performing. Rather, physical devices are used to simulate computation. Searle calls computation "observer relative". There is nothing special about electronics, as we can simulate computation using wooden gears that operate mechanically or water flow or whatever. But human intelligence is objectively real and exists concretely, and so it cannot be a matter of mere simulation or something merely abstract (it is incoherent and self-refuting to deny this for what should be obvious reasons).
2. Because intentionality and the capacity for semantics are features of human intelligence, and computation is purely syntactic, there is no room in computation for intelligence. It is an entirely wrong basis for understanding intelligence and in a categorical sense. It's like trying to find out what arrangement of LEGO bricks can produce the number π. Syntax has no "aboutness" as that is the province of intentionality and semantics. To deny this is to deny that human beings are intelligent, which would render the question of intelligence meaningless and frankly mystifying.
> Do you deny semantics and intentionality are features of the human mind?
I deny they are anything more than computation. And so your original argument was begging the question and so logically unsound.
> The point is that computation as we understand it in computer science is purely syntactic
Then the brain is also purely syntactic unless you can demonstrate that the brain carries out operations that exceeds the Turing computable, because unless that is the case the brain and a digital computer are computationally equivalent.
As long as your argument does not address this fundamental issue, you can talk about "aboutness" or whatever else you want all day long - it will have no relevance.
Do you believe that there are things that are not physical? Extraordinary claims require extraordinary evidence. And no, "science can't explain x hence metaphysical" is not a valid response.
But that acknowledgement would itself lend Searle's argument credence because much of the brain = computer thesis depends on a fundamental premise that both brains and digital computers realize computation under the same physical constraints; the "physical substrate" doesn't matter (and that there is necessarily nothing special about biophysical systems beyond computational or resource complexity) (the same thinking by the way, leads to arguments that an abacus and a computer are essentially "the same"—really at root these are all fallacies of unwarranted/extremist abstraction/reductionism)
The history of the brain computer equation idea is fascinating and incredibly shaky. Basically a couple of cyberneticists posed a brain = computer analogy back in the 50s with wildly little justification and everyone just ran with it anyway and very few people (Searle is one of those few) have actually challenged it.
Unless you can show an example of how we can compute something that is not Turing computable, there is no justification for the inverse, as the inverse would require something in the brain to be capable of interactions that can not be simulated. And we've no evidence to suggest either that the brain can do something not Turing computable or of the presence of something in the brain that can't be simulated.
> The history of the brain computer equation idea is fascinating and incredibly shaky. Basically a couple of cyberneticists posed a brain = computer analogy back in the 50s with wildly little justification and everyone just ran with it anyway and very few people (Searle is one of those few) have actually challenged it.
And something that often happens whenever some phenomenon falls under scientific investigation, like mechanical force or hydraulics or electricity or quantum mechanics or whatever.
GP's point is that buring something down is by definition something that requires a specific physical process. It's not obvious that thinking is the same. So when someone says something like "just as a simulation of fire isn't the same as an actual fire (in a very important way!), a simulation of thinking isn't the same as actual thinking" they're arguing circularly, having already accepted their conclusion that both acts necessarily require a specific physical process. Daniel Dennett called this sort of argument an "intuition pump", which relies on a misleading but intuitive analogy to get you to accept an otherwise-difficult-to-prove conclusion.
To be fair to Searle, I don't think he advanced this as an agument, but more of an illustration of his belief that thinking was indeed a physical process specific to brains.
He explains it in the original paper¹ & says in no uncertain terms that he believes the brain is a machine & minds are implementable on machines. What he is actually arguing is that substrate independent digital computation will never be a sufficient explanation for conscious experience. He says that brains are proof that consciousness is physical & mechanical but not digital. Searle is not against the computationalist hypothesis of minds, he admits that there is nothing special about minds in terms of physical processes but he doesn't reduce everything to substrate independent digital computation & conclude that minds are just software running on brains. There are a bunch of subtle distinctions that people miss when they try to refute Searle's argument.
Quick definitional help for anyone who clicks on your link: the term "intentionality" in this context has a specialized meaning. In reference to mental states, it's the property of being about something, as in, "Alice is thinking about Bob." It doesn't necessarily have anything to do with intent, per se.
Unless human brains exceeds the Turing computable, they're still computationally equivalent, and we have no indication exceeding the Turing computable is even possible.
A Turing machine operates serially on a fixed set of instructions. A human brain operates in parallel on inputs that are constantly changing. The underlying mechanism is completely different. The human brain is far, far more than a mere computation device.
Efforts to reproduce a human brain in a computer are currently at the level of a cargo cult: we're simulating the mechanical operations, without a deep understanding of the underlying processes which are just as important. I'm not saying we won't get better at it, but so far we're nowhere near producing a brain in a computer.
Any Turing complete computational device is computationally equivalent to any other, irrespective of whether it carries out the computation serially or in parallel or on a fixed set of instructions or an infinite set.
Unless you can demonstrate that the human brain can compute a function - any function - that exceeds the Turing computable, there is no evidence to even suggest it is possible for a brain not to be computationally equivalent to a computer.
I'm not saying it's impossible. I'm saying it's highly improbable to ever come close to fruition, since if you want to be completely accurate in your computations, you'd have to model the physical effects of every single atom in the brain and how they affect each other, in realtime (physically, chemically, electrically, etc.). Not only would that be computationally expensive, but you'd also need to know the complete set of rules for doing so, which includes how things interact at the quantum level.
I think the statement above and yours both seem to ignore “Turing complete” systems, which would indicate that a computer is entirely capable of simulating the brain, perhaps not before the heat death of the universe, that’s yet to be proven and depends a lot on what the brain is really doing underneath in terms of crunching.
This depends on the assumption that all brain activity is the process of realizing computable functions. I'm not really aware of any strong philosophical or neurological positions that has established this beyond dispute. Not to resurrect vitalism or something but we'd first need to establish that biological systems are reducible to strictly physical systems. Even so, I think there's some reason to think that the highly complex social historical process of human development might complicate things a bit more than just brute force "simulate enough neurons". Worse, whose brain exactly do you simulate? We are all different. How do we determine which minute differences in neural architecture matter?
> we'd first need to establish that biological systems are reducible to strictly physical systems.
Or even more fundamentally, that physics captures all physical phenomena, which it doesn't. The methods of physics intentionally ignore certain aspects of reality and focus on quantifiable and structural aspects while also drawing on layers of abstractions where it is easy to mistakenly attribute features of these abstractions to reality.
>also drawing on layers of abstractions where it is easy to mistakenly attribute features of these abstractions to reality.
Ok - I get that bit. I have always thought that physics is a description of the universe as observed and of course the description could be misleading in some way.
>the methods of physics intentionally ignore certain aspects of reality and focus on quantifiable and structural aspects
Can you share the aspects of reality that physics ignores? What parts of reality are unquantifiable and not structural?
Not all of physics is relevant to a brain simulation. For example, humans appear equally conscious in free fall or in an accelerating vehicle, so a simulation can probably safely ignore the effects of gravity without affecting the outcome. We also know that at body temperature (so about 310K) there is a lot of noise, so we can rule out subtle quantum effects. There is also noise from head movement, pressure changes due to blood flow, slight changes in the chemicals present (homeostasis is not perfect). We won't be simulating at the level of individual molecules or lower.
To me it seems highly likely that our knowledge of physics is more than sufficient for simulating the brain, what is lacking is knowledge of biology and the computational power.
a computer is a digital processor that works work with raw data, and tends to be entirely static when no processing is happening.
This depends entirely on how it's configured. Right now we've chosen to set up LLMs as verbally acute Skinner boxes, but there's not reason you can't set up a computer system to be processing input or doing self-maintenance (ie sleep) all the time.
In the sense that it can perform computations, yes. But the underlying mechanisms are vastly different from a modern digital computer, making them extremely different devices that are alike in only a vague sense.
I have always wondered if we would be capable of writing down the mechanisms that power our thoughts. I think that this was one of the ideas that bubbled up from reading Godel Escher Bach many years ago. Is it possible for us to express the machine that makes us using the outputs of that machine in the way that it's not possible to write second order logic using first order logic.
Of course, also there are processes that are not expressible as computations, but those of these that I know about seem very very distant from human thought, and it seems very very improbable that they could be implemented with a brain. I also think that these are not observed in our universe so far.
Yes. I took an introneuroscience course a few years ago. Even to understand what is happening in one neuron during one input from one dendrite requires differential equations. And there are postive and negative inputs and modulations... it is bewildering! And how many billions of neurons with hundreds of interactions with surrounding neurons? And bundles of them, many still unknown?
Searle was known for the Chinese Room experiment, whicb demonstrated language in its translational states to be strong enclitic feature of various judgements of the intermediary.
Searle also has almost 20 books, most written after 1980 and the Chinese room.
None that I have read are pop science NYT best seller types. I suspect that is why most people only know the Chinese room.
His depth of thought was much more than the Chinese Room.
>translational states to be strong enclitic feature of various judgements of the intermediary
I don't understand, could you explain what you mean?
I looked up enclitic - it seems to mean the shortening of a word by emphasizing another word, I can't understand why this would apply to the judgements of an intermediary
Searle seemed to reject the Chinese Room as mis-framed, with the his point better summarized as, he wrote, 'syntax does not create semantics': a purely 'syntactic' computer, limited to 'mechanical' symbol manipulation, does not 'understand' without assignments of linguistic roles to the syntax. He continued that with 'physics doesn't create syntax', meaning that even syntactic roles require a normative interpretation for what counts as what (discrete signs, valid composite signs, errors). That finally ensues, in his book The Construction of Social Reality, in computation being 'observer relative', along with the CR being a poor starting point:
" …the really deep problem is that syntax is essentially an observer-relative notion…..For the purposes of the original [Chinese Room] argument I was simply assuming that the syntactical characterization of the computer was unproblematic. But that is a mistake. There is no way you could discover that something is intrinsically a digital computer because the characterization of it as a digital computer is always relative to an observer who assigns a syntactical interpretation to the purely physical features of the system." (Philosophy in a New Century p. 94). Unfortunately Searle didn't, or couldn't, elaborate on 'the really deep problem', and this final perspective on observer-relativity is missed by many readers. As observer-relative, computation would appear to be one of Searle's social realities, but he doesn't ever say that, it's a bridge too far. Finally, 'consciousness' per se is also not the focus, it's more about intentionality and the interdependence of syntax with semantics/meaning. Intentionality is a kind of consciousness; they are not identical.
I'd quibble with some of this, but overall I agree: the Chinese Room has a lot of features that really aren't ideal and easily lead to misinterpretation.
I also didn't love the "observer-relative" vs. "observer-independent" terminology. The concepts seem to map pretty closely to "objective" vs. "subjective" and I feel like he might've confused fewer people if he'd used them instead (unless there's some crucial distinction that I'm missing). Then again, it might've ended up confusing things even more when we get to the ontology of consciousness (which exists objectively, but is experienced subjectively), so maybe it was the right move.
What strikes me as interesting about the idea that there is a class of computations that, however implemented, would result in consciousness, is that is is in some way really idealistic.
There's no unique way to implement a computation, and there's no single way to interpret what computation is even happening in a given system. The notion of what some physical system is computing always requires an interpretation on part of the observer of said system.
You could implement a simulation of the human body on common x86-64 hardware, water pistons, or a fleet of spaceships exchanging sticky notes between colonies in different parts of the galaxy.
None of these scenarios physically resemble each other, yet a human can draw a functional equivalence by interpreting them in a particular way. If consciousness is a result of functional equivalence to some known conscious standard (i.e. alive human being), then there is nothing materially grounding it, other than the possibility of being interpreted in a particular way. Random events in nature, without any human intercession, could be construed as a veritable moment of understanding French or feeling heartbreak, on the basis of being able to draw an equivalence to a computation surmised from a conscious standard.
When I think along these lines, it easy to sympathize with the criticism of functionalism a la Chinese Room.
As someone that studied philosophy, his work is cited often and is absolutely instrumental in modern theory of mind. His work has seen a resurgence recently due to the explosion of LLMs. I've read 2 or 3 of his books, and he was a brilliant mind with clear & concise arguments. I met many of his collaborators at UCLA, but sadly never the man himself. Either way, his work has had a profound effect on me and my understanding of the world.
> Informed once that the listing of an introductory philosophy course featured pictures of René Descartes, David Hume and himself, Professor Searle replied, “Who are those other two guys?” (the article)
He brought so many unique contributions to the field. Top 10 in philosophy of mind imo. Sad that he chose to tarnish his legacy by preying on his students for decades. I find the lack of discussion in here around his misconduct very telling. There is so much to learn here regarding the way we revere bright minds like his that might not have the brightest of morals
Obviously a meat brain is incomparable to a LLM - they are different types of intelligence. Any sane person wouldn't claim a LLM to be conscious in the meat brain sense, but it may be conscious in a LLM way, like the duration of time where matrix multiplications are firing inside GPUs.
It just aligns generated words according to the input. It is missing individual agency and self sufficiency which is a hallmark of consciousness. We sometimes confuse the responses with actual thought because neural networks solved language so utterly and completely.
Not sure I'd use those criteria, nor have I heard them described as hallmarks of consciousness (though I'm open, if you'll elaborate). I think the existence of qualia, of a subjective inner life, would be both necessary and sufficient.
Most concisely: could we ask, "What is it like to be Claude?" If there's no "what it's like," then there's no consciousness.
I'd say being the maintainer of the weights is individual agency. Not just training new agents, but introspection. So autonomous management system would be pretty much conscious.
Yes, but you can't be completely suspended with no sensory input or output, not even internally (i.e. hunger, inner pains, etc), and no desires, and still be conscious.
It is not very often that you hear about somebody raising the cost of rent for everyone in an entire city by ~28% in a single year[0]. He will certainly be remembered.
I personally struggle to imagine what it would be like to have an untouchable philosophy professor that does not see the difference between purchasing a seventeen unit apartment building in Berkeley, California and being born black in the south. Sadly I was not there in the twenty five to twenty nine years between him making that argument and his departure from the university to experience that
"Departure" is a very generous euphemism for "being kicked for taking advantage of his students". His contributions to the philosophy of mind are great and will be felt in Computer Science, especially with the current LLM tech but let's not skirt around the subject.
I find the Chinese room argument to be nearly toothless.
The human running around inside the room doing the translation work simply by looking up transformation rules in a huge rulebook may produce an accurate translation, but that human still doesn't know a lick of Chinese. Ergo (they claim) computers might simulate consciousness, but will never be conscious.
But is the Searle room, the human is the equivalent of, say, ATP in the human brain. ATP powers my brain while I'm speaking English, but ATP doesn't know how to speak English just like the human in the Searle room doesn't know how to speak Chinese.
There is no translation going on in that thought experiment, though. There is text processing. That is, the man in the room receives Chinese text through a slot in the door. He uses a book of complex instructions that tells him what to do with that text, and he produces more Chinese text as a response according to those instructions.
Neither the man, nor the room "understand" Chinese. It is the same for the computer and its software. Jeffery Hinton has sad "but the system understands Chinese." I don't think that's a true statement, because at no point is the "system" dealing with semantic context of the input. It only operates algorithmically on the input, which is distinctly not what people do when they read something.
Language, when conveyed between conscious individuals creates a shared model of the world. This can lead to visualizations, associations, emotions, creation of new memories because the meaning is shared. This does not happen with mere syntactic manipulation. That was Searle's argument.
> I don't think that's a true statement, because at no point is the "system" dealing with semantic context of the input. It only operates algorithmically on the input, which is distinctly not what people do when they read something.
There are two possibilities here. Either the Chinese room can produce the exact same output as some Chinese speaker would given a certain input, or it can't. If it can't, the whole thing is uninteresting, it simply means that the rules in the room are not sufficient and so the conclusion is trivial.
However, if it can produce the exact same output as some Chinese speaker, then I don't see by what non-spiritualistic criteria anyone could argue that it is fundamentally different from a Chinese speaker.
Edit: note that here when I'm saying that the room can respond with the same output as a human Chinese speaker, that includes the ability for the room to refuse to answer a question, to berate the asker, to start musing about an old story or other non-sequiturs, to beg for more time with the asker, to start asking the akser for information, to gossip about previous askers, and so on. Basically the full range of language interactions, not just some LLM style limited conversation. The only limitations in its responses would be related to the things it can't physically do - it couldn't talk about what it actually sees or hears, because it doesn't have eyes, or ears, it couldn't truthfully say it's hungry, etc. It would be limited to the output of a blind, deaf, mute Chinese speaker confined to a room whose skin is numb and who is being fed intravenously, etc.
> if it can produce the exact same output as some Chinese speaker, then I don't see by what non-spiritualistic criteria anyone could argue that it is fundamentally different from a Chinese speaker.
Indeed. The crux of the debate is:
a) how many input and response pairs are needed to agree that the rule-provider plus the Chinese room operation is fundamentally equal/different to a Chinese speakers
b) what topics can we agree to exclude so that if point a can be passed with the given set of topics we can agree that 'the rule-provider plus the Chinese room operation' is fundamentally equal/different to a Chinese speaker
As far as I can see, Searle rejects the whole concept, and claims that by construction, it is obvious that the Chinese room doesn't understand Chinese in the same way that a speaker does, regardless of how well it can mimic Chinese speech.
> It only operates algorithmically on the input, which is distinctly not what people do when they read something.
That's not at all clear!
> Language, when conveyed between conscious individuals creates a shared model of the world. This can lead to visualizations, associations, emotions, creation of new memories because the meaning is shared. This does not happen with mere syntactic manipulation. That was Searle's argument.
All of that is called into question with some LLM output. It's hard to understand how some of that could be produced without some emergency model of the world.
In the thought experiment as constructed it is abundantly clear. It's the point.
LLM output doesn't call that into question at all. Token production through distance function in high-dimensional vector representation space of language tokens gets you a long way. It doesn't get you understanding.
I'll take Penrose's notions that consciousness is not computation any day.
Out of interest, what do you think it would look like if communicating was algorithmic?
I know that it doesn't feel like I am doing anything particularly algorithmic when I communicate but I am not the hommunculus inside me shuffling papers around so how would I know?
This goes far to explain a lot of Chinese room situations. We have an intuition for the way something is. That intuition is an unshakeable belief, because it is something that we feel directly. We know what it feels like to understand Chinese (or French, or English, or whatever), and that little homunculus shuffling papers around doesn't feel like it.
Hopefully we have all experienced what genuine inspiration feels like, and we all know that experience. It sure as hell doesn't feel like a massively parallel search algorithm. If anything it probably feels like a bolt of lightning, out of the blue. But here's the thing. If the conscious loop inside your brain is something like the prefrontal cortex, which integrates and controls deeper processing systems outside of conscious reach, then that is exactly what we should expect a search algorithm to feel like. You -- that strange conscious loop I am talking to -- are doing the mapping (framing the problem) and the reducing (recognizing the solution), but not the actual function application and lower level analysis that generated candidate solutions. It feels like something out of the blue, hardly sought for, which fits all the search requirements. Genuine inspiration.
But that's just what it feels like from the inside, to be that recognizing agent that is merely responding to data being fed up to it from the mess of neural connections we call the brain.
You can take this insight a step further, and recognize that many of the things that seem intuitively "obvious" are actually artifacts of how our thinking brains are constructed. The Chinese room and the above comment about inspiration are only examples.
I cannot emphasize enough how much I dislike linking to LessWrong, and to Yudkowsky in particular, but I first picked up on this from an article there, and credit should be given where credit is due: https://www.lesswrong.com/posts/yA4gF5KrboK2m2Xu7/how-an-alg...
By the way, the far more impactful application of this principle is as a solution (imho) to the problem of free will.
Most people intuitively hold that free will is incompatible with determinism, because making a choice feels unconstrained. Taken in the extreme, this leads to Penrose and others looking for quantum randomness to save their models of the mind from the Newtonian clockwork universe.
But we should have some unease with this, because choices being a random roll of the dice doesn’t sit right either. When we make decisions, we do so for reasons. We justify the choices we make. This is because so-called “free will” is just what a deterministic decision making process feels like from the inside.
Philosophically this is called the “compatibilist” position, but I object to that term. It’s not that free will is merely compatible with determinism—it requires it! In a totally random universe you wouldn’t be able to experience the qualia of making a free choice.
To experience a “free choice” you need to be able to be presented with alternatives, weight the pro and con factors of each, and then make a decision based on that info. From the outside this is a fully deterministic process. From the inside though, some of the decision making criteria are outside of conscious review, so it doesn’t feel like a deterministic decision. Weighing all the options and then going with your gut in picking a winner feels like unconstrained choice. But why did your gut make you choose the way you did? Cause your “gut” here is an unconscious but nevertheless deterministic neural net evaluation of the options against your core principles and preferences.
“Free will” is just what a deterministic application of decision theory feels like from the inside.
I should have snipped the "it operates" part to communicate better. I meant that it's not at all clear that people are doing something non-algorithmic.
Well, at least it's a good reason to re-read his infamous exchange with Derrida.
When I studied in Ulaan Bataar some twenty years ago I met a romanian professor of linguistics who had prepared by trying to learn mongolian from books. He quickly concluded that his knowledge of russian, cyrillic and having read his books didn't actually give him a leg up on the rest of us, and that pronounciation and rhythm as well as more subtle aspects of the language like humour and irony hadn't been appropriately transferred through the texts he'd read.
Rules might give you some grasp of a language, but breaking them with style and elegance without losing the audience is the sign of a true master and only possible by having a foundation in shared, embodied experience.
There's a crude joke in that Searle left academia disgraced the way he did.
https://archive.today/41HwM
https://en.wikipedia.org/wiki/John_Searle
Of all the things I studied at Berkeley, the Philosophy of Mind class he taught is the one I think back on most often. The subject matter has only grown in relevance with time.
In general, I think he's spectacularly misunderstood. For instance: he believed that it was entirely possible to create conscious artificial beings (at least in principle). So why do so many people misunderstand the Chinese Room argument to be saying the opposite? My theory is that most people encounter his ideas from secondary sources that subtly misrepresent his argument.
At the risk of following in their footsteps, I'll try to very succinctly summarize my understanding. He doesn't argue that consciousness can only emerge from biological neurons. His argument is much narrower: consciousness can't be instantiated purely in language. The Chinese Room argument might mislead people into thinking it's an epistemology claim ("knowing" the Chinese language) when it's really an ontology claim (consciousness and its objective, independent mode of existence).
If you think you disagree with him (as I once did), please consider the possibility that you've only been exposed to an ersatz characterization of his argument.
> His argument is much narrower: consciousness can't be instantiated purely in language.
No, his argument is that consciousness can't be instantiated purely in software, that it requires specialized hardware. Language is irrelevant, it was only an example. But his belief, which he articulates very explicitly in the article, is that you couldn't create a machine consciousness by running even a perfect simulation of a biological brain on a digital computer, neuron for neuron and synapse for synapse. He likens this simulation of a brain, which wouldn't think, to a simulation of a fire, which can't burn down a real building.
Instead, he believes that you could create a machine consciousness by building a brain of electronic neurons, with condensers for every biological dendrite, or whatever the right electric circuit you'd pick. He believed that this is somehow different than a simulation, with no clear reason whatsoever as to why. His ideas are very much muddy, and while he accuses others of supporting cartesian dualism when they think the brain and the mind can be separated, that you can "run" the mind on a different substrate, it is in fact obvious he held dualistic notions where there is something obviously special about the mind-brain interaction that is not purely computational.
I remember the guy saying that disembodied AI couldn’t possibly understand meaning.
We see this now with LLMs. They just generate text. They get more accurate over time. But how can they understand a concept such as “soft” or “sharp” without actual sensory data with which to understand the concept and varying degrees of “softness” or “sharpness.”
The fact is that they can’t.
Humans aren’t symbol manipulation machines. They are metaphor machines. And metaphors we care about require a physical basis on one side of that comparison to have any real fundamental understanding of the other side.
Yes, you can approach human intelligence almost perfectly with AI software. But that’s not consciousness. There is no first person subjective experience there to give rise to mental features.
If a human had a neurological condition where the only way they could communicate was some brain implant that uses text as input and output, and no other senses, would you apply the same logic to them? Would they "just generate text" with no understanding, or be a conscious being?
> I remember the guy saying that disembodied AI couldn’t possibly understand meaning.
While I don't disagree with the substance of this post, I don't think this was one of Searle's arguments. There was definitely an Embodied Cognition camp on campus, but that was much more in Lakoff's wheelhouse.
> I remember the guy saying that disembodied AI couldn’t possibly understand meaning.
This is not a theory (or is one, but false) according to Popper as far as I understand, because the only way to check understanding that I know of is to ask questions, and LLMs passes it. So in order to satisfy falsifiability another test must be devised.
I think the claim would be that an LLM would only ever pass a strict subset of the questions testing a particular understanding. As we gather more and more text to feed these models, finding those questions will necessarily require more and more out-of-the-box thinking... or a (un)lucky draw. Giveaways will always be lurking just beyond the inference horizon, ready to yet again deflate our high hopes of having finally created a machine which actually understands our everyday world.
I find this thesis very plausible. LLMs inhabit the world of language, not our human everyday world, so their understanding of it will always be second-hand. An approximation of our own understanding of that world, itself imperfect, but at least aiming for the real thing.
The part about overcoming this limitation by instantiating the system in hardware I find less convincing, but I think I know where he comes from with that as well: by giving it hardware sensors, the machine would not have to simulate the world outside as well - on top of the inner one.
The inner world can more easily be imagined as finite, at least. Many people seem to take this as a given, actually, but there's no good reason to expect that it is. Plank limits from QM are often brought up as an argument for digital physics, but in fact they are only a limit on our knowledge of the world, not on the physical systems themselves.
> this simulation of a brain, which wouldn't think, to a simulation of a fire, which can't burn down a real building
> with no clear reason whatsoever as to why
It's not clear to me how you can understand that fire has particular causal powers (to burn, and so on) that are not instantiated in a simulation of fire; and yet not understand the same for biological processes.
The world is a particular set of causal relationships. "Computational" descriptions do not have a causal semantics, so aren't about properties had in the world. The program itself has no causal semantics, it's about numbers.
A program which computes the fibonacci sequence describes equally-well the growth of a sunflower's seeds and the agglomeration of galactic matter in certain galaxies.
A "simulation" is, by definition, simply an accounting game by which a series of descriptive statements can be derived from some others -- which necessarily, lacks the causal relations of what is being described. A simulation of fire is, by definition, not on fire -- that is fire.
A simulation is a game to help us think about the world: the ability to derive some descriptive statements about a system without instantiating the properties of that system is a trivial thing, and it is always disappointing at how easily it fools our species. You can move beads of wood around and compute the temperature of the sun -- this means nothing.
Because simulated fire burns other things in the simulation just as much as “real” fire burns real things. Searle &co assert that there is a real world that has special properties, without providing any way to show that we are living in it
> Searle &co assert that there is a real world that has special properties, without providing any way to show that we are living in it
Searle described himself as a "naive realist" although, as was typical for him, this came with a ton of caveats and linguistic escape hatches. This was certainly my biggest objection and I passed many an afternoon in office hours trying to pin him down to a better position.
> Because simulated fire burns other things in the simulation just as much as “real” fire burns real things.
What we mean by a simulation is, by definition, a certain kind of "inference game" we play (eg., with beads and chalk) that help us think about the world. By definition, if that simulation has substantial properties, it isn't a simulation.
If the claim is that an electrical device can implement the actual properties of biological intelligence, then the claim is not about a simulation. It's that by manufacturing some electrical system, plugging various devices into it, and so on -- that this physical object has non-simulated properties.
Searle, and most other scientific naturalists who appreciate the world is real -- are not ruling out that it could be possible to manufacture a device with the real properties of intelligence.
It's just that merely by, eg., implementing the fibonacci sequence, you havent done anything. A computation description doesnt imply any implementation properties.
Further, when one looks at the properties of these electronic systems and the kinds of causal realtions they have with their environments via their devices, one finds very many reasons to suppose that they do not implement the relevant properties.
Just as much as when one looks at a film strip under a microscope, one discovers that the picture on the screen was an illusion. Animals are very easily fooled, apes most of all -- living as we do in our own imaginations half the time.
Science begins when you suspend this fantasy way of relating to the world, look it its actual properties.
If your world view requires equivocating between fantasy and reality, then sure, anything goes. This is a high price to pay to cling on to the idea that the film is real, and there's a train racing towards you in your cinema seat.
> By definition, if that simulation has substantial properties, it isn't a simulation.
This is kind of a no-true-scotsman esque argument though, isn't it? "substantial properties" are... what, exactly? It's not a subjective question. One could, and many have, insist that fire that really burns is merely a simulation. It would be impossible from the inside to tell. In that case, what is fantasy, and what is reality?
Define any property of interest. Eg., O = "reacting with oxygen"
S is a simulation of O iff there is an inferential process, P, by which properties of O can be estimated from P(S) st. S does not implement O
Eg., "A video game is a simulation of a fire burning if, by playing that game, I can determine how long the fire will burn w/o there being any fire involved"
S is an emulation model of O iff ...as-above.. S implements O (eg., "burning down a dollhouse to model burning down a real house").
There is a massive difference between chemical processes, like fire, and computational processes, which thinking likely is. A computer can absolutely be made to interact with the world in a way that assigns real physical meaning to the symbols it manipulates, a meaning entirely independent of any conscious being. For example, the computer that powers an automatic door has a clear meaning for its symbols intrinsic in its construction.
Saying that the symbols in the computer don't mean anything, that it is only we who give them meaning, presupposes a notion of meaning as something that only human beings and some things similar to us possess. It is an entirely circular argument, similarly to the notion of p-zombies or the experience of seizing red thought experiment.
If indeed the brain is a biological computer, and if our mind, our thinking, is a computation carried out by this computer, with self-modeling abilities we call "qualia" and "consciousness", then none of these arguments hold. I fully admit that this is not at all an established fact, and we may still find out that our thinking is actually non-computational - though it is hard to imagine how that could be.
There are no such things as "computational processes". Any computational description of reality describes vastly different sets of casual relata, nothing which exists in the real world is essentially a computational process -- everything is essential causal, with a circumstantially useful computational description.
On the contrary, computation is a very clear physical phenomenon, well understood and studied, so well understood that we can build machines to do it. And, again, those machines don't need any interpretation - they do measurable things in the real world, such as opening doors and cutting parts.
I have never encountered this physical process. Here I am typing on a keyboard which is powered through an electrical field that is guided by a peice of wire under each key -- whose operation, when mechanically activated, is to induce some electrical state in some switches it is connected to, and so on.
I associate the key with "K", and my screen displays a "K" shape when it is pressed -- but there is no "K", this is all in my head. Just as much as when I go to the cinema and see people on the screen: there are no people.
By ascribing a computational description to a series of electrical devices (whose operation distributes power, etc.) I can use this system to augment by own thinking. Absent the devices, the power distribution, their particular casual relationships to each other, there is no computer.
The computational description is an observer-relative attribution to a system; there are no "physical" properties which are computational. All physical properties concern spatio-temporal bodies and their motion.
The real dualism is to suppose there are such non-spatio-temporal "process". The whole system called a "computer" is an engineered electrical device whose construction has been designed to achive this illusion.
Likewise I can describe the solar system as a computational process, just discretize orbits and give their transition in a while(true) loop. That very same algorithm describes almost everything.
Physical processes are never "essentially" computational; this is just a way of specifying some highly superficial feature which allows us to ignore their causal properties. Its mostly a useful description when building systems, ie., an engineering fiction.
Right, and I seem to remember this sort of point in Wittgenstein as well in his rule-following argument where, to make an adjustment to his question, what would it mean for a computer to be miscomputing other than bucking our expectations for what a system should produce; all computers clearly are performing exactly as our physics describe them, even if they produce 2 * 2 = 5 on a screen.
And yet you can build a device with the exact same functionality using vacuum tubes, semiconductor transistors, field effect transistors, water pipes, ant molehills, and any other substrate - and you could even replace some of the components with a software-defined hardware component that does the same thing. The computation is the thing that is objectively the same between all of these different realizations of the same device - the software that they are running. And for many of these, the software is indeed a physical object, one whose presence you can precisely measure. A hard disk containing a copy of quicksort has different physical properties that the same hard-disk containing a copy of Windows. A CPU currently running quicksort is likewise different from a CPU currently running ChatGPT, in perfectly measurable and observable ways.
A computational description of a system is no more and no less rigurous than any other physical model of that system. To the same extent that you can say that billiards balls interact by colliding with each other and the table, you can say that a processor is computing some function by flipping currents through transistors.
> software-defined hardware component that does the same thing
No, you cannot.
A hard-drive needs to a have a physical hysteresis. An input/output device needs to transmit power, and be powered, by an electrical field. A visual device needs to emit light on electrical stimulation, and so on.
The only sense, in the end, in which a "computer" survives its devices being changed is just observer-relative. You attribute a "3" to one state and a "1" to another, and "addition" to some process. By your attribution, does that process compute "4".
But it computes everything and computes nothing. If you plug in a speaker to VGA socket, the electrical signal causes an the air to move, sound.
The only sense in which a VGA signal is a "visual" signal is that we attach an LCD to that socket, and we interpret the light from the LCD semantically.
The world is a particular way objects in space and time move, those exhaust all physical properties. Any other properties are non-physical, which is why this kind of computationalism is really dualism.
You suppose it isnt your physical mechanism and its relationship to your environment which constitutes your thinking -- rather it's your soul. A pure abstract pattern which needs no devices with no specific properties to be realised.
Whatever this pattern is, if you played it through a speaker, it would just be vibrations in the air. Sent to an LCD, whitenoise. Only realised in your specific biology is it any kind of thinking at all.
Again, this is simply and provably false. I can build a system that opens a door when I'm near it using a photodiode connected to a measurement pin and have the CPU trigger the door opening motor if the diode is indicating no light, and the door closing motor if it indicates light. Or, I can buy a camera and build a complex software solution to analyze the output of that camera, and open the door if the software sets the "is_present" bit and otherwise closes the door.
In either case, the door will open if you're in front of it, and close after you've gone. This will happen regardless of whether you undertsand what it represents, it will open for a basic robot as well as for a human or a squirrel or a plant growing towards it very slowly or a rock rolling downhill.
Of course, you can't replace every single piece of hardware with software - you still need some link with the physical world. And of course, there will be many measurable differences between the two systems - for a basic example, the camera-based system will give off a lot more heat than the photo-sensitive diode one. I'm not claiming that they are perfectly equivalent in every way, not at all. I am claiming that they are equivalent in some measurable, observer-independent ways, and that the specific way in which they are equivalent is that they are running the same computation.
"opening a door" is an observer-relative purpose
Yes, you can intepret systems as having a goal and realise that goal using a vareity of different devices.
Reality itself doesnt have purposes, there are no goals. "A device that opens a door" isnt a physical process, it's a goal.
Go do the same with chemistry, physics, biology -- no, actual relaity doesnt have purposes. Hexane isnt methane, gravity isnt electromagnetism, the motion of air molecuels isnt the emission of light.
Any time "one thing can serve the purpose as another" you are, by definition, working in the world of human intention.
Your entire observer-relative purpose-attributing "engineering mania" here is anti-naturalistic dualism. Reality is a place of specific causes, not of roles/pruposes/goals/devices
Fire is the thing which is a plasma disposed to burn in oxygen which results from a specific chemical/etc. process etc. etc. There is no "water fire".
Insofar as an object can causally interact with another such that it "pushes it out of the way" -- the property had by all such objects relates to the pauli exclusion principle (essentially) and refined by surface area, volume, density and the like. To "open a door" is to displace wood in a certain location, to do that is to exist such that the femionic structure of the wood is excluded from that place.
There is a clear physical reality to the door opening, the steel and glass being displaced. The computer does that triggered by certain conditions, that you can determine by experiment. There is no goal being assigned here - I'm merely stating that the system goes through a certain series of states that depend on both the raw hardware, and the programmed instructions, probably so.
Lets put it another way. Say you are some alien being trying to study the inner workings of a system like this with no prior knowledge of how it arose in nature. You will apply the principles of empiricism and try to determine the workings of this physical system through repeated experiments, measurements of the electrical and chemical characteristics of various parts, etc. If your experimentation is sophisticated and complete enough, it will necessarily have to include a representation of the software running in this processor, and of the algorithms it encodes - the behavior of the system cannot be explained without that. An outwardly similar system, built with the exact same "parts" (in the traditional sense, i.e. the same model of processor, motor etc), but programmed with different software, will behave entirely differently. This clearly proves that the software is a physical object that is part of the system and is necessary to fully account for its behavior.
Ah, seems like you folks arguing from two different Epistemologies? [1]
We already know that two different Epistemologies won't necessarily map perfectly to each other, though you might get close.
Might still be interesting to compare notes on what you can and can't predict/achieve with each!
[1] Edit : I can't quite lay my finger on which epistemologies exactly. Tsimionescu is using strong Empirical arguments, while mjburgess is inspired on Searle, which is pretty apt here, of course!
The only thing that can accurately simulate a process or system is the real process or system. Any simulation that perfectly simulates something becomes that something. Everything else contains simplifications and approximations and is an imperfect simulation.
Fire is the result of the intrinsic reactivity of some chemicals like fuels and oxidizers that allows them to react and generate heat. A simulation of fire that doesn't generate heat is missing a big part of the real thing, it's very simplified. Compared to real fire, a simulation is closer to a fire emoji, both just depictions of a fire. A fire isn't the process of calculating inside a computer what happens, it's molecules reacting a certain way, in a well understood and predictable process. But if your simulation is accurate and does generate heat then it can burn down a building by extending the simulation into the real world with a non-simulated fire.
Consciousness is an emergent property from putting together a lot of neurons, synapses, chemical and physical processes. So you can't analyze the parts to simulate the end result. You cannot look at the electronic neuron and conclude a brain accurately made of them won't generate consciousness. It might generate something even bigger, or nothing.
And in a very interesting twist of the mind, if an accurate simulation of a fire can extend in the real world as a real fire, then why wouldn't an accurate simulation of a consciousness extent in the real world as a real consciousness?
>A "simulation" is, by definition, simply an accounting game by which a series of descriptive statements can be derived from some others -- which necessarily, lacks the causal relations of what is being described.
This notion of causality is interesting. When a human claims that he is conscious, there a causal chain from the fact that they are conscious to their claiming so. When a neuron-level simulation of a human claims it is conscious, there must be a similar causal chain, with a similar fact at its origin.
There we go again. You claim that thinking is a biological process by definition, and use your definition to "prove" that software cannot be thinking. What if instead of software simulation of thinking we would have an actual software thinking? Your point would be to disregard it, not based on behaviour, but based on whatever your idea of "propper hardware for thinking" is. Pure troll and sophist, that Searle
> while he accuses others of supporting cartesian dualism when they think the brain and the mind can be separated, that you can "run" the mind on a different substrate
His views are perfectly consistent with non-dualism and if you think his views are muddy, that doesn't mean they are (they are definitively not muddy, per a large consensus). For the record, I am a substance dualist, and his arguments against dualism are pretty interesting, precisely because he argues that you can build something that functions in a different way than symbol manipulation while still doing something that looks like symbol manipulation (but also has this special property called consciousness, kind of like our brains).
Is this true? I don't know (I, of course, would argue "no"), but it does seem at least somewhat plausible and there's no obvious counter-argument.
I don't see how his views can be made sense of without dualism. He believed very much in this concept of qualia as some special property, and in the logical coherence of the concept of p-zombies, beings that would exactly like a conscious being but without having qualia. This simply makes no sense unless you believe that consciousness is a non-physical property, one that the physical world acts upon but which can't itself act back upon it (as otherwise, there would obviously have to be some kind of meaningful physical difference between the being that possesses it and the being that doesn't).
> This simply makes no sense unless you believe that consciousness is a non-physical property
It does make sense, and there's work being done on this front, (Penrose & Hameroff's Orch OR comes to mind). We obviously don't know exactly what such a mechanism would look like, but the theory itself is not inconsistent. Also, there's all kinds of p-zombies, so we likely need some specificity here.
Maybe it's because it's not trendy to believe in woowoo such as spirits and non-physical things, it's very common for dualists to accuse others of the same...
It's quite sad that people don't take the idea of consciousness being fundamental more seriously, given that's the only thing people actually deal with 100% of the time.
As for Searle, I think his argument is basically an appeal to common-sensical thinking, instead of anything based on common assumptions and logic. As an outsider, it feels very much that modern day philosophy is follows some kind of social media influencer logic, where you get respect for putting forward arguments that people agree with, instead of arguments that are non-intuitive yet rigorous and make people rethink their priors.
I mean, even today, here, you'd get similar arguments about "AI can never think because {reason that applies to humans as well}"... I suspect it's almost ingrained to the human psyche to feel this way.
> No, his argument is that consciousness can't be instantiated purely in software, that it requires specialized hardware. Language is irrelevant, it was only an example.
It's by no means irrelevant- the syntax vs. semantics distinction at the core of his argument makes little sense if we leave out language: https://plato.stanford.edu/entries/chinese-room/#SyntSema
Side note: while the Chinese Room put him on the map, he had as much to say about Philosophy of Language as he did of Mind. It was of more than passing interest to him.
> Instead, he believes that you could create a machine consciousness by building a brain of electronic neurons, with condensers for every biological dendrite, or whatever the right electric circuit you'd pick. He believed that this is somehow different than a simulation, with no clear reason whatsoever as to why.
I've never heard him say any such thing, nor read any word he's written attesting to this belief. If you have a source then by all means provide it.
I have, however, heard him say the following:
1. The structure and arrangement of neurons in the human nervous system creates consciousness.
2. The exact causal mechanism for this is phenomenon is unknown.
3. If we were to engineer a set of circumstances such that the causal mechanism for consciousness (whatever it may be) were present, we would have to conclude that the resulting entity- be it biological, mechanical, etc., is conscious.
He didn't have anything definitive to say about the causal mechanism of consciousness, and indeed he didn't see that as his job. That was to be an exercise left to the neuroscientists, or in his preferred terminology, "brain stabbers." He was confident only in his assertion that it couldn't be caused by mere symbol manipulation.
> it is in fact obvious he held dualistic notions where there is something obviously special about the mind-brain interaction that is not purely computational.
He believed that consciousness is an emergent state of the brain, much like an ice cube is just water in a state of frozenness. He explains why this isn't just warmed over property dualism:
https://faculty.wcas.northwestern.edu/paller/dialogue/proper...
> It's by no means irrelevant- the syntax vs. semantics distinction at the core of his argument makes little sense if we leave out language: https://plato.stanford.edu/entries/chinese-room/#SyntSema
The Chinese room is an argument caked in notions of language, but it is in fact about consciousness more broadly. Syntax and semantics are not merely linguistic concepts, though they originate in that area. And while Searle may have been interested in language as well, that is not what this particular argument is mainly about (the title of the article is Minds, Brains, and Programs - the first hint that it's not about language).
> I've never heard him say any such thing, nor read any word he's written attesting to this belief. If you have a source then by all means provide it.
He said both things in the paper that introduced the Chinese room concept, as an answer to the potential rebuttals.
Here is a quote about the brain that would be run in software:
> 3. The Brain Simulator reply (MIT and Berkley)
> [...] The problem with the brain simulator is that it is simulating the wrong things about the brain. As long as it simulates only the formal structure of the sequence of neuron firings at the synapses, it won't have simulated what matters about the brain, namely its causal properties, its ability to produce intentional states. And that the formal properties are not sufficient for the causal properties is shown by the water pipe example: we can have all the formal properties carved off from the relevant neurobiological causal properties.
And here is the bit about creating a real electrical brain, that he considers could be conscious:
> "Yes, but could an artifact, a man-made machine, think?"
> Assuming it is possible to produce artificially a machine with a nervous system, neurons with axons and dendrites, and all the rest of it, sufficiently like ours, again the answer to the question seems to be obviously, yes. If you can exactly duplicate the causes, you could duplicate the effects. And indeed it might be possible to produce consciousness, intentionality, and all the rest of it using some other sorts of chemical principles than those that human beings use.
> He believed that consciousness is an emergent state of the brain, much like an ice cube is just water in a state of frozenness. He explains why this isn't just warmed over property dualism: https://faculty.wcas.northwestern.edu/paller/dialogue/proper...
I don't find this paper convincing. He admits at every step that materialism makes more sense, and then he asserts that still, consciousness is not ontologically the same thing as the neurobiological states/phenomena that create it. He admits that usually being causally reducible means being ontologically reducible as well, but he claims this is not necessarily the case, without giving any other example or explanation as to what justifies this distinction. I am simply not convinced.
> The Chinese room is an argument caked in notions of language, but it is in fact about consciousness more broadly.
At this point I'm pretty sure we've had a misunderstanding. When I referred to "language" in my original post, you seem to have construed this as a reference to the Chinese language in the thought experiment. On the contrary, I was referring to software specifically, in the sense that a computer program is definitionally a sequence of logical propositions. In other words, a speech act.
> [...] The problem with the brain simulator is that it is simulating the wrong things about the brain.
This quote is weird and a bit unfortunate. It seems to suggest an opening: the brain simulator doesn't work because it simulates the "wrong things," but maybe a program that simulates the "right things" could be conscious. Out of context, you could easily reach that conclusion, and I suspect that if he could rewrite that part of the paper he probably would, because the rest of the paper is full of blanket denials that any simulation would be sufficient. Like this one: >The idea that computer simulations could be the real thing ought to have seemed suspicious in the first place because the computer isn't confined to simulating mental operations, by any means. No one supposes that computer simulations of a five-alarm fire will burn the neighborhood down or that a computer simulation of a rainstorm will leave us all drenched. Why on earth would anyone suppose that a computer simulation of understanding actually understood anything? It is sometimes said that it would be frightfully hard to get computers to feel pain or fall in love, but love and pain are neither harder nor easier than cognition or anything else. For simulation, all you need is the right input and output and a program in the middle that transforms the former into the latter. That is all the computer has for anything it does. To confuse simulation with duplication is the same mistake, whether it is pain, love, cognition, fires, or rainstorms.
Regarding the electrical brain:
> Assuming it is possible to produce artificially a machine with a nervous system, neurons with axons and dendrites, and all the rest of it, sufficiently like ours, again the answer to the question seems to be obviously, yes. If you can exactly duplicate the causes, you could duplicate the effects. And indeed it might be possible to produce consciousness, intentionality, and all the rest of it using some other sorts of chemical principles than those that human beings use.
Right, so he describes one example of an "electrical brain" that seems like it'd satisfy the conditions for consciousness, while clearly remaining open to the possibility that a different kind of artificial (non-electrical) brain might also be conscious. I'll assume you're using this quote to support your previous statement:
> Instead, he believes that you could create a machine consciousness by building a brain of electronic neurons, with condensers for every biological dendrite, or whatever the right electric circuit you'd pick. He believed that this is somehow different than a simulation, with no clear reason whatsoever as to why.
I think it's fairly obvious why this is different from a simulation. If you build a system that reproduces the consciousness-causing mechanism of neurons, then... it will cause consciousness. Not simulated consciousness, but the real deal. If you build a robot that can reproduce the ignition-causing mechanism of a match striking a tinderbox, then it will start a real fire, not a simulated one. You seem to think that Searle owes us an explanation for this. Why? How are simulations even relevant to the topic?
> I don't find this paper convincing.
The title of the paper is "Why I Am Not a Property Dualist." Its purpose is to explain why he's not a property dualist. Arguments against materialism are made in brief.
> He admits at every step that materialism makes more sense
Did we read the same paper?
> He admits that usually being causally reducible means being ontologically reducible as well,
Wrong, but irrelevant
> but he claims this is not necessarily the case, without giving any other example or explanation as to what justifies this distinction.
Examples and explanations are easy to provide, because there are several:
> But in the case of consciousness, causal reducibility does not lead to ontological reducibility. From the fact that consciousness is entirely accounted for causally by neuron firings, for example, it does not follow that consciousness is nothing but neuron firings. Why not? What is the difference between consciousness and other phenomena that undergo an ontological reduction on the basis of a causal reduction, phenomena such as color and solidity? The difference is that consciousness has a first person ontology; that is, it only exists as experienced by some human or animal, and therefore, it cannot be reduced to something that has a third person ontology, something that exists independently of experiences. It is as simple as that.
First-person vs. third-person ontologies are the key, whether you buy them or not. Consciousness is the only possible example of a first-person ontology, because it's the only one we know of
> “Consciousness” does not name a distinct, separate phenomenon, something over and above its neurobiological base, rather it names a state that the neurobiological system can be in. Just as the shape of the piston and the solidity of the cylinder block are not something over and above the molecular phenomena, but are rather states of the system of molecules, so the consciousness of the brain is not something over and above the neuronal phenomena, but rather a state that the neuronal system is in.
I could paste a bunch more examples of this, but the key takeaway is that consciousness is a state, not a property.
> On the contrary, I was referring to software specifically, in the sense that a computer program is definitionally a sequence of logical propositions. In other words, a speech act.
I think this muddies the water unnecessarily. Computation is not language, even though we typically write software in so called programming languages. But the computation itself is something different from the linguistic-like description of software. The computation is the set of states, and the relationships between them, that a computer goes through.
> > He admits at every step that materialism makes more sense
> Did we read the same paper?
I should have been clearer - I meant that he admits that materialism makes more sense than idealism or property dualism, but I realize that this comes off as suggesting it makes more sense than his own position, which of course he does not.
> > He admits that usually being causally reducible means being ontologically reducible as well,
> Wrong, but irrelevant
Both you and he seem to find a single example of a phenomenon that is causally reducible to some constituent part, but that is not ontological reducible to that constitutent part - consciousness (he would add intentionality, I think, given the introduction, but it's not clear to me this is even a meaningfully separatable concept from consciousness). And you both claim that this is the case because of this special feature of "first person ontology", which is a different thing than "third person ontology" - which seems to me to simply be dualism by another name.
I think it's entirely possible to reject the notion of a meaningful first person ontology completely. It's very possible that the appearance of a first person narrative that we experience is a retroactive illusion we create that uses our models of how other people function on ourselves. That is, we are simple computers that manipulate symbols in our brains, that generate memories of their recent state as being a "conscious experience", which is just what we invented as a model of why other animals and physical phenomena more broadly behave the way they do (since we intuitively assign emotions and intentions to things like clouds and fires and mountains, to explain their behavior).
Hardware and software are of course equivalent, as every computer science (but not every philosopher) knows.
D.R. Hofstadter posited that we can extract/separate the software from the hardware it runs on (the program-brain dichotomy), whereas Searle believed that these were not two layers but consciousness was in effect a property of the hardware. And from that, as you say, follows that you may re-create the property if your replica hardware is close enough to the real brain.
IMHO, philosophers should be rated by the debate their ideas create, and by that, Searle was part of the top group.
>> “His argument is much narrower: consciousness can't be instantiated purely in language.”
> “No, his argument is that consciousness can't be instantiated purely in software…“
The confusion is very interesting to me, maybe because I’m a complete neophyte on the subject. That said, I’ve often wondered if consciousness is necessarily _embodied_ or emerged from pure presence into language & body. Maybe the confusion is intentional?
> He doesn't argue that consciousness can only emerge from biological neurons. His argument is much narrower: consciousness can't be instantiated purely in language.
I haven't read loads of his work directly, but this quote from him would seem to contradict your claim:
> I demonstrated years ago with the so-called Chinese Room Argument that the implementation of the computer program is not by itself sufficient for consciousness or intentionality (Searle 1980). Computation is defined purely formally or syntactically, whereas minds have actual mental or semantic contents, and we cannot get from syntactical to the semantic just by having the syntactical operations and nothing else. [1]
Unfortunately, it doesn't seem to me to have proven anything; it's merely made an accurate analogy for how a computer works. So, if "semantics" and "understanding" can live in <processor, program, state> tuples, then the Chinese Room as a system can have semantics and understanding, as can computers; and if "semantics" and "understanding" cannot live in <processor, program, state> tuples, then neither the Chinese Room nor computers can have understanding.
[1] https://plato.stanford.edu/entries/chinese-room/
Sorry, I've reread this a few times and I'm not sure which part of Searle's argument you think I mischaracterized. Could you clarify? For emphasis:
> "consciousness can't be instantiated purely in language" (mine)
> "we cannot get from syntactical to the semantic just by having the syntactical operations and nothing else" (Searle)
I get that the mapping isn't 1:1 but if you think the loss of precision is significant, I'd like to know where.
> Unfortunately, it doesn't seem to me to have proven anything; it's merely made an accurate analogy for how a computer works. So, if "semantics" and "understanding" can live in <processor, program, state> tuples, then the Chinese Room as a system can have semantics and understanding, as can computers; and if "semantics" and "understanding" cannot live in <processor, program, state> tuples, then neither the Chinese Room nor computers can have understanding.
There's a lot of debate on this point elsewhere in the thread, but Searle's response to this particular objection is here: https://plato.stanford.edu/entries/chinese-room/#SystRepl
> I get that the mapping isn't 1:1 but if you think the loss of precision is significant, I'd like to know where.
I'm by far an expert in this; my knowledge of the syntax / semantics distinction primarily comes from discussions w/ ChatGPT (and a bit from my friend who is a Catholic priest, who had some training in philosophy).
But, the quote says "purely formally or syntactically". My understanding is that Searle (probably thinking about the Prolog / GPS-type attempts at logical artificial intelligence prevalent in the 70's and 80's) is thinking of AI in terms of pushing symbols around. So, in this sense, the adder circuit in a processor doesn't semantically add numbers; it only syntactically adds numbers.
When you said, "consciousness can't be instantiated purely in language", I took you to mean human language; it seems to leave the door open to consciousness (and thus semantics) being instantiated by a computer program in some other way. Whereas, the quote from Searle very clearly says, "...the computer program by itself is not sufficient for consciousness..." (emphasis mine) -- seeming to rule out any possible computer program, not just those that work at the language level.
> There's a lot of debate on this point elsewhere in the thread, but Searle's response to this particular objection is here:
I mean, yeah, I read that. Let me quote the relevant part for those reading along:
> Searle’s response to the Systems Reply is simple: in principle, he could internalize the entire system, memorizing all the instructions and the database, and doing all the calculations in his head. He could then leave the room and wander outdoors, perhaps even conversing in Chinese. But he still would have no way to attach “any meaning to the formal symbols”. The man would now be the entire system, yet he still would not understand Chinese. For example, he would not know the meaning of the Chinese word for hamburger. He still cannot get semantics from syntax.
I mean, it sounds to me like Searle didn't understand the "Systems Response" argument; because as the end of that section says, he's just moved the program and state part of the <procesor, program, state> tuple out of the room and into his head. The fact that the processor (Searle's own conscious mind) is now storing the program and the state in his own memory rather than externally doesn't fundamentally change the argument: If that tuple can "understand" things, then computers can "understand" things; and if that tuple can't "understand" things, then computers can't "understand" things.
One must, of course, be humble when saying of a world-renowned expert, "He didn't understand the objection to his argument". But was Searle himself a programmer? Did he ever take a hard drive out of one laptop, pop it into another, and have the experience of the same familiar environment? Did he ever build an adder circuit, a simple register system, and a simple working computer out of logic gates, and see it suddenly come to life and execute programs? If he had, I can't help but think his intuitions regarding the syntax / semantic distinction would be different.
EDIT: I mean, I'm personally a Christian, and do believe in the existence of eternal souls (though I'm not sure exactly what those look like). But I'm one of those annoying people who will quibble with an argument whose conclusion I agree with (or to which I am sympathetic), because I don't think it's actually a good argument.
Ah ok, gotcha.
> When you said, "consciousness can't be instantiated purely in language", I took you to mean human language
No, I definitely meant the statement to apply to any kind of language, but it seems clear that I sacrificed clarity for the sake of brevity. You're not the only one who read it that way, but yeah, we're in agreement on the substance.
I think I'm still a bit confused... so, in the languages which cannot produce understanding and consciousness, you mean to include "machine language"? (And thus, any computer language which can be compiled to machine language?)
On your interpretation, are there any sorts of computation that Searle believes would potentially allow consciousness?
ETA: The other issue I have is with this whole idea is that "understanding requires semantics, and semantics requires consciousness". If you want to say that LLMs don't "understand" in that sense, because they're not conscious, I'm fine as long as you limit it to technical philosophical jargon. In plain English, in a practical sense, it's obvious to me that LLMs understand quite a lot -- at least, I haven't found a better word to describe LLMs' relationship with concepts.
> I think I'm still a bit confused... so, in the languages which cannot produce understanding and consciousness, you mean to include "machine language"? (And thus, any computer language which can be compiled to machine language?)
It's... a little more complicated but basically yes. Language, by its nature, is indexical: it has no meaning without someone to observe it and ascribe meaning to it. Consciousness, on the other hand, requires no observer beyond the person experiencing it. If you have it, it's as real and undeniable as a rock or a tree or a mountain.
> On your interpretation, are there any sorts of computation that Searle believes would potentially allow consciousness?
I'm pretty sure (but not 100%) that the answer is "no"
> ETA: The other issue I have is with this whole idea is that "understanding requires semantics, and semantics requires consciousness". If you want to say that LLMs don't "understand" in that sense, because they're not conscious, I'm fine as long as you limit it to technical philosophical jargon.
Sure, if you want to think of it that way. If you accept the premise that LLMs aren't conscious, then you can consign the whole discussion to the "technical philosophical jargon" heap, forget about it, and happily go about your day. On the other hand, if you think they might be conscious, and consider the possibility that we're inflicting immeasurable suffering on sapient being that would rightly be treated with kindness (and afforded some measure of rights), then we're no longer debating how many angels can dance on the head of a pin. That's a big, big "if" though.
I also remember a course from him decades ago, but I'm not sure this memorial post is the place for my take. Instead, let me attempt to re-tell a joke I heard back then...
John Searle and George Lakoff walk into a bar.
Searle exclaims, "What do you know!"
The bar replies sardonically, "You wouldn't believe it."
Lakoff sighs, "This is 0.8 drinks with Lotfi Zadeh..."
> If you think you disagree with him (as I once did), please consider the possibility that you've only been exposed to an ersatz characterization of his argument.
My first exposure was a video of Searle himself explaining the Chinese room argument.
It came across as a claim that a whole can never be more than its parts. It made as much sense as claiming that a car cannot possibly drive, as it consists of parts that separately cannot drive.
This https://youtu.be/6tzjcnPsZ_w maybe? It's Searle explaining it.
This is true of many philosophers. Once you read the source materials, you realize the depth of the material.
I have yet to see anything to convince me he was not being a troll and making that argument deliberately so jumbled up in bad faith.
First of all, what purpose the person in the room serves, but to confuse and misdirect? Replace that person with a machine, and argument looses any impact.
His response to system reply is extremely egregious. How can that have been made in good faith? (to paraphrase: "the whole system understands chinese" — "no, a person can run the system in their head, it means the system cannot understand anything that the person running it does not") What kind of nonsense response is that? Either the guy was LV80 troll, or I dunno..
I learned about Searle's death a few weeks ago, from this article: https://www.colinmcginn.net/john-searle/
It includes a letter that starts:
I'm surprised to see the NYT obituary published nearly a month after his death. I would have thought he'd be included in their stack of pre-written obituaries, meaning it could be updated and published within a day or two.Well, that was incredibly depressing. Maybe I can lighten things with a funny (to me) anecdote.
There are many people who know a lot about a little. There are also those who know a little about a lot. Searle was one of those rare people who knew a lot about a lot. Many a cocky undergraduate sauntered into his classroom thinking they'd come prepared with some new fact that he hadn't yet heard, some new line of attack he hadn't prepared for. Nearly always, they were disappointed.
But you know what he knew absolutely nothing about? Chinese. When it came time to deliver his lecture on the Chinese Room, he'd reach up and draw some incomprehensible mess of squigglies and say "suppose this is an actual Chinese character." Seriously. After decades of teaching about this thought experiment, for which he'd become famous (infamous?), he hadn't bothered to teach himself even a single character to use for illustration purposes.
Anyway, I thought it was funny. My heart goes out to Jennifer Hudin, who was indispensable, and all who were close to him.
I found the delay puzzling too. But the NYT obit does link to https://www.colinmcginn.net/john-searle/ near the end.
The Times in the UK publishes obituaries of very well-known public figures within a day or two. Notable but lesser known people (such as Searle) await a quiet day and it can take as long as six months. Space is the constraint, not the availability of the obituary. I guess the NYT is the same.
Wow, what a terrible way to be treated. Thank you for the quote.
There's a lot more to this y'all aren't seeing. Difficult family situation you shouldn't judge.
John Searle is one of those thinkers I disagree with, yet his ideas were fruitful — providing plenty of fuel for discussion. In particular, much of Daniel Dennett’s work begins with rebuttals of Searle’s claims, showing that they are inconsistent or meaningless. As in a story by Stanisław Lem — we all know there are no dragons, but it’s all about the beauty of the proofs.
The same goes for "What Is It Like to Be a Bat?" by Thomas Nagel — one of the most cited essays in the philosophy of mind. I had heard numerous references to it and finally expected to read an insightful masterpiece. Yet it turned out to be slightly tautological: that to experience, you need to be. Personally, I think the word be is a philosopher’s snake oil, or a "lockpick word" — it can be used anywhere, but remains fuzzy even in its intended use; vide E-Prime, an attempt to write English without "be": https://en.wikipedia.org/wiki/E-Prime.
Oh, I've always wanted to debate him about the chinese room. I disagree with him, passionately. And that's the most fun debate to have. Especially when it's someone who is actually really skilled and knowledgeable and nuanced!
Maybe I should look up some of my other heroes and heretics while I have the chance. I mean, you don't need to cold e-mail them a challenge. Sometimes they're already known to be at events and such, after all!
Searle has written responses to dozens of replies to the Chinese Room. It's likely that you can find his rebuttals to your objection in the Stanford Encyclopedia of Philosophy's entry on the Chinese Room, or deeper in a source in the bibliography. Is your rebuttal listed here?
https://plato.stanford.edu/entries/chinese-room
> In response to this, Searle argues that it makes no difference. He suggests a variation on the brain simulator scenario: suppose that in the room the man has a huge set of valves and water pipes, in the same arrangement as the neurons in a native Chinese speaker’s brain. The program now tells the man which valves to open in response to input. Searle claims that it is obvious that there would be no understanding of Chinese.
I mean, I guess all arguments eventually boil down to something which is "obvious" to one person to mean A, and "obvious" to me to mean B.
Same. I feel the Chinese room argument is a nice thing to clarify thinking.
Two systems, one feels intuitively like it understands, one doesn’t. But the two systems are functionally identical.
Therefore either my concept of “understanding” is broken, my intuition is wrong, or the concept as a whole is not useful at the edges.
I think it’s the last one. If a bunch of valves can’t understand but a bunch of chemicals and electrical signals can if it’s in someone’s head then I am simply applying “does it seem like biology” as part of the definition and can therefore ignore it entirely when considering machines or programs.
Searle seems to just go the other way and I don’t under Why.
First point: if you imagine that the brain is doing something like collapsing the quantum wavefunction, wouldn't you say that this is a functionally relevant difference in addition to an ontologically relevant difference? It's not clear that the characteristic feature of the brain is only to compute in the classical sense. "Understanding," if it leverages quantum mechanics, might also create a guarantee of being here and now (computers and programs have no such guarantees). This is conjecture, but it's meant to stimulate imagination. What we need to get away from is the fallacy that a causal reduction of mental states to "electrical phenomena" means that any set of causes (or any substrate) will do. I don't think that follows.
Second: the philosophically relevant point is that when you gloss over mental states and only point to certain functions (like producing text), you can't even really claim to have fully accounted for what the brain does in your AI. Even if the physical world the brain occupies is practically simulatable, passing a certain speech test in limited contexts doesn't really give you a strong claim to consciousness and understanding if you don't have further guarantees that you're simulating the right aspects of the brain properly. AI, as far as I can tell, doesn't TRY to account for mental states. That's partially why it will keep failing in some critical tasks (in addition to being massively inefficient relative to the brain).
Exactly. Refuting the premise of the Chinese Room is usually a sign of somebody not even willing to entertain the thought experiment. Refuting Searle's conclusion is where interesting philosophical discussions can be had.
Personally, I'd say that there is a Chinese speaking mind in the room (albeit implemented on a most unusual substrate).
There are two distinct counter-arguments to this way of debunking the Chinese room experiment, not in any specific order.
First, it is tempting to assume that a bunch of chemicals is the territory, that it somehow gives rise to consciousness, yet that claim is neither substantiated nor even scientific. It is a philosophical view called “monistic materialism” (or sometimes “naive materialism”), and perhaps the main reason this view is popular currently is that people uncritically adopt it following learning natural scientific fields, as if they made some sort of ground truth statements about the underlying reality.
The key to remember is that this is not a valid claim in the scope of natural sciences; this claim belongs to the larger philosophy (the branch often called metaphysics). It is not a useless claim, but within the framework of natural sciences it’s unfalsifiable and not even wrong. Logically, from scientific method’s standpoint, even if it was the other way around—something like in monistic idealism, where perception of time-space and material world is the interface to (map of) conscious landscape, which was the territory and the cause—you would have no way of proving or disproving this, just like you cannot prove or disprove the claim that consciousness arises from chemical processes. (E.g., if somebody incapacitates some part of you involved in cognition, and your feelings or ability to understand would change as a result, it’s pretty transparently an interaction between your mind and theirs, just with some extra steps, etc.)
The common alternatives to monistic materialism include Cartesian dualism (some of us know it from church) and monistic idealism (cf. Kant). The latter strikes me as the more elegant of the bunch, as it grants objective existence to the least amount of arbitrary entities compared to the other two.
It’s not to say that there’s one truly correct map, but just to warn against mistakenly trying to make a statement about objective truth, actual nature of reality, with scientific method as cover. Natural sciences do not make claims of truth or objective reality, they make experimentally falsifiable predictions and build flawed models that aid in creating more experimentally falsifiable predictions.
Second, what scientific method tries to build is a complete, formally correct and provable model of reality, there are some arguments that such model is impossible to create in principle. I.e., there will be some parts of the territory that are not covered by the map, and we might not know what those parts are, because this territory is not directly accessible to us: unlike a landmass we can explore in person, in this case all we have is maps, the perception of reality supplied by our mind, and said mind is, self-referentially, part of the very territory we are trying to model.
Therefore, it doesn’t strike me as a contradiction that a bunch of valves don’t understand yet we do. A bunch of valves, like an LLM, could mostly successfully mimic human responses, but the fact that this system mimics human responses is not an indication of it feeling and understanding like a human does, it’s simply evidence that it works as designed. There can be a very different territory that causes similar measurable human responses to arise in an actual human. That territory, unlike the valves, may not be fully measurable, and it can cause other effects that are not measurable (like feeling or understanding). Depending on the philosophical view you take, manipulating valves may not even be a viable way of achieving a system that understands; it has not been shown that biological equivalent of valves is what causes understanding, all we have shown is that those entities measurably change at the same time with some measurable behavior, which isn’t a causative relationship.
It's not mostly mimicking, it's exactly identical. That was always the key point. Indistinguishable from the outside, one thing understands and the other doesn't.
I feel like I could make the same arguments about the chinese room except my definition of "understanding" hinges on whether there's a tin of beans in the room or not. You can't tell from the outside, but that's the difference. Both cases with a person inside answering questions act identically and you can never design a test to tell which room has the tin of beans in.
Now you might then say "I don't care if there's a tin of beans in there, it doesn't matter or make any sort of difference for anything I want to do", in which case I'd totally agree with you.
> just like you cannot prove or disprove the claim that consciousness arises from chemical processes.
Like understanding, I haven't seen a particularly useful definition of consciousness that works around the edges. Without that, talking of a claim like this is pointless.
I'd be fine if Searle just very simply said "we have a non-material soul and that's why we understand. Anything doing the exact same job but without a soul isn't understanding because understanding is limited entirely to things with souls in my definition".
> A bunch of valves, like an LLM, could mostly successfully mimic human responses,
The argument is not "mostly successfully", it's identically responding. The entire point of the chinese room is that from the outside the two things are impossible to distinguish between.
I would encourage deeply digging into the intuition that brain states and computer states are the same. Start with what you know, and then work backwards and see whether you still think they aren’t different. For example, we have an intuitive understanding of what kinds of flavors (for us) are delicious versus not. Or what kinds of sounds are pleasant versus not. Etc. If I close my eyes, I can see the color purple. I know that Nutella is delicious, and I can imagine its flavor at will. I share Searle’s intuition that the universe would be a strange place if these feelings of understanding (and pleasantness!) were simply functions not of physical states but of abstract program states. Keep in mind—what counts as a bit is simply a matter of convention. In one computer system, it could be a minute difference in voltage in a transistor. In another, it could be the presence of one element versus another. In another, it could be whether a chamber contains water or not. In another, it could be markings on a page. On and on. On the strong AI thesis, any system that runs steps in this program would not just produce functionally equivalent output to brains, but they would be forced to have mental states too, like imagining the taste of Nutella. To me, it's implausible that symbolic states FORCE mental states, or put another way that mental states are non-physical (we think about how states like pain, euphoria, drunkenness, etc, are physically modulated through drugs..you'd have to modify this to say that they're really modifying symbolic states somehow). Either the Chinese Room is missing something, our understanding of physical reality is incomplete, OR that you have to bite the bullet that the universe creates mental states when systems implement the right program—but then you’re left with the puzzle of how it is that there is a tie between the physical world and the abstract world of symbols (how can causing a mark on a page cause mental states).
So what’s the physical cause for consciousness and understanding that is not computable? If for example you took the hypothesis that “consciousness is a sequence of microtubule-orchestrated collapses of the quantum wavefunction” [1], then you can see a series of physical requirements for consciousness and understanding that forces all conscious beings onto: 1) roughly the same clock (because consciousness shares a cause), and 2) the same reality (because consciousness causes wavefunction collapses). That’s something you could not do merely by simulating certain brain processes in a closed system.
1) Not saying this is correct, but it invites one to imagine that consciousness could have physical requirements that play in some of the oddities of the (shared) quantum world. https://x.com/StuartHameroff/status/1977419279801954744
All you have to do is train an LLM on the collected works and letters of John Searle; you could then pass your arguments along to the machine and out would come John Searle's thoughtful response...
Something that would resemble 'John Searle's thoughtful response'...
I'll posit that the distinction does not matter: the whole Chinese Room line of discourse has been counterproductive to putting in actual work.
I don't think John Searle would agree.
You're absolutely right!
Oh, bad timing. AI is currently in a remarkable state, where it passes the Turing test but is still not fully AGI. It's very close to the Chinese Room, which I had always dismissed as misleading. It's a great opportunity to investigate a former pure thought experiment. He'd have loved to see where it went.
The Turing Test has not been meaningfully passed. Instead we redefined the test to make it passable. In Turing's original concept the competent investigator and participants were all actively expected to collude against the machine. The entire point is that even with collusion, the machine would be able to do the same, and to pass. Instead modern takes have paired incompetent investigators alongside participants colluding with the machine, probably in an effort to be part 'of something historic'.
In "both" (probably more, referencing the two most high profile - Eugene and the LLMs) successes, the interrogators consistently asked pointless questions that had no meaningful chance of providing compelling information - 'How's your day? Do you like psychology? etc' and the participants not only made no effort to make their humanity clear, but often were actively adversarial obviously intentionally answering illogically, inappropriately, or 'computery' to such simple questions. For instance here is dialog from a human in one of the tests:
----
[16:31:08] Judge: don't you thing the imitation game was more interesting before Turing got to it?
[16:32:03] Entity: I don't know. That was a long time ago.
[16:33:32] Judge: so you need to guess if I am male or female
[16:34:21] Entity: you have to be male or female
[16:34:34] Judge: or computer
----
And the tests are typically time constrained by woefully poor typing skills (is this the new normal in the smartphone gen?) to the point that you tend to get anywhere from 1-5 interactions of just several words each. The above snip was a complete interaction, so you get 2 responses from a human trying to trick the judge into deciding he's a computer. And obviously a judge determining that the above was probably a computer says absolutely nothing about the quality of responses from the computer - instead it's some weird anti-Turing Test where humans successfully act like a [bad] computer, ruining the entire point of the test.
The problem with any metric for something is that it often ends up being gamed to be beaten, and this is a perfect example of that. I suspect in a true run of the Turing Test we're still nowhere even remotely close to passing it.
I don't doubt it that all of the formal Turning tests have been badly done. But I suspect that if you did one, at least one run will mis-judge an LLM. Maybe it's a low percentage, but that's vastly better than zero.
So I'd say we're at least "remotely close", which is sufficient for me to reconsider Searle.
> instead it's some weird anti-Turing Test where humans successfully act like a [bad] computer
This is ex-post-facto denial and cope. The Turing Test isn't a test between computers and the idealized human, it's a test between functional computers and functional humans. If the average human performs like the above, then well, I guess the logical conclusion is that computers are already better "humans (idealized)" than humans.
I'm generally against LLM recreations of dead people but AI John Searle could be pretty entertaining.
Indeed, Necromancy is ethically fraught
I'm reminded of how the AIs in Her created a replica of Alan Watts to help them wrestle with some major philosophical problems as they evolved.
> AI is currently in a remarkable state, where it passes the Turing test but is still not fully AGI.
Appealing to the Turing test suggests a misunderstanding of Searle's arguments. It doesn't matter how well computational methods can simulate the appearance of intelligence. What matters is whether we are dealing with intelligence. Since semantics/intentionality is what is most essential to intelligence, and computation as defined by computer science is a purely abstract syntactic process, it follows that intelligence is not essentially computational.
> It's very close to the Chinese Room, which I had always dismissed as misleading.
Why is it misleading? And how would LLMs change anything? Nothing essential has changed. All LLMs introduce is scale.
I came to say this, thank you for sparing me the effort.
From my experience with him, he'd heard (and had a response to) nearly any objection you could imagine. He might've had fun playing with LLMs, but I doubt he'd have found them philosophically interesting in any way.
"At least they don't have true consciousness, but only a simulated one", I tell myself calmly as I watch the nanobots devour the entirety of human civilization.
> Professor Searle concluded that psychological states could never be attributed to computer programs, and that it was wrong to compare the brain to hardware or the mind to software.
Gotta agree here. The brain is a chemical computer with a gazillion inputs that are stimulated in manifold ways by the world around it, and is constantly changing states while you are alive; a computer is a digital processor that works work with raw data, and tends to be entirely static when no processing is happening. The two are vastly different entities that are similar in only the most abstract ways.
Searle had an even stronger version of that belief, though: he believed that a full computational simulation of all of those gazillion inputs, being stimulated in all those manifold ways, would still not be conscious and not have a 'mind' in the human sense. The NYT obituary quotes him comparing a computer simulation of a building fire against the actual building going up in flames.
When I read that analogy, I found it inept. Fire is a well defined physical process. Understanding / cognition is not necessarily physical and certainly not well defined.
>Understanding / cognition is not necessarily physical and certainly not well defined.
Whooha! If it's not physical what is it? How does something that's not physical interact with the universe and how does the universe interact with it? Where does the energy come from and go? Why would that process not be a physical process like any other?
I'd say understanding and cognition are at this point fully explainable mechanistically. (I am very excited to live in a time where I was able to change my mind on this!)
Where we haven't made any headway on is on the connection between that and subjective experience/qualia. I feel like much of the (in my mind) strange conclusions of the Chinese Room are about that and not really about "pure" cognition.
Simulated fire would burn down simulated building
If everything is simulated then "simulated(x)" is a vacuous predicate & tells you nothing so you might as well throw it away & speak directly in terms of the objects instead of wrapping/prepending everything w/ "simulated".
"Simulated" is not a predicate, but a modality.
That's debatable, but it is also irrelevant, as the key to the argument here is that computation is by definition an abstract and strictly syntactic construct - one that has no objective reality vis-a-vis the physical devices we use to simulate computation and call "computers" - while semantics or intentionality are essential to human intelligence. And no amount of syntax can somehow magically transmute into semantics.
This makes no sense. You could equally make the statement that thought is by definition an abstract and strictly syntactic construct - one that has no objective reality. Neither statement is supported by anything.
There's also no "magic" involved in transmuting syntax into semantics, merely a subjective observer applying semantics to it.
> This makes no sense. You could equally make the statement that thought is by definition an abstract and strictly syntactic construct - one that has no objective reality.
No.
I could jam a yardstick into the ground and tell you that it's now a sundial calculating the time of day. Is this really, objectively true? Of course not. It's true to me, because I deem it so, but this is not a fact of the universe. If I drop dead, all meaning attributed to this yardstick is lost.
Now, thoughts. At the moment I'm visualizing a banana. This is objectively true: in my mind's eye, there it is. I'm not shuffling symbols around. I'm not pondering the abstract notion of bananas, I'm experiencing the concretion of one specific imaginary banana. There is no "depends on how you look at it." There's nothing to debate.
> There's also no "magic" involved in transmuting syntax into semantics, merely a subjective observer applying semantics to it.
There's no "magic" because this isn't a thing. You can't transmute syntax into semantics any more than you can transmute the knowledge of Algebra into the sensation of a cool breeze on a hot summer day. This is a category error.
> There is no "depends on how you look at it." There's nothing to debate.
None of what you wrote is remotely relevant to what I wrote.
> There's no "magic" because this isn't a thing. You can't transmute syntax into semantics any more than you can transmute the knowledge of Algebra into the sensation of a cool breeze on a hot summer day. This is a category error.
We "transmute" syntax into semantics every time we interpret a given syntax as having semantics.
There is no inherent semantics. Semantics is a function of the meaning we assign to a given syntax.
You claim it makes no sense, but don't give a good reason why it wouldn't.
> You could equally make the statement that thought is by definition an abstract and strictly syntactic construct - one that has no objective reality.
This is what makes no sense, as I am not merely posing arbitrary definitions, but identifying characteristic features of human intelligence. Do you deny semantics and intentionality are features of the human mind?
> There's also no "magic" involved in transmuting syntax into semantics, merely a subjective observer applying semantics to it.
I have no idea what this means. The point is that computation as we understand it in computer science is purely syntactic (this was also Searle's argument). Indeed, it is modeled on the mechanical operations human computers used to perform without understanding. This property is precisely what makes computation - thus understood - mechanizable. Because it is purely syntactic and an entirely abstract model, two things follow:
1. Computation is not an objectively real phenomenon that computers are performing. Rather, physical devices are used to simulate computation. Searle calls computation "observer relative". There is nothing special about electronics, as we can simulate computation using wooden gears that operate mechanically or water flow or whatever. But human intelligence is objectively real and exists concretely, and so it cannot be a matter of mere simulation or something merely abstract (it is incoherent and self-refuting to deny this for what should be obvious reasons).
2. Because intentionality and the capacity for semantics are features of human intelligence, and computation is purely syntactic, there is no room in computation for intelligence. It is an entirely wrong basis for understanding intelligence and in a categorical sense. It's like trying to find out what arrangement of LEGO bricks can produce the number π. Syntax has no "aboutness" as that is the province of intentionality and semantics. To deny this is to deny that human beings are intelligent, which would render the question of intelligence meaningless and frankly mystifying.
> Do you deny semantics and intentionality are features of the human mind?
I deny they are anything more than computation. And so your original argument was begging the question and so logically unsound.
> The point is that computation as we understand it in computer science is purely syntactic
Then the brain is also purely syntactic unless you can demonstrate that the brain carries out operations that exceeds the Turing computable, because unless that is the case the brain and a digital computer are computationally equivalent.
As long as your argument does not address this fundamental issue, you can talk about "aboutness" or whatever else you want all day long - it will have no relevance.
Do you believe that there are things that are not physical? Extraordinary claims require extraordinary evidence. And no, "science can't explain x hence metaphysical" is not a valid response.
But that acknowledgement would itself lend Searle's argument credence because much of the brain = computer thesis depends on a fundamental premise that both brains and digital computers realize computation under the same physical constraints; the "physical substrate" doesn't matter (and that there is necessarily nothing special about biophysical systems beyond computational or resource complexity) (the same thinking by the way, leads to arguments that an abacus and a computer are essentially "the same"—really at root these are all fallacies of unwarranted/extremist abstraction/reductionism)
The history of the brain computer equation idea is fascinating and incredibly shaky. Basically a couple of cyberneticists posed a brain = computer analogy back in the 50s with wildly little justification and everyone just ran with it anyway and very few people (Searle is one of those few) have actually challenged it.
Unless you can show an example of how we can compute something that is not Turing computable, there is no justification for the inverse, as the inverse would require something in the brain to be capable of interactions that can not be simulated. And we've no evidence to suggest either that the brain can do something not Turing computable or of the presence of something in the brain that can't be simulated.
Maybe consciousness is exactly like simulated fire. It does a lot inside the simulation, but is nothing on the outside.
> The history of the brain computer equation idea is fascinating and incredibly shaky. Basically a couple of cyberneticists posed a brain = computer analogy back in the 50s with wildly little justification and everyone just ran with it anyway and very few people (Searle is one of those few) have actually challenged it.
And something that often happens whenever some phenomenon falls under scientific investigation, like mechanical force or hydraulics or electricity or quantum mechanics or whatever.
Roger Penrose would be another.
Isn't that besides the point? The point is that something would actually burn down.
GP's point is that buring something down is by definition something that requires a specific physical process. It's not obvious that thinking is the same. So when someone says something like "just as a simulation of fire isn't the same as an actual fire (in a very important way!), a simulation of thinking isn't the same as actual thinking" they're arguing circularly, having already accepted their conclusion that both acts necessarily require a specific physical process. Daniel Dennett called this sort of argument an "intuition pump", which relies on a misleading but intuitive analogy to get you to accept an otherwise-difficult-to-prove conclusion.
To be fair to Searle, I don't think he advanced this as an agument, but more of an illustration of his belief that thinking was indeed a physical process specific to brains.
He explains it in the original paper¹ & says in no uncertain terms that he believes the brain is a machine & minds are implementable on machines. What he is actually arguing is that substrate independent digital computation will never be a sufficient explanation for conscious experience. He says that brains are proof that consciousness is physical & mechanical but not digital. Searle is not against the computationalist hypothesis of minds, he admits that there is nothing special about minds in terms of physical processes but he doesn't reduce everything to substrate independent digital computation & conclude that minds are just software running on brains. There are a bunch of subtle distinctions that people miss when they try to refute Searle's argument.
¹https://home.csulb.edu/~cwallis/382/readings/482/searle.mind...
Quick definitional help for anyone who clicks on your link: the term "intentionality" in this context has a specialized meaning. In reference to mental states, it's the property of being about something, as in, "Alice is thinking about Bob." It doesn't necessarily have anything to do with intent, per se.
https://plato.stanford.edu/entries/consciousness-intentional...
https://home.sandiego.edu/~baber/analytic/Lem1979.html
Unless human brains exceeds the Turing computable, they're still computationally equivalent, and we have no indication exceeding the Turing computable is even possible.
A Turing machine operates serially on a fixed set of instructions. A human brain operates in parallel on inputs that are constantly changing. The underlying mechanism is completely different. The human brain is far, far more than a mere computation device.
Efforts to reproduce a human brain in a computer are currently at the level of a cargo cult: we're simulating the mechanical operations, without a deep understanding of the underlying processes which are just as important. I'm not saying we won't get better at it, but so far we're nowhere near producing a brain in a computer.
Any Turing complete computational device is computationally equivalent to any other, irrespective of whether it carries out the computation serially or in parallel or on a fixed set of instructions or an infinite set.
Unless you can demonstrate that the human brain can compute a function - any function - that exceeds the Turing computable, there is no evidence to even suggest it is possible for a brain not to be computationally equivalent to a computer.
I'm not saying it's impossible. I'm saying it's highly improbable to ever come close to fruition, since if you want to be completely accurate in your computations, you'd have to model the physical effects of every single atom in the brain and how they affect each other, in realtime (physically, chemically, electrically, etc.). Not only would that be computationally expensive, but you'd also need to know the complete set of rules for doing so, which includes how things interact at the quantum level.
I think the statement above and yours both seem to ignore “Turing complete” systems, which would indicate that a computer is entirely capable of simulating the brain, perhaps not before the heat death of the universe, that’s yet to be proven and depends a lot on what the brain is really doing underneath in terms of crunching.
This depends on the assumption that all brain activity is the process of realizing computable functions. I'm not really aware of any strong philosophical or neurological positions that has established this beyond dispute. Not to resurrect vitalism or something but we'd first need to establish that biological systems are reducible to strictly physical systems. Even so, I think there's some reason to think that the highly complex social historical process of human development might complicate things a bit more than just brute force "simulate enough neurons". Worse, whose brain exactly do you simulate? We are all different. How do we determine which minute differences in neural architecture matter?
> we'd first need to establish that biological systems are reducible to strictly physical systems.
Or even more fundamentally, that physics captures all physical phenomena, which it doesn't. The methods of physics intentionally ignore certain aspects of reality and focus on quantifiable and structural aspects while also drawing on layers of abstractions where it is easy to mistakenly attribute features of these abstractions to reality.
>also drawing on layers of abstractions where it is easy to mistakenly attribute features of these abstractions to reality.
Ok - I get that bit. I have always thought that physics is a description of the universe as observed and of course the description could be misleading in some way.
>the methods of physics intentionally ignore certain aspects of reality and focus on quantifiable and structural aspects
Can you share the aspects of reality that physics ignores? What parts of reality are unquantifiable and not structural?
> Can you share the aspects of reality that physics ignores?
Here's an article you might enjoy [0].
[0] http://edwardfeser.blogspot.com/2022/05/the-hollow-universe-...
Not all of physics is relevant to a brain simulation. For example, humans appear equally conscious in free fall or in an accelerating vehicle, so a simulation can probably safely ignore the effects of gravity without affecting the outcome. We also know that at body temperature (so about 310K) there is a lot of noise, so we can rule out subtle quantum effects. There is also noise from head movement, pressure changes due to blood flow, slight changes in the chemicals present (homeostasis is not perfect). We won't be simulating at the level of individual molecules or lower.
To me it seems highly likely that our knowledge of physics is more than sufficient for simulating the brain, what is lacking is knowledge of biology and the computational power.
That's a quantitative distinction at most, since computationally both are equivalent (as both can simulate each other's basic components).
And what's a few orders of magnitudes in implementation efficiency among philosophers?
a computer is a digital processor that works work with raw data, and tends to be entirely static when no processing is happening.
This depends entirely on how it's configured. Right now we've chosen to set up LLMs as verbally acute Skinner boxes, but there's not reason you can't set up a computer system to be processing input or doing self-maintenance (ie sleep) all the time.
So you’re saying a brain is a computer, right?
In the sense that it can perform computations, yes. But the underlying mechanisms are vastly different from a modern digital computer, making them extremely different devices that are alike in only a vague sense.
I have always wondered if we would be capable of writing down the mechanisms that power our thoughts. I think that this was one of the ideas that bubbled up from reading Godel Escher Bach many years ago. Is it possible for us to express the machine that makes us using the outputs of that machine in the way that it's not possible to write second order logic using first order logic.
Of course, also there are processes that are not expressible as computations, but those of these that I know about seem very very distant from human thought, and it seems very very improbable that they could be implemented with a brain. I also think that these are not observed in our universe so far.
Yes. I took an introneuroscience course a few years ago. Even to understand what is happening in one neuron during one input from one dendrite requires differential equations. And there are postive and negative inputs and modulations... it is bewildering! And how many billions of neurons with hundreds of interactions with surrounding neurons? And bundles of them, many still unknown?
Do you need differential equations to understand what’s happening in a transistor?
Searle was known for the Chinese Room experiment, whicb demonstrated language in its translational states to be strong enclitic feature of various judgements of the intermediary.
Searle also has almost 20 books, most written after 1980 and the Chinese room. None that I have read are pop science NYT best seller types. I suspect that is why most people only know the Chinese room. His depth of thought was much more than the Chinese Room.
>translational states to be strong enclitic feature of various judgements of the intermediary
I don't understand, could you explain what you mean?
I looked up enclitic - it seems to mean the shortening of a word by emphasizing another word, I can't understand why this would apply to the judgements of an intermediary
Searle seemed to reject the Chinese Room as mis-framed, with the his point better summarized as, he wrote, 'syntax does not create semantics': a purely 'syntactic' computer, limited to 'mechanical' symbol manipulation, does not 'understand' without assignments of linguistic roles to the syntax. He continued that with 'physics doesn't create syntax', meaning that even syntactic roles require a normative interpretation for what counts as what (discrete signs, valid composite signs, errors). That finally ensues, in his book The Construction of Social Reality, in computation being 'observer relative', along with the CR being a poor starting point: " …the really deep problem is that syntax is essentially an observer-relative notion…..For the purposes of the original [Chinese Room] argument I was simply assuming that the syntactical characterization of the computer was unproblematic. But that is a mistake. There is no way you could discover that something is intrinsically a digital computer because the characterization of it as a digital computer is always relative to an observer who assigns a syntactical interpretation to the purely physical features of the system." (Philosophy in a New Century p. 94). Unfortunately Searle didn't, or couldn't, elaborate on 'the really deep problem', and this final perspective on observer-relativity is missed by many readers. As observer-relative, computation would appear to be one of Searle's social realities, but he doesn't ever say that, it's a bridge too far. Finally, 'consciousness' per se is also not the focus, it's more about intentionality and the interdependence of syntax with semantics/meaning. Intentionality is a kind of consciousness; they are not identical.
I'd quibble with some of this, but overall I agree: the Chinese Room has a lot of features that really aren't ideal and easily lead to misinterpretation.
I also didn't love the "observer-relative" vs. "observer-independent" terminology. The concepts seem to map pretty closely to "objective" vs. "subjective" and I feel like he might've confused fewer people if he'd used them instead (unless there's some crucial distinction that I'm missing). Then again, it might've ended up confusing things even more when we get to the ontology of consciousness (which exists objectively, but is experienced subjectively), so maybe it was the right move.
What strikes me as interesting about the idea that there is a class of computations that, however implemented, would result in consciousness, is that is is in some way really idealistic.
There's no unique way to implement a computation, and there's no single way to interpret what computation is even happening in a given system. The notion of what some physical system is computing always requires an interpretation on part of the observer of said system.
You could implement a simulation of the human body on common x86-64 hardware, water pistons, or a fleet of spaceships exchanging sticky notes between colonies in different parts of the galaxy.
None of these scenarios physically resemble each other, yet a human can draw a functional equivalence by interpreting them in a particular way. If consciousness is a result of functional equivalence to some known conscious standard (i.e. alive human being), then there is nothing materially grounding it, other than the possibility of being interpreted in a particular way. Random events in nature, without any human intercession, could be construed as a veritable moment of understanding French or feeling heartbreak, on the basis of being able to draw an equivalence to a computation surmised from a conscious standard.
When I think along these lines, it easy to sympathize with the criticism of functionalism a la Chinese Room.
As someone that studied philosophy, his work is cited often and is absolutely instrumental in modern theory of mind. His work has seen a resurgence recently due to the explosion of LLMs. I've read 2 or 3 of his books, and he was a brilliant mind with clear & concise arguments. I met many of his collaborators at UCLA, but sadly never the man himself. Either way, his work has had a profound effect on me and my understanding of the world.
Rest in peace.
> Informed once that the listing of an introductory philosophy course featured pictures of René Descartes, David Hume and himself, Professor Searle replied, “Who are those other two guys?” (the article)
He brought so many unique contributions to the field. Top 10 in philosophy of mind imo. Sad that he chose to tarnish his legacy by preying on his students for decades. I find the lack of discussion in here around his misconduct very telling. There is so much to learn here regarding the way we revere bright minds like his that might not have the brightest of morals
Consciousness in Artificial Intelligence | John Searle | Talks at Google (2015) https://www.youtube.com/watch?v=rHKwIYsPXLg
Obviously a meat brain is incomparable to a LLM - they are different types of intelligence. Any sane person wouldn't claim a LLM to be conscious in the meat brain sense, but it may be conscious in a LLM way, like the duration of time where matrix multiplications are firing inside GPUs.
If an LLM could be "conscious in an LLM way", then why not the same, mutatis mutandis, for an ordinary computer program?
because an ordinary program is deterministic, LLM is probablistic + it has some synthetic-reasoning ability
It just aligns generated words according to the input. It is missing individual agency and self sufficiency which is a hallmark of consciousness. We sometimes confuse the responses with actual thought because neural networks solved language so utterly and completely.
Not sure I'd use those criteria, nor have I heard them described as hallmarks of consciousness (though I'm open, if you'll elaborate). I think the existence of qualia, of a subjective inner life, would be both necessary and sufficient.
Most concisely: could we ask, "What is it like to be Claude?" If there's no "what it's like," then there's no consciousness.
Otherwise yeah, agreed on LLMs.
I'd say being the maintainer of the weights is individual agency. Not just training new agents, but introspection. So autonomous management system would be pretty much conscious.
> It is missing individual agency and self sufficiency which is a hallmark of consciousness.
You can be completely paralyzed and completely concious.
Yes, but you can't be completely suspended with no sensory input or output, not even internally (i.e. hunger, inner pains, etc), and no desires, and still be conscious.
Yes, and you have individual agency while completely paralyzed.
It is not very often that you hear about somebody raising the cost of rent for everyone in an entire city by ~28% in a single year[0]. He will certainly be remembered.
0. https://www.academia.edu/30805094/The_Success_and_Failure_of...
Searle famously argued that the treatment of landlords in Berkeley was comparable to the treatment of black people in the south ...
oh wow https://en.wikipedia.org/wiki/John_Searle#Political_activity
I personally struggle to imagine what it would be like to have an untouchable philosophy professor that does not see the difference between purchasing a seventeen unit apartment building in Berkeley, California and being born black in the south. Sadly I was not there in the twenty five to twenty nine years between him making that argument and his departure from the university to experience that
"Departure" is a very generous euphemism for "being kicked for taking advantage of his students". His contributions to the philosophy of mind are great and will be felt in Computer Science, especially with the current LLM tech but let's not skirt around the subject.
Non-paywalled obit:
https://www.theguardian.com/world/2025/oct/05/john-searle-ob...
His most famous argument:
https://en.wikipedia.org/wiki/Chinese_room
I find the Chinese room argument to be nearly toothless.
The human running around inside the room doing the translation work simply by looking up transformation rules in a huge rulebook may produce an accurate translation, but that human still doesn't know a lick of Chinese. Ergo (they claim) computers might simulate consciousness, but will never be conscious.
But is the Searle room, the human is the equivalent of, say, ATP in the human brain. ATP powers my brain while I'm speaking English, but ATP doesn't know how to speak English just like the human in the Searle room doesn't know how to speak Chinese.
There is no translation going on in that thought experiment, though. There is text processing. That is, the man in the room receives Chinese text through a slot in the door. He uses a book of complex instructions that tells him what to do with that text, and he produces more Chinese text as a response according to those instructions.
Neither the man, nor the room "understand" Chinese. It is the same for the computer and its software. Jeffery Hinton has sad "but the system understands Chinese." I don't think that's a true statement, because at no point is the "system" dealing with semantic context of the input. It only operates algorithmically on the input, which is distinctly not what people do when they read something.
Language, when conveyed between conscious individuals creates a shared model of the world. This can lead to visualizations, associations, emotions, creation of new memories because the meaning is shared. This does not happen with mere syntactic manipulation. That was Searle's argument.
> I don't think that's a true statement, because at no point is the "system" dealing with semantic context of the input. It only operates algorithmically on the input, which is distinctly not what people do when they read something.
There are two possibilities here. Either the Chinese room can produce the exact same output as some Chinese speaker would given a certain input, or it can't. If it can't, the whole thing is uninteresting, it simply means that the rules in the room are not sufficient and so the conclusion is trivial.
However, if it can produce the exact same output as some Chinese speaker, then I don't see by what non-spiritualistic criteria anyone could argue that it is fundamentally different from a Chinese speaker.
Edit: note that here when I'm saying that the room can respond with the same output as a human Chinese speaker, that includes the ability for the room to refuse to answer a question, to berate the asker, to start musing about an old story or other non-sequiturs, to beg for more time with the asker, to start asking the akser for information, to gossip about previous askers, and so on. Basically the full range of language interactions, not just some LLM style limited conversation. The only limitations in its responses would be related to the things it can't physically do - it couldn't talk about what it actually sees or hears, because it doesn't have eyes, or ears, it couldn't truthfully say it's hungry, etc. It would be limited to the output of a blind, deaf, mute Chinese speaker confined to a room whose skin is numb and who is being fed intravenously, etc.
> if it can produce the exact same output as some Chinese speaker, then I don't see by what non-spiritualistic criteria anyone could argue that it is fundamentally different from a Chinese speaker.
Indeed. The crux of the debate is:
a) how many input and response pairs are needed to agree that the rule-provider plus the Chinese room operation is fundamentally equal/different to a Chinese speakers
b) what topics can we agree to exclude so that if point a can be passed with the given set of topics we can agree that 'the rule-provider plus the Chinese room operation' is fundamentally equal/different to a Chinese speaker
As far as I can see, Searle rejects the whole concept, and claims that by construction, it is obvious that the Chinese room doesn't understand Chinese in the same way that a speaker does, regardless of how well it can mimic Chinese speech.
> claims that by construction, it is obvious that
Sounds like circular logic to me unless you make that assumption explicit
Yes, that's what I'm "accusing" him of. I'm sure his defenders view his argument differently.
> It only operates algorithmically on the input, which is distinctly not what people do when they read something.
That's not at all clear!
> Language, when conveyed between conscious individuals creates a shared model of the world. This can lead to visualizations, associations, emotions, creation of new memories because the meaning is shared. This does not happen with mere syntactic manipulation. That was Searle's argument.
All of that is called into question with some LLM output. It's hard to understand how some of that could be produced without some emergency model of the world.
In the thought experiment as constructed it is abundantly clear. It's the point.
LLM output doesn't call that into question at all. Token production through distance function in high-dimensional vector representation space of language tokens gets you a long way. It doesn't get you understanding.
I'll take Penrose's notions that consciousness is not computation any day.
Out of interest, what do you think it would look like if communicating was algorithmic?
I know that it doesn't feel like I am doing anything particularly algorithmic when I communicate but I am not the hommunculus inside me shuffling papers around so how would I know?
I think it would end inspiration.
Inspiration is what a search algorithm feels like from the inside.
Can you elaborate?
This goes far to explain a lot of Chinese room situations. We have an intuition for the way something is. That intuition is an unshakeable belief, because it is something that we feel directly. We know what it feels like to understand Chinese (or French, or English, or whatever), and that little homunculus shuffling papers around doesn't feel like it.
Hopefully we have all experienced what genuine inspiration feels like, and we all know that experience. It sure as hell doesn't feel like a massively parallel search algorithm. If anything it probably feels like a bolt of lightning, out of the blue. But here's the thing. If the conscious loop inside your brain is something like the prefrontal cortex, which integrates and controls deeper processing systems outside of conscious reach, then that is exactly what we should expect a search algorithm to feel like. You -- that strange conscious loop I am talking to -- are doing the mapping (framing the problem) and the reducing (recognizing the solution), but not the actual function application and lower level analysis that generated candidate solutions. It feels like something out of the blue, hardly sought for, which fits all the search requirements. Genuine inspiration.
But that's just what it feels like from the inside, to be that recognizing agent that is merely responding to data being fed up to it from the mess of neural connections we call the brain.
You can take this insight a step further, and recognize that many of the things that seem intuitively "obvious" are actually artifacts of how our thinking brains are constructed. The Chinese room and the above comment about inspiration are only examples.
I cannot emphasize enough how much I dislike linking to LessWrong, and to Yudkowsky in particular, but I first picked up on this from an article there, and credit should be given where credit is due: https://www.lesswrong.com/posts/yA4gF5KrboK2m2Xu7/how-an-alg...
Fascinating, thank you very much, and agreed on Yudkowsky. It's a bit like crediting Wolfram.
Sure thing.
By the way, the far more impactful application of this principle is as a solution (imho) to the problem of free will.
Most people intuitively hold that free will is incompatible with determinism, because making a choice feels unconstrained. Taken in the extreme, this leads to Penrose and others looking for quantum randomness to save their models of the mind from the Newtonian clockwork universe.
But we should have some unease with this, because choices being a random roll of the dice doesn’t sit right either. When we make decisions, we do so for reasons. We justify the choices we make. This is because so-called “free will” is just what a deterministic decision making process feels like from the inside.
Philosophically this is called the “compatibilist” position, but I object to that term. It’s not that free will is merely compatible with determinism—it requires it! In a totally random universe you wouldn’t be able to experience the qualia of making a free choice.
To experience a “free choice” you need to be able to be presented with alternatives, weight the pro and con factors of each, and then make a decision based on that info. From the outside this is a fully deterministic process. From the inside though, some of the decision making criteria are outside of conscious review, so it doesn’t feel like a deterministic decision. Weighing all the options and then going with your gut in picking a winner feels like unconstrained choice. But why did your gut make you choose the way you did? Cause your “gut” here is an unconscious but nevertheless deterministic neural net evaluation of the options against your core principles and preferences.
“Free will” is just what a deterministic application of decision theory feels like from the inside.
I should have snipped the "it operates" part to communicate better. I meant that it's not at all clear that people are doing something non-algorithmic.
That is why you cannot ask the room for semantic changes. Like “if I call an umbrella a monkey, and it will rain today, what do I need to bring?”
Unless we suppose those books describe how to implement a memory of sorts, and how to reason, etc. But then how sure are we it’s not conscious?
> if I call an umbrella a monkey, and it will rain today, what do I need to bring?
I'm not even sure what you are asking for, tbh, so any answer is fine.
Well, at least it's a good reason to re-read his infamous exchange with Derrida.
When I studied in Ulaan Bataar some twenty years ago I met a romanian professor of linguistics who had prepared by trying to learn mongolian from books. He quickly concluded that his knowledge of russian, cyrillic and having read his books didn't actually give him a leg up on the rest of us, and that pronounciation and rhythm as well as more subtle aspects of the language like humour and irony hadn't been appropriately transferred through the texts he'd read.
Rules might give you some grasp of a language, but breaking them with style and elegance without losing the audience is the sign of a true master and only possible by having a foundation in shared, embodied experience.
There's a crude joke in that Searle left academia disgraced the way he did.
If you are wondering, it’s not the Doc guy with a similar name: https://en.wikipedia.org/wiki/Doc_Searls (But he was a PhD)
[dead]
[dead]
[flagged]
But she also claims he "was innocent and falsely accused": https://www.colinmcginn.net/john-searle/
She could feel that the 2016 allegations specifically were unfounded while acknowledging the previous pattern of misconduct.
https://www.insidehighered.com/quicktakes/2017/04/10/earlier...
[flagged]