> If you want to be really good, you have to understand what’s being said. That’s the only way.
This is simply not true. Predicting the next letter or word, or id you abstract it away from things that mean something to you, like the next color of a block in a long chain of colored blocks. You would realize that all we are doing is using statistics to predict what the next item might be. There simply is no need or requirement for the system doing the prediction to understand.
> You would realize that all we are doing is using statistics to predict what the next item might be.
So what does "understanding" really means then?
"Understanding" is not really well defined. Either we (humans) do it, but then LLMs might just do it as well, depending on the definition of "understanding", or we both don't do it. But if no-one is really "understanding" anything, then this definition of "understanding" is maybe not useful.
Nobody actually understands how humans actually understand something, just like nobody actually understands how LLMs do what they do.
Everybody opining about it is doing just that: offering an opinion. Geoff Hinton’s opinion is worth more than someone else’s, but it is still an opinion.
We also understand, down to a very very very microscopic level, how neurons work. We also know a helluva lot about the architecture of the brain. Does that mean we can explain our own intelligence, how our minds actually work? Nope.
I feel that LLMs raise some very interesting challenges for anyone trying to figure out what it means to understand something and how we do it, but I am not yet ready to agree with Hinton.
For example, we are aware that some, but by no means all, of what people say is about an external world that may or may not conform to what the words say. We can also doubt that we have understood things correctly, and take steps to either confirm or refute our opinions. We see ourselves as entities in an external reality containing other individuals who also do this, and that we, and they, have a limited ability to influence what happens in that world. Do LLMs do these things, or is what they produce a result of having a lot of information about the purely formal properties of human language use, independently of semantics?
>I feel that LLMs raise some very interesting challenges for anyone trying to figure out what it means to understand something and how we do it, but I am not yet ready to agree with Hinton.
Agreed. What LLMs say about understanding deserves a lot more attention than it has received. I wrote down some of my thoughts on the matter:
>Do LLMs do these things, or is what they produce a result of having a lot of information about the purely formal properties of human language use, independently of semantics?
These two points aren't necessarily in opposition, and understanding why is I think key to solving a lot of important problems around intelligence, sentience, etc. To compute is to operate on formal properties. But this doesn't exclude semantic properties from having causal relevance to the behavior of the system. What we need is a way to conceptualize how a system can have multiple related descriptions at different levels. A description at the level of semantics doesn't exclude a description in terms of formal properties or vice versa. I think of it in terms of constraints: the higher level descriptions constrain the lower level behavior. What the computational description does is ensures the higher level semantic constraint is maintained. It does this by the particular space of computational dynamics it follows. Essentially, the information that picks out this programs space of branching dynamics embeds the semantic description in question, and this description realizes the computational dynamic necessary to maintain the higher level semantic constraint. Rather than semantics being in opposition to formal properties, they are two sides of the same coin.
I agree with a lot of what you say in the linked article, and I particularly agree that it is not helpful to define understanding in a way that would, a priori, make it a category error to propose that a suitably-programmed computer might understand things. I do, however, have a few words to say about the relationship between modeling and understanding. I can easily accept that an ability to model is necessary in order to understand something, but I feel the idea that it is sufficient would leave something out.
For example, meteorologists understand a lot about the weather in terms of the underlying physics, representing it as a special application of more general laws, but they are not very good at predicting it. Machine learning produces models which are much better predictors, but it does not seem to follow that they have a superior understanding of the weather.
One problem in assessing whether a token predictor has some sort of understanding is that if its training material is consistent with the supposition that, broadly speaking, it was produced by people who do have a reasonable understanding of what they were writing about, then it seems likely that the productions of a good predictor would unavoidably have that feature as well - but maybe that just is how most human understanding works? I am on the fence on this one.
>Machine learning produces models which are much better predictors, but it does not seem to follow that they have a superior understanding of the weather.
Fair points, and I agree. I don't recall if I made this point in the linked piece, but I think the extra function is a model embedded within some dynamic such that the capacity for modelling is in service to some goal. The goal can be simple like answering questions or something more elaborate. But the point is to engage the model as to influence the dynamic in a semantically rich way. The model itself doesn't represent understanding, but a process that understands will have a model that can be queried and manipulated in various ways corresponding to the process' goals.
>then it seems likely that the productions of a good predictor would unavoidably have that feature as well
Yeah, assessment is hard because of the sheer size of the training data. We can't be sure that some seemingly intelligent response isn't just recalling a similar query from training. One of the requirements for understanding is the counterfactual capacity, being able to report accurate information that is derivative of the training data but not explicitly in the training data. The Sparks of AGI paper, assuming it can be believed, demonstrates this capacity IMO. Particularly where GPT-4 draws a graph of a room after having been given navigation instructions. But its hard to make a determination in particular cases.
As a translator I have come to think of it like this- (without the need for defining what “understanding” means) the ability to translate correctly depends on the size of your context. So humans and LLMs are both on the same spectrum with different context sizes of their training set and all facts known in the universe (for a theoretical perfect human) respectively.
Doing moderately technical translations, state of the art LLMs may get it about 99% right but you find about 1 word per page where they would have needed to “zoom out” to a wider context to choose the word correctly. Often its an important word that is missed.
From the article, Gelman's money quote is this: " So I’m not knocking auto-complete; I’m just disagreeing with Hinton’s statement that “by training something to be really good at predicting the next word, you’re actually forcing it to understand.” As a person who does a lot of useful associative reasoning and also a bit of logical understanding, I think they’re different, both in how they feel and also in what they do."
And while AI gets better and better and we will remain as touchy as ever about abstract concepts that make us oh so human, how about we say it just can't be understanding, unless a human does, eh, it.
How about this: understanding is the ability to generalize knowledge and apply it to novel scenarios.
This definition is something that humans, and animals for that matter, do every day - both in small and large ways. And this is something that current language models aren't very good at.
I taught it Firefly, which is an undocumented programming language I'm working on, through conversion.
I find it's a lot quicker than any human at picking up syntax and semantics, both in real time and in number of messages, and makes pretty good attempts at writing code in it, as much as you could expect from a human programmer.
That is, until you run out of context - is this what you mean?
The key complication is "once you've opened the door, you may no longer touch a switch." It gets this. There are many examples of it written out on the web. When I give it a variation and say "you can open the door to look at the bulbs and use the switches all you want" and it is absolutely unable to understand this. To a human it's simple: look at the bulbs and flick the switches. It kept giving me answers about using a special lens to examine the bulbs, using something to detect heat. I explained it in many ways and tried several times. I was paying for GPT-4 at the time as well.
I would not consider this thinking. It's unable to make this simple abstraction from its training data. I think 4 looks better than 3 simply because it's got more data, but we're reaching diminishing returns on that, as has been stated.
GPT-4 on platform.openai.com says this on the first try:
Switch on the first switch and leave it on for a few minutes. Then, switch it off and switch on the second switch. Leave the third switch off. Now, walk into the room.
The bulb that is on corresponds to the second switch. The bulb that is off and still warm corresponds to the first switch because it had time to heat up. The bulb that is off and cool corresponds to the third switch, the one you never turned on.
GPT-4-0314:
1. Turn on the first switch and leave it on for about 5 minutes.
2. After 5 minutes, turn off the first switch and turn on the second switch.
3. Open the door and enter the room.
Now observe the lights:
- The bulb that is on is connected to the second switch (which is currently on).
- The bulb that is off but warm to the touch is connected to the first switch (it was on long enough to heat up the bulb).
- The bulb that is off and cool to the touch is connected to the third switch (it was never turned on).
----
But– It's also trained on the internet. GPT-4 paper 'sparks of AGI' had a logical puzzle it most likely never encountered in the training data that it could solve.
Also– I encourage you to ask these types of logical puzzles on the street to rando's. They're not easy to solve.
My question to you would be: What would convince you that it actually can 'think' logically?
I think your comment misunderstands the comment you're responding to.
The point is that while LLMs can solve the puzzle when the constraints are unchanged -- as you said, there are loads of examples of people asking and answering variations of this puzzle on the internet -- but when you change the constraints slightly ("you can open the door to look at the bulbs and use the switches all you want") it is unable to break out of the mold and keeps giving complicated answers, while a human would understand that under the new constraints, you could simply flip each switch and observe the changes in turn.
A similar example that language models used to get stuck on is this: "Which is heavier, a pound of feathers or two pounds of bricks?"
There are plenty of results supporting my assertion; but the tests must be carefully designed. Of course, LLMs are not databases that store exact answers - so it's not enough to ask it something that it hasn't seen, if it's seen something similar (as is likely the case with your programming language).
One benchmark that I track closely is ConceptARC, which aims to test generalization and abstraction capabilities.
Here is a very recent result that uses the benchmark: https://arxiv.org/abs/2311.09247. Humans correctly solved 91% of the problems, GPT-4 solved 33%, and GPT-4V did much worse than GPT-4.
Someone sufficiently fast and skilled at googling can explain and use in context a lot of things that they don't really properly understand.
So unless you're saying that the composite of the googler and of google understand something that neither does individually, your definition has some holes.
I would say that there is a stronger consensus that a human being can be reasonably described as a single entity than a human being using a reference resource.
A more apt comparison to my mind would be if a human being can be described as personally exerting strong nuclear force, just because their subatomic particles do, which I would happily answer "no."
Isn't that exactly how humans learn to respond to stimuli? Don't we just try to predict the best next response to everything?
Yes, It's statistics but the fun part is nobody is writing this statistical function by hand.
LLMs don't have a concept of "best". Only most likely in what they've been trained on.
I think LLMs ultimately just take imitation to a creative and sophisticated extreme. And imitation simply doesn't comprise the whole of human intelligence at all, no matter how much it is scaled up.
The sophistication of the imitation has some people confused and questioning whether everything can be reduced to imitation. It can't.
The ability to imitate seeking a goal isn't identical to the ability to seek a goal.
The ability to imitate solving a problem isn't identical to the ability to solve a problem.
Imitation is very useful, and the reduction of everything to imitation is an intriguing possibility to consider, but it's ultimately just wrong.
There are levels of sophistication in "imitation". It follows a gradient. At the low end of this gradient is a bad imitation.
At the high end of this gradient is a perfect imitation. Completely indistinguishable from what it's imitating.
If an imitation is perfect than is it really an imitation?
If I progressively make my imitation more and more accurate am I progressively building an imitation or am I progressively building the real thing?
See what's going on here? You fell for a play on words. It's a common trope. Sometimes language and vocabulary actually tricks the brain into thinking in a certain direction. This word "imitation" is clouding your thoughts.
Think about it. A half built house can easily be called an imitation of a real house.
Ok, so now we need an example that separates humans from LLMs?
I struggle to think of one, maybe someone on HN has a good example.
Eg if I'm in middle school and learning quadratic equations, am I imitating solving the problem by plugging in the coefficients? Or am I understanding it?
Most of what I see coming out of chatGPT and copilot could be said to be either. If you're generous, it's understanding. If not, it's imitation.
Some humans created maths. And it took thousands of years of thinking and interaction with the real world.
Seems like goalpost moving to me.
I think the real things that separate LLMs from humans at the moment are:
* Humans can do online learning. They have long term memory. I guess you could equate evolution to the training phase of AI but it still seems like they don't have quite the same on-line learning capabilities as us. This is what probably prevents them from doing things like inventing maths.
* They seem to be incapable of saying "I don't know". Ok to be fair lots of humans struggle with this! I'm sure this will be solved fairly soon though.
* They don't have a survival instinct that drives proactive action. Sure you can tell them what to do but that doesn't seem quite the same.
Interestingly some humans will admit to not knowing but are allergic to admitting being wrong (and can get fairly vindictive if forced to admit being wrong).
LLM’s actually admit to being wrong easily, but aren’t great at introspection and confabulate too often. also their Meta cognition is poor still.
I guess LLM's don't have the social pressure to avoid admitting errors. And those sort of interactions aren't common in text so they don't learn them strongly.
Also ChatGPT is trained specifically to be helpful and subservient.
About this goalpost moving thing. It's become very popular to say this, but I have no idea what it's supposed to mean. It's like a metaphor with no underlying reality.
Did a wise arbiter of truth set up goalposts that I moved? I guess I didn't get the memo.
If the implied claim is "GPT would invent math too given enough time", go ahead and make that claim.
> Did a wise arbiter of truth set up goalposts that I moved?
Collectively, yes. The criticism of AI has always been "well it isn't AI because it can't do [thing just beyond its abilities].
Maybe individually your goalpost hasn't moved, and as soon as it invents some maths you'll say "yep, it's intelligent" (though I strongly doubt it). But collectively the naysayers in general will find another reason why it's not really intelligent. Not like us.
No. I don't think anyone seriously believes that. AGI requires human level reasoning and it hasn't achieved that, despite what benchmarks show (they tend to focus on "how many did it get right" more than "how many did it fail in stupid ways").
The issue with most criticism of LLMs wrt AGI is that they come up with totally bogus reasons why it isn't and can't ever be real intelligence.
It's just predicting the next word. It's a stochastic parrot. It's only repeating stuff it has been trained on. It doesn't have quantum microtubules. It can't really reason. It has some failure modes that humans don't. It can't do <some difficult task that most humans can't do>.
Seems to be mostly people feeling threatened. Very tedious.
You can ask ChatGPT to solve maths problems which are not in its training data, and it will answer an astonishing amount of them correctly.
The fact that we have trained it on examples of human-produced maths texts (rather than through interacting with the world over several millennia) seems like more of an implementation detail and not piece of evidence about whether it has “understood” or not.
They also get problems wrong, in the most dumb way possible. I've tested it out many times where the LLM got most of the more 'difficult' part of the problem right, but then forgot to do something simple in the final answer--and not like a simple error a human would make. It's incredibly boneheaded, like forgetting to apply the coefficient it solved for and just returning the initial problem value. Sometimes for coding snippets, it says one thing, and then produces code which does not even incorporate the thing it was talking about. It is clear that there is no actual conceptual understanding going on. I predict the next big breakthroughs in physics will not be made by LLMs--even if they have the advantage of being able to read every single paper ever published, because they cannot think.
> Isn't that exactly how humans learn to respond to stimuli?
Maybe it is, maybe it isn't. Maybe we are "just" an incredibly powerful prediction engine. Or maybe we work from a completely different modus operandi, and our ability to predict things is an emergent capability of it.
The thing is, no one actually knows what makes us intelligent, or even how to define intelligence for that matter.
> You would realize that all we are doing is using statistics to predict what the next item might be.
Two questions:
1. How can humans come up with novel ideas then? Statistically predicting the next item means I am limited to the training set. How do humans formulate new concepts in this scenario?
2. Why can humans learn from their own output then? If all we do is statistically predicting the next token, then, by necessity, feeding our own output back into the model (aka. letting humans learn from it) should degrade human capabilities by overfitting to the training set. But we observe exactly the opposite.
> Statistically predicting the next item means I am limited to the training set
Not at all. Extrapolation is perfectly possible in a purely predictive model. It’s one of the things GPTs are best at.
In the stream of tokens output by an LLM it’s completely possible for new concepts to emerge, and for it then to continue to use and build on them in the remainder of the stream. You see this simply executed in programming contexts with LLMs where it is able to declare a novel function and then use it.
> with LLMs where it is able to declare a novel function and then use it.
Novel as in "implements a new algorithm that has never seen in any form and is actually an improvement over existing methodology"?
Here is a little thought experiment: If all the training data in the set says that manned, powered flight is impossible, is a statistical prediction engine trained on that data capable of developing an airplane?
In other words, could a very powerful stochastic parrot do what the Wright Brothers did?
Why is "using statistics" mutually exclusive with "understanding"? It would help to carefully define terms. Note that "understanding" doesn't mean it's necessarily conscious.
These systems learn high-level representations/abstractions of concepts we humans also use as part of our cognition. The concept of an object, an intuitive physics, the role of specific objects. I don't criticize him for using the word "understanding" to describe this.
The best “statistic” you could compute to predict the next token would be the meaning of the prompt. A predictor computing meaning would outperform one that is not.
By "understanding" he means it builds a model of higher order abstractions over the text in order to model the meaning and context of each token in relation to other tokens. This is more sophisticated than a Markov process.
> This is more sophisticated than a Markov process.
Nothing prevents a markov process from having a world model. A markov process lacks state and generates a sequence based on statistics and previous entries, so LLMs as they are now are markov processes.
Many of the dumb behaviors we see from LLMs today comes from their lack of internal state between tokens, so it don't remember what reason it had for generating the previous token and that means it can easily generate inconsistent answers. So LLMs being a markov process is an important point to highlight since it makes its thinking very different from how humans think.
The transformer architecture is NOT a Markov process, by mathematical definition of a Markov process. This is not even debatable. It's a mathematical fact.
> Many of the dumb behaviors we see from LLMs today comes from their lack of internal state between tokens, so it don't remember what reason it had for generating the previous token and that means it can easily generate inconsistent answers
The attention mechanism in the transformer architecture models relations between tokens within the context window, and does the exact opposite of what you are describing here. This is one aspect of LLMs that violates the Markov property.
> The transformer architecture is NOT a Markov process, by mathematical definition of a Markov process. This is not even debatable. It's a mathematical fact.
What, yes it is.
> The attention mechanism in the transformer architecture models relations between tokens within the context window, and does the exact opposite of what you are describing here. This is one aspect of LLMs that violates the Markov property.
The context window is finite, so that is the previous step. You know the dumb markov chains that are based on bag of words? They also look several words back, they don't just go based on a single word. LLMs are just that but with a way larger lookback and some extra logic there, but none of that changes the fundamental parts to make it no be a markov process.
With a large enough context size you could argue it is now fundamentally different in practice, but in theory it is the same. There is no "hidden state", its just the previous n words defines the next word.
If you define the state broadly enough to include the state of the entire machine itself, including all of its internal representations, weights, activations, etc, then you are playing a funny trick here.
By the same reasoning, a human brain is also Markov process.
What you are doing here is a vast oversimplification and it is practically useless for understanding how LLMs work.
> You would realize that all we are doing is using statistics to predict what the next item might be.
I agree that Hinton's original quote doesn't make sense to me either. I suspect he would leverage the phrase "really good" to explain the difference between ChatGPT and, say, a Markov chain. I think that's a little disingenuous, if that's how he means it, but I don't know if I'm right about that.
But I also do not agree that humans use statistics to predict what the next item in a series might be. As evidence, there is the classic example of asking people to predict the next coin toss in the series: "heads, heads, heads, heads, heads, heads, heads...". They'll either guess heads because it's come up so many times already, or because they assume the coin isn't fair, or tails because it's "overdue" to come up, but none of those are based on statistics per se.
This is simply not true. Predicting the next letter or word, or id you abstract it away from things that mean something to you, like the next color of a block in a long chain of colored blocks. You would realize that all we are doing is using statistics to predict what the next item might be. There simply is no need or requirement for the system doing the prediction to understand.