YouSaid · the spoken record

Jay McClelland

lines on the record

125

first

2021-09-20

most recent

2021-09-20

sittings or episodes

1

sources

podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

1. “Yeah. And because they give you the ability to. Be exact about Like, how many sheep you have? Like, you know, I sent you out this morning. There were 23 sheep. You came back with only 22. What happened?”

   2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

2. “Gives human beings the incredible leverage that they didn't have without these concepts. And I think this is actually already true. When we think about Just, you know, the natural numbers I always like to include zero. So I'm going to say the Non negative integers, but That's a place where some people prefer not to include zero.”

   2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

3. “Is something about Triangles. It's not a matter of formulas. These are idealist objects. In fact, we built bridges out of triangles and we understand how to measure the height of something we can't climb by extending these ideas about triangles a little further. All of the ability to get a tiny speck of matter launched from The planet Earth to intersect with some tiny, tiny little body way out and way beyond Pluto somewhere at exactly a predicted time and date is something that depends on these ideas, right? And it's actually... Happening in the real physical world that these ideas make contact with it in those kinds of instances. So, but you know, there are these idealized objects, these triangles or these distances or these points, whatever they are that allow for this set of tools to be created that then”

   2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

4. “Well, I think of mathematics as a set of tools for. Exploring Idealized worlds that often turn out to be extremely relevant to the real world, but need not. But there are worlds in which Objects exist with idealized properties. And in which Relationships among them can be characterized with precision. so as to allow implications of Certain facts to then allow you to derive other facts with certainty. So... If you have two triangles and you know that there is An angle in the first one that has the same measure as an angle in the second one. And you know that the lengths of the sides Adjacent to that angle in each of the two triangles. Corresponding sides adjacent to that angle also have the same measure. Then you can then conclude that the triangles are Congruent, that is to say, they have all of their properties in common. And that.”

   2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

5. “Yes. And I think that, you know, obviously that's the. That's being played out massively at Google Brain, at OpenAI, and to some extent at DeepMind as well. I guess I shouldn't say to some extent the massive scale of Computations that are used to succeed at games like Go or to solve the protein folding problems that they've been solving and so on.”

   2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

6. “So I think of, I started using the phrase computational intelligence at some point as to characterize the field that I thought people like Jeff Hinton and many of the people I know at DeepMind. Are working in and where I feel like I'm. I'm a kind of a human-oriented computational intelligence researcher in that I'm actually kind of interested in the human solution. But at the same time, I feel like that's where a huge amount of the The excitement of deep learning actually lies is in the idea that We may be able to even go beyond what we can achieve with our own nervous systems when we build Computational intelligence that Know not limited in the ways that we are by our own biology”

   2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

7. “On something called the Bolson machine was his way of connecting with that Boolean tradition and bringing it into the more continuous probabilistic graded constraint satisfaction realm. And it was beautiful set of ideas linked with theoretical physics and as well as with logic. It's always been, I mean, I've always been inspired by the Bolson Machine, too. It's like, well, if the neurons are probabilistic rather than, you know, deterministic in their computations, then maybe this somehow is part of the... Serendipity or adventitiousness of the moment of insight, right? It might not have occurred at that particular instant. It might be sort of partially the result of a stochastic process. And that too is part of the magic of the emergence of. Some of these things”

   2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

8. “Yeah, Jeff is a descendant of the logician bull. He comes from a long line of English academics. And together with the Deeply intuitive thinking ability that he has, he also It's been clear he's described this to me, and I think he's mentioned it from time to time in other interviews that he's had with people, he's wanted to be able to sort of think of himself as contributing to the To the understanding of Reasoning itself not just human reasoning, like Boole is about logic, right? It's about What can we conclude from what else and how do we formalize that? And as a computer scientist, logician, philosopher, you know, the goal is to Understand how we derive truth from givens and things like this. The work that Jeff was doing in the Early to mid 80s”

   2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

9. “Effort to understand human mathematical cognition is that he doesn't write too many equations. And people tell stories like, oh, in the Hints and Lab meetings, you don't get up at the board and write equations like you do in everybody else's machine learning lab. What you do is you draw a picture. And he explains aspects of the way deep learning works by putting his hands together and showing you the shape of a ravine. Using that as a geometrical metaphor for what's happening as this gradient descent process, you're coming down the wall of a ravine. If you take too big a jump, you're going to jump to the other side. And so that's why we have to turn down the learning rate, for example. It speaks to me of the fundamentally intuitive character of deep insight. Together with To really understanding”

   2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

10. “When you call a subroutine, you need to save the state that you had. When you called it so you can get back to where you were when you're finished with the subroutine. And the idea was that you would save the state of the calling routine by making fast changes to connection weights. And then when Finished with the subroutine call, those fast changes in the connection weights would allow you to go back to where you had been before and reinstate the previous context so that you could continue on with the top level of the computation. Anyway, that was part of the idea. And I always thought, okay, that's really, you know, He had extremely creative ideas that were quite a lot ahead of his time, and many of them in the 1970s and early 1980s. So, another thing about Jeff Hinton's way of thinking, which has profoundly influenced my”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

11. “Well, so Jeff has said many things to me that had a profound impact on my thinking. And he's written several articles Way ahead of their time. He had two papers in 1981 just to give one example. One of which was essentially the idea of transformers, and another of which was an early paper on semantic cognition, which Inspired Him and Rummel Hart and me throughout the 80s and Know still, I think, sort of grounds my own thinking about the semantic aspects of cognition. He also In a small paper that was never published, he wrote in 1977 before he actually arrived at UCSD or maybe a couple years even before that. I don't know when he was a PhD student. described how a neural network could do recursive computat It was a very clever idea that he's continued to explore over time, which was sort of the idea that”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

12. “Yeah. So it came from Hinton having introduced the concept of, you know, define your objective function, figure out how to take the derivatives so that you can adjust the connections so that they make progress towards your”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

13. “Yeah. Probably that name is opaque to me. What does that mean? What it meant was that in order to figure out what the changes you needed to make to the connections from the input to the hidden layer, you had to backpropagate the error signals. From the output layer through the connections from the hidden layer to the output. Get the signals that would be the error signals for the hidden layer. And that's how Rummelhart formulated it. It was like, well, we know what the air signals are at the output layer. Let's see if we can get a signal at the hidden layer that tells each hidden unit what its error signal is, essentially. So it's backpropagating through the connections. From the hidden to the output to get the signals to tell the hidden units how to change their weights from the input. And that's why it's called back prop.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

14. “So as to be able to change the connections from earlier layers of units to the ones at a hidden layer between the input and the output. And so he first called the algorithm the generalized delta rule because it's just an extension of it. The gradient descent idea. And interestingly enough, Hinton was thinking that this wasn't going to work very well. So Hinton had his own alternative algorithm at the time. Based on the concept of the Bolson machine, that he was pursuing. So the paper on the Bolson machine came out in Learning in Bolson Machines came out in 1985. But it turned out that backprop worked better than the Boltzmann machine learning algorithm.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

15. “Professor in the engineering electrical engineering department at Stanford, Bernie Woodrow, and a collaborator named Hoff. I never met him. Anyway, so. So, gradient descent in continuous neural networks with multiple neuron like processing units was already understood. For a single layer of connection weights, we have some inputs over a set of neurons. We want the output to produce a certain pattern. We can define the difference between our target and what the narrow network is producing, and we can figure out how to change the connection weights to reduce that error. So what Romelhard did was to generalize that.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

16. “And Rummelhart heard that and said to himself, okay, so I'm going to start thinking about it that way. I'm going to Essentially, imagine that I have some objective function, some goal of the computation. I want my machine to correctly classify all of these images. And I can score that. I can measure how well they're doing on each image. And I get some measure of error or loss it's typically called in deep learning. And I'm going to figure out how to adjust the connection weight so as to minimize my loss or reduce the error. That's called gradient descent. And engineers were already familiar with the concept of gradient descent. And in fact, there was an algorithm called the Delta rule that had been invented.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

17. “And several Spolensky was one of the other postdocs. He was still there as a postdoc. A few other people. But anyway Talk to us about learning and how we should think about Learning occurs in a neural network. And he said, the problem. But the way you guys have been approaching this is that you've been looking for inspiration from biology to tell you how, what the rules should be for how the synapses should change the strengths of their connections, how the connections should form. He said, that's the wrong way to go about it. What you should do is you should think in terms of How you can Adjust connection weights. Solve a problem. So, you define your problem And then you figure out how the adjustment of the connection weights will solve the problem.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

18. “Started a research group, which we called the PDP research group, and several other people came. Francis Crick, who was at the SOC Institute, heard about it from Jeff, because Jeff was known among Brits to be brilliant in Francis, was well connected with his British friends. So Francis Crick came.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

19. “Yeah, yeah. And, but just to say something more about the scientist and the back propagation idea that you mentioned. So in eighty two Hinton had been there as a postdoc and organized that conference. He'd actually gone away and gotten an assistant professorship. And then there was this opportunity to bring him back. So Jeff Hinton was back. On a sabbat San Diego And Rummelhardt and I had decided we wanted to do this, you know, we thought it was really exciting. Are the papers on the interactive activation model that I was telling you about had just been published, and we both sort of saw huge potential for this work. And Jeff was there. And so the three of us.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

20. “Partial competencies still exist in the absence of other aspects of these competencies. So this is what always fascinated me about what used to be called cognitive neuropsychology, you know, the effects of brain damage on cognition. But in particular, this gradual disintegration part.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

21. “After bowling, I took him to lunch and I said, where would you like to go? You want to go to Wendy's? And he said, nah. And I said, okay, well, where do you want to go? And he just pointed. He said, turn here. So he still had a certain amount of spatial cognition and he could get me to the restaurant. And then when we got to the restaurant, I said, what do you want to order? And he couldn't come up with any of the words, but he knew where on the menu the thing was that he wanted. It's, you know, and he couldn't say what it was, but he knew that that's what he wanted to eat. And so it's like it isn't monolithic at all. Our cognition is. You know, first of all, graded in certain kinds of ways, but also multi-partite. There's many elements to it and things, certain sort of”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

22. “But it's so I would say he didn't and this was part of what from the outside was a profound tragedy but but on the other hand at some level he sort of did because Know there was a period of time when you finally was realized that he had really become Profoundly impaired. This was clearly a biological condition, and he wasn't just like he was distracted that day or something like that. So he retired, you know, from his professorship at Stanford, and he became Lived with his brother for a couple years, and then he moved into a facility for people with Cognitive impairment One that many elderly people end up in when they have cognitive impairments. I would spend time with him during that period. This was like in the late 90s, around 2000 even. And, you know, I would, we would go bowling and he could still bowl.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

23. “We can contrast Hawking and Rummelhart in this way. And I like to do that to honor Rummelhart because I think Rummelhart is sort of like the hawking of cognitive science to me in some ways. Both of them suffered from a degenerative condition. In Hawking's case, it affected the motor system. In Romromelhart's case, it's affecting the semantics. And not just the pure object semantics, but maybe the self-semantics as well. And we don't understand that”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

24. “Thoughtful. Person who was willing to work for years to solve a hard problem, you know, he He starts to disappear. There was a period of time when it's like, Hard for any of us to really appreciate that he was sort of in some sense not fully there anymore.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

25. “Distributed representation, a pattern of activation represents the concepts really similar ones. As you degrade them, they start being, you lose the differences. So the difference between the dog and the goat sort of is no longer part of the pattern anymore. And since dog is really familiar, that's the thing that remains. And we understand that in the way the models work and learn. But Rummelheart underwent this condition. So on the one hand, it's a fascinating aspect of parallel distributed processing to be, and it reveals this sort of texture of distributed representation in a very nice way. I've always felt. But at the same time, it was extremely poignant because Is exactly the condition that Rummel Hart was undergoing. And there was a period of time when he was. This man who had been the most focused Goal directed. Competitive.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

26. “So you give the patient a picture of pyramids and they have a choice. Which goes with the pyramids? Palm trees or pine trees? And she showed that this wasn't just a matter of language because the patient's loss of disability shows up whether you present the material with words or with pictures. The pictures They can't put the pictures together with each other properly anymore. They can't relate the pictures to the words either. They can't do word picture matching, but they've lost the conceptual grounding. from either modality of input. And so that's why it's called semantic dementia. The very semantics is disintegrating. And we understand this in terms of our”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

27. “Pigs and goats and sheep, and cause all middle sized animals dogs and all can't recognize rabbits and rodents anymore. They call all the little ones cats and they can't recognize hippopotamuses and cowsing where they call them all horses, you know. So there was this one patient who went through this progression where at a certain point Any four-legged animal he would call it either a horse or a dog or a cat. And if it was big, he would tend to call it a horse. If it was small, he'd tend to call it a cat, middle-sized ones he called dogs. This is just a part of the syndrome, though the patient loses the ability to relate concepts to each other. So my collaborator in this work, Carolyn Patterson, developed a test called the pyramids and palm trees test.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

28. “The disorder is something my colleagues and collaborators have chosen to call semantic dementia. So it's a specific form of Loss of mind related to meaning semantic dementia. And it's progressive in the sense that the patient Loses the ability Appreciate the meaning of the experiences that they have, either from touch, from sight, from sound. From language, I hear sounds, but I don't know what they mean kind of thing. So, as this illness progresses, it starts with The patient being unable to differentiate similar breeds of dog or remember, you know, the lower frequency unfamiliar categories that they used to be able to remember. But as it progresses, it becomes more and more striking. And the patient loses the ability to recognize. You know, things like”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

29. “Absolutely. He passed away 15 ish years ago now His demise was actually one of the most Poignant and Relevant. Relevant to our conversation. He Started to undergo a progressive neurological condition. Isn't fully understood. That is to say his particular course isn't fully understood Because brain scans weren't done at certain stages and no autopsy was done or anything like that. The wishes of the family. So we don't know as much about the underlying pathology as we might. I had begun to get interested in this neurological condition that might have been the very one that he was succumbing to as my own efforts to understand another aspect of this mystery that we've been discussing. While he was beginning to get progressively more and more affected. So I'm going to talk about the disorder and not about Rummelhart for a second, okay?”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

30. “Well, who knows whether there aren't some illusory characteristics there. And I think that philosophically many people have confronted that possibility over time. But it's still important to accept it as magic, right? Know, I think of Fellini and this context, I think of others who have appreciated the role of magic, of actual trickery in creating illusions that move Move us. And Plato was on to this too. It's like somehow or other these shadows, you know, give rise to something much deeper than that. And that's. So, you know, we won't try to figure out what it is. We'll just accept it as given that that occurs. Know, but he was still onto the magic of it.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

31. “Yeah, no, I agree. I'm not, that's why I got rid of eliminative, right? Yeah, because it seemed like that was trying to say that it's all. Completely”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

32. “Sort of all of the stuff that allows them to change in the ways that they do. And that's where I think the connectionist feeds into the cognitive. It's like, okay, so if the substrate is parallel distributed connectionist, then it doesn't mean that the contents of thought isn't abstract and symbolic. It's more fluid, maybe than is easier to capture with a set of logical expressions.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

33. “Don't exist as such. And so there was an example that Doug Hofsteder used to use that I thought was helpful in this respect. Just the idea that We could think about sand dun As entities. And talk about how many there are even. But we also know that a sand dune is a very fluid thing. It's a pile of sand that is capable of moving around under the wind. And Reforming itself in somewhat different ways. And if we think about our thoughts as like sand dunes as being things that emerge from Just the way all the lower level elements sort of work together and are constrained by external forces. Then we can say yes, they exist as such, but they also. You know. We shouldn't treat them as completely monolithic entities that we can understand without understanding”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

34. “I pay $64,000. Maybe it's the $64 billion question now. I think that From Emergentist side, which you know I place myself on. So I used to sometimes tell people I was a radical eliminative connectionist because I didn't want them to think that I wanted to build like anything into the machine. But I don't like the word eliminative anymore because it makes it seem like It's wrong to think that there is this emergent level of understanding. I disagree with that. So I think I would call myself an radical emergentist connectionist rather than eliminative connectionist, right? Because I want to acknowledge that. These higher level kinds of aspects of our cognition are Are real, but they're not, they don't.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

35. “It never worked, right? You couldn't ever write down a set of propositions for visual recognition. And so in that space, it sort of always seemed very natural that something more implicit, you know, you don't have access to what the details of the computation were in between. You just get the result. So that's the other part of connectionism. You cannot, you don't read the contents of the connections. The connections only cause outputs to occur based on inputs.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

36. “Or there's my own dog and I recognize my dog Which is a member of the same species as many other dogs, but I know this one because of some slightly unique characteristics. I don't know how to describe it. What it is that makes me know that I'm looking at Lex or at my particular dog, right? Or even that I'm looking at a particular brand of car, like I can say a few words about it, but I wrote you a paragraph about the car. You would have trouble figuring out which car is he talking about, right? So, the idea that we have propositional knowledge of what it is that allows us to recognize that this is an actual instance of this particular natural kind has always been...”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

37. “Yeah, but let's go back to that CNN, right? That CNN with all those layers of neuron like processing units that we were talking about before, it's going to come out and say, this is a cat, that's a dog. It has no idea why it said that. It's just got all these connections between all these layers of neurons, like from the very first layer to the whatever these layers are, they just get numbered after a while because they somehow further in you go, the more The more abstract the features are, but it's a graded and continuous sort of process of abstraction anyway. And it goes from very local, very, very specific to much more sort of global, but it's still another sort of pattern of activation over an array of units. And then at the output side, it says it's cat or it's a dog. And when I open my eyes and say, oh, that's Lex.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

38. “The word time isn't written anywhere inside the bottle. It's only written there in the picture we drew of the model to say that's the unit for the word time, right? And if somebody wants to tell me, well, how do you spell that word? You have to use the connections from that out to then get those letters, for example.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

39. “Think it's useful It highlights the notion that the knowledge that the system exploits is in the connections between the units, right? There isn't a separate dictionary. There's just the connections between the units. So I already sort of laid that on the table with the connections from the letter units to the unit for the word time. The unit for the word time isn't a unit for the word time for any other reason than it's got the connections to the letters that make up the word time. Those are the units on the input that excite it when it's excited that in a sense represents in the system that There's support for the hypothesis that the word time is present in the input. But it's not”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

40. “Because these connections are bidirectional. You have prior knowledge that it might be the word time that starts to prime the feature, the letters and the features. And if you don't, then it has to start bottom up. But the directionality just depends on where the information comes in first. And if you have context together with features at the same time, they can convergently result in an emergent perception. That was the Piece of work that we did together that sort of got us both completely convinced that this neural network way of thinking was going to be able to actually address the questions that we were interested in cognitive psychology.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

41. “And what he realized kind of after reading Hinton's dissertation and hearing about Jim Anderson's linear algebra-based neural network models that I was telling you about before was that he could replace those experts with neuron-like processing units, which just would have their connection weights that would do this job. What ended up happening was that Rommelhardt and I got together and we created a model called the Interactive Activation Model of Letter Perception. Which is takes these little pixel level inputs, constructs line segment features, letters, and words, but now we built it out of a set of neuron-like processing units that are just connected to each other with connection weights. So the unit for the word time has a connection to the unit for the letter T in the first position and the letter I in the second position, so on.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

42. “Yeah. But the thing is, though, I want to just sort of relate this to the earlier part of the conversation. When Rommel Hart was first thinking about it, there were these experts on the side. One for the features and one for the letters and one for how the letters make the words and so on. And they would each be working sort of evaluating various propositions about, you know, is this combination of features here going to be one that looks like the letter T and so on?”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

43. “Construct a line out of pixels, and another expert about which sets of lines go together to make which letters, and another one about which letters go together to make mich words, and another one about what the meanings of the words are, and another one about how the meanings fit together and, you know, things like that. And all these experts are looking at this data and they're updating hypotheses at other levels. So the word expert can tell the letter expert, oh, I think there should be a T there because I think there should be a word the here and the bottom up sort of feature to letter expert could say, I think there should be a T there too. And if they agree, then you see a T, right? And so there's a top-down, bottom-up, interactive process, but it's going on at all layers simultaneously. So everything can filter all the way down from the top as well as all the way up from the bottom. And it's a completely interactive bider.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

44. “At every level of analysis you can think of actually depends on all the other levels of analysis. So What are the actual pixels making up each letter? What did those pixels signify about which letters they are? And what are those letters tell us about What words are there? And what are those words tell us about what ideas the author is trying to convey and what? So he had this model where we have these little tiny. Elements that represent each of the pixels, each of the letters and then other ones that represent the line segments in them and other ones that represent the letters and other ones that represent the words. At that time, his idea was there's this set of experts. There's an expert about how to”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

45. “Yeah, yeah, by the late 70s, it was kind of old fashioned and it hadn't really panned out, you know, and people were beginning to recognize that, and Rummel Hart was, you know, like, yeah, he's part of the recognition that this wasn't all working. Anyway, so he started thinking in terms of The idea that we needed systems that allowed us to integrate multiple simultaneous constraints in a way that would be mutually influencing each other So, um He wrote a paper that just really, first time I read it, I thought, oh, well, you know, yeah. But is this important? But after a while it just got under my skin and it was called an interactive model of reading. And in this paper, he laid out the idea that every aspect Interpretation of What's coming off the page when we read.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

46. “Whatever the heck that is. Yeah, he was grappling that this was something that they grappled with at the end of that book that I was describing, Explorations and Cognition.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

47. “Well, there's a couple of ideas you could have, but the most natural one is that the good humor man brings ice cream. She likes ice cream. She knows she needs money to buy ice cream, so she's going to run into the house and get her money so she can buy herself an ice cream. It's a huge amount of inference that has to happen to get those things to link up with each other. And he was interested in how the hell that could happen. And he was trying to build good old-fashioned AI style models of representation of language and content of. Things like has money.”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

48. “Doing mathematical modeling. But he had gotten interested in Cognition. He'd gotten interested in. Understanding And understanding, I think, remains. What does it mean to understand anyway? It's an interesting sort of curious, you know, like how would we know if we really understood something? But he was interested in building machines that would hear a couple of sentences and have an insight about what was going on. So for example, one of his favorite things at that time was. Um. Marky was sitting on the front step when she heard the familiar jingle of the good humor man. She remembered her birthday money and ran into the house. What is Margie doing? Why?”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

49. “Probability that the subject will be correct on the seventh trial or the experiment is or something like that, right? So it's a use of mathematics to descriptively characterize aspects of behavior. And Stanford at that time was the place where there were several really, really strong mathematical thinkers who were also connected with three or four others around the country who brought a lot of really exciting ideas onto the table. And it was a very, very prestigious part of the field of psychology at that time. So Rummelhart comes into this. He was a very strong student within that program. And he got this job at this brand new university in San Diego in 1967. He's one of the first. Assistant professors in the Department of Psychology at UCSD I got there in seventy four, seven years later. Rumhardt at that time”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source

50. “Yeah, it's mathematical in the sense that You say This is true, and that is true, then I can derive that this should follow. And so you say these are my stipulations about the fundamental principles, and this is my prediction about behavior. And it's all done with equations. It's not done with a computer simulation. So you solve the equation and that tells you what the”

    2021-09-20 · Lex Fridman Podcast · #222 – Jay McClelland: Neural Networks and the Emergence of Cognition · IDENTIFIED FROM THE TRANSCRIPT · source