YouSaid · the spoken record

Andrej Karpathy

lines on the record
266
first
2022-10-29
most recent
2022-10-29
sittings or episodes
1
sources
podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

  1. “And especially when you get into the details of it, I used to think Origin of Life was very. It was this magical rare event, but then you read books like, for example, Nick Lane, The Vital Question, Life Ascending, etc. And he really gets in and he really makes you believe that this is not that rare.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source

  2. “Yeah, I've been preoccupied with this question quite a bit recently, basically the Fermi paradox and just thinking through. And the reason actually that I am very interested in the origin of life is fundamentally trying to understand how common it is that there are technological societies out there in space. And the more I study it, the more I think that There should be quite a lot.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source

  3. “I'm hesitant to say that it is rare, by the way, but it definitely seems like it's kind of like a punctuated equilibrium where you have lots of exploration and then you have certain leaps, sparse leaps in between. So of course like Origin of Life would be one, DNA, sex, eukaryotic life, the endosymbiosis event where the Archeon ate little bacteria, just the whole thing. And then, of course, emergence of consciousness and so on. So it seems like definitely there are sparse events where massive amount of progress was made. But yeah, it's kind of hard to pick one.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source

  4. “For me as an artificial intelligence researcher, it's probably the last piece. We have lots of animals that are not building technological society, but we do. And it seems to have happened very quickly. It seems to have happened very recently. And something very interesting happened there that I don't fully understand. I almost understand everything else, I think, intuitively, but I don't understand exactly that part and how quick it was.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source

  5. “Certainly, I would say it's an extremely remarkable story that I'm only like briefly learning about recently, all the way from actually you almost have to start at the formation of Earth and all of its conditions and the entire solar system and how everything is arranged with Jupiter and Moon and the habitable zone and everything. And then you have an active Earth that's turning over material. And then you start with biogenesis and everything. And so it's all like a pretty remarkable story. I'm not sure that I can pick like a single unique piece of it that I find most interesting.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source

  6. “That said, evolution has found that it is very useful to predict and have a predictive model in the brain. And so I think our brain utilizes something that looks like that as a part of it. But it has a lot more gadgets and gizmos and value functions and ancient nuclei that are all trying to make you survive and reproduce and everything else.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source

  7. “Are complicated alien artifact. I do not make analogies to the brain because I think the optimization process that gave rise to it is very different from the brain. So there was no multi-agent self-play kind of setup and evolution. It was an optimization that is basically what amounts to a compression objective on a massive amount of data.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source

  8. “I would say I'm definitely on, I'm much more hesitant with the analogies to the brain than I think you would see potentially in the field. And I kind of feel like Certainly the way neural networks started is everything stemmed from inspiration by the brain. But at the end of the day, the artifacts that you get after training, they are arrived at by a very different optimization process than the optimization process that gave rise to the brain. And so I think I kind of think of it as a very complicated alien artifact. It's something different. The neural nest that we're training.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source

  9. “A lot of knobs, and somehow. So speaking concretely, one of the neural nets that people are very excited about right now are GPTs, which are basically just next word prediction networks. So you consume a sequence of words from the internet and you try to predict the next word. Once you train these on a large enough data set, you can basically prompt these neural nets in arbitrary ways and you can ask them to solve problems and they will. So you can just tell them you can make it look like you're trying to. Solve some kind of a mathematical problem, and they will continue what they think is the solution based on what they've seen on the internet. And very often those solutions look very remarkably consistent, look correct, potentially.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source

  10. “It's kind of interesting because I'm simultaneously underselling them, but I also feel like there's an element to which I'm over, like it's actually kind of incredible that you can get so much emergent magical behavior out of them despite them being so simple mathematically. So I think those are kind of like two surprising statements that are kind of juxtaposed together. And I think basically what it is is we are actually fairly good at optimizing these neural nets. And when you give them a hard enough problem, they are forced to learn very interesting solutions in the optimization. And those solutions basically have these immersion properties that are very interesting.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source

  11. “Yeah, could be. I mean, I'm using phrases that are common, et cetera, but I'm remixing it into pretty sort of unique sentence at the end of the day. But you're right, definitely, there's like a ton of remixing.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source

  12. “It definitely feels like you're referencing some kind of a declarative structure of memory and so on. And then you're putting that together with your prompt and giving away some aspects.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source

  13. “Is definitely some kind of a generative model that's a GPT like and prompted by you. Yeah, so you're giving me a prompt and I'm kind of like responding to it in a generative way.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source

  14. “Yeah, I think that's fair. So basically, I'm underselling it by a lot because you definitely do get very surprising emergent behaviors out of these neural nets when they're large enough and trained on complicated enough problems, like say, for example, the next word prediction in a massive data set from the internet. And then these neural nets take on pretty surprising magical properties. Yeah, I think it's kind of interesting how much you can get out of even very simple mathematical formalism.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source

  15. “Many knobs, and these knobs are loosely related to basically the synapses in your brain, they're trainable, they're modifiable. And so the idea is like we need to find the setting of the knobs that makes the neural net do whatever you want it to do, like classify images and so on. And so there's not too much mystery, I would say, in it. You might think that basically don't want to endow it with too much meaning with respect to the brain and how it works. It's really just a complicated mathematical expression with knobs, and those knobs need a proper setting for it to do something desirable.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source

  16. “What is a neural network? It's a mathematical abstraction of Brain, I would say that's how it was originally developed. At the end of the day, it's a mathematical expression. It's a fairly simple mathematical expression when you get down to it. It's basically a sequence of matrix multiplies, which are really dot products mathematically, and some nonlinearity is thrown in. And so it's a very simple mathematical expression, and it's got knobs in it.”

    2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source