YouSaid · the spoken record

Andrej Karpathy

lines on the record: 266
first: 2022-10-29
most recent: 2022-10-29
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“I sold fast, but I do intermittent fasting. But really, what it means at the end of the day is I skip breakfast. So, I do 18.6 roughly by default when I'm in my steady state. If I'm traveling or doing something else, I will break the rules. But in my steady state, I do 18.6. So I eat only from 12 to 6. Not a hard rule and I break it often, but that's my default. And then, yeah, I've done a bunch of random experiments. For the most part, right now, where I've been for the last... I don't actually know the differences, but it sounds better in my mind. But it just means I prefer plant based food”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“I was thinking about it. Like, suppose I did all these things but did not share them. I don't think I would have the same amount of motivation that I can build up”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Thing to exist. Yeah, it needs to exist. And then I think to me also a big factor is are other humans going to appreciate it? Are they going to like it? A big part of my motivation if I'm helping humans and they seem happy, they say nice things, they tweet about it or whatever, that gives me pleasure because I'm doing something useful.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“I can probably go a small few hours and then I need some breaks in between for food and stuff. And yeah, but I think it's still really hard to accumulate hours. I was using a tracker that told me exactly how much time I spent coding any one day. And even on a very productive day, I still spent only like six or eight hours. And it's just because there's so much padding, commute, talking to people. There's like the cost of life just living and sustaining and homeostasis and just maintaining yourself as a human is very high”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“I do find it interesting to know about the world. I don't know that it's useful or good, but it's part of my routine right now. So I do read through a bunch of news articles and I want to be informed. And I'm suspicious of it. I'm suspicious of the practice, but currently that's where I am.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“On and so on And I mean, it can take some time off for distractions and in between, but I think it can't be too much. Most of your day is sort of like spent on that problem. And then I... Coffee, I have my morning routine, I look at some news Twitter, hacker news, Wall Street Journal, et cetera”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Basically, I need to load my working memory with the problem and I need to be productive because there's always a huge fixed cost to approaching any problem. I was struggling with this, for example, at Tesla because I want to work on small side project. But okay, you first need to figure out, oh, okay, I need to SSH into my cluster. I need to bring up a VS Code editor so I can work on this. I need to run into some stupid error because of some reason. You're not at a point where you can be just productive right away. You are facing barriers. And so it's about really removing all of that barrier and you're able to go into the problem and you have the full problem loaded in your memory.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“A month. So I can't talk about one day basically in isolation because it's a whole process. When I want to get productive in the problem, I feel like I need a span of a few days where I can really get in on that problem. And I don't want to be interrupted. And I'm going to just be completely obsessed with that problem. And that's where I do most of my good work, I would say.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“When you're falling asleep, you need to be obsessed with the problem and it's fully in your memory, and you're ready to wake up and work on it right there.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Stable or not, it's semi stable, like eight or nine or something like that. During my PhD, it was even later. I used to go to sleep usually at 3 a.m. I think the AM hours are precious and very interesting time to work because everyone is asleep. At 8 a.m. or 7 a.m., the East Coast is awake. So there's already activity. There's already some text messages, whatever. There's stuff happening. You can go on some news website and there's stuff happening. It's distracting. At 3 a.m. everything is totally quiet. And so you're not going to be bothered and you have solid chunks of time to do work. So I like those periods, night owl by default. And then I think like productive time basically, what I like to do is you need to build some momentum on the problem without too much distraction. And you need to load your RAM, your working memory with that problem. And then you need to be obsessed with it when you're taking shower.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Sudden perspective. Yeah, I'm not super huge fan of, I think, all these interfaces that look very different. I would want everything to be normalized into the same API. So for example, screen pixels, very same API, instead of having different world environments that have very different physics and joint configurations and appearances and whatever, and you're having some kind of special tokens for different games that you can plug, I'd rather just normalize everything to a single interface. So it looks the same to the neural net, if that makes sense.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“I think so. So Gatto is very much a kitchen sink approach to reinforcement learning lots of different environments with a single fixed transformer model. I think it's a very sort of early result in that realm. But I think, yeah, it's along the lines of what I think things will eventually look like.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Not just text, right? You're giving it gadgets and gizmos. You're teaching some kind of a special language by which it can save arbitrary information and retrieve it at a later time. And you're telling about these special tokens and how to arrange them to use these interfaces. It's like, hey, you can use a calculator. Here's how you use it. Just do 5, 3 plus 41 equals. And when equals is there, a calculator will actually read out the answer and you don't have to calculate it yourself. And you just tell it in English. This might actually work.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yes, but I don't know to what extent it will be explicitly constructed. It might take unintuitive forms where you are telling the GPT, like, hey, you have a declarative memory bank to which you can store and retrieve data from. And whenever you encounter some information that you find useful, just save it to your memory bank. And here's an example of something you have retrieved and how you say it. And here's how you load from it. You just say load, whatever, you teach it in text in English. And then it might learn to use a memory bank from that.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“I think it's just like the hardware for long term memory is just not fully developed. Sure. I kind of feel like the first few years of infants is not actually like learning. It's brain maturing. We're born premature. And there's a theory along those lines because of the birth canal and the swallowing of the brain. And so we're born premature. And then the first few years we're just that the brain's maturing. And then there's some learning eventually. That's my current view on it.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“I think humans definitely, I mean, obviously, we learn a lot during our lifespan, but also we have a ton of hardware that helps us initial at initialization coming from sort of evolution. And so I think that's also a really big component. A lot of people in the field, I think they just talk about the amounts of like seconds and that a person has lived pretending that this is a Tabul Arasa, sort of like a zero initialization of a neural net. And it's not. You can look at a lot of animals, like, for example, zebras. zebras get born and they see and they can run there's zero train data in their lifespan they can just do that so somehow i have no idea how evolution has found a way to encode these algorithms and these neural net initializations that are extremely good into atcgs and i have no idea how this works but apparently it's possible because here's a proof by existence”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“100%. I just think at some point you need a massive data set. And then when you pre train your massive neural nut and get something that is like a GPT or something, then you're able to be very efficient at training any arbitrary new task. So a lot of these GPTs, you know, you can do tasks like sentiment analysis or translation or so on just by being prompted with very few examples. Here's the kind of thing I want you to do. Here's an input sentence. Here's the translation into German. Input sentence, translation to German. Input sentence blank and the neural network will complete the translation to German just by looking at sort of the example you've provided. And so that's an example of a very few shot learning in the activations of the neural net instead of the weights of the neural nut. And so I think basically just like humans, neural nets will become very data efficient at learning any other new task. But at some point you need a massive data set to pre-train your network.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Maybe not. I don't see it as a fundamental really important part of training neural nets currently. But I think as neural nets become more and more powerful, I think you will need fewer examples to train additional behaviors. And simulation is, of course, there's a domain gap in a simulation that is not the real world. There's slightly something different. But with a powerful enough neural net, you need the domain gap can be bigger, I think, because neural net will sort of understand that even though it's not the real world, it like has all this high level structure that I'm supposed to be able to learn from.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“I Humans use simulators. For humans use simulators and they find them useful. And so computers will use simulators and find them useful.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, but that's independent from the use of simulation in the sense of computer games or using simulation for training set creation.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“I think as neural nets converge to humans The value of simulation to neural nets will be similar to the value of simulation to humans. So, people use simulation because they can learn something in that kind of a system without having to actually experience it.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Was the right amount of difficult and simple and interesting enough? It just kind of like it was the right time for that kind of a data set.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Unfortunately, I don't think academics currently have the next image net. We've obviously, I think we've crushed MNIST, we've basically kind of crushed ImageNet, and there's no next sort of big benchmark that the entire community RAL is behind and uses for further development of these networks.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, ImageNet has been crushed. I mean, you know, the error rates are. We're getting like 90% accuracy in 1000 classification way prediction. And I've seen those images. And that's like really high. That's really good. If I remember correctly, the top five error rate is now like 1% or something.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“I don't know that I recall the specific instance where I was unhappy or criticizing ImageNet. I think ImageNet has been extremely valuable. It was basically a benchmark that allowed the deep learning community to demonstrate that deep neural networks actually work. There's a massive value in that. So I think ImageNet was useful, but basically it's become a bit of an MNIS at this point. So MNIST is like the little 28 by 28 grayscale digits. There's kind of a joke data set that everyone just crushes.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Some people are also very negative and very vocal, so they're very prominently featured. But actually, there's a ton of people who are cheerleaders, but they're silent cheerleaders. And when you talk to people just in the world, they will tell you it's amazing, it's great, especially like people who understand how difficult it is to get this stuff working. Like people who have built products and makers and entrepreneurs, like making this work and changing something is incredibly hard. Those people are more likely to cheerlead you.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I think the tricky thing is like some people really love you, some people unfortunately you're working on something that you think is extremely valuable, useful, et cetera. Some people do hate you. There's a lot of people who hate me and the team and the whole project.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yes, 100%. You're deploying this. People like it. People drive it. People pay for it. They care about it. There's all these YouTube videos. Your grandma drives it. She gives you feedback. People like it. People engage with it. You engage with it. Huge.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Want to set up your data engine, your improvement loops, the telemetry, the evaluation, the harness, and everything. And you want to improve the product over time incrementally, and you're making revenue along the way. That's extremely important because otherwise you cannot build these large undertakings just like don't make sense economically. And also from the point of view of the team working on it, they need the dopamine along the way. They're not just going to make a promise about this being useful. This is going to change the world in 10 years when it works. This is not where you want to be. You want to be in a place like I think Autopald is today where it's offering increased safety and convenience of driving today. People pay for it. People like it. People purchase it. And then you also have the greater mission that you're working towards.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“I think it's going to take a long time, but the way you want to structure the development is you need to say, okay, it's going to take a long time. How can I set up the product development roadmap so that I'm making revenue along the way? I'm not setting myself up for a zero one loss function where it doesn't work until it works. You don't want to be in that position. You want to make it useful almost immediately. And then you want to slowly deploy it and at scale.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“It thinks it's a car. Some of the earlier demos, actually, we were talking about potentially doing them outside in the parking lot because that's where all of the computer vision that was like working out of the box instead of inside. But all the operating system, everything just copy pastes computer vision, mostly copy paste. I mean, you have to retrain the neural nets, but the approach and everything in data engine and offline trackers and the way we go about the occupancy tracker and so on, everything copy paste. You just need to retrain the neural nuts. And then the planning control, of course, has to change quite a bit. But there's a ton of copy-paste from what's happening at Tesla. And so if you were to go with goal of like, okay, let's build a million human robots and you're not Tesla, that's a lot to ask. If you're Tesla, it's actually like it's not that crazy.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“And the reason that happened very quickly is, as you alluded to, there's a ton of copy-paste from what's happening in the autopilot a lot. The amount of expertise that came out of the woodworks at Tesla for building the human robot was incredible to see. Basically, Elon said at one point we're doing this. And then next day, basically, like all these CAD models started to appear. And people talking about the supply chain and manufacturing. And people showed up with like screwdrivers and everything the other day and started to put together the body. And I was like, whoa, like all these people exist at Tesla. And fundamentally, building a car is actually not that different from building a robot. And that is true, not just for the hardware pieces. And also, let's not forget hardware, not just for demo, but manufacturing of that hardware at scale is a whole different thing. But for software as well, basically this robot currently thinks it's a car.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I think it's a very hard project. I think it's going to take a while. But who else is going to build human robots at scale? And I think it is a very good form factor to go after because, like I mentioned, the world is designed for human and form factor. These things would be able to operate our machines. They would be able to sit down in chairs, potentially even drive cars. Basically, the world is designed for humans. That's the form factor you want to invest into and make work over time. I think there's another school of thought, which is, okay, pick a problem and design a robot to it. But actually designing a robot and getting a whole data engine and everything behind it to work is actually incredibly hard problem. So it makes sense to go after general interfaces that, okay, they are not perfect for any one given task, but they actually have the generality of just with a prompt with English able to do something across. And so I think it makes a lot of sense to go after a general interface in the physical world. And I think it's a very difficult project. I think it's going to take time. But I see no other company that can execute on that vision. I think it's going to be amazing.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“And I like Terminator 1 as well. So, okay, so few exceptions, but by and large, for some reason, I don't like movies before 1995 or something. They feel very slow. The camera is like zoomed out. It's boring. It's kind of naive. It's kind of weird.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“I think I will make a strong statement. I don't know why. I don't know why, but I basically don't like any movie before 1995, something like that.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Human robots are going to be amazing. I think autonomous transportation is going to be amazing. All this is happening at Tesla. So I think it's just a really amazing organization. So being part of it and helping it along, I think was very basically I enjoyed that a lot. Yeah, it was basically difficult for those reasons because I love the company. But I'm happy to potentially at some point come back for Act 2. But I felt like at this stage, I built the team. It felt autonomous. And I became a manager and I wanted to do a lot more technical stuff. I wanted to learn stuff. I wanted to teach stuff. And I just kind of felt like it was a good time for a change of pace a little bit.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, it was hard because obviously I love the company a lot and I love Elon, I love Tesla. So it was hard to leave. I love the team, basically. But yeah, I think actually I would be potentially interested in revisiting it, maybe coming back at some point, working in Optimus, working in AGI at Tesla. I think Tesla is going to do incredible things. It's basically like... It's a massive large scale robotics kind of company for the ton of in-house talent for doing really incredible things. And I think.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, and so Kind of like grew that into what I think is a fairly respectable deep learning team, a massive compete cluster, a very good data annotation organization. And I was very happy with where that was. It became quite autonomous. And so I kind of stepped away. I'm very excited to do much more technical things again. Yeah. And kind of like we focus on AGI.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Basically, as I described, Ed Ren, I think over time during those five years, I've kind of gotten myself into a little bit of a managerial position. Most of my days were meetings and growing the organization and making decisions about sort of high-level strategic decisions about the team and what it should be working on and so on. And it's kind of like a corporate executive role. And I can do it. I think I'm okay at it, but it's not fundamentally what I enjoy. And so I think when I joined, there was no computer vision team because Tesla was just going from the transition of using Mobile, a third-party vendor for all of its computer vision, to having to build its computer vision system. So when I showed out, there were two people training deep neural networks and they were training them at a computer at their legs.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“You're making progress and you see what the next directions are, and you're looking at some of the remaining challenges, and they're not like perturbing you and they're not changing your philosophy and you're not contorting yourself. You're like, actually, these are the things that we still need to do.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I would say definitely to use a game analogy, there's some fog of war, but you definitely also see the frontier of improvement. And you can measure historically how much you've made progress. And I think, for example, at least what I've seen in roughly five years at Tesla, when I joined, it barely kept lane on the highway. I think going up from Palo Alto to SF was like three or four interventions. Anytime the road would do anything geometrically or turn too much, it would just like not work. And so going from that to a pretty competent system in five years and seeing what happens also under the hood and what the scale at which the team is operating now with respect to data and compute and everything else is just massive progress.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“I would say what's easy to say is that this problem is tractable, and that's an easy prediction to make it's tractable, it's going to work Yes, it's just really hard. Some things turn out to be harder. But it definitely feels tractable and it feels like at least the team at Tesla, which is what I saw internally, is definitely on track to that.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“I think the tough thing with timelines of self driving, obviously, is that no one has created self-driving. So it's not like, what do you think is a timeline to build this bridge? Well, we've built million bridges before. Here's how long that takes. No one has built autonomy. It's not obvious. Some parts turned out to be much easier than others. So it's really hard to forecast. You do your best based on trend lines and so on and based on intuition. But that's why fundamentally it's just really hard to forecast this. No one has done it.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yep. I mean, I think a good example here is the deep learning revolution in some sense because you could be in computer vision at that time during the deep learning sort of revolution of 2012 and so on. You could be improving a computer vision stack by 10% or it can just be saying actually all this is useless. And how do I do 10x better computer vision? Well, it's not probably by tuning a hog feature detector. I need a different approach. I need something that is scalable going back to Richard Sutton's and understanding sort of like the philosophy of the bitter lesson and then being like, actually, I need much more scalable system like a neural network that in principle works and then having some deep believers that can actually execute on that mission and make it work. The 10X solution.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Say that setting impossible goals exactly is a good idea, but I think setting very ambitious goals is a good idea. I think there's what I call sublinear scaling of difficulty, which means that 10x problems are not 10x hard. Usually 10x harder problem is like 2 or 3x harder 2x execute on. Because if you want to actually improve a system by 10%, it costs some amount of work. And if you want to 10x improve the system, it doesn't cost 100x amount of work. And it's because you fundamentally change the approach. And if you start with that constraint, then some approaches are obviously dumb and not going to work. And it forces you to reevaluate. And I think it's a very interesting way of approaching problem solving.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“I do think you need someone in a powerful position with a big hammer like Elon, who's like the cheerleader for that idea and ruthlessly pursues it. If no one has a big enough hammer, everything turns into committees, democracy within the company, process, talking to stakeholders, decision making, just everything just crumbles. If you have a big person who is also really smart and has a big hammer. Things move quickly.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“In the form of meetings and that kind of stuff. Yeah, meetings. He hates meetings. He keeps telling people to skip meetings. If they're not useful, he basically runs the world's biggest startups, I would say. Tesla SpaceX are the world's biggest startups. Tesla actually is multiple startups. I think it's better to look at it that way. And so I think he's extremely good at that. And yeah, he's a very good intuition for streamlining processes, making everything efficient. Best part is no part, simplifying, focusing, and just kind of removing barriers, moving very quickly, making big moves. All this is a very startup-y sort of seeming things, but at scale.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“The most I've learned is about how to sort of run organizations efficiently and how to create efficient organizations and how to fight entropy in an organization.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“Humans don't need it. So it's very useful to have a low level map of like, okay, the connectivity of your road, you know that there's a fork coming up when you drive an environment, you sort of have that high level understanding. It's like a small Google map. And Tesla uses Google Map, like similar kind of resolution information in its system, but it will not pre-map environments to send me a level accuracy. It's a crutch. It's a distraction. It costs entropy and it diffuses the team. It dilutes the team. And you're not focusing on what's actually necessary, which is the computer vision problem.”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source
“You've mentioned they pre-map all the environments and they need to refresh the map. And they have a perfect centimeter level accuracy map of everywhere they're going to drive. It's crazy. How are you going to, when talking about autonomy actually changing the world, we're talking about a deployment on a global scale of autonomous systems for transportation? And if you need to maintain a centimeter accurate map for Earth or for many cities and keep them updated, it's a huge dependency that you're taking on, huge dependency. It's a massive, massive dependency. And now you need to ask yourself, do you really need it?”
2022-10-29 · Lex Fridman Podcast · #333 – Andrej Karpathy: Tesla AI, Self-Driving, Optimus, Aliens, and AGI · IDENTIFIED FROM THE TRANSCRIPT · source