YouSaid · the spoken record

Jim Keller

lines on the record
215
first
2021-02-18
most recent
2021-02-18
sittings or episodes
1
sources
podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

  1. “Well, it's worse than that. It'll understand physics in ways that we can't understand. I liked your Stephen Wolfram talk where he said, you know, there's three generations of physics. There was physics by reasoning. Well, big things should fall faster than small things, right? That's reasoning. And then there's physics by equations. But the number of programs in the world that are solved with the single equations is relatively low. Almost all programs have more than one line of code, maybe 100 million lines of code. So you said now we're going to physics by equation, which is his project, which is cool.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  2. “The more data you have, the better it gets. So then you start to wonder, well, is that a fundamental thing or is this just another step to some fundamental understanding about this kind of computation? Which is really interesting”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  3. “And the other thing is just like the Tesla Autonomous Driving hardware, it was only serving one software stack. And the hardware team and the software team were tightly coupled. You know, if you're building a general purpose AI solution and there's so many different customers with so many different needs. Now, something Andre said is, I think this is amazing. 10 years ago, vision, recommendation, language were completely different disciplines. He said the people literally couldn't talk to each other. And three years ago, it was all neural networks, but the very different neural networks. And recently it's converging on one set of networks. They vary a lot in size. Obviously, they vary in data, vary in outputs, but the technology has converged a good bit.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  4. “First, I think it's great. So we need lots of experiments, right? And there's lots of startups working on this and they're pursuing different things. I was there when we started Dojo and it was sort of like, what's the unconstrained computer solution to go do very large training problems? And then there's fun stuff like, you know, we said, well, we have this 10,000 watt board to cool. Well, you go talk to guys at SpaceX and they think 10,000 watts is a really small number, not a big number. And there's brilliant people working on it. I'm curious to see how it'll come out. I couldn't tell you. I know it pivoted a few times since I left.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  5. “Yeah, that's probably what happens autonomous cars will have a small number of accidents. Humans would have avoided, but they'll wipe. Get rid of the bulk of them.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  6. “It's going to be one of these compensating things. So when you're driving, you have an intuition about what humans are going to do, but you don't have 360 cameras and radars and you have an attention problem. So the self-driving car comes in with no attention problem, 360 cameras, a bunch of other features So they'll wipe out a whole class of accidents. And emergency braking with radar and especially as it gets AI enhanced will eliminate collisions. But then you have the other problems of these unexpected things where you think your human intuition is helping, but then the cars also have a set of hardware features that you're not even close to.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  7. “We'll see. There's another funny thing you don't learn to drive with infinite amounts of data. You learn to drive with an intellectual framework that understands physics and color and horizontal surfaces and laws and roads and, you know, all your. Your experience from manipulating your environment. Like, look, there's so many factors go into that. So then when you learn to drive, like driving is a subset of this conceptual framework that you have. And so with self-driving cars right now, we're teaching them to drive with driving data. You never teach a human to do that. You teach a human all kinds of interesting things, like language. Like, don't do that. You know, watch out. You know, there's all kinds of stuff going on.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  8. “Right. It'll create unlimited amount of data, which then they'll scale. Now, the networks that may use that data might be way too big for cars, but then there'll be the transformation from now we have unlimited data. I know exactly what I want. Now can I turn that into something that fits in the car. And that process is going to happen all over the place. Every time you get to the place where you have unlimited data, and that's where it software 2.0 is about, unlimited data training networks to do stuff. Without humans writing code to do it”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  9. “And there's two pieces of it. You find edge cases that don't work, and then you define something that goes get you data for that. But then the other constraint is whether you have to label it or not. Like the amazing thing about the GPT-3 stuff is it's unsupervised. So there's essentially infinite amount of data. Now, there's obviously infinite amount of data. Available from cars if people successfully driving. But the current pipelines are mostly running on labeled data, which is human limited. So when that becomes unsupervised.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  10. “Making progress, it's taken longer than anybody thought. My wonder was, is hardware three, is it enough computing, off by two, off by five, off by ten, off by 100? And I thought it probably wasn't enough, but they're doing pretty well with it now. And one thing is. The data set gets bigger, the training gets better, and then there's this interesting thing you sort of train and build an arbitrary size network that solves the problem, and then you refactor the network down to the thing that you can afford to ship. Right. So the goal isn't to build the network that fits in the phone. It's to build something that actually works. And then how do you make that most effective on the hardware you have? And they seem to be doing that much better than a couple years ago.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  11. “Well, they're making serious progress. I like the videos of people driving the beta stuff. Like, it's taken some pretty complicated intersections and all that, but it's still an intervention per drive. I mean, I have autopilot, the current autopilot, my Tesla. I use it every day.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  12. “And even Tesla started with a lot of CV stuff in it, and Andre's basically been eliminating it. Move everything into the network.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  13. “But that's because they think, well, 5,000 watts and $10,000 is okay because it's replacing a driver. Elon's approach was that port has to be cheap enough to put it in every single Tesla, whether they turn on an autonomous driving or not, which immobili was like, we need to fit in the bomb and, you know, cost structure that car companies do so they may sell you a GPS for $1,500. But to bomb for that is like $25.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  14. “They didn't build a ground up solution. Like the Gypsy Tesla are pretty cheap. Like MobileI has been doing this. They're doing the classic work from the simplest thing. They were building 40 square millimeter chips in Nvidia, their solution, had 800 millimeter chips and two 200mm chips. And, you know, like boatloads of really expensive DRAMs. And it's a really different approach. Mobile i fit the let's say automotive cost and form factor and then they added features as it was economically viable and nvidia said take the biggest thing and we're going to go make it work And that's also influenced Waymo. There's a whole bunch of autonomous startups where they have a 5,000 watt server in their trunk.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  15. “Well, they've innovated several times, but they've also worked really hard on mobile. They worked really hard on radios. They're fundamentally a GPU company.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  16. “Could do floating point programs on convolutions and matrices. And Nvidia invested for years in CUDA, first for HPC, and then they got lucky with the AI trend.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  17. “Yeah, but there is. So you build a reflection map, which also has some pixelated thing. And then when the pixel is looking at the reflection map has to calculate what the normal off the surface is, and it does it per pixel. By the way, there's boatloads of hacks on that. You may have a lower resolution light map, reflection map. There's all these hacks they do. But at the end of the day, it's per pixel computation.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  18. “Figure it out. So that pixel says I'm this pixel. I know the angle of the light. I know the occlusion. I know the color I am. Every single pixel here is a different color. Every single pixel gets a different amount of light. Every single pixel has a subtly different translucency. So to make it look realistic, the solution was you run a separate program on every pixel.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  19. “Is because you have 8 million pixels in every single, so when you have a light, right? Comes down, the angle, you know, the amount of light, like say this is a line of pixels across this table, right? The amount of light on each pixel is subtly different.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  20. “Look like little simple floating point programs or complicated ones. You can have 8,000 instructions in a shader program.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  21. “Well, no, that's just, well, like in modern A small piece of modern GPUs. What they did is that they still rasterize triangles when you're running a game, but for the most part, most of the computation in the area, the GPU is running shader programs, but they're single threaded programs on pixels, not graphs.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  22. “That's an example of that operation as a software program is really bad. I've written a program that did rasterization, the hardware that does it, it's actually less code than the software program that does it, and it's way faster. Right. So there are certain times when the abstraction you have rasterize a triangle, execute a graph, the components of a graph, the right thing to do in the hardware software boundary is for the hardware to naturally do it.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  23. “If the line of the triangle is like half on the pixel, what's the pixel color? Because it's half of this pixel and half the next one. That's called rasterization.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  24. “Well, GPUs were built around shader programs on millions of pixels, not to run graphs. So there's a hypothesis that says the way the graphs are built is going to be really interesting to be inefficient on computing this. And then the primitives is not a SIMD program, it's matrix multiply convolution. And then the data manipulations are fairly extensive about how do you do a fast transpose with a program. I don't know if you've ever written that transpose program. They're ugly and slow, but in hardware you can do really well. I'll give you an example. When GPU accelerators first started doing triangles, like so you have a triangle which maps on the set of pixels. So it's very easy, straightforward to build a hardware engine that'll find all those pixels. And it's kind of weird because you walk along the triangle to get the edge and then you have to go back down to the”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  25. “So each one of those things has been as the world of software gets more and more complicated, how do we create the right abstraction levels to simplify it in a way that people can now work independently on different levels of it. So I would say all three of those projects, LOVM, SWIFT, and MLIR did that successfully. So I'm interested in what he's going to do next in the same kind of way.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  26. “Because its development slowed down YTorch started a little later and then passed it. So he did a lot of work on that. And then his idea about LLIR, which is what people started to realize is the complexity of the software stack above, the low level IR was getting so high that forcing the features of that into the level was putting too much of a burden on it. So he's splitting that in the multiple pieces. And that was one of the inspirations for our software stack where we have several intermediate representations that are all executable. And you can look at them and do transformations on them before you lower the level. So that was, I think we started before MLIR really got far enough along to use, but we're interested in that.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  27. “Miguel of Yan became Know the de facto platform for compilers. It's amazing. And it was good code quality, good design choices. He hit the right level of abstraction. There's a little bit of the right time, the right place. And then he built a new programming language called Swift, which after, let's say, some adoption resistance became very successful. I don't know that much about his work at Google, although I know that, you know, that was the typical They started TensorFlow stuff and they, you know, it was new is, you know, they wrote a lot of code and then at some point it needed to be refactored to be”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  28. “They both get stuff done. They only get stuff done to get their own projects done. They talk about it clearly. They educate large numbers of people and they've created platforms for other people to go do their stuff on.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  29. “But the people we talk to want to say if I buy the car so we have a PC Express card with our chip on it. If you buy the card, you plug it in your machine, you download the driver, how long does it take me to get my network to run? Right. No, that's a real question.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  30. “And then lots of people are working on it, but there's lots of technical reasons why some of them aren't going to work out that well. And that's interesting. And there's also the same problem about getting the basics right. Like we've talked to customers about exciting features. And at some point, we realized that realizing they want to hear first about memory bandwidth, local bandwidth, compute intensity, programmability. They want to know the basics, power management, how the network ports work, what are the basics, do all the basics work. Because it's easy to say, we've got this great idea that, you know, the crack GPT-3.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  31. “It's also a really interesting place to be. Like the AI world is exploding. I looked at some other opportunities like build a faster processor, which people want. But that's more on an incremental path than what's going to happen in AI in the next 10 years So, this is kind of an exciting place to be part of.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  32. “Well, the power efficiency of local memory, local computation, and the way we built it is pretty good. And then there's a lot of efficiency on being able to do conditional graphs and sparse. I think for complicated networks, I want to go into small factor, it's quite good. But we have to prove that Fun pro”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  33. “One of the goals is to scale from 100 milliwatts to a megawatt. And the same kind of AI programs work at all different levels. So that's a goal. Since the natural data item is a packet that we can move around, it's built to scale. So many people have small problems.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  34. “There's all kinds of problems. Like there's small inference problems or small training problems or big training problems.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  35. “Well, the protocol at the bottom, we send it an Ethernet phi, but the protocol basically says send the packet from here to there. It's all point to point. The header bit says which processor to send it to. And we basically take a packet off our on-chip network, put an Ethernet header on it, send it to the other end to strip the header off and send it to the local thing. It's pretty straightforward.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  36. “It's built to scale naturally now. My experience with scaling is as you scale, you run into lots of interesting problems. So scaling is a mountain to climb. So the hardware was built to do this, and then we're in the process of.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  37. “Our goal is if you write PyTorch, that's good PyTorch, you can do it. Now as the networks are evolving, they've changed from convolutional to matrix multiply. People are talking about conditional graphs. They're talking about very large matrices. They're talking about sparsity. They're talking about problems that scale across many, many chips. So the native... Data item is a packet. So you send a packet to a processor, it gets processed. It does a bunch of work and then it may send packets to other processors and they execute like a data flow graph kind of methodology. We have a big network on Chep and then the next second chip has 16 Ethernet ports to hook lots of them together. And it's the same graph compiler across multiple chips.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  38. “Get to the optimal point, you either write use a pre written library, which is a good strategy for some things, or you have to be an expert in microarchitecture to program it.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  39. “On a GPU, if you write a large matrix multiply naively, you'll get 5 to 10% of the peak performance of the GPU. And then there's a bunch of people published papers on this, and I read them about what steps do you have to do. And it goes from pretty reasonable, well, transpose one of the matrices. So you do rotor, not column ordered, you know, block it so that you can put a block of the matrix on different SMs, you know, groups of threads. But some of it gets into little details. Like you have to schedule it just so you don't have register conflicts. So they call them Kuda ninjas.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  40. “But there is scheduling for that. So one of the goals is if you write a piece of PyTorch code that looks pretty reasonable, you should be able to compile it, run it on the hardware without having to tweak it and do all kinds of crazy things to get performance.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  41. “There's more love for that. And that may change. So the first thing is when they write their programs, can the hardware execute it pretty much as it was written? So PyTorch turns it into a graph. We have a graph compiler that makes that graph. Then it fractions the graph down. So if you have a big matrix multiply, we turn it into right size chunks that run on the processing elements. It hooks all the graph up. It lays out all the data. There's a couple mid-level representations of it that are also simulatable. So that if you're writing the code, you can see how it's going to go through the machine, which is pretty cool. And then at the bottom, it schedules kernels, like math, data manipulation, data movement kernels, which do this stuff. So we don't have to run right a little program to do matrix multiply because we have a big matrix multiplier. Like there's no SIMT program for that.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  42. “Not an expert on that. I know many people who have switched from TensorFlow to PyTorch. And there's technical reasons for it. I use both.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  43. “So, the native language of people who write AI network programs is PyTorch now, PyTorch TensorFlow. There's a couple others.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  44. “Hardware piece. And then the other cool thing is for a relatively small amount of money, they did a test chip and two production chips. So it's like a super effective team. And unlike some AI startups where if you don't build the hardware to run the software that they really want to do, then you have to fix it by writing lots more software. So the hardware naturally does matrix multiply, convolution, the data manipulations, and the data movement between processing elements that you can see in the graph. Which I think is all pretty clever. And that's what I'm working on now.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  45. “And then Chris has been working, he worked on LLVM, the low-level virtual machine, which became the intermediate representation for all compilers. And now he's working on another project called MLIR, which is mid-level intermediate representation, which is. Essentially under the graph about how do you represent that kind of computation and then coordinate large numbers of potentially heterogeneous computers. And I would say technically tense torrents. Know two pillars of those two ideas software 2.0 and mid level representation. But it's in service of executing graph programs. The hardware is designed to do that.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  46. “I think so. 10 storeents started by a friend of mine, Labisha Bajak, and I was his first investor. So I've been kind of following him and talking to him about it for years in the fall when I was considering things to do. I decided we held a conference last year with a friend organized it and we wanted to bring in thinkers and two of the people were Andre Carpathi and Chris Lattner and Andre gave this talk on YouTube called Software 2.0 which I think is great which is we went from programmed computers where you write programs to data program computers you know like the future is you know of software is data programs the networks. And I think that's true.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  47. “First Fundamental elements in the graphs are things like matrix multiplies, convolutions, data manipulations, and data movements. So, GPUs emulate those things with their little singles, basically running the single threaded program. And then there's an NVIDIA calls it to Orp where they group a bunch of programs that are similar together. So for efficiency and instruction, use. And then at a higher level, you kind of, you take this graph and you say this part of the graph is a matrix multiplier which runs on these 32 threads. The model at the bottom was built for running programs on pixels, not executing graphs.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  48. “You have a petabyte data space spread across some huge array of computers. When you do a computation somewhere, you send the result of that computation or maybe a pointer to the next program to some other piece of data and do it. But I think a better word might be graph and all the AI neural networks are graphs. Do some computations, send a result here, do another computation, do a data transformation, do emerging, do a pooling, do another computation.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  49. “So Raja Gaddori and I have been having this conversation about given versus found parallelism and then the kind of walk as we got more transistors like computers way back when did stuff on scalar data than we did on vector data, famous vector machines. Now we're making computers that operate on matrices. And then the category, we said it was next was spatial, like imagine you have so much data that you want to do the compute on this data. And then when it's done, it says send the result to this pile of data, run some software on that. And it's better to think about it spatially than to move all the data to a central processor and do all the work.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source

  50. “Parallelism, yeah, it might be because when you run the GPU program on all the pixels, you're running, you know, depends, this group of pixels say it's background blue and it runs a really simple program. This pixel is some patch of your face, so you have some really interesting shader program to give you an impression of translucency. But the pixels themselves don't talk to each other. There's no graph. You do the image and then you do the next image and you do the next image and you run 8 million pixels, 8 million programs every time in modern GPUs have like 6,000 thread engines in them. So to get 8 million pixels, each one runs a program on 10 or 20 pixels. And that's how they work. There's no graph.”

    2021-02-18 · Lex Fridman Podcast · #162 – Jim Keller: The Future of Computing, AI, Life, and Consciousness · IDENTIFIED FROM THE TRANSCRIPT · source