YouSaid · the spoken record

Reiner Pope

lines on the record: 143
first: 2026-04-29
most recent: 2026-04-29
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“Let's sort of zoom in on this and look at the wire density. Draw this diagram just run some more so we have a bit of a cleaner version to work with and a larger version. Let's say I have some switches in the middle. And let's say I'm going to have initially I'm going to start with just two GPUs on each side or two trays of GPUs on each side. And let's say maybe each tray wants to have two cables coming out of it. So I get some kind of, I physically run vertical cables that look like this running to the switches. Now, if I want to double the number of GPUs in a rack, I need to run literally twice the density of cables. So I need to run. These as well.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“From Hopper to Blackwell is mostly just the decision to switch from trays as the form factor. One of these is a tray to switching to racks as the form factor. That's a product decision. There wasn't a substantial technical barrier there. Switching from the 64 to 500 or so, there's a bit of tension math there, but there is at least a genuine 4x increase, which is coming from a much more complicated and difficult rack design. And so that is actually new physical design to run more cables.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“In Baduins. So, the challenge, if you want to, for example, lay out a mixture of expert layer across two racks is that Half of the GPUs here are going to be wanting to talk to the GPUs here. And so just on average, when I look at where the tokens on these GPUs want to go, half of the tokens I want to go inside the rack, that's great. They can use the fast scale up network. But half the tokens are going to want to leave the rack and go to the other rack. And that's not as good. They're going to need to use a much slower network. And so that becomes the bottleneck on the all-to-all pattern. A different choice would be, well, why don't I have a big switch here and sort of and connect everything to some big switching, like a much bigger switch that actually combines the two racks together? There are many ideas in this direction, but in general, it becomes the reason you have this sort of hierarchy of switches rather than one big switch is to manage the cabling congestion. You just need to run a large number of cables.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So every GPU goes to the switches in the middle, and then the switches have connections to all the GPUs. So all of the GPUs can talk to all the other GPUs in just like two hops, going to switch, going to the other GPU. Now, when I want to leave the rack, I end up going via a different path. The GPUs have also a much slower connectivity, which is typically about eight times slower, which is the green that I drew here in GPU cases is the NVLink. More generally, it's called the scale up network. This is the scale up network. You will typically also have a scale out network, which allows you to connect to some data center switch, so data center switch. And then all of the GPUs will have some connectivity up to some data center switch somewhere. But this is our times. This is the scale out. And it tends to be about eight times slower.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“In NVIDIA's case, the communication topology is actually, they put the GPUs on the outside of the rack, and then they put these switches on the inside of the rack. So what this ends up being is that there's a set of switches in here. These are the NV switches And then they run a bunch of cables every single GPU has cables going to the switches in the middle.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, and this is a place where it starts to be very different, in fact, between NVIDIA, for example, and Google and then others, including us. So generally, a rack is a It is a physical structure. It's a few meters tall, meter or two wide, it depends on configuration, and it stores some number of GPUs or XPUs, which is typically about 64. What constrains it being a certain size is power delivery weight, and cooling ability. It ends up being about this size in many cases because of these physical constraints. So then when I deploy a data center, a data center may have thousands of these racks. So I've got one of these tall racks. It's got a bunch of GPUs in it and so on. And then I put another rack next up.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“And I no longer, in fact, have all to all communication between all the GPUs in two racks. And so the rack to rack communication ends up being a substantial bottleneck. So this sort of, like the fundamental thing here is that one rack is actually the bounds the size of an expert layer you can do. And so this has been part of what's been driving towards larger and larger interconnector mains.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah. So these are 64 GPUs. These are 64 GPUs. It's actually the same GPUs we just draw them as separate because they're serving different purposes. So at this point, any GPU can be sending to any GPU. So this all to all pattern of communication that shows up how the Blackwell racks are configured is a perfect fit for the communication pattern that the MOE actually wants to do. However, if you think maybe I want to do maybe one rack is too slow and I want to do two racks, then I have this challenge that maybe I've got some sort of rack boundary drawn outside here like this”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, the route, so I drew this as one router. In reality, you would actually have many copies of the router, and so you would have as many routers as GPUs, in fact.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“And then we can look at the communication cost. We had some experts, some tokens stored centrally here. They get routed to all of these experts. And so there's some communication cost paid here. There's the same communication cost paid on the output. And then the hope is that this does not become communication limited. Now, what is the traffic pattern here? The traffic pattern here is that any GPU, in fact, will be talking to any other GPU, depending on the decisions made by their model. So, this is an all to all traffic pattern.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So the standard practice here, and it is the best solution, is to use expert parallelism. So that means different experts go on different GPUs. So if we take something like a deep seek model, they have 256 experts. Say we want to run that on a Blackwell rack. So there are 72 GPUs. We have a divisibility problem. This is not a power of two. So we'll just simplify and say we're only going to use 64 of them. Just ignore the other eight. It's not a big deal. And so we have four experts per GP. Very simple. For the sake of the diagram, I'll actually just say, let's say we have two experts per GPU. So we end up just putting, these are the GPU boundaries, every pair of experts is on its own GPU.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“These experts, so each expert itself is a normal MLP. It has an up projection and then a down projection or a nonlinearity in between. And then finally, we sort of do the inverse operation. So where we were broadcasting things out here, we're going to bring them back in and sum them up. In a man And then finally, we have our residual connections, the token is also passed through here and it gets added. Result of the HimoE layer. So, this is a normal MOE layer. What I want to talk through is how this is mapped to a GPU rack and what this means for communication. Because I think this will start to show some of the limits of how sparse we can go.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Spouse Mixture of Experts. Maybe how we lay that out on a GPU. So let's zoom in on the mixture of experts layer first and sort of draw what that looks like. So we typically will have some kind of a router layer Which is making the decision where we route the experts, the tokens to. So tokens coming in here, they go through a router layer, and then we have a bunch of different experts. I'll draw a few more to line some up. And then the router will make a decision which experts am I going to route to? And it'll be a small fraction of them, maybe one in 32. So maybe it'll make a decision to route to this one. Maybe this one, and maybe this one.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah. Yeah, so I mean, maybe this would be a good point to actually talk about how a mixture of experts layer is typically layered out on a rack of GPUs or something like that.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So, is that good or bad, actually? Even from a memory point of view, keep in mind you are doubling this portion of the memory fetches, which is amortized by batch. And so just keep running a larger batch size. From the point of view of the analysis we've done here, this is pure win. Keep doing it. Keep doing it until you run out of available users, basically. So there's actually this equivalence between if I want to go sparse or if I have a lot of users, I can go to a much sparser model. So from that point of view, it's a reasonable trade-off. The other trade-off that shows up here is that it also consumes memory capacity, which we've only reasoned about memory bound with here, but it also consumes memory capacity.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Active parameters. Yeah, I mean, actually, even more so. Yeah, it's a huge increase in parameter count for a modest increase in.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“The techniques have changed a lot. Deepseek mixture of experts was a big change in how it worked. There have been older papers which are G-Shard, Switch Transformer. So the actual empirical results are going to depend on all of that. But on one of the older techniques that is shown here, you can see if I hold constant the number of active parameters at a certain size, and then I increase the sporosity, which they call expert count here. The quality keeps increasing. And then if you imagine drawing a horizontal line from 1.3B dense across, you end up seeing that, for example, in this case, the 64 expert, 370 million activated parameters model is as good as a dense 1.3 billion model.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So this paper, this is unified laws for routed language models. It's a somewhat old paper by this stage, but one of the things that they did is looked at if I keep increasing sparsity, what is the model quality impact? This answer is very sensitive to the actual choice of a mixture of experts, mixture of experts has been around for a really long time, I think it was maybe even back in 2017.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so equality of the model, I mean, rather than speed of the model. Yeah, yeah. So unfortunately, we're not able to answer that analytically. That is an empirical question of model quality. I can do is pull up a paper and answer that empirically.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“When you look at some of the announcements, sometimes the API providers will brag about how much traffic they have. The numbers that I've remembered from some announcements of Gemini last year were in the hundreds of millions of tokens per second worldwide. About it, that like this is one thousandth of that footprint.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“We can do a bit of analysis on this, which would be actually it's like you can think of it in terms of number of users, but maybe a more productive way to think of it is in terms of number of tokens per second. So, what does this batch size mean in terms of tokens per second of the system? So tokens per second, tokens per second is going to be equal to the batch size. We run a batch many tokens, and then we do that every t so every time in tools, which is, let's say, this thing is equal to the 15 milliseconds longer. So this ends up being batch size itself times About 60, so like 64 times B. And so this ends up being around. 2,000 times 64, so like 128k token specific. So this is sort of in more digestible units. It's hard to reason about concurrent users, but what is the global traffic for a system?”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, there's sort of two scenarios. Why don't we pick a latency that is bigger than 15 milliseconds? If I think what that means, it means I actually have time to read the HPM twice. By the way, most of HPM accesses is reads, not writes. It's like most all reads because the weight matrices are read-only, and then almost all of the KB cache accesses are reads. So let's say I run 30 milliseconds, I can read all of HPM twice. What's the point of that? Like, I don't want to read the white matrices twice. I don't want to read the KVs twice.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Per second. Yeah, so for example, I mean, on I think the urban generation, it is something like 288 gigabytes divided by 20 terabytes per second. This looks like it comes out to about 15 milliseconds.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, rule of thumb, but where it comes from is not fully explained yet, but So far, we've focused on memory bandwidth and compute time. When we look at memory, the other consideration is that we want to use all of the memory capacity we have. And so generally we're going to use all of that memory capacity to store the weights or the KBs. And so we just want to read in the time of doing a forward pass, maybe we want to read all of the memory capacity into the chip. And so that is capacity divided by bandwidth. That tends to be 20 milliseconds on many different generations of HPM.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“You can think of this as a schedule for the train, a new train departs over 20 milliseconds. Any passengers who are ready board the train. If the train is full, then they wait to the next train. If the train is not full, the train's going to go anyway. And so in terms of what that means for queuing latency, it means that the worst cases that you a request arrives just after the train departed, it has to wait for the next train, so that's after 20 milliseconds, and then it has to wait for that train to complete. And so the worst case latency is 40.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, the way to think about this, I guess we think of it as when does the train depart as a model? So let's say I've picked a batch size that I'm going to run at. Maybe I pick this batch size. And so, and by the way, this intersection point is the same intersection point here. So I picked this batch size. I know that it's going to take, for example, maybe it's something like 20 milliseconds is a common place that sends up landing. What I'm going to produce is, so this is a timeline of what is running on the GPU. It's going to start a new batch every 20 milliseconds, regardless. And so this is 20.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, okay. So I guess this is, keep in mind that I'm talking about the number of tokens that I'm generating one more token for. So it's like it's actually 2,000 unique sequences again.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Should grow larger. So we solve for the equivalence between when compute time is equal to memory time. If I add in more memory bandwidth, like something that consumes more memory bandwidth, then I have less available for the weight loads. And so I need to grow the memory bandwidth more, and therefore the batch size more.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So, this actually gives you a ballback, which is remarkably accurate to practice. Generally, people will go a little bit larger than this. They don't really want to be exactly at the balance point because real-world efficiencies aren't as good as a roofline analysis would say. But take this and maybe double it or triple it.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So there's a hardware parameter to what extent has the hardware changed? So from A100 to A100 to B100, the flops has increased substantially. The memory management has also increased substantially, and it has remained reasonably stable. And we can express this one as well. This is a sparsity parameter And I might even phrase it slightly different. Let's solve for batch size in total. We end up with, so we're just moving this back over to the other side. We end up with batch size needs to be bigger than approximately 300 times sparsity. So, for example, if I have, I activate in 32 out of 256 experts. So this would be eight for deep sick.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“The fact that HFP4 is half a byte. And so I can actually make this ending up being dimensionless. And this ends up being on most GPUs. Around 300 Somewhere around”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Divided by the compute performance. So, looking over here, everything on the top, these are model parameters. Everything on the bottom, these are hardware parameters. It turns out to be nice to rearrange them such that we have the hardware parameters on one side. So this is equivalent to. Over memory bandwidth Being equal to match size times number of active parameters Divided by the number of total parameters So, this is a hardware parameter. Actually, this actually ends up being a dimensionless constant if you look in terms of flop What are the dimensions of this? This is multiplies per second. This is bytes per second, so that's not quite dimensionless. But what do you do is you say multiplies per second times, let's say I'm doing fp4. So I do how many fp4 multiplies per second.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“You can just solve for that, actually. And it's not even particularly sensitive to model architecture. So let's go ahead and do that. So, what we're talking about is we're going to say when the memory time is equal to the compute time. That's what our question is. For now, I'm going to discard the because we're focused on what the batch size is, and really there's a question of when the weights are amortized over the multipliers. I'm going to focus on comparing the wait fetch time to the weight multiply time. I'm going to disregard the KB fetch term, just to simplify the analysis so we can get a kind of a clean answer out. So we're going to equate... This portion This with these two Writing that out, we get n number of total parameters A memory bandwidth. Is equal to Batch size times number of active parameters”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, yeah, they're unique per batch. The compute is also unique per batch. And so, what is the minimum work you can do per batch after amortizing everything else away?”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So, again, we see some limiting behavior. The cost initially starts very high at batch size of one, actually. It almost goes to infinity. It's because we've got so many weight fetches which are not amortized over a large batch size. But then as we increase the batch size, the weight fetches amortized over so many different batch elements that their cost grows very small. And eventually the compute time ends up driving the cost. So there is a limiting Lower bound on cost. Which is this”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“This is the cost per token. So We have to imagine dividing each of these three curves by B, so multiplying by this reciprocal. And so what we end up with there is the compute curve is going to, it was linear. We divide by B. That makes it a constant here. And this is t compute The KV fetch was linear. Now it becomes a constant as well. KB fetch. And then The weight fetch was constant, and now we're divided by B, and so it becomes this hyperbolic. And so again, we're going to compute the max of the sum. So, the sum of these two terms shifts the parabola up, some of the KV fetch and the weight fetch gives us a sort of a Higher parabola that's And then we're going to take the max with the compute. So we end up with this being the overall shape that we care about.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“I'm pretty excited about spouse attention. It's hard to know what the labs are using. DeepSeek has published a spouse attention mech I'll just put a plug in that sparse attention. Some of the deep-seek papers that have published sparse attention end up putting a square root in this term. OK, so far we've done, we've looked at the latency. It's kind of hard to read off cost from this. So if I think, what does cost mean? To run this inference, I'm going to use the GPU for a certain number of seconds, like one millisecond or 20 milliseconds or something like that. And I have to pay the rental time for that time. So it's $2 an hour per GPU or something like that. So that's the cost of this inference, but how much value, how many tokens have I processed during that inference? That is the batch size. And so what we actually on the plot is going to be the cost versus batch size, which is like t over b versus batch size.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“That's right. So that is true as modeled here. There's a key point here that I'm modeling this context length as, or I'm modeling the memory fetch as linear in context length. That actually depends on model architecture. It is true for many of the, all of the model architectures with dense attention. There's a sparse attention actually scales much better than that.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, whenever we have balance points, it kind of says that you're getting it exactly right. And so for the particular context length where the slopes match, that says I am equally memory bound and compute bound, which is a really desirable place to do it.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, this is really sensitive to the context length. So I think we should come back and explore this. The KB fetch time will go up and up. And so that'll cause a transition from compute limited to memory limit.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Okay, so what does this mean, actually? So, this is a latency plot. So, if I grow my batch size, I get initially some not very strong dependence on batch size. And so there's some lower bound on latency here, latency lower bound. Lower So this already partially answers the question. For a given hardware configuration, and then we can talk about varying hardware configuration, but for a given hardware configuration, there is a lower bound on latency, which is simply the I need to read all of my total parameters from memory into the chips. And that takes a certain amount of time. If I use all of my memory bandwidth, I can't do any better than that.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Which is the KB fetch, which we're going to draw as. There's the KB Which are linear and bash size. So it looks like that. So the sum of this plus this maxed with this. So let's at least first draw the sum So, the true memory times in conjunction end up looking on this curved slope like this.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“We said that the big effect you can get is some trade off in latency versus cost in batch size. So let's draw them out. I think there's just really two graphs we want to draw. We'll first just draw batch size versus time here. So when we look at the shape of this, we've got a maximum of a sum and then another term. So let's look at these terms one by one and how they scale the time for compute and memory and how they show up. So let's first look at this compute time. This is just purely linearly linear in batch size with no. No offset, so it is Curve like this. This is this is t compute. And then on the memory side, we've got some portion here that is just this constant that is constant in some base offset here, which is the weight fetch. Wait, fetch. And then finally, we have this term here.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source