YouSaid · the spoken record

Reiner Pope

lines on the record: 143
first: 2026-04-29
most recent: 2026-04-29
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“That's in general, right? Because RL is less efficient in terms of machine time. And so if you're trying to equalize the RL and pre-training time, then you should have fewer tokens in order to have the same wall time.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“You might notice when the money's going down the drain. Yeah, yeah. So, yeah, so I think this all of these end up being close as modeled here. This 30% may have been a little bit too generous. So let's say something like 1.5 here and made this as a 1 here. So I think at this point, you can almost read it off. The number of inference tokens should be about the same as the number of pre-training tokens should be about the same as the number of RL tokens within factors that we're not able to reason about.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, okay. We can make this like two in 10. Make it bigger. So, yeah, like just write it out once more. Like, this is two over 10. This is 1 over 10. So the number of inference tokens you have, and this is just a function of like, I've got hundreds of millions of tokens per second times my model is deployed for, I don't know, two months before I shift to the next version, that should determine The number of tokens in RLN pre-training, and then I guess we didn't do the equivalence between pre-training and RL, so we'll do that here. Pre-training should be equal to like two over 10 times data in RL for them to be caustic equivalent. Sorry, this one over. I got it backwards. Like, we pay more cost when it's inefficient. So this needs to be one over. So tracing this back forward.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“And then let's just roughly size the alpha. This alpha, it's going to be... This is like the, it's maybe somewhere in the range of two to six to six over six from this term compared to this term. And then we've got an inefficiency term, which I would say is maybe in the range of like 30% or something like that. I'm sorry. So, this alpha is going to be something like 1 and 10 In this beta here is actually the same. It's a third, it's one third times 30%. So it's also equals one in ten, something like that.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“That's also a valid answer as well. Because this is heuristic, I can't really argue for one versus the other, they don't differ by that much. Like 33 versus 25 is only a small factor off. So let's pick one of them. All equal seems simple enough. And so we're just going to solve for equality of them. It's pretty straightforward. We can immediately see that the number of activated parameters totally disappears. And so let's factor that out. And we're going to just say that data in pre-training. Decided to do it your way. It's a little bit nicer, actually. So, data in pre training plus this, oh, I didn't have the inefficiency over here either. Efficiency data in pre training plus some multiple of alpha times the data in RL. Just going to be end up equal to some beta times the data inference.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yes. Yeah. Thank you. And then inference is just true. So we're going to solve for essentially maybe a qualitative all three of these terms that is ballpark where people are going to be. Labs have more information on what is productive in doing more RL, for example, than versus doing more pre-training. I don't have that information. But I think a good ballpark is 33% split between each of them.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, somewhere in the range of two to six. So we'll just like say somewhere into the range of two to six and leave it at that. And then we can add in the inference cost. The inference cost is two number of active times the data in inference.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, yeah, there's that. And then the other perhaps even bigger inefficiency is. This involves a substantial amount of decode and often decode runs at less MFU than training.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Times the data on pre training. That's the cost of pre-training. There's a factor of six out here, which is the number of flops. There's the famous 6ndD formula. And then in RL, we have approximately the same thing. We've got like same number of active parameters, but now the amount of data is the RL data. There's this extra efficiency multiplier, which is, or inefficiency, like the inefficiency.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“It's also true for e to the x and like e to the minus x and all kinds of other things. So basically I've got some curve that's going down, some other curve that's going up, and they tend to be minimized at this equal point. Heuristically, I will conjecture that that is true for the setup you described as well. Actually, showing that that would be true would require looking at the scaling laws and like fitting these weird exponents. But things that do follow power laws tend to have this property. So I'll just make that claim and move on. So we're going to say that the cost of training. Plus the cost of inference, we want to equalize these. We'll do pre training only first because it's a little. Well, actually, we can do all of it in general. So actually, we'll cost it that. Cost of pre-training. So number of active params.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“This is a place where we have to do a bit of guesswork because the updated scaling laws and the model traffics are not reported. And so we have to guess there. But one way to look at it. First, just make a sort of a general heuristic claim. If I had some cost and I've got a total cost, which is a sum of cost A and cost B, Maybe this is the training cost and this is the inference cost. And so I want to minimize this sum. For many curves that tend up being the case, the minimum tends to be where the costs are equalized. That's something of a heuristic claim, but there are many examples where it's true where is one over x and the other one is x, for example. They tend to be minimized at the point where they equal each other.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, it lets you just run the model at lower latency as a first thing. If I just do a very sparse model and it's on a little H100 box The latency will be really high.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, yeah, time is GPU bandwidth. And so this term doesn't increase a lot. It maybe increases 1.5 or 2x per generation. But this one increased by like a factor of eight from hubbu.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so I mean, we talked about latency of the hop of this hop. There is also just the same TMM latency, the memory time latency is actually Substantially, like massively improved by larger scale up domains. So I'll recall TMM down here. For the weights TM of weights. This was equal to the number of total parameters. Divided by the memory bandwidth. Which memory bandwidth are we talking about here? Is it just one GPU or it's in fact it is the number of GPUs that I can use in parallel to load these weights? So I can't use different pipeline stages in parallel because they're not running at the same time, but I can use all the GPUs in my scale up domain in parallel to load the weights. And so, this is actually extremely effective. So basically, I end up with a term here, this memory bandwidth term itself is equal to scale up size.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, if it goes from 20 to 30, right? Or something like that. Yeah So, just to chart the path that it goes through, here you're going from your GPU or TPU or whatever to a network card, which then goes to a top-of-rack switch. And then hops over to the other act and does the same thing in reverse. So you sort of have to sum up the latencies of these different things.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Actually, I make the pipelining Doesn't help with context length. It totally helps with model size. And so because of the ability to do pipelining, At least a rack should not be a constraint on your ability to fit the model parameters. I guess the other consideration you're asking why hasn't it scaled up more and why did bigger scale up domains help? So we talked through one aspect of that, which is we kind of said it. It's not because of memory capacity. We have a solution to the memory capacity, at least with respect to model size, not with respect to KV cache size, but at least with respect to model size, we have a solution to memory capacity. The other issue that shows up is latency”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yes. Yeah. I mean, you can look at how it depends on model size You could have a very large model like. One that exceeds the memory of a rack. And they're doing a bit of pipelining. Maybe it's extremely sparse, for example. And that would be a reason to do it”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So, I mean, the deep stake paper reports what they do, which is they just do a lot of expert parallelism. You should, in effect, you should increase your expert parallelism up to your scale up domain size. And then do very little pipelining. Maybe not at all, maybe two. Just enough to make the wait storage not too big of an issue. Those are the only two parallelisms that really make sense. In the past, there was tensor parallelism, which was cutting up within an expert. But the experts are so small now that that is not a profitable optimization.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, so first we did you can amortize KV caches across batch size. And now we're saying you also can't shard it across pipeline stages. It sucks from both of those points of”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, you only need to store like one layer rather than two layers of KVs, right? Yeah. So it helps from that perspective. You're right. What's competing with that though is that you need to be keeping all of the racks usefully busy at a time. And so the number of sequences that are in flight simultaneously has gone up.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“And so, what we find if you increase the number of pipeline stages, the memory footprint for the number of weights keeps going down and down and down, but the memory footprint for the number of activations stays constant. So it doesn't actually work. Most of your memory ends up once you do enough pipelining and it's really not much, like even two is often enough, this term becomes very small. This becomes the dominant term. The KB cache becomes the dominant term.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“For sure during massive scale training, this is done, it can be done for inference. I'm actually going to make the case for why it is less attractive. It is useful for weights, but not so useful for KVs. The big challenge is so let's fill this in. The micro batch size here ends up being equal to the number of pipeline stages. When we go back and substitute all of that into here. You get a number of pipeline stages times this little b. Showing up And then when we factor this out, I'm going to split this into this plus into two terms. We get the full division by E times P over here. We still have division by E times P over here, but the P's cancel. This P and this P.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Right, yes, this is going to be the 20 milliseconds train So, the global batch size is the number of micro batches times the local batch size. Local batch size is set by this hardware parameter. The number of micro batches, well, the number of micro batches is as small as possible such that we can wrap around and not leave any idle time when we wrap around. So if we had fewer, we would have this idle time when we wrap around. And so you can sort of just visually see that it is equal to the number of pipeline stages. I mean, a sort of proof by visual here. It is four and it's for this way as well. But you can sort of look and see that it goes along here and then it wraps around number of pipeline stages.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So let's split this batch. This batch will be the global batch size. So B is going to be the number of micro batches. Times the batch of the batch size per micro batch. So how many micro batches do we need? So the number of micro batches in this diagram is four, zero, one, two, three. And then the batch size per the micro batch size, this is still this 2000-ish number. This is the one that he's like. This is the 2000 times sporosity. No, this is the 300 times sparsity, 300 times sparsity.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, for example. So this is the place where we actually need to go back and analyze this batch size B. And you were making this comment that there's micro batching versus global batching. So let's come back to this pipelining diagram here. We've got one batch going forward here. And then as I drew it, it kind of just like disappeared. That's not really correct. If you think about how decode is working, I have a bunch of tokens that I have. Do one forwards password, I generate a new token. And then I push, like, then I write that to my KV cache, and then I do another forwards pass that generates the next token. So I'm actually going to be running this batch zero in a loop. So in fact Go forwards. Once I finish, I can start the next iteration of the loop up here So we'll just fill this in. Yeah, so we've got the two or three.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“And well, obviously we just take all of these numbers and divide it by A and B. Really easy. So it's this n total plus the batch times length of context. Bytes. Toke, all of this is divided by the timespeed. Okay, so this is like, why is this correct divided this way? Well, we're saying we knew that the parameters were perfectly divided amongst all the GPUs in a rack. The layers are perfectly divided amongst the different racks. So that works here. And somehow we're going to arrange, I'll hand wave exactly how. Somehow we can arrange the same perfect sharding of the contexts across GPUs in Arak and based on layer across racks.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So, we're going to run this on some number of GPUs, and we're going to say we're going to have one extent, which is E is going to be the expert parallelism. How many, when we had this charting of expert layer across many GPUs, how much of that, to what extent do we do that? How many GPUs? So we're going to say that this is fact, for example, 64. And then P is going to be the extent of pipeline. And so this is a number of racks, which who knows, maybe we'll pick four or something like that. What we want to calculate, so this is like the total memory requirement across the system. But now I'm going to calculate a memory requirement per GPU. So per GPU memory requirement. We're going to have, I guess I'll use a Lovercase C mem.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“You don't need it. Yeah, so we've like in the equations we had here before we raised them, we were doing memory time. So memory bandwidth and compute bandwidth. Let's now start looking at memory capacity. So we'll start off with just memory capacity without even thinking about parallelism scheme. And so the The capacity of memory or the demand on memory is the number of total parameters. So, this is what we need to fit the weights in some system that we are using And then we need to fit the KVs as well. So KVs go as batch size times the length of the context. Times the bytes per hour. Okay, so What I was arguing about in this context, and the case I was making for pipelining is that we will actually, there are some techniques that allow us to solve this. Are there techniques that allow us to solve this? So let's consider”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So, even in inference, in fact, it is not used a ton. It reduces your memory capacity requirements. There's actually a huge surplus, like, I think you're saying that a rack of Blackwell has many, many terabytes, maybe tens of terabytes. That's much bigger than the trillion parameter model. The trillion parameter model only needs one terabyte. And so it already fits, in fact. And so there's not a huge benefit from... From pipelining because you're reducing a number that's already pretty small. But it does say that theoretically maybe you had too much memory and maybe you could have done a different, like build a different hardware that has less memory, in fact. If you were designing your hardware and you said, I actually didn't need that much memory because I don't need the weights to fit in one rack. I can fit the weights in eight racks.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, right, r More usefully, you can do the weight gradient step, but you can also wave it. So in inference, actually, the effect of pipelining on anything you care about, like batch size or latency, actually is neutral. It doesn't improve it. It doesn't make it worse. So if you look at the latency of this inference, running it, if it were pipelined versus if it were all on one rack, if it were all on one rack, we would just slide all of the boxes down and still put them in a row. And the latency would be the same. So pipelining is neither better nor worse for latency, but it does mean that you just use less memory per rack, like memory capacity, because now instead of needing the whole model, you only need a quarter of the model.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Total training time perspective, it's smaller is worse from a system's perspective. And so the optimum is the trade off between these two. So you pick a batch size and then for that batch size you do some amount forwards and then a sum amount backwards. You asked why there is even a hard stop? Pipeline parallelism, because of this, the like. There are so many techniques in the literature for how to lay this out differently and avoid that. There are more complicated schemes called zero bubble or one forward, one backward, which sort of interleave the forwards and the backwards in complicated ways.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I mean, smaller is always better, actually, is a way to put it, but it's like from a ML convergence rate perspective, smaller is always better because basically you're getting the freshest information from the gradient descent.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Should we draw the training diagram? Yeah, let's do that. So, this is the inference diagram, and I'll call this four just so we don't have the wrong thing showing up there. So let's do the same thing for training now. We've got a forwards pass, but at some stage we're going to have to transition to a backwards pass. We'll do some number of batches in the forwards pass. And then we're going to transition to the backwards path for everyone all in one go. So the inference part is the same here, but then we do a hard stop at this point and then transition everyone to backwards pass. Similar numbering like this”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So we don't actually run inference one here. We run it as soon as we can, which is immediately after inference zero finishes like this. And we keep going. So if we hadn't filled this in, we would call this the pipeline bubble. When I've drawn it in this inference context where we're only going in a forwards pass, it's obvious, like, why would you do the stupid thing? But in a training context, it's maybe less obvious. But in the inference context, it's sort of really natural to make this change.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so we draw the pipeline bubble. Yeah. Okay, so why do we do, what is this micro batching that shows up in pipeline parallelism? So I'll focus on inference first. It's a slightly simpler problem. So, and I'm going to draw. So, this is time, and then this is which rack we're on. And so the idea is that maybe I'll have like four racks. So I've got an inference that is going to step through these four racks in some time like this. This is inference number zero It runs at a certain batch size and it steps through all the pipeline stages like this. Now, if we were to say, well, we're going to run inference number one here, like this is clearly a massive waste, right? Like three quarters of the time, each of the racks is doing nothing”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so, and I guess we didn't really fully articulate even what the benefit that we're getting from pipelining. And so these complexities are real. Pipelining is a massive hassle, but it does give you some benefits. And then you can then decide whether those benefits are worth the costs. The biggest benefit that shows up, so it has some benefits and influence, maybe bigger benefits in training. In inference, what are we saving on? Are we saving on memory time or compute time? Not really. We're just moving the memory time from one chip to another chip or one rack to a different rack. There's no actual benefit in runtime. However, what we are saving on is that the memory capacity The amount of memory used per rack, if we think that the memory in a rack is a bottleneck, then there's a constraint on how sparse we can go.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah. So, I mean, I think a way to think of it is, I mean, okay, the Galaxy Brain way to think of it is, what are all the different dimensions in which a model is scaled up? And so it is scaled up by layers. It is scaled up by the demodel dimension. It is scaled up by the DFF dimension. It is scaled up by the number of experts. Every single one of those numbers you can choose to cut along. And if those numbers are big enough, it eventually becomes profitable to cut along there. And we have selected two of them. The other two, in the way typical models are typically sized, are not profitable.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So, what this ends up looking like is that I can, in fact, have an entire pipeline of racks where one rack does one layer, and then I move on to the next rack and I do another layer, and then I move on to the next rack, I can do another layer.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yes, yes. And there's a factor of two. Thank you. So, what we would like is for the scale up time to be greater than the scale-out time, because the scale up time is the more important and precious resource. And so we want this one. We would like this number to be greater than or equal to one. And this really doesn't seem hard. There's just a factor of H that we need to overcome. So we need the product of these three things to be bigger than eight. Typically, we have a fairly large number of activated experts. It could be H by itself. And then we can increase the number of layers per stage a lot until we satisfy this.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“The same thing applies on multiple different layers. So maybe I'm going to run two layers. So there's also multiple Number of Leas”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Okay, so we're going to talk through sort of how much more, what is the slowdown of, to what extent is scale up a bottleneck over scale out? So we will directly jump to the ratio of the time spent on scale up. Time on scale Over the time spent on scale out. This is the quantity we're talking about. And the first consideration is that the scale up is eight times faster than scale out generally. And so at a baseline, if the bandwidths were the same, we would have this 1 over h which is coming from bandwidth. Have some amount of expansion in how much data we're sending. So if one token comes in here, This one token gets routed to in the deep seat case, it'll get routed to maybe 32 experts or 16 experts. It gets routed to some number of experts. So this is the number of activated experts, number of activated Experts And then it also.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so a number of activated GPUs, right? So I don't send to this GPU at all, right? So there's an explosion from one to three times larger here in this diagram. The key thing is that I didn't even need to send to this GPU at all, and so that's a big saving.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So let's try this. I mean, the hint is going to be that there's a lot more sense here, like we're sending many things here, whereas we're only sending one thing here. And then we're also maybe doing it many times. So that's going to be what makes the difference.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Now, is that going to become a communication bottleneck? So we can actually just solve for when this becomes a communication bottleneck. But before we do that algebraically, let's just sort of visualize it out and sketch the path. So we're going to have a bunch, this is another MOE layer. We're going to have another MOE layer of the apple and so on. So let's say I change rack here, and then some number of layers later, I change rack here as well. So our methodology that we're going to use to determine whether we have a communication bottleneck in this point where we change rack is we're going to compare this is the scale out bandwidth requirements to the scale up bandwidth requirements.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“The literature is tensor parallelism. This is with a trend towards smaller experts. This has become much less relevant, so we can ignore that. But the other two things that we have available are data parallelism and pipeline parallelism. And they are actually, they can be a much better fit for using multiple racks. So let's focus on pipeline parallels specifically. This is one layer of MOE. I'm going to have like 100 more layers up above. I could decide at this point, for example, to move to a different rack, change rack.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, really interesting. So Okay, so to answer that question, we're going to need to talk about the communication patterns. So we've talked about the mixture of expert communication pattern. That is this all to all. This all to all. To all. Alterall very strongly favors full connectivity, which is what we've kind of just shown here, and favors being within one rack. There are other kinds of parallelism besides expert parallelism, which we just showed here.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Not having been there at the time, I'm not sure how much is coming from successfully deploying higher sparsity ratios, which could be. It could also be, I mean, there's a whole bunch of actual modeling things of specifically how do you do the mixture of experts. We've seen the deep seek mixture of experts has said actually activate more experts, but fineer grained experts was a big innovation. I'm sure there are many other innovations on the model architecture as well as on the training data. It's kind of hard to disentangle all of them. But what shows up in terms of the limits of what you can do? The active parameters, as we saw, is limited by the compute cost. And then the total parameters is limited by the scale-up size.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, deploying in larger scale updomains is a huge unlock. I mean I've drawn here the sort of NVIDIA Blackwell deployment, the Google deployment has actually had very large scallop domains for”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so I mean, Rack Design is not my expertise, but when I talk to folks and what are the constraints they're up against, it's a combination of, so what are the big physical things you're optimizing for space, weight of the rack? It's actually really heavy. And so you need enough metal top to not sag and fall. But then you add more metal and it's heavier and then power and cooling. And so all of those are competing for like. Modern racks are pushing all of those to very extreme physical”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so there is space outside the rack, inside the rack. Like these racks are like, I mean, as they become more optimized, these racks are very tight. So there's connector density going from the tray into the rack and the rack's backplane. And then the backplane itself has a really high density. There are other physical constraints, including bend radius of cables. You don't want to snap them and so on.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source