YouSaid · the spoken record

Reiner Pope

lines on the record: 143
first: 2026-04-29
most recent: 2026-04-29
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“So this actually can be the largest memory footprint during training. And so this is normal training. And then I run the backwards pass and I read it kind of in reverse. Like I run them sort of forward pass goes forward, backward pass goes backwards, and I have to read them back out. The idea of this RevNets paper is that because it's invertible, I don't need to store this at all. I can completely rematerialize it. When I'm running my backwards pass, so I run my forwards pass, and then when I'm running my backwards pass, I'm simultaneously in lockstep undoing all of the forwards pass steps that I did in order to have the activations that I need here. So this ends up being a memory saving, which is a nice idea.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“And then this becomes the new X, the output X. And then this x. Comes the output So, really, what this is doing, this is actually sort of doing, if you think of two layers back, this is actually the thing you mentioned before, it's actually doing the residual connection from two layers back, like this why it came from the previous layer and was the residual connection there. But because of this construction, the whole thing is invertible. Why do I care? What does Invertible Metaphor? The big thing that it can be interesting for is for training. If I think of a forward passive training, so I will, let's say I have four layers, I run them in 0, 1, 2, 3 order. I have to write all of the activations to HBM. And so I get an HPM footprint here that is kind of like linear in a number of layers”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“This has actually been ported over into Into neural nets. There's a 2017-18 paper called RevNets, Reversible Networks. And what it does is it actually makes the entire, like you can apply it to any network, like a transformer network. I do a forwards pass, but then I can actually run the entire pass backwards as well. So the whole neural network is invertible. With exactly this construction. And so this paper reversible networks applied to some layer, like a transformer layer, for example. We've got this function f, which is our transformer layer. Now, normally we would have just an input and then a residual connection coming out. And it gets added like this over here. But now the variation of this is going to be we've got two input x and y. So we've got x. Y input X goes through the function, gets added to y”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“We need to actually remember what x was. So we're going to stick x over here so that we can work backwards. And then we also count drop y. So we're going to remember y, and we're going to add them together. And so we form this tuple. So, the way to invert this, like if you think I have this output and I want to recover x and y, well, I can easily recover x. That's right there. I just read it off. And then to recover y, if this thing was called z, I can recover y by z minus f of x because I've already recovered x. So that means that this construction is invertible. This was used in Cypher's a ton. Still is used. It's one of the main mechanisms of constructing ciphers. Often you want ciphers to be invertible, especially the layers of ciphers you want to be invertible, because that has better cryptographic properties.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so using neural nets for cryptography, well, in general, cryptography like creating a new cipher is a very, very dangerous proposition. Like almost all of them are broken, like 99% of them are broken. Totally a bad place to start. But the other direction has been very, in at least one very clear case, quite productive. So there's this construction that exists in ciphers and then was imported into neural nets called a Feistel cipher network. So the idea is that you may have some function f which is not invertible. But you like the function because it does interesting things, for example, or it mixes in an interesting way. You'd like to build something out of this that is invertible. So, the construction we're going to make is going to actually be a two input function rather than a one input function. We're going to apply.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“This is like an adversarial. So, yeah, I mean, in fact, this is actually a place where you get exactly the sort of avalanche property that ciphers have as well. Adversarial attacks Models, right? Can I find a perturbation of the image that a very, very small perturbation of the image that totally changes the class location? Don't totally change the That is the common case in ciphers, whereas the undesired case in neural nets for sure”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“The whole job of a well designed cipher is to make the difference in very large. So, I guess the distinction is that the optimization goals at that point are about complexifying. They don't have the same residual connections or layer norms.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“One of the biggest attacks against cryptographic ciphers is also to differentiate the cipher. Ciphers run in a different number field. They run in the field of two elements, so just binary, whereas neural nets run in theory in the field of real numbers. And so you have to differentiate with respect to binary numbers. You can absolutely differentiate a cipher. This is called differential cryptanalysis. And basically what it says is that if you take a small difference of the input, it's quite difficult to make the difference of the output be small.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“The thing that makes it interpretable is the gradient descent. So you can differentiate a neural network and get a meaningful derivative. And we do a lot of work to Like not overcomplicate the derivative. So the residual connection keeps it contained and simple, and so does the layer norm stuff that we do.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So, I mean, the mixing, I try to look for other examples where mixing, like scrambling, mixing shows up as well. There's actually almost even a physical example where you're stirring something, you're making a cake and you want to stir the batter. And literally the idea, like first stir it this way and then stir it this way is like actually not too bad of an approach. But beyond that, like back to the digital world. There are some differences. And the one you call out is a pretty strong difference. The way it shows up, what makes neural nets, like if you just randomly initialize a neural network, actually, maybe it's a reasonable cryptography cipher as well, because the random initialization is going to jumble stuff in a complicated way. It may even do what you want, who knows”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, basically. I think that probably indicates that the two tiers are flash and spinning disc. I'm kind of shocked to see spinning disc being used at all because it's such an old technology.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Because, like, this is the retrieval time. Yeah, I mean, this is. This is how many other things I can hold simultaneously.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So you've got a bunch of different memory tiers like we've listed four of them. Your choice, like your choice of which memory tier is like, you want to minimize the cost. And so What fraction of the device are you using? You're using some fraction of the device for the holding onto it, and then you're using some fraction of the device to retrieve it. And so let's say I'm using 10% of the device. And I want to equalize those two fractions. That's a sign that I've hit the right thing. So let's say I've got some runtime here. I'm going to hold on for all of this time and then so this is the time hold. And then there's going to be some amount of time here, which is time to retrieve. And I want, I mean, basically to equalize the costs, these two costs. I want the retrieval time to be equal to the hold time. Times the fraction of capacity.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“DDR could be about an order of magnitude or too often this, and so this is probably in the order of like, actually, I think it might even be in the seconds, like 1 to 10 seconds. This is really, I don't have these numbers memorized, but generally as you go to slower tiers, Flash is plausibly in the order of one minute. And then spinning disk, which is massively different, I think is on the order of one hour. So this might actually identify that the tiers are probably flash and spinning disk.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, exactly. I think this will probably end up being, it's going to be the drain time of the memory tier that you're in. And so what that means is, like, given that I know I'm going to be holding something for five minutes, I would like to pick a memory that I can read every five minutes. Like I can read the whole memory once per five minutes ballpark. So that is the drain time of the memory. So if I take all the storage storage capacity over storage bandwidths. I would like this to be equal to five minutes or something like that. And so, actually, we did this calculation for HPM. For HPM, we know that this number is 20 milliseconds. So HPM is much too short, like much too small.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So, actually, we might actually be able to determine which memory terrier it is by the durations, actually. The duration probably tells it to you, actually.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“I think that's like a pretty good assumption. If you look at the numbers, it might also turn out that it's one tier down and it's DDR versus Flash is there.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, for example So these are three things, and I guess I ordered them wrong. In general, if you're balancing two costs and you've got different memory, different tiers in the memory hierarchy, you should expect as this cost goes up, this cost should go down. So you can kind of see where the zeros are and like I should have ordered them this one first, this one second, and this one third. If you're going to hold on to it for a very short amount of time. Then all of this is like multiplied by the hold time. One is, and so is this one”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I mean, it depends what you define a retrieve to be here. I'm defining retrieve to be move it into HVM so that you can start actually doing inference on it. And so sort of by definition.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“And so this is like, this is a cost per second, whereas this is like an instantaneous cost. So rematerialization has a cost to retrieve and has zero cost to storage because we've deleted it. This is the one that I put in the wrong location. This is actually the cost to hold on. So I will rewrite it. So we have this, like if we're just storing it in HBM, it has this sort of cost profile. And then if we store in DDR, it's actually going to take some time. So it's like we get the same thing here by... Per token over DDR capacity Cost. Second, but now this has a cost to retrieve that is higher than the HPM because we need to copy it into the HPM. And so this is a token. Over DJ bandwidth. And then this consumes some amount of the DDR as well.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“And then I need to store this in the HPM. So it's going to use up some of my HPM capacity So, a way to think of this is that if I have too many of these things sitting in HPM, if I fill up my HPM with just KB caches that I'm not using, I can't use that GPU. And so how do I price that? Maybe I say that the cost of it is proportional to the fraction of the HPM I'm using. So there's also times GPU dollars. And then let's just do one more memory tier and say something like DDR, store in DDR instead. The same kind of thing goes up for Flash and for DDR. I put these in the wrong columns, actually. I meant to make two columns. The distinction I want to make is that there is cost to retrieve. And then there's cost to store, cost to hold on.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Is a big improvement. Maybe that is priced in already, perhaps. It's not an infinite improvement because if you go too sparse, you lose too much quality. But yeah, I mean, the empirical result is that the context things haven't been increasing that much. And I think it's because there is no solution to the memory wall here. So going too sparse just means like you're attending to a very small subset of the tokens and the quality will get So, what is the cost of these different ways of producing resynthesizing the KV cache? Computing it from scratch is based on my GPU time. I have to do a certain amount of multipliers in order to, of GPU time that I spend in order to produce it. Storing HBM. This really goes as my, I think I had a number here, which was the bytes per So, I need to just have some number of bytes per token.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“The memory bandwidth cost, yeah. I actually don't see a very good path to solving that. The HPM is where it's at where it is. It's not getting hugely better.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, sparse attention gives you a get out for sure because you get this square root gives you a big improvement. But I think it's like if you look at the history of context lengths of models, From earlier models like GPT-3, maybe to GPT-4, I don't remember when the transition happened exactly. They shot up from about 8K to 100K to 100K. And then for the last year or two, they've all been hovering around there. I think that actually indicates... That is sort of the reasonably balanced cost point and going massively beyond that would be cost prohibitive.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so there are two costs of long contact. One is the memory bandwidth cost, which we've spent a lot of time analyzing. That's this thing. And then the other one is the compute cost. The compute cost is almost always and sort of actually forced by Fundamental principles to be a much smaller slope than the memory bandwidth. The primary thing that limits you to have really large contexts are memory bandary capacity. Which is exactly this effect.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so there is a quadratic term. It shows up in the compute. As an approximation, I chose to remove it. I will just show you sort of quickly what that looks like. It's because you have the If you look at the cost per token or the number of flops per token, there is the flops that are coming from doing the weight matrix multipliers as a function of context length. And then there is the number of multipliers that comes from doing the KV cache, which goes up linearly with the amount of stuff you attend to. The slope on this is so low that when you draw it like this, it's like it's very well approximated by a flat. So, like it starts to, like, you start to notice the effect of the quadratic or the linear term up in the millions of tokens or so. So just not super relevant.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So we rematerialize it. And so basically, this is going to cost the length of the context. Actually, we'll look at cost per token so that we don't need to carry around this length of context everywhere. So to rematerialize one token of KVC, I just need to run a forward pass on a whole model. And then, so this is going to be the compute time. I have to rerun the compute. Whenever speed my GPU does it, and then I multiply it by my GPU. Per second”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So the cost ratio is really talking about the ratio between those two mechanisms for producing it. A cash miss means you've deleted it from all your memories and you have to recompute it on the tokens directly. In fact, maybe even take that a step further and think about which memory tier do you store it in. So you could store it in HPM. There are other slower and cheaper memories than HPM, like DDR on your host or flash as well. And so one of the things you can do is a calculation of where it makes sense to be in each memory tier. And this is related to how long you're going to store for. So we want to look at the cost of storage in a few different memory tiers and also the cost of free materialization. So remap means the cost to rebuild all of the KV cache from scratch after you delete it.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Right. So there's two ways you can produce tokens or the KV cache for a token. You can just produce it from scratch by computing it from the underlying token IDs, which are tiny. Or you can previously have produced it and stored it in a memory somewhere.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Why is it cost higher? Yeah, y So, I mean, we're going to, it's this division by length path that actually makes it all. This is going to divide out, but then we're going to get all of this is going to divide the length of pass. And it's going to make the memory cost cheaper.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, actually, for processing the entire batch, so in this, like at this cost, we have processed this many tokens like letter prefill. I guess all the paths. Yeah, not this prefix, but it's this cost.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So, yeah, there has the time for one pass, but actually the amount of tokens is that much larger. So I guess we want the COSPR token, in fact, or the time per token.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“I think it's sort of less draw actually how prefill shows up here, if I may clarify. So we do a bit of decode like this. We may actually come back and do more prefill. If you think this is a chat session, the user says something, the AI generates response, and then the user says something else and we prefill this. So maybe this is the more common, like this is the general case rather than this.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Of the past in general. Yeah. If we say, if we can think of decoders being a past with one and then prefill being a pass with many.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh, this is, this number is actually the same for. Oh, sorry, this D head is the dimension of the vector. A number of KV heads is typically in the range of one to eight. So It is totally plausible to get this by, for example, having eight kV heads and a D head of 128. That gives you exactly this number. Or you could have fewer KV heads, a bit more layers. So this is one way to get there via dense attention. There's also a way to get there via sparse attention where you increase all of these numbers, but then you have a run over sparsely term. So, yeah, I mean, I think this number is plausible, if maybe a little bit small.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“The KV heads are the heads that are stored in memory, like store the contents of the previous tokens. The Q heads are the retrieval heads there. They only use temporarily and they're used by the attending token. So in this auto-aggressive context. I've got KV heads associated with all of the context, and then Q heads associated with this new token here.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So this is the number of unique contexts per layer. Do you share the context across many layers or do you use it only once? So in character AI-like models, this number is one. We said this is 128. And this is a choice which typically ranges from KV heads, I meant.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“That is plausible, actually. So you said around two kilobytes. So let's just do a sanity check for this could be. There are two mechanisms that people do attention with a small number of bytes per token. One is dense attention with a lot of reuse across layers. So character AI has a blog post talking about that, alternating long and short context. And in the character AI kind of model, which also showed up in the GEMA models, the global context, which is really what we're talking about here, global context, was shared across all the layers. And so to get this two kilobytes, you could get that, for example, as a D head of 128 is typical. And then the number of bytes is typically Number of attention layers times Two times d head. Times number of Q heads.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Shows up here, and then it cancels out by the time we get to here. And I dropped the land con So we can plug in numbers. This number, this is this reciprocal of the number that we saw before. This is like 1 over 300, which is reasonably stable across many different hardware platforms. We conjecturally said that maybe a number of activated tokens. And length of the context we said was 200K. Something is wrong here. The length of the context should be on the denominator of the numerator.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So we can actually probably even complete that calculation just to see where it lands out. We can solve for the number of bytes per token if we sort of make some assumptions about the number of active parameters. Solving for the number of bytes per token, we're going to assume that the point where we equalize the time of memory and the time of compute is at, let's say, 200k tokens. So we equalize these two. We're also going to just assume that the batch size is large enough, that the memory time spent on weights is negligible, so we'll forget about this, and we'll focus on the actual memory time spent on KB cache. That ends up saying copying this term over batch terms land context times bytes. Token over mem bandwidth. Is going to be equal to number of activated Over Then we're going to solve for bytes per token Size was”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So, I think it says something about, given that the bump is at 200K, it probably means that this is somewhat aligned with this crossover point, maybe not exactly aligned with.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“We'll draw the compute. The cost of the compute is actually constant as a function of context length. There's no dependence here on context length. In reality, there is some dependence, but it is very mild dependence, so we'll ignore it. So this is the time for the compute. One, and then we'll also draw the dependence of the memory fetch on context length. And this starts at a large number for the weights and then grows gradually with the context length. So maybe here and then go gradually with context length. And so you take the maximum and you see there is this inflection point here. So now this is the costs that, for example, Gemini and I might be paying. And then you think how might you put a pricing structure on top of that? You would like to ensure that no matter what the context length is, you are still profitable.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah. So, I mean, why specifically 50%? Let's sort of, so the high level, even in the first place is... There is some amount of increasing cost with context length. And we can bring that back up. That was the memory time. This is the compute time. OK. So we've put out these same equations from before of the time for memory fetches, which is the weights and the KB cache, and then the time for the compute, which is just the matrix multiplications for the weights. I will also draw the cost curve. But this time I'll do it as a function of context length instead of as a function of patch size So, this is time over just time. And so this is the cost curve as a function of context next.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“This is why you should just approximate everywhere because there's so big error bars on this. But yeah, no, it's kind of like empowering to just set A equal to B and figure it out.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, like the ratio of this 200 trillion or 100 trillion parameters over the chinchilloptimal of two trillion, that's the amount it's overtrained, which is like a factor of 100 overtrained, perhaps.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Want me to not remove that? That's fine as well. And I think often active params, number of active params, could be in the range of 100 billion, something like that. Maybe a bit larger. So I'm assuming active prames of about 100 billion. And so multiply by 20 to get the chinchilla token count. So Chinchilla, D Chinchilla, would be around 2 trillion. And yeah, and we see we're at 110 times larger than”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“Okay, 2.6 times 10 to the 15th. Okay. This number is probably too large because this is going to be multiple models in a family. Let's make it Five times smaller or ten times smaller or something like that. Okay, so we're estimating maybe 50 million tokens per second per specific model. The model is live for two months. And so this comes out to around 200 trillion tokens. And then we want to compare that to active parameters on a frontier model. I don't actually know the latest rumors, but some... Do you know”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So, I think we just have to make some real world assumptions here in order to do that. So the inference tokens, we should totally be able to catch, right? So let's say a few hundred million, I don't know, maybe it's like 500 million tokens a second just now. I don't really know 500 million tokens a second times a model is deployed for two months before it becomes obsolete. I don't really know. I can't do this in my head. Can you type into your computer?”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“So, I mean, which way are people going to err? Like, if you think that people's power of prediction is not perfect. And also, you run the risk that you make a model that is not a frontier model and then you just throw it away. Then that kind of changes the cost trade off because there's some probability that applies to the inference and you should derate the inference tokens by some amount.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, I think starting with equalizing and cost is right, but depending on how you model the cost this comes close to equalizing in data.”
2026-04-29 · Dwarkesh Podcast · Reiner Pope – The math behind how LLMs are trained and served · IDENTIFIED FROM THE TRANSCRIPT · source