YouSaid · the spoken record

Leopold Aschenbrenner

lines on the record: 332
first: 2024-06-04
most recent: 2024-06-04
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“This person was intentional about it, they could get a lot. I mean, they couldn't get the sort of like, I mean, actually, you could probably just also exfiltrate the code. They could get a lot of the key ideas. Again, like, you know, up until recently stuff was published, but they could get a lot of the key ideas if they tried. I think there's a lot of people who don't actually kind of like look around to see what the other teams are doing. But, you know, I think you kind of can. Yeah, I mean, they could, it's scary”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“What is the state of mind? What is the state of political debate of a sort of average Chinese person or like an average Chinese leader? And yeah, I think that I find that distance kind of worrying. And there's some people who do this and they do really great work where they kind of go through the party documents and the party speeches. And it seems to require kind of a lot of interpretive ability where there's like very specific words and mendering that mean we'll have one connotation and not the other connotation Yeah, I think it's sort of interesting given how globalized everything is. And like, I mean, now we have basically perfect translation machines and it's still so, so impenetrable.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“On the China point, you know, just having this experience of reading German newspapers, and I think how much more poorly I would understand the sort of German debate and sort of the sort of state of mind from just kind of afar, I worry a lot about, or I think it is interesting just how kind of impenetrable China is to me. It's a billion people, right? And like, you know, almost everything else is really globalized. You have a globalized internet. And I kind of a sense of what's happening in the UK. I probably even if I didn't read German newspapers, I sort of would have a sense of what's happening in Germany. But I really don't feel like I have a sense of what like.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“But it's interesting that there's major mistakes, right? Like the sort of defense spending, right? And then Russia invades Ukraine. And you're like, wow, what did we do?”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Right. In China, we're going to get Yeah, yeah, yeah. I mean, I think there's something really important about the sort of like very raucous political debate. And yeah, in general, kind of like, you know, there's the sense in which in America, lots of people live in their kind of like own world. I mean, like we live in this kind of bizarre little bubble in San Francisco and people. But I think that's important for the sort of evolution of idea of error correction, that sort of thing. There's other ways in which the terminal system is more functional.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, exactly, and sort of complete imposition of a new political system and on both sides. Yeah, so it was, but in some sense that worked out better than the post World War I piece, where then there was this kind of resurgence of German nationalism. And, you know, in some sense, the thing that has”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Know, maybe it was sort of a counterreaction against the sort of whole Aryan race and that sort of thing. I mean, I also just think there was a certain amount in what certain Look at World War I, end of World War I versus end of World War II for Germany, right? And sort of a common narrative is that the peace of Versailles was too strict on Germany. But the peace imposed after World War II was much more strict, right? It was a complete, you know, I mean, the whole country was destroyed. Most of the major cities, you know, over half of the housing stock had been destroyed, right? In some birth cohorts, like 40% of the men had died. Half the population.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“I think I kind of think there's a sort of very constrained view of the world in some sense. And that includes kind of, you know, I think after World War II, there's a real backlash against anything like elite. Again, no, no elite high schools or elite colleges and sort of excellence isn't cherished.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“The roads are good and they're clean and they're well maintained. In some sense, the sort of, a lot of this is the sort of flip side of things that I think are bad about Germany, right? So in the US, it's a little bit like there's a bit more of a sort of Wild West feeling to the United States, right? And it includes the kind of like crazy bursts of creativity. It includes like, you know, Political candidates that are sort of, you know, there's a much broader spectrum, and both in Obama and Trump as somebody you just wouldn't see in the sort of much more confined kind of German political debate. I wrote this blog post at some point, you're a political stupor about this. But anyway, and so there's this sort of punctilious sort of rule following that is like good in terms of keeping your kind of state capacity functioning, but that is also”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, yeah, yeah. I mean, look, I'm generally very bearish on Germany. I think in this context, I'm kind of like, you know, it's a little bit, you know, I think you're underrating a little bit. I think it's probably still one of the top five most important countries in the world. Europe overall has, I mean, it's a GDP that's close to the United States the size of the GDP and there's things actually that Germany is kind of good at, right? Like state capacity, right?”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“If alignment is an issue, which we don't fully know yet, but sort of the science will develop, we're going to get better measurements of alignment, and the case will be clear and obvious. Worry that evidence is ambiguous. And I think a lot of this is the most scary kind of intelligence explosion scenarios are worlds in which evidence is ambiguous. But again, it's sort of like if evidence is ambiguous, then that's the world in which you really want the safety margins. And that's also the worlds in which kind of like running the intelligence explosion is sort of like running a war, right? It's like the evidence isn't big US. We have to make these really tough trade-offs and you better have a really good chain of command for that. And it's not just like yo-lowing it. Let's go. It's cool. Yeah.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, they might not be. Know, I agree. I think A few thoughts, right? First of all, I think the private world, even if they sort of nominally care, is extremely tough for alignment. A couple of reasons. One, you just have the race between the sort of commercial labs, right? And it's like you don't have any headroom there to be like, ah, actually, we're going to hold back for three months, like get this right. And we're going to dedicate 90% of our compute to automate alignment research instead of just pushing the next Zoom. The other thing, though, is like in the private world, you know, China has stolen your H. China has your secrets. They're right on your tails. You're in this fever struggle. No room at all for maneuver. They're like the way it's like absolutely essential to get alignment right and to get it during this intelligence explosion to get it right is you need to have that room to maneuver and you need to have that clear lead and you know again maybe you've made the deal or whatever but I think you're an incredibly tough space tough spot if you don't have this clear lead so I think the sort of private world is kind of rough there on like whether people take it seriously you know I don't know I have some faith in sort of sort of normal mechanisms of a liberal society”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Longer chains of thought. And so it's very plausible to me that for the initial automated alignment researchers, we don't need to do any complicated mechanistic interpretability. Which is great. It's a huge advantage, right? However, I'm very likely not the most efficient way to do it, right? There's probably some way to have a recurrent architecture. It's all internal states. There's a much more efficient way to do it. That's what you get by the end of the year. You're going in this year from RHF++. Some extension works to it's vastly superhuman. It's like, it's...”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Many ooms, it ends up being like by the end of it, you have a thing that's vastly smarter than humans. Think the intelligence explosion is really scary from an alignment point of view because basically if you have this rapid intelligence explosion less than a year, two years or whatever, you're going, say in the period of a year from systems where failure would be bad, but it's not catastrophic to saying a bad word, it's like something goes awry to like, you know, failure is like, you know, it extradited itself. It starts hacking the military. It can do really bad things. You're going less than a year from sort of a world in which it's some descendant of current systems and you kind of understand it. And it's like, you know, has good properties. There's something that potentially has a very sort of alien and different architecture after having gone through another decade of maladvances. I think one example there that's very salient to me is legible and faithful chain of thought, right? So a lot of the time when we're talking about these things, we're talking about, you know, it has tokens of thinking and then it uses many tokens of thinking. And maybe we bootstrap ourselves by it's pre-trained, it learns to think in English, then we do something else on top so it can do the sort of”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Of critical issue that comes in is that these I systems are getting superhuman, and they're going to be able to do things that are too complex for humans to evaluate, right? So again, even early on, in the intelligence explosion, the automated AI researchers and engineers, they might write millions, billions, trillions of lines of complicated code. They might be doing all sorts of stuff. You just don't understand anymore. And so in the million lines of code, is it somewhere kind of like hacking or exfiltrating itself or trying to go for the nukes or whatever? You don't know anymore, right? And so this sort of like thumbs up, thumbs down, pure RLHF doesn't fully work anymore. Second part of the picture, maybe we can talk more about this. First part of the picture, I think it's going to be like there's a hard technical problem of what do you do sort of post RHF, but I think it's a solvable problem. And it's like, you know, there's various things in bullish on. I think there's like ways in which deep learning has shaked out favorably. The second part of the problem is you're going from your initial systems and intelligence explosion to superintelligence. And you know, it's like.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Successful if that gets rewarded, that's just reinforced. So basically, I think there's sort of more serious misalignments kind of like misaligned long-term goals that could arise between now, or that sort of necessarily have to be able to arise if you're able to get long horizon system. That's one. You want to do in that situation is you want to add side constraints, right? So you want to add don't lie, don't deceive, don't commit fraud. And so how do you add those side constraints? The basic idea you might have is like RLHF, right? You're kind of like, yeah, it has this goal of make money or whatever, but you're watching what it's doing. If it starts trying to lie or deceive or fraud or whatever, or break the law, you're just kind of like thumbs down, don't do that, you anti-reinforce that.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“They do long term plans, right? They have long, long, you know, they're somehow they're able to act over long horizons, right? You need that, right? That's the sort of prerequisite to be able to do the sort of automated AI research. And so I think there's basically, you know, I basically think sort of pre-training is sort of alignment neutral in the sense of like it has all these representations. It has good representations as representations of doing bad things. But it's not like scheming against you or whatever. I think the sort of misalignment can arise once you're doing more kind of long horizon training, right? And so you're training, again, two simplified example, but to kind of illustrate, you know, you're training an AI to make money. And if you're just doing that with reinforcement learning, it might learn to commit fraud or lie or deceive or seek power simply because those are successful strategies in the real world, right? So maybe RL is basically it explores. Maybe it figures out like, oh, it tries to hack. And then it gets some money. And that made more money.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, yeah, yeah. So, I don't know if I believe in this concept of sharp left turn, but I do think there's basically, I think there's important quality of changes that happen between now and kind of like somewhat superhuman systems, kind of like early on the intelligence explosion, and then important qualitative changes that happen from early in intelligence explosion to kind of true superintelligence and all its power and might. And let's talk about both of those. And so, okay, so the first part of the problem is one, we're going to have to solve ourselves, right? We have to have to line the initial AIs and the intelligence explosion, you know, the sort of automated. I think there's kind of like, I mean, two important things that change from GPD4, right? So one of them is, you know, if you believe the story on like, you know, synthetic data or L or self-play to get past the data wall. And if you believe this on hobbling story, at the end, you're going to have things, you know, they're agents, right? Including.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Sorry, the other important thing is a bunch of different factions need their own AIs, right? And so it's really important that each political party gets to have their own, you know, and like whatever crazy, you might totally disagree with their values, but it's like, it's really important that they get to have their own kind of like superintelligence. And again, I think it's that these sort of like classical liberal processes play out, including different people of different persuasions and so on. And I don't know, the AI advisors might not make them, you know, wise. They might not follow the advice or whatever. But I think it's important.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Podcast. When I run out of high quality content, I mean, I think there's going to be a process of figuring out what the Constitution should be. I think this Constitution has worked for a long time. You start with that. Maybe eventually things change enough that you want edits to that. But anyway, you want them to, for example, for the checks and balances, they really love the Constitution and they believe in it and they take it really seriously. And look, at some point, yeah, you are going to have AI police and AI military. But I think sort of like, you know, being able to ensure that they believe in it in the way that a Supreme Court justice does or like in the way that like a Federal Reserve official takes their job really seriously. Yeah.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I mean, look, I think sentient AI is a whole nother topic. I don't know if we want to talk about it. I agree that it's going to be very important how we treat them. You know, in terms of what you're actually programming these systems to do, again, it's like alignment is just, it's a technical problem, a technical solution enables the CCP bots. I mean, in some sense, I think the I almost feel like the sort of model and also about talking about checks and balances is sort of like the federal.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“And so, in the world where alignment is salt Yeah, I mean, by removing the worlds in which the AIs take over, then the remaining worlds are the ones where it's like the humans decide what happens. And then as we talked about, there's a whole lot of worlds on how that could go.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, I think in the world where you get the democratic coalition, yeah. I mean, also just alignment is often dual use, right? Like RHF, you know, it's like alignment team developed. It was great. It was a big win for alignment, but it's also, you know, obviously makes these models useful. But yeah, so yeah, alignment enables the CCP bots. Alignment also is what you need to get the sort of whatever USAIs to like follow the Constitution and disobey unlawful orders and respect separation of powers and checks and balances. And so yeah, you need alignment for whatever you want to do. It's just the sort of underlying technique.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Who was that famous author that was? So, yeah, just you know, again, people are predicting deep learning will hold a wall every year. Maybe one year they're right, but it's like gone a long way and it hasn't hit a wall. We don't have that much more to go. And so, yeah”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“I think actually, I mean, the other thing I'll say is I say this 2027 timeline, I think it's unlikely, but I do think there's worlds that are like AGI next year. And that's basically if the test time compute overhang is really easy to crack. If it's really easy to crack, then you do four rooms of test time compute from a few hundred tokens to a few million tokens quickly. And then again, maybe only takes one or two 3.5 to 4 jumps per token. One or two of those jumps per token plus uses test time compute and you basically have the proto-automated engineer.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Say MLPR on the benchmark or whatever, right? And so, you know, like, you know, we mentioned the sort of the LMS differences, you know, RLHF, again, as good as 100x chain of thought, right? Chain of just going from this prompting change, a simple algorithmic change can be like 10x effective compute increases on math benchmarks. I think this is like, you know, I think this is useful to illustrate that on hobblings are large, but I think they're like, I kind of think of them as slightly separate things. And kind of the way I think about it is that at a per token level, I think GP4 is not that far away from a token of my internal monologue, right? Even like 3.5 to 4 took us kind of from like the bottom of the human range to the top of the human range on like a lot of like high school tests. And so it's like a few more 3.5 to 4 jumps per token basis, like per token intelligence. And then you got to unlock the test time. You've got to solve the onboarding problem, make it use a computer. And then you're getting real close.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“The hobblings are just a separate thing. Yeah, exactly. So, this is sort of like, you know, it's, yeah. So, I mean, a few things here, right? Okay, so on the effect of compute scaling, in some sense, I think it's like people center the scaling laws because they're easy to explain and the sort of like why is scaling matter. The scaling laws came way after people, at least, you know, like Gario Ilya realized.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so I mean, I think these cases are pretty different. I don't know. I don't think there is a sort of clear, I don't know how rocket science works, but I didn't get the impression that there's some clear scaling behavior with like, you know, the amount of jet fuel. I think the I think in AI, you know, I mean, first of all, the scaling laws, you know, they've just held, right? And so if you, a friend of mine pointed this out, and I think it's a great point, if you kind of concatenate both these sort of original Kaplan scaling laws paper that I think went from 10 to the negative 9 to 10 petaflop days, and then concatenate additional compute from there to kind of GPD4 assume some algorithmic progress. It's like the scaling laws have held probably over 15 ooms. Probably maybe even more held for a lot of ooms.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“You have a much bigger influence street anyway than you're training. Okay. But I think the experiment compute is a constraint. Yeah. Okay.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“We'll see. We'll see. I mean, the other thing maybe, even if they get like 10x more expensive, then you have 10 million instead of 100 million. So it's like, it's not really”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“I don't know. I think it's not crazy baseline assumption that actually these models, frontier models are not necessarily going to get more expensive per token.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“That's sort of wild. I think it's worth appreciating. And I think it gestures that sort of an underlying pace of algorithmic progress. I think there's a sort of more theoretically grounded way to why inference costs would stay constant. And it's the following story, right? So on Chichilla's scaling laws, half of the additional compute you allocate to bigger models and half of it you allocate to more data. But also if we go with the sort of basic story of half an order of year more compute and half an order of magnitude a year of algorithmic progress, you're also kind of like you're saving half an order of magnitude a year. And so that kind of would exactly compensate for making the model bigger. The caveat on that is obviously not all training efficiencies are also inference efficiencies. A bunch of the time they are. Separately you can find inference efficiencies. So I don't know given this historical trend, given the sort of like baseline sort of theoretical reason.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, we're just doing per token, right? And then I'm just saying, you know, if suppose each model token was the same as sort of a human token thing at 100 tokens a minute. So it's like, yeah, it'll use more, but the sort of, if you just, the token calculations is already pricing that in. The question is per token pricing, right? And so GB3, when it launched, was like, actually, more expensive than GPD4 now. And so over just fast increases in capability gains, inference costs remain constant.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Great. Okay, great question. So I actually don't think inference costs for sort of frontier models are necessarily going to go up that much. So, I mean, one historical data point isn't.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Outputted And if you Super than a human worker, huh? Cheaper than a human worker, but it can't do the job yet. That's right, that's right.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I mean, it depends. I think, again, and it's all the stuff, you know, I have a lot of uncertainty, right? So, a lot of the time I'm trying to kind of tell the modal story. I think it's important to be kind of concrete and visceral about. And I have a lot of uncertainty basically over how the 2030s play out. And basically, the thing I know is it's going to be fucking crazy. But exactly what the bottlenecks are and so on, I think that'll be kind of like, so.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“By the time it gets to equilibrium here, it's like way faster. At least, you know, and it depends on the sort of exponents, but it's basically it's the increase. Like, suppose you need to like 10x effective research effort in AI research in the last four or five years to keep the You know, there's this sort of long run hyperbolic trend. It manifested in the sort of like sort of change in growth mode in the revolution, but there's just this long run hyperbolic trend. And now you have this sort of now you have that another sort of change in growth mode.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“You know, we talked about a 10x of research effort or maybe 10 30x over a decade. Even without any kind of self-improvement type loop, even in the sort of GP40 AGI story, we're talking about an order of magnitude of effective compute increase a year, right? Half an order of magnitude of compute, half an order of magnitude of algorithmic progress. That sort of translates into effective compute. And so you're doing a 10x a year, right? Basically on your labor force, right? So it's like it's a radically different world if you're doing a 10x or 30x in a century versus a 10x a year on your labor force. So the magnitudes really matter. They also really matter on the sort of intelligence explosion, right? So like just the automated AI research part. So one story you could tell there is like, well, ideas get harder to find, right? Algorithmic progress is going to get harder. Yeah, right now you have the easy wins, but in like four or five years, there'll be fewer easy wins. And so the sort of automated eye researchers are just going to be what's necessary to just keep it going, right? Because it's gotten harder. But that's sort of, it's like a really weird knife edge assumption economics where you assume it's just”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Goods have gotten cheaper, or they can manufacture more things. Yeah, it's both. It's both. So it's both that are important. The other thing I'll say is all of this stuff. I think the magnitudes are really, really important, right?”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“You start with one robot factory as producing more robots, and basically this cumulative process because you've taken labor out of the equation.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Again, I agree about there are a lot of real world bottlenecks, right? And so, I don't know, it's quite possible that we're going to have crazy drone swarms, but also lawyers and doctors still need to be humans because of regulation. But I think you kind of start narrowly, you broaden, and then the worlds in which you kind of let them loose, which again, because of, I think, these competitive pressures, we will have to let them loose in some degree on various national security applications. I think quite rapid progress is possible. The other thing, though, is it's sort of, you know, basically in the sort of an explosion after there's kind of two components. There's the A, right, in the production function, like growth of technology. And that's massively accelerated by you. Now you have a billion superintelligent scientists and engineers and technicians, superbly competent everything. You also just automated labor, right? And so it's like even without the whole technological explosion thing, you have this industrial explosion, at least if you let them loose, which is like now you can just build, you know, you can cover Nevada and like.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“A million instances of GPT 6, and you come out the other end I think you're exaggerating the timelines a little bit, but I think decades worth of progress in a year or something, I think that's the reasonable prompt. So I think this is where basically the sort of automate AI researcher comes in because it gives you this enormous headwind on all the other stuff, right? So it's like automate AI research with your sort of automated Alec Radfords, you come out the other end, you've done another five ooms, you have a thing that is vastly smarter. Not only is it vastly smarter, you've been able to make it good at everything else, right? You're like, you're solving robotics. The robots are important, right? Because for a lot of other things, you do actually need to try things in the physical world. I mean, I don't know, maybe you can do a lot in simulation. Those are the really quick worlds. I don't know if you saw the last NVIDIA GTC. You know, it was all about the digital twins and just like having all your manufacturing processes and simulation. I don't know. Again, if you have these super intelligent cognitive workers, can they just make simulations of everything off of style? And then, you know, make a lot of progress.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, but I think it indicates that clearly there's stuff to be done on hobbling. I think, yeah, I think the interesting question is this time a year from now, you know, is there a model that is able to think for a few thousand tokens coherently, cohesively, authentically? And I think probably there's, you know, again, this is what I'd feel better if we had an um or two more data because it's like the scaling just gives you this sort of like tailwind, right? Where like, for example, tools, right? Tools, I think, you know, talking to people who try to make things work with tools actually sort of GPT-4 is really when tools start to work. And it's like, you can kind of make them work with GP 3.5, but it's just really tough. And so it's just like having GP4, you can kind of help it learn tools in a much easier way. And so just a bit more tailwind from scaling. And then, yeah, and does, does, I don't know if it'll work. It's a key question.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“GPT 4 has gotten a lot better since it's actually so the GPD4 gains since launch I think are a huge indicator that there's like you know so you talked about this with Johnson on the podcast John said this was mostly post training gains right you know if you look at the sort of LM sys scores you know it's like 100 elo or something it's like a bigger gap than between cloud 3 opus and cloud 3 haiku and the price difference between those is 60x but it's not more general”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“What they said, right? Like, even like Gemini has like a million contacts, and the million of contacts is actually great for consumption, and it solves one important on hobbling, which is the sort of onboarding problem, right? Which is, you know, a new coworker in your first five minutes, like a new smart high school intern, first five minutes, not useful at all. A month in, you know, much more useful, right? Because they've looked at the monorepo and understand how the code works and they've looked at read your internal docs. And so being able to put that in context, great, solves this onboarding problem. Yeah, but they're not good at sort of the production of a million tokens yet.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, we'll see, right? I mean, on the sort of sample efficiency thing, again, sort of first principles, but I think, again, there's this clear sort of missing middle. And so, you know, in sort of like, you know, People hadn't been trying. Now people are really trying. And so it's sort of, you know, I think often again in deep learning, something like the obvious thing works. And there's a lot of details to get right. So it might take some time, but it's now what people are really trying. So I think we get a lot of signal in the next couple years. On hobbling, I mean, what is the signal on hobbling that I think would be interesting? I think the question is basically are you making progress on this test time compute thing? Is this thing able to think longer horizon than just a couple hundred tokens? That was unlocked by chain of thought.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“I guess earlier we were talking about this sort of like when in the sort of human trajectory are you able to learn from yourself? And so if you go with that analogy, again, like if you'd only gotten the preschooler model, can't learn from itself. If you'd only gotten the elementary schooler model, can't learn from itself. And maybe GP4, smart high school is really where it starts. Ideally, you have a somewhat better model than it really is able to kind of learn from itself or learn by itself. So yeah, I think there's an interesting, I think, I mean, I think maybe one UMLS data I would be like more iffy, but maybe still doable. Yeah, I think it would feel chiller if we had, you know, like one or two. It would be an interesting exercise.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Kick off the RL explosion with. Figure it out The amount of data we happen to have in the world. I mean, three of them is pretty rough, right? Like three oms, if less data means like six rooms, smaller less compute model and chatilla scaling laws. It's basically capping out at like GP2. So I think that would be really rough. I think you do make an interesting point about the contingency.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Which is like 100x effective compute from GP4 is not that much if you do half an order of magnitude a year of compute, half an order of magnitude, a year of algorithmic progress, that's kind of like two years from GP4. So GP4 finished pre-training in 22. So, I think one thing that really matters, I think we won't quite know by end of the year, but 25, 26, are we cracking the data wall?”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Some numbers on this, you know, suddenly you do common crawl online is like 30 trillion tokens, llama three trained on 15 trillion tokens. So you're basically already using all the data. And then you can get somewhat further by repeating it. So there's an academic paper by Boaz Barak and some others that does scaling laws for this. And they're basically like, yeah, you can repeat it sometime. After 16 times of reputation, it just returns basically go to zero. You're just completely screwed. And so I don't know. Say you can get another 10x on data from, say like llama three and llama 3 is already kind of like at the limit of all the data. Maybe we can get 10x more by repeating data. I don't know. Maybe that's like at most 100x better model than GPD4.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source