YouSaid · the spoken record

Leopold Aschenbrenner

lines on the record: 332
first: 2024-06-04
most recent: 2024-06-04
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“Yeah, I mean, I think there's two kinds of key things. There's the unhobbling and there's the data wall, right? And I think we should talk about the data wall for a moment. I think the data wall is, even though kind of like all this stuff has been about crazy eye progress, I think the data wall is actually sort of underrated. I think there's like a real scenario where we just stagnate. Because we've been running this tailwind of just like, it's really easy to bootstrap and you just do unsupervised learning next token prediction. It learns these amazing world models like bam, you know, great model. And you just got to buy some more compute, you know, do some simple efficiency changes. And again, like so much of deep learning, all these big gains on efficiency have been pretty dumb things, right? Like, you know, you add a normalization layer, you fix the scaling laws, and these already have been huge things, let alone kind of like obvious ways in which these models aren't good yet. Anyway, so data wall, big deal. I don't know, some like.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“So, I think the story is sort of like basically, I think it's a little bit more continuous, you know, right? I think already, you know, like I talked about, you know, 25, 26, you're basically going to have models as good as a college graduate. I don't know where the unhobbling is going to be, but I think it's possible that even then you have kind of the proto-automated engineer. So I think there is a bit of a smear kind of an AGI smear or whatever, where it's like there's sort of unhobblings that you're missing. There's kind of like ways of connecting them you're missing. There's like some level of intelligence you're missing. But then at some point you are going to get the thing that is like 100% automated Alec Bradford. And once you have that things really take off, I think.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“And, you know, so there won't just be quantitatively superhuman. They'll pretty quickly become kind of qualitatively superhuman. It's sort of like, look, you're a high school student, you're like trying to wrap yourself your mind around kind of standard physics. And then there's some super smart professor who is like, quantum physics all makes sense to him. And you're just like, what is going on? And sort of, I think pretty quickly, you kind of enter that regime. Just given even the underlying pace of AI progress, but even more quickly than that, because you have this sort of accelerated force of now this automated AI research.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“And then when it does algorithmic progress, it'll figure out how to flip a burger. But, you know, look, the other thing is about, you know, again, these are the quantities are lower bound, right? So it's like this is just like, we can definitely run 100 million of these. Probably what will happen is one of the first things we're going to try to figure out is how to, again, run translate quantity into quality. And so it's like even at the baseline rate of progress, you're quickly getting smarter and smarter systems, right? If we said it was like, you know,”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Okay, great. This is great. Okay, so a few responses to that. First of all, I mean, I totally agree with the kind of real world bottlenecks type of thing. I think this is sort of, you know, easy to underrate. Basically, what we're doing is we're removing the labor constraint. We automate labor, and we kind of explode technology. But there's still lots of other bottlenecks in the world. And so I think this is part of why the story is that kind of like starts pretty narrow at the thing where you don't have these bottlenecks and then only over time as we let it kind of expand to sort of broader areas. This is part of why I think it's like initially this sort of AI research explosion, right? It's like AI research doesn't run into these real world bottlenecks. It doesn't require cloud field or dig up coal. It's just you're just doing AI research. The other thing about...”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Sense it's a sort of there's lots of clear metrics, it's all virtual, there's code, it's things you can kind of develop and train for.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Basically, agree, right? I think my story here is like, I talk about, I think there's going to be some long tail, right? And so maybe it's like 26, 27, you have like the proto-automated engineer and it's really good at engineering. He doesn't have the research intuition yet. You don't quite know how to put them to work. But the sort of even the underlying pace of AI progress is already so fast, right? In three years from not being able to do any kind of like math at all to now crushing, crushing these math competitions. And so you have the initial thing in like 26, 27, maybe the sort of, it's an automated research engineer. It speeds you up by 2x. You go through a lot more progress in that year. By the end of the year, you figured out the remaining kind of unhobbling. You've got a smarter model. And maybe then that thing, or maybe it's two years, that thing just like that thing really can do automate 100%. And again, they don't need to be doing everything. They don't need to be making coffee. They don't need to like, you know, maybe there's a bunch of tacit knowledge and a bunch of other fields. But, you know, AI researchers at AI Labs really know the Java of an AI researcher.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Initial creation. Internet tokens are usually final output, right? A lot of these tokens, if we talk, we talked about the unhobbling, right? And I think of a kind of like, you know, a GPDN token as sort of like one token of my internal monologue. And so that's how I do this math on human equivalence. It's like 100 tokens a minute. And then, you know, humans working for X hours. And what is the equivalent there?”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Final thing I'll say is the 100 million human equivalent AI researchers, that is just a way to visualize it. So that doesn't mean you're going to have literally 100 million copies. So there's trade-offs you can make between serial speed and in parallel. So you might make the trade-off is, look, we're going to run them at 10x, 100x serial speed. It's going to result in fewer tokens overall because of sort of inherent trade-offs. But then we have, I don't know, I don't know what the numbers would be, but then we have 100,000 of them running at 100x human speed and thinking. And there's other things you can do on coordination. They can kind of share latent space, attend to each other's context. There's basically this huge range of possibilities of things you can do. The 100 million thing is more, I mean, another illustration of this is, you know, if you kind of, I run the math in my series and it's basically 27, 28, you have this automated AI researcher, you're going to be able to generate an entire internet's worth of tokens every single day. So there's clearly sort of a huge amount of intellectual work they can do.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“I don't know what else. It's like, what does it take to be Nalc Redford? Well, there's a sort of cultural acclimation aspect of it, right? You know, if you hire somebody new, there's like politicking, maybe they don't fit in. Well, in the AI case, you just make replicas, right? There's motivation aspect for it, right? So it's like, you know, Alec, you know, if I could just duplicate Alec Radford, and before I run every experiment, I have him spend like, you know, a decade's worth of human time double checking the code and thinking really carefully about it. I mean, first of all, I don't have that many Bradfords. And he wouldn't care and he would not be motivated. But the AIs, it can just be like, look, I have 100 million of you guys. I'm just going to put you on just really making sure this code is correct. There are no bugs. This experiment is thought through. Every hyperparameter is correct.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Had Dario on the podcast Extremely selective. We better make sure training is not as easily scalable, right? So training is very hard. You know, if you just hired 100,000 people, it's like, I mean, you couldn't train them all. It would be really hard to train them all. You wouldn't be doing any eye research. There's huge costs to bringing on a new person training them. This is very different with AIs, right? And I think it's really important to talk about the sort of like advantages AIs will have. So it's like, you know, training, right? It's like, what does it take to be an Alec Bradford? You know, you need to be in a really good engineer, right? The AIs, they're going to be an amazing engineer. They're going to be amazing at coding. You can just train them to do that. They need to have, you know, not just be good engineer, but have really good research intuitions and really understand deep learning. And this is stuff that, you know, Alec Radford or somebody like him has acquired over years of research over just like being deeply immersed in deep learning, having tried lots of things himself and failed. AIs, you know, they're going to be able to read every research paper.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Come online in 2026 or something. Some of them won't work out, some of them won't have the traits we like Yeah, and so sometimes this happens, right? Like smart physicists have been really good at AI research. It's like all the anthropic co-founders.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“In future models A genius in the world. All right, great. So let's talk about the Open AI example and let's talk about the automated AI researchers. So, I mean, the Open AI case, I mean, just, you know, just kind of like look at the inflation of AI researcher salaries over the last year. I mean, I think like, I don't know, I don't know what it is, you know, 4x, 5x. It's kind of crazy. So they're clearly really trying to recruit the best AR researchers in the world. And, you know, I don't know. They do find the best AR researchers in the world. And I think my response to your thing is almost all of these 150 IQ people, you know, if you just hire them tomorrow, they wouldn't be good AI researchers. They wouldn't be in Alec Radford.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“So, I think there's some process like this. I think it's totally plausible that institutions have gotten worse by some factor. Obviously, there's some sort of exponent of diminishing returns on more people. So serial time is better than just parallelizing. But still, I think it's clearly inputs matter.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Isn't it a coincidence that supply equals demand on the market clears? And the same thing here, right? So it's getting how much ideas have gotten harder to find as a function of how much progress you've made. And then what the overall growth rate has been is a function of how much ideas have gotten harder to find in ratio to how much you've been able to increase research effort. What is the sort of growth of log cumulative research effort? So in some sense, I think the story is sort of like fairly natural. And you see this, you see this not just economy-wide. You see it in kind of experience curve for all sorts of individual technologies.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Sense, I think this is a natural story. Now, one objection kind of people then make is like, oh, isn't it suspicious, right? That like ideas, you know, well, we increased research effort 10x and ideas also just got 10x harder to find. And so it perfectly equilibrates. And there I say, you know, it's just an equilibrium. It's an endogenous equilibrium, right? So it's like, you know,”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Poverty, whatever Same sort of level is no longer deliberative to the 100x point, right? So I don't know if it's 100x. I think it's easy to inflate these things, it's probably at least 10x. And so people are sometimes like, ah, you know, like, you know, come on, ideas haven't gotten that much harder to find. You know, why would you have needed this 10x increase in research effort? Whereas to me, I think this is an extremely natural story. And why is it a natural story? It's a straight line on a log log plot. This is sort of a deep learning researcher's dream, right? What is this log log plot? On the x-axis, you have log cumulative research effort. On the y-axis, you have log GDP or ooms of algorithmic progress or log transistors per square inch or in the sort of experience curve for solar kind of like whatever the log of the price for a gigawatt of solar and it's extremely natural for that to be a straight line you know this is sort of a class yeah it's a classic and um you know it's basically the first thing is very easy then basically you know you have to have log increments of cumulative research effort to find the next thing um and so you know”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, yes, if you can highly select all the best AI researchers in the world, he might only need a few hundred. But if that's the talent pool, it's like you have 300 best AI researchers in the world.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Okay, great. So, this is getting into some good stuff. I had classic disagreement I have with Patrick and others. All right, so obviously inputs matter, right? So it's like the United States produces a lot more scientific and technological progress than Liechtenstein or Switzerland. And even if I made Patrick Collison dictator of like Liechtenstein or Switzerland and Patrick Collison was able to implement his utopia of ideal institutions, keeping the talent pool fixed, he's not able to do some crazy high school immigration thing or whatever, some crazy genetic breeding scheme or whatever he wants to do, keeping the town pool fixed, but amazing institutions. I claim that still, even if you made Patrick Collison Dictator Switzerland, maybe you get some factor, but Switzerland is not going to be able to outcompete the United States in scientific and technological partners. Obviously, magnitude is better.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Would tell me security is our number one priority. But then invariably when it came time to sort of invest serious resources, when it came time to make trade-offs, to sort of take some pretty basic measures, security would not be prioritized. And so, yeah, I think it's the cognitive dissonance, and I think it's the sort of unreliability that causes a bunch of the drama.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“And then, I mean, I think the thing that Know, really gets people is the sort of tendency to kind of then make commitments and sort of like, you know, they say they take these issues really seriously, they make big commitments on them, but then sort of frequently don't follow through, right? So, you know, again, as mentioned, there's this commitment around Super Lime and Compute, so 20% of compute for this long-term safety research effort. And I think you and I could have a totally reasonable debate about what is the appropriate level of compute for superalignment. But that's not really the issue. The issue is that this commitment was made and it was used to recruit people. And it was very public. And it was made because there's a recognition that there would always be something more urgent than a long-term safety research effort, like some new product or whatever. Then in fact, they just really didn't keep the commitment. And so there was always something more urgent than long-term safety research. I mean, I think another example of this is, you know, when I raised these issues about security,”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Where a lot of the drama comes from is open AI really believes they're building AGI, right? And it's not just a claim they make for marketing purposes, whatever. There's this report that Sam is raising $7 trillion for chips. And it's like, that stuff only makes sense if you really believe in AGI. And so I think what gets people sometimes is sort of the cognitive dissonance between sort of really believing in AGI, but then sort of not taking some of the other implications seriously. This is going to be incredibly powerful technology, both for good and for bad. And that implicates really important issues like the national security issues we spoke about, like, you know, are you protecting the secrets from the CCP? Does America control the core AGI infrastructure or does it a Middle Eastern dictator control the Corey GI infrastructure?”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“What's going on? Yeah, there's a lot of trauma. So, why is there so much drama? I think there would be a lot less drama if all OpenAI claimed to be was sort of building ChatGPT or building business software.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, I think to be clear, the leak allegation was just that sort of document I should get for feedback. This is just sort of a separate thing that they cited and they said, I wouldn't have been fired if not for the security memo.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Look, I don't know what the whole situation is. I certainly think sort of vested equity is pretty rough if you're conditioning that onto NDA. It might be a somewhat different situation if it's a sort of severance agreement.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“No, my situation was a little different in that I was sort of, I was basically right before my cliff. But then they still offered me the equity, but I didn't want to sign the non-disparagement. Freedom is priceless.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“All that being said, I think I really want to emphasize there's just a lot of really incredible people at OpenAI and it was an incredible privilege to work with them. And overall, I'm just extremely grateful for my time there.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“I think in some sense, I think it would have been perfectly reasonable for them to come to me and say, look, we're taking the company in a different direction. We disagree with your point of view. We don't trust you enough to sort of toe the company line anymore. Thank you so much for your work at OpenAI, but I think it's time to part ways. I think that would have made sense. I think we did start sort of materially diverging on sort of views on important issues. I'd come in very excited and aligned with OpenAI, but that sort of changed over time. Look, I think that would have been a very amicable way to part ways. And I think it's a bit of a shame that this is the way it went down.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“I guess the sort of two senior most people didn't sign were Andre and yeah. And you know, I mean, on the letter, by the way, by the time on sort of Monday morning, when that letter was going around, I think probably it was appropriate for the board to resign. I think they kind of like lost too much credibility and trust with the employees. But I thought the letter had a bunch of issues. I mean, I think one of them was it just didn't call for an independent board. I think it's sort of like basics of corporate governance to have an independent board. Anyway, it's other things in sort of other discussions, I pressed leadership for sort of opening eye to abide by its public commitments. I raised a bunch of tough questions about whether it was consistent with the Open AI mission and consistent with the national interest to sort of partner with authoritarian dictatorships to build the core infrastructure for AGI. It's a free country, right? That's what I love about this country. We talked about it. And so they have no obligation to keep me on staff.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Anyway, so that's what they allege. That's what happened. I've spoken to kind of a few dozen former colleagues about this since I think the sort of universal reaction is kind of like that's insane. I was sort of surprised as well. I had been promoted just a few months before. I think Ilya's comment for the promotion case at the time was something like Leopold's amazing. We're lucky to have him. Look, I mean, I think the thing I understand, and I think in some sense is reasonable, is like, you know, I think I ruffled some feathers, and I think I was probably kind of annoying at times. It's like security stuff and I kind of like repeatedly raised that and maybe not always in the most diplomatic way. I didn't sign the employee letter during the board events”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“And then they also claim that I was engaging on policy in a way that they didn't like. And so what they cited there was that I had spoken to a couple external researchers, you know, somebody got a think tank about my view that AGI would become a government project, as we discussed. In fact, I was speaking to lots of sort of people in the field about that at the time. I thought it was a really important thing to think about. Anyway, and so they found, you know, they found a DM that I'd written to a friendly colleague five or six months ago where I relayed this and they cited that. And I had thought it was well within OpenIN Norms to kind of talk about high-level issues on the future of AGI with external people in the field.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Employees, the security memo. There's a couple other allegations they threw in. One thing they said was that I was unfortunate during the investigation because I didn't initially remember who I had shared the doc with, the sort of preparedness brainstorming doc only that I had sort of spoken to some external researchers about these ideas. And look, the doc was over six months old. I'd spent a day on it. It was a Google Doc. I shared with my OpenAI email. It wasn't a screenshot or anything. I was trying to hide. It simply didn't stick because it was such a non-issue.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Warning I'd gotten for the security memo. Anyway, and I mean, some other, you know, what might also be helpful context is the sort of questions they ask me when they fired me. So this was a bit over a month ago. I was pulled aside for a chat with a lawyer that quickly turned very adversarial. The questions were all about my views on AI progress, on AGI, on the level of security appropriate for AGI, on whether government should be involved in AGI, on whether I and superalignment were loyal to the company on what I was up to during the Open AI board events, things like that. And, you know, then they chatted to a couple of my colleagues and then they came back and told me I was fired. They'd gone through all of my digital artifacts from the time at OpenAI, messages docs. And that's when they found the leak. Yeah. And so anyway, so the main claim they made was this leaking allegation. That's what they told.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“It was sort of unconstructive You know, look, I think I probably wasn't at my most diplomatic. I definitely could have been more politically savvy. But I thought it was a really, really important issue. The security incident had made me really worried. Anyway, and so I guess the reason I bring this up is when I was fired, it was sort of made very explicit that the security memo was a major reason for my being fired. I think it was something like the reason that this is a firing and not a warning is because of the security memo.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“It's pretty thin Yeah, I mean, so that was the leaking claim. I mean, say a bit more about sort of what happened in. So, one thing was last year I had written a memo, internal memo, about opening i security. I thought it was egregiously insufficient. I thought wasn't sufficient to protect the theft of model weights or key algorithmic secrets from foreign actors. So I wrote this memo, I shared it with a few colleagues, a couple members of leadership who sort of mostly said it was helpful. But then a couple weeks later, sort of major security incident occurred. And that prompted me to share the memo with a couple members of the board. And so after I did that, days later, it was made very clear to me that leadership was very unhappy with me having shared this memo with the board. Apparently the board had hassled leadership about security. And then I got sort of an official HR warning for this memo, you know, for sharing it with the board. The HR person told me it was racist to worry about CCP espionage.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“A line in the dock about planning for AGI by 2728, and that setting timelines for preparedness. Know, I wrote the stock a couple months after the super alignment announcement. We had put out this sort of four year planning horizon. I didn't think that planning horizon was sensitive. It's the sort of thing Sam says publicly all the time. I think sort of John said on my podcast a couple weeks ago. Anyway, so that's it”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Know sometime last year I had written a sort of brainstorming document on preparedness, on safety and security measures we need in the future on the path to AGI. And I shared that with three external researchers for feedback. So that's it. That's the leak. I think for context it was totally normal at OpenAI at the time to share sort of safety ideas with external researchers for feedback. It happened all the time. The doc was sort of my ideas before I shared it. I reviewed it for anything sensitive. The internal version had a reference to a future cluster, but I redacted that for the external copy. There's a link in there to some slides of mine, internal slides. But that was a dead link to the external people I shared it with. The slides weren't shared with them. So look, obviously I pressed them to sort of tell me what is the confidential information in this document. And what they came back with was”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah. Look, why don't I tell you what they claim I leaked in and you can tell me what you think? Yeah, so OpenAI did claim to employees that I was fired for leaking. And, you know, I and others have sort of pushed them to say what the leak is. And so here's their response in full.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“No, I mean, look, obviously, sort of after the November board events, you know, there are personnel changes. I think Ilya Leaving was just incredibly tragic for OpenAI. And I think some amount of reprioritization. I think some amount of, I mean, there's been some reporting on the Super Lime Compute commitment. There's this 20% compute commitment as part of how a lot of people recruited. It's like, we're going to do this ambitious effort and alignment. some amount of not keeping that and deciding to go in a different direction.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“So, look, OpenAI wanted to do this really ambitious effort on alignment, and Eliot was backing it, and I liked a lot of other people there. And so I was, you know, I was really excited. And I was kind of like, you know, I think there was a lot of people. Sort of alignment, there's always a lot of people kind of making hay about it. And I appreciate people highlighting the importance of the problem. And I was just really into like, let's just try to solve it. Let's do the ambitious effort. Let's do the operation warp.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Thought this was an important problem, and I thought it was just a really solvable problem, right? I thought this was basically, I think there's a, I still do, I mean, even more so do. I think there's a lot of just really promising sort of ML research on alignment on sort of aligning superhuman systems. And maybe we should talk about that a bit more later. It was so solvable.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“We were trying to do was basically kind of be the basic research bat to figure out what the successor to RLHF. And the reason we needed that is basically RLHF probably won't scale to superhuman systems. RLHF relies on sort of human raiders who kind of thumbs up, thumbs down. The model said something, it looks fine, looks good to me. At some point, the superhuman models, the superintelligence, it's going to write a million lines of crazy complex code. You don't know at all what's going on anymore. And so how do you kind of steer and control these systems? How do you add side constraints? The reason I joined was”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, totally. What was the goal of the Super Alignment team? You know, the alignment team at OpenAI, you know, at other labs sort of like several years ago kind of had done sort of basic research and they developed RHF, reinforcement learning from UN feedback. And that was sort of a, you know, ended up being a really successful technique for controlling sort of current generation of AI models.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“He didn't tolerate disagreement well, you know, sort of by the end, he and I just didn't get along well. And I think the reason for that was like there are some biosecurity grants he really liked because they're kind of cool and flashy. And at some point I'd kind of run the numbers and it didn't really seem that cost effective. And I pointed that out. And he was pretty unhappy about that. And so I knew his character. And I think one takeaway for me was I think it's really worth paying attention to people's character and including people you work for and successful CEOs. And that can save you a lot of pain down the line.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Look, obviously, I didn't know he was a fraud, and the whole, you know, I would have never worked. We were a separate thing. We weren't with the business. I mean, I think I do think there are some takeaways for me. I think one takeaway was, I think there's a, I had this tendency, I think people in general have this tendency to kind of like, you know, give successful CEOs a pass on their behavior because they're successful CEOs and that's how they are and that's just successful CEO things. And I didn't know Sam Bankman Fried was a fraud, but I knew SBF and I knew he was extremely risk-taking, right? I knew he was narcissistic.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Closer to home, you know, all that grantees wanted to help them, and we thought they were doing amazing projects. But instead of helping them, we ended up saddling them with a giant problem. Personally, it was a startup, right? And so I had worked 70 hour weeks every week for basically a year on this to kind of build this up. We were a tiny team. And then from one day to the next, it was all gone and not just gone. It was associated with this giant fraud. And so that was incredibly tough. Yeah. And”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Are on a team of four people Yeah. I mean, just, I mean, yeah, future fund, you know, the, yeah, I mean, so that was sort of the heyday, right? I mean, then obviously, you know, when in sort of November of 22, it was kind of revealed that Sam was this giant fraud. And from one day to the next, the whole thing collapsed. That was just really tough. I mean, obviously, it was devastating. It was devastating, obviously, for the people at their money in FTX.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“A little bit of empowerment, kind of like removing excuses, making the process easy. You can kind of get people to do great things.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Future Fund was a foundation that was funded by Sam Bankman Fried. I mean, we were our own thing. We were based in the Bay. At the time, this was in sort of early 2022. It was just incredibly exciting opportunity, right? It was basically like a startup foundation, which is like, you know, it doesn't come along that often that we thought would be able to give away billions of dollars. Thought would be able to kind of remake how philanthropy is done from first principles thought would be able to have this great impact. The causes we focused on were biosecurity, AI, finding exceptional talent and putting them to work on hard problems. A lot of the stuff we did, I was really excited about academics who usually take six months would send us emails like, ah, this is great. This is so quick and easy and straightforward. In general, I feel like I've often find that with a little bit of encouragement.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source
“Just trying to do it. And then it was fine and it worked. That kind of reinforced you don't just have to kind of conform to what the Overton window is. You can just kind of try to do the thing, the thing that seems right to you. And like, you know, most people can be wrong. I don't know, things like that. And I think that was kind of a valuable kind of early experience that was sort of formative.”
2024-06-04 · Dwarkesh Podcast · Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history · IDENTIFIED FROM THE TRANSCRIPT · source