YouSaid · the spoken record

Holden Karnofsky

lines on the record: 131
first: 2023-01-03
most recent: 2023-01-03
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“I think it's really hard to know how noble the future was in the past and what that means for today. I do think that like when you look back and again, this is something I've looked at a fair amount, when you look back at people trying to predict the future in the past, it just looks like deeply unserious. You could say that future people will say the same about us. I'm sure they'll think we look less serious than they are, but I think there's a difference. I really do think that attempts to kind of like rigorously make predictions about the future, there haven't been a lot of them historically. And I don't think it's obvious that people were doing the best they can and that we can't do better today. So, you know, this population is an example. I'm just, it doesn't seem necessarily true to me that you couldn't have been at that time and said, gosh, you know, population's been going up for a while now and people keep inventing new ways to come up with more resources. Maybe that'll keep.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I think historically people have been overly pessimistic about future technologies. And I think by default, the picture with AI looks just like really scary. It just looks like it would be really easy to get a bad future in a lot of different ways if we just didn't move cautiously. So these two considerations kind of balance each other out a little bit for me. I know a lot of people who just who believe that we're in just deep, deep enormous trouble, that this outcome where you get AI with its own goals, wiping humanity off the map is just like almost surely going to happen. I don't believe that. And this is part of the reason I don't believe it. It's just like I actually think the situation looks very challenging, very scary by default. And I think we're tending to overestimate how bad and how dire things are. And so they balance out a little bit for me.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I don't remember super well. I mean, the book is like a really broad survey of lots of stuff. And I think, you know, an example I might give is he talks about, it's like he talks about the risk of stagnation, for example. So the risk that like growth might just stop or something, or growth might slow to very low levels. And then that kind of implies that what we should be trying to do is make sure we continue to innovate and continue to have growth. But then there's like other parts of the book that kind of make it sound like we shouldn't move too fast and we shouldn't innovate too much because we don't want”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Into preventing it, but that's kind of, you know, that's crossing my threshold. Most things don't. And so if you make a list of ways to make the next million years go well, I'll look at most of them and be like, I don't know. I don't really believe in this. I wouldn't really invest in this. So I think Will is a bit broader than I am in a sense. He's interested in more things and I am pickier than he is because I think it's so hard to know what's really going to affect the long run future that I'm just looking for like a really short list of things that are worth paying special attention to.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I like Will's book. I mean, I think it's worth reading Will McCaskill's book about how the future could be very large and very important. I think he presents a picture in my opinion if you want to talk about the long run future and how to make the long-run future go well, you're starting from a place of like by default almost nothing I can do will actually make sense. Like I do really believe it's hard to understand the long run future and it's hard to make specific plans about it. So I would say compared to Will I am very picky about which issues I think are like big enough and serious enough to actually pay attention to. And so I feel the issue of AI, AI would be transformative enough. It looks likely enough that it'll be soon if it's soon that means there might actually be things we can do that have predictable effects. I think this misaligned AI thing is a real threat the way people design AI systems today would be really bad. I am ready to like put some resources”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“If we got these systems to behave as intended, we still might have problems because we might have humans doing really stupid things and locking in really bad futures. I think that's an issue too. I feel reasonably comfortable, though not 100% confident, saying that we'd like to avoid that just like slip up. We'd like to avoid having these systems that have these random goals we gave them by accident, and they're very powerful and they're better at kind of like setting up the world than we are. And so we get this world that's just doing this random thing that we did by accident. I think that's a thing worth avoiding.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I mostly think of AI alignment as just trying to avoid a really bad thing happening. So I mostly think of it like the thing we don't want to happen is we have some AI system. We thought we were designing it to help us. We were actually designing it to do something extremely random thing. It's just like, again, these systems work by trial and error, by encouragement, discouragement, or positive and negative reinforcement. And so we might have not even noticed that through the pattern of reinforcement we were giving, trained some system to want to put as much money as possible into one bank account or gain as much power as possible or control as much energy as possible or something like that or just set its own reward number, its own score to the highest possible number. I think that would be a form of lock-in if we had systems more powerful than humans that had these kind of random goals. That would be like locking in kind of a future that is not related to the things that humans value and care about. That's an example of a thing I think would be really bad.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“You had this kind of world where you completely control the environment, you might want to lock in the fact that you never have one person with all the power. That might be a thing you might want to lock in, and that prevents other kinds of lock-in.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, I don't think a lot about what I would do with unrealistic buttons where I have crazy amounts of power that I'll never have and shouldn't have. I think of lock-in by default is like mostly a bad thing. I mean, I feel like mostly we want to at least kind of preserve optionality and have a world where it's not just like one person running the show with their values set up the way they want forever. I think of it mostly that way. I can imagine some future world where the civilization's been around long enough we've learned what there is to learn and we know what a good world looks like and most people feel pretty confident about that and they're right to feel confident and maybe lock-in's not so bad. But I mostly think of Lockin as a bad thing. I also imagine that you could sort of like lock in some things about the world in order to avoid lock-in others. So I can imagine if you had this enormous amount of power over how the world works and again this comes down some of this is more explained in my digital people series how this could work.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“On it, I think when you think about if you imagine that we're going to get this explosion in scientific and technological advancement, I think you have to take pretty seriously the idea that that could end by hitting a wall, that there could not be a lot of room for more dynamism, and that we could have these kind of very stable societies. What does seriously mean? I don't know, a quarter, a third, a half, something like that. I don't know. I'm making up numbers. I think it's serious enough to think about it and think about it as something that affects the stakes of what we're talking about.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Everything changes. And a point I'm making the most important century series is that you can imagine a level of technological development where there just aren't new things to find and there isn't a lot of new growth to have and people aren't dying because for whatever reason that seems like it should be medically possible for people not to age or die. And so you can imagine a lot of the sources of dynamism in the world actually going away if we had enough technology. You could imagine a government that was able to actually surveill everyone, which is not something you can do now with a dictator who actually doesn't age or die, who knows everything going on, who's able to respond to everything. And then you can imagine that world just being completely stable. I think this is a very scary thought, and it's something we have to be mindful of, that if the rate of technological progress speeds up a lot, we could quickly get to a world that doesn't have a lot more dynamism and is a lot more stable. What do I think are the odds of this? I don't know. It's very hard to put a probability.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“So Lockin is my term, or I don't know if it's my term, but it's a term I use to talk about the possibility that we could end up with a very stable civilization. And so I talk about that as another. The idea is basically throughout history so far, let's say someone becomes in charge of a government and they're very powerful and they're very bad. Usually this is like generally considered to be temporary in at least some sense, like the kind of thing that's not going to go on forever. There's a lot of reasons the world is just dynamic and the ways the world is tend to just not stay that way completely. The world just has changed a lot throughout history. It's kind of a dumb thing to say, but I'll get to why this might be important. So, you know, if someone is running a country in a really cruel corrupt way, I mean, for one thing, at some point they're going to get old and die and someone else is going to take over and that person will probably be different from them. For another thing, the world is changing all the time. There's new technologies, new things are possible. There's new ideas. And so, you know, the most powerful country today might not be the most powerful tomorrow. The people in power today might not be the ones in power tomorrow. And I think this gets us used to the idea that everything is temporary.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“This isn't our unique in one of my more obscure cold takes posts, but I say AI that could actually take everyone's job, like every human's job, might be a lot harder than AI that could radically transform the galaxy via new technology. Because it might be in some ways, like it might be easier to take a scientist job than a teacher's job or a doctor's job because the teachers and the doctors are regulated. And people might just say, I want human teachers. I don't want an AI teacher. Whereas you can sit there in your lab with your AI scientists and find new theories that change the world. So some of this stuff I think is very counterintuitive, but I can imagine worlds where you get really wacky stuff before you get self-driving cars out on the road just because the way the regulation is working.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Are not making the same kind of progress on like robotics. So weirdly, a task that might be among the hardest ones to automate, or especially things go fast. It might be hard to automate the task of taking this bottle of water and taking off the cap because I have this, you know, I have this hand that is just like well designed for that. Well, it's clearly not designed for that, but these hands that can do a lot of stuff. And we aren't seeing the same kind of progress there. So I think there are a lot of places where AI systems might have kind of their brains can do roughly everything human brains can, but there's some other reason they can't do some key economic task. And I think these are not the tasks I see likely to bottom like the R&D as much.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I think some of the stuff that might be hardest to automate would just be stuff that in some sense doesn't have anything to do with software or capabilities. So an example of something that might just be extremely hard to automate might be like trust, like making a business deal or providing care for someone who's sick. It might just be that even if an AI system has all the same intellectual capabilities as a human, you know, can write poetry just as well, can have just as many ideas, can have just as good a conversation. It just doesn't look like a human, so people don't want that. Maybe it can create a perfect representation of a human on a screen, but it's still on a screen. And in general, I see the progress in AI as being mostly on the software front, not the hardware front. So AIs are able to do a lot of incredible things with language, things with math, things with board games. I wouldn't be surprised if they could write hit music in the next decade or two, but people really...”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“But once you have that, I think you should figure there's just a lot of ways to get around all your other bottlenecks because you have this potentially massive population of thinkers looking for them. And so an example is, you know, you could with enough firepower, enough energy, enough AI, enough analysis, you could probably find a way to simulate a lot of the experiments you need to run, for example.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Called weak point in the most important century, full automation. Part of this is just like you don't need to automate the entire economy to get this crazy growth loop. You can automate just a part of it that are specifically parts that have to do with very important tech, like energy and AI itself. And those actually seem in many ways just less bottlenecked than a lot of other parts of the economy. So you could have, you could be like developing better AI algorithms and AI chips, manufacturing them mostly using robots, using those to come up with even better designs. And then you could also be designing more and more efficient solar panels using those to collect more and more energy to power your AIs so you get like a lot of the crucial pieces here just actually don't seem all that likely to be bottlenecked. And then also, I mean, at the point where you have something that has the ability to have kind of like creative new scientific hypotheses the way a human does, which is a debate over whether we should ever expect that and when.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I think the single best criticism and my biggest point of skepticism on this most important century stuff is the idea that you could build an AI system that's very impressive, that could do pretty much everything humans can do, but there might be like one step that you still have to have humans do, and that could model like everything. And then you could have the world not speed up that much and science and technology not advance that fast because AIs are doing almost everything, but humans are still slowing down this one step or the real world is slowing down one step. So you have to do, you know, let's say real world experiments to invent new technologies and they just take how long they take. I think this is the best objection to this whole thing and the one that I most like to look into more. But I ultimately think that like there's enough reason to think that if you had AI systems that had kind of like human reasoning and analysis capabilities, I think you shouldn't count on this kind of bottleneck causing everything to go really slow. And a lot of that, I mean, I write about that in this piece.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“So the innovation mining metaphors is from another bit of cold takes. It's an argument I make that you should think of ideas as being somewhat like natural resources in the sense of once someone discovers a scientific hypothesis or even once someone writes a certain great symphony, that's something that can only be done once. That's an innovation that can only be done once. And so it gets harder and harder over time to have revolutionary ideas because the most revolutionary, easiest to find ideas have already been found. So there's an analogy to mining. I don't think it applies super importantly to AI thing because all I'm saying is that success by person one makes success by person two harder. I'm not saying that has no impact or that it doesn't speed things up. So just to use a literal mining metaphor, let's say there's like a bunch of gold in the ground, it is true that if you rush and go get all that gold, it'll be harder for me to now come in and find a bunch of gold. That is true. What's not true is that it doesn't matter if you do it. I mean, you might do it.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I think it's common for people who become convinced that AI could be really important to just jump straight to, well, I want to make sure that people I trust build it first. And that could mean my country, that could mean my friends, people I'm investing in. And I have generally called that the competition frame. Want to win a competition to develop AI and have contrasted it with a frame that I also think is important, which is the caution frame, which is that we need to all work together to be careful to not build something that spins out a control and has all these properties and behaves in all these ways we didn't intend. I do think we're likely, if we do develop these very powerful AI systems, I do think we're likely to end up in a world where there's multiple players trying to develop it, and they're all hot on each other's heels. And I am very interested in ways to find ways for us all to work together to avoid disaster as we're doing that, and I am maybe less excited than the average person who first learns about this is about picking the one I like best and helping them race ahead.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, I think in some sense the effects are permanent in that if you cause someone to live a healthier, better life, that's a thing that happened and nothing will ever erase that life or make that life unimportant. But I think in terms of the effects on the future, I do expect it mostly to wash out. I expect mostly whatever we do to make the world better in that way. It will not persist in any kind of systematic, predictable way past these kind of crazy changes. And I think that's probably how things look pre and post-industrial revolution. There's probably some exceptions, but that's my guess.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Treating children for intestinal parasites, you know, open philanthropy does a lot of advocacy for more money going to foreign aid or for better land use policies to have a stronger economy. We do a bunch of like scientific research work that is more aimed at just direct medical applications, especially in poor countries. So I support all that stuff. I'm glad we're doing it. And it's just a matter of how real and how imminent do you think this transformative AI stuff is, the more real and more imminent, the more relatively of our resources should go into it.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, I don't know about Utopia. I mean, my general take is that anything could happen, and I think my general take on this most important century stuff, the reason it's so important is because it's easy to imagine a world that is really awesome and is free from scarcity and that we end up in a really, you know, we see more of the progress we've seen over the last 200 years and end up in a really great place. It's also easy to imagine like a horrible dystopia. But I do think it's, you know, my take is that the more likely you think all of this is, the more likely you think transformative AI is, the more you should think that that should be the top priority that we should be trying to make that go well instead of trying to solve more direct problems at a more short term. I'm not an extremist on this. So, you know, open philanthropy does both. Open philanthropy works on speculative far-off future risks and open fill also does a bunch of more direct work. Again, we do direct and recommend a lot of money to give us top charities, which do things like distributing bed nets in Africa to help prevent malaria.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“And, you know, one of the results that I found most ridiculous and strange and weird was this thing called Minerva, where people took one of these language models and with very little intervention, very little special intervention. They got it to do these difficult math problems and explain its reasoning and get them right about half the time. And this is just not like it wasn't really trained in a way that was very specialized for these math problems. And so I think that's like we just see AI systems kind of having all these unpredictable human likabilities just from having this very simple training procedure. And that is something I find kind of wild and kind of scary. I don't know exactly where it's going or how fast.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I definitely think that recent developments in AI have made me a bit more freaked out. And even since I wrote the most important century series and before that, I mean, there were years when open philanthropy was very interested in AI risk, but it's become more so as we've seen the progress in AI. So I think what we're seeing is we're seeing these very simple systems. It's kind of the same type of system over and over again, very general, and is able to do a lot of different tasks. One of the most interesting, so I think people are interested in this. I think you should maybe Google GPT-3. There's a lot of compilations of what this very simple language model that, by the way, my wife and brother-in-law both worked on, so that's a disclosure. But this very simple language model that just predicts the next word it's going to see in a stream of text, people have gotten it to tell stories, people have got it similar, though not identical models to analyze and explain jokes. People have gotten it to play kind of role-playing games, write poetry, write lyrics, answer multiple choice questions, answer trivia questions.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, I mean, one thing that I would say is that humans have a lot in common with each other. So I think some of history contains cases of humans just kind of learning more about the world, learning more about themselves, debating each other. Maybe I think in a lot of moral progress has just come from humans, like getting to know other humans who they previously were stereotyping and judging negatively and afraid of. So I think there's some way in which humans learning about the world and learning about themselves leads them to have kind of conclusions that are that are more reflective and more intelligent for their own goals. But if you brought in something into the picture that was not a human at all, it might be very intelligent and reflective about its goals. And those goals might have zero value from our point of view.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Of would have the same evolution that I've had in reflecting on morality or would come to the same conclusions I've come to or would come up with moralities that seem good to me. I don't have any reason to think any of that. I do think historically there have been some cases of moral progress.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I kind of don't in particular, so I've kind of used the term moral progress as just a term to refer to changes in morality that are good. I think there has been moral progress, but I don't think that means moral progress is something inevitable or something that happens every time you are intelligent or something like that. So I just, you know, an example I use a lot is just like attitude towards homosexuality, it's a lot more accepted today than it used to be. I call that moral progress because I think it's good. And some people will say, well, you know, I don't believe that morality is objectively good or bad. I don't believe there is any such thing as moral progress. I just think things change randomly. And that will often be an example I'll pull out. And I say, but do you think that was a neutral change? I just think it was good. I think it was good. But that's not because I believe there's some underlying objective reality. It's just my way of tagging or using language to talk about moral changes that seem like they were positive to me. I don't particularly expect that an AI system would have the same”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“You were kind of giving it encouragement every time it was, you know, getting more money into your bank account. If you were encouraging it to do that, you might get something that's very, very good at getting money into your bank account to the point where it's going to disrupt the whole world to do that. And you will not automatically get something that thinks, gosh, is this a good thing to do? And I think a lot of human goals are just like there's not really a right answer about whether our goals actually make sense. They're just the goals we have.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I am not that comforted by that. Pretty much, this is a very, very old debate in the world of AI alignment. Elizer Yakowskowski has something called the orthogonality thesis. I don't remember exactly what it says, but it's something like you could be very intelligent about any goal. You could have the stupidest goal and be very intelligent about how to get it. And in many ways, a lot of human goals.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“You know, if it's in three months, I would probably be trying to hammer out a reasonable test of whether we can demonstrate that the AI system is either safe or dangerous. And if we can demonstrate it's dangerous, use that demonstration to really advocate for a broad slowing of AI research to buy more time to figure out how to make it less dangerous. But I don't know that I feel that much optimism. If AI is like five, you know, this kind of AI is 500 years off, then I'm kind of inclined to just ignore it and just try and make the world better and more robust and wiser. But I think if we've got, you know, 10 years, 20 years, 50 years, 80 years, something in that range, I think that is kind of the place where supporting early careers and supporting people who are going to spend their lives thinking about this, then we flash forward to this crucial time and there's a lot more people who spent their lives thinking about it. And I think that would be a big deal.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I mean, I think the further out things are, the more I think it's valid to say that humans have trouble making predictions on long timeframes. And the more I'm interested in focusing more on other causes or on very broad things we do, such as trying to grow the set of people who thinks about issues like this rather than trying to specifically study, for example, how to get AI systems like today's to behave as intended. So I think that's a general shift. But I would say that I tend to feel a bit more optimistic on longer timeframes because I do think that the world just isn't ready for this and isn't thinking seriously about this. And a lot of what we're trying to do at open philanthropy is create support that doesn't exist in traditional institutions for people to think about these topics. And, you know, that includes doing the AI alignment research. That also includes thinking through how we want politically, what regulations we might want to prevent disaster. I think those are a lot of the things. So it's kind of a spectrum. I would say.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Do and I'm like, I don't know. We did what we could do. Um, you know, like go back in time and like try to prepare for this better. So that would be a lot of my answer. I mean, I could say more specific things about what I'd say in the one to three month timeframe, but a lot of it would be like flailing around and freaking out, frankly.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I would find that extremely scary. And I think I would feel like that's one of the worlds in which I might not even be able to offer an enormous amount. I think so my job is in philanthropy. And a lot of what philanthropists do historically or have done well historically is we help fields grow. We help do things that operate on very long timescales. So an example of something open philanthropy does a lot of right now is we fund people who do research on AI alignment and we fund people who are thinking about what it would look like to get through the most important century successfully. And a lot of these people right now are like very early in their career and just figuring stuff out. And so a lot of the world I picture is like it's 10 years from now. It's 20 years from now. It's 50 years from now. And there's this whole field of expertise that got support when traditional institutions wouldn't support it. And now it's because of us. And then you come to me and you say, we've got one week left.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Didn't have time. So let's say you get a I need to distinguish you may not have time to avoid some of these catastrophes, like huge concentration of power or AI systems that don't behave as intended and have their own goals. If you can prevent those catastrophes from happening, you might then get more time after you build the AIs to have these tools that help us invent new technologies and help us perhaps figure things out better and ask better questions. And then you could have a lot of time or you could figure out a lot in a little time if you had those things. But if someone said, how long did you give me?”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh, yeah. Yeah, yeah. So, yeah, that's a problem that would have to be solved. I mean, we have a lot of today, we have a lot of attention paid to how voting system works, who gets to vote, and how we avoid things being unfair. And yeah, I mean, it's definitely true that if we had decided there was some kind of digital entity that it should have the right to vote and that digital entity was able to copy itself, well, you could get some havoc right there. you'd want to come up with some system that maybe restricts how many copies you can make of yourself or restricts how many of those copies can vote these are problems that i that i'm hoping can be handled in a in a way that while not perfect could be non-catastrophic uh by a society that hasn't been derailed by some huge concentration of power or misaligned ai systems”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, a world that could easily imagine, although that doesn't mean it's realistic at all, is a world where we do build these AI systems. They do what they're supposed to do. And we kind of use them to gain more intelligence and wisdom. I've talked a little bit about this hypothetical idea of digital people. Maybe we develop something like that. And then after a hundred years of this, we've been around and we've been, people have been having discussions in the public sphere and people kind of start to talk about whether the AIs themselves do have rights of their own and should be sharing the world with us. And then maybe they do get rights. And maybe AI, you know, maybe some AI systems end up voting or maybe we decide they shouldn't and they don't. And either way, you have this kind of world where there's a bunch of different beings that all have rights and interests that matter and they vote on how to set up the world so that we can all hopefully thrive and have a good time. We have less and less material scarcity, so fewer and fewer trade-offs need to be made. That would be great. I don't know exactly where it ends or what it looks like, but that does, I don't know. I mean, what does anything?”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Study ourselves, understand what makes us happy, what makes us thrive, and hopefully the world just gets better over time, and we have to make sure that”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Look like to me. I mean, success could look like a lot of things, but one thing success would look like to me would frankly just be that we get something not too different from the traditory we're already on. So in other words, if we can have AI systems that behaved as intended, acted as tools and amplifiers of humans, did the things they're supposed to do. And if we could avoid a world where those AI systems got sort of like, I don't know, all controlled by one government or one person avoid a world where that caused a huge concentration of power. If we could just have a world where AI systems are just another technology, they help us do a lot of stuff. We invent lots of other technologies, and everything is relatively broadly distributed, and everything works roughly as it's supposed to work, then you might be in a world where we can continue the trend we've seen over the last couple hundred years, which is that we're all getting richer, we're all getting more tools, we all hopefully get increasing ability to kind of understand ourselves.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, I think we've talked a lot about the difficulty of predicting the future, and I think I do want to emphasize that I really do believe in that. So my attitude took the most important century is not at all, hey, I know exactly what's going to happen and I'm making a plan to get us through it. It's much more like there's a general fuzzy outline of a big thing that might be approaching us. There's maybe like two or three things we can come up with that seem good to do. Everything else we think about, we're not going to know if it's good to do or bad to do. And so I'm just trying to find the things that are good to do so that I can make things go a little bit better or help things go a little bit better. That is my general attitude. So it's, you know, it's, I don't know. It's like if you were, if you were on a ship in a storm and you saw some like very large, fuzzy object obscured by the clouds, you might want to steer away from it. You might not want to say, well, what I think that is, is it's an island and I think there's probably a tiger on it. And if we go and train the tiger in the right way, blah, blah, blah, blah, blah, you don't want to get into that, right? So that is the general attitude I'm taking. So what a success.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Like a lot of signs that this century is not just a random century picked from a sample of millions of centuries. So that's the basic structure of the argument. As far as the growth rate in zero AD, I mean, I think it matters. I think you're asking this question, why do the dynamics of growth in zero AD matter at all for this argument? And I think it's because it just, it's a question of like how does economic growth work generally and how has it worked and what is the trend that we're on and what happens if that trend continues? So if around zero AD, growth was very low but accelerating and if that was also true at 100 AD and 1,000 AD and negative 1,000 or 1000 BC, however you want to put it, if all those things are true, then it starts to point to a general pattern, the growth is accelerating and maybe accelerating for a particular reason and therefore you might expect more acceleration.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“It's just one argument in the series. It just kind of, you know, I think the way that I would come at it is I would just say, hey, look at AI systems, look at what they're doing, look at how fast the rate of progress is, look at these like five different angles on imagining when AI might be able to do all the things humans do to advance science and technology and just imagine that we get there this century, wouldn't it be crazy to have AI that could do all the things humans do to advance science and technology? Wouldn't that lead to just a lot of crazy stuff happening? Like there's only ever been one species in the history of the universe that we know of that can do the kinds of things humans do. Wouldn't it be weird if there were two? That would be crazy. And one of them was a new one we built that could be like copied at will, run at different speeds, run on any hardware you have. That would be crazy. And then you might come back and say, yeah, that would be crazy. This is too crazy. I'm ruling this out because this is too crazy. And then I would say, okay, well, we have a bunch of evidence that we live in an unusual crazy time and you actually should think that there's.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“That were one of the 80 most important centuries, it shouldn't take me nearly as much further evidence to say, actually, this one might be number one out of 80, because your starting odds are more than 1%. So to get you up to 10% or 20% or 30%, it doesn't necessarily require a massive update the way that it would if we're just starting from nowhere.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh, sure. Yeah, yeah, no, totally. No, the thing about the rapid growth is not supposed to be on its own, an argument that this century is the special one. I mean, by growth standards, this century looks less special than the last one or two. It's just, it's saying this century is one of a handful. Or I think when I say like one of 80 of the most significant centuries or something by economic growth standards. And that's only one argument. And then I look at a lot of other ways in which this century looks unusual. But just to be clear, I mean, to say that something is the most important century of all time sounds totally nuts because there's so many centuries. There's just so many centuries in the history of humanity even, and especially if you want to think about it on galactic timescales. And even once you narrow it down to 80, it's just way less weird. It's just like, well, if I've already convinced you using kind of non-controversial reasoning,”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Gosh, this looks like this looks like it's probably going to end up looking like a very special time or a very weird time.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“That doesn't sound right. I don't think this Galaxy calculation was very close. There's also going to be a bunch of Dead Space as you get to the outer reach of the galaxy. There's not going to be as much there. That doesn't sound super right, but let's just roll with it. Let's just say that you had 2% today, and then growth went down to 0.5% and just stayed there forever. Yeah, I'm pretty sure that's still too big. I'm pretty sure you're still going to hit limits in some reasonable period of time. But also that would still be just weird on its own. It would just be like, well, we lived in the 200-year period when we had 2% growth, and then we had 0.5% growth forever. That would still make this kind of an interesting time. It would be like the most dynamic, fastest changing time in all of human history, not by a ton, but it's also like you pick the number that's like the closest and the most perfectly optimized here. So, you know, if it went down to 0.1% or even down to 0.01%, then it would take longer to run out of stuff, but it would be even stranger, the 2% versus the 0.01%. So I don't really think there's any way out of like.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I think we might want to give a little bit more context here or something. So one of the key arguments and the most important century is it's just hard. It's just one of the arguments that we live in a strange time. But I'm arguing the current level of economic growth just looks too high to go on for another 10,000 years or so. And one of the points I make, which is a point I got from Robin Hansen, it's not my own originally, is that if you just take the current level,”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Trying to predict the future and do helpful things. So I see my main role in the world as trying to predict the future and do helpful things. I can do that without doing a bunch of harmful or common sense on ethical stuff. Maybe someday, you know, there will be one of these intense trade-offs. I haven't really felt like I've run into them yet. And if I ever ran into one of those intense trade-offs, I'd have to ask myself how confident I really am. The current level of information and confidence I have is, in my opinion, not enough to do really unsustified the mean sea stuff.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Pretty much. I'm really not a fan of Ends Justify the Means Reasoning. I just think it's historically looks that's the thing that looks really, really bad is people kind of saying, you know, it's worth doing horrible things and coercing each other and using force to accomplish these things that the ends we're trying to get to are more important than everything else. So I'm against that stuff. I think that stuff looks a lot worse historically than people like.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Reasoning, you know, I think whatever people are trying to do to help the most important century go well, it's totally good and fine to do research on AI alignment, but people shouldn't be like going and telling lies or breaking the law in order to further their ends. That would be my proposed rule is that we should, you know, when we have these high stakes crazy thoughts, we should do what we can about them and not go so crazy about them that we break all the rules of society. And that seems like a better rule. That's the rule I'm trying to follow.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source