YouSaid · the spoken record

Holden Karnofsky

lines on the record: 131
first: 2023-01-03
most recent: 2023-01-03
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“A really bad rule or something, or a really bad norm that every time you find yourself thinking the stakes are really high or that you're in a really important position, that you just decide to ignore the thought. I think that would be very bad. Like if you just imagine a universe where there actually are some people who live in an especially important time, and then there's a bunch of other people who like tell stories to themselves about whether they do, how would you want all those people to behave? And it's like, to me, the worst possible rule is all those people should just be like, now this is crazy and forget about it. That's like the worst possible rule. Because the people who are living at the important time will then do the wrong thing. I think another bad rule would be that everyone should take themselves completely seriously and literally and just promote their own interests ahead of everyone else's. And a rule I would propose over either of them is that all these people should take their beliefs reasonably seriously and try to do the best thing according to their beliefs, but should also adhere to common sense standards of ethical conduct and not do too much ends justify the means.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, I mean, so first off, I don't know that it's really true that it's that common for people who've said they're living in the most important time in history. I mean, this would be like an interesting thing to look at, but just from, you know, from stuff I've read about whatever past works on political philosophy and stuff, I don't exactly see this claim all over the place. It definitely happens. It's definitely happened. And I think a way of thinking about it is that, you know, there's kind of two reasons that you might think you are especially important. One is that you actually are and you made reasonable observations about it. And another is that you're, you know, you want to be or you want to think you are or you're self-deceiving. And so, you know, over the long sweep of history, a lot of people will come to this conclusion for the second reason. And most of the people who think they're the most important will be wrong. So that's all true. And that certainly could apply to me and it certainly could apply to others. But I mean, so I think that's just completely fine and completely true. And I think we should have some skepticism when we find ourselves making these kind of observations. And at the same time, I think it would be like.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh, it's totally not obvious. I mean, I think my basic position here is I'm not sitting here highly confident. I'm not saying, you know, there's tons of precedent and we know exactly what to do. That's not what I believe. I believe we should be giving it a shot. I think we should be trying and I don't think we should be totally defeatist and saying, well, it's so obvious that there's never anything you could have come up with throughout history and humans have been helpless to predict the future. I don't think that is true. And so yeah, I think that's like enough of an example to kind of illustrate that. And I mean, gosh, like you could make the same statement today as you could say, look, doing research on how to get AI systems to behave as intended is a perfectly fine thing to do at any period in time. It's not like a bad thing to do. And I think John Locke was doing his stuff because he felt it was a good thing to do at any period in time. But the thing is that if we are at this crucial period of time, it becomes an even better thing to do and it becomes magnified to the point where it could be more important than other things.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Are actually doing. I don't think they were saying this could be the most important millennium, so let's do this. But it is interesting that you, it doesn't look like there was nothing to be had there. It doesn't look like there's nothing you could have come up with. And in many ways, it looks like what the Enlightenment thinkers were up to had the same kind of esoteric, strange, overly cerebral feel at the time and ended up mattering a huge amount. So it doesn't feel like there's zero precedent either.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“What a government owes to its citizens or when a citizen has a right to overthrow a government or when it's acceptable to kind of enforce certain beliefs and not. And it's like the other person in the dialogue is just like, this is the weirdest, most esoteric question. Why does this matter? Why aren't you helping poor people? But, you know, these are the questions that the Enlightenment thinkers were thinking about. And I think there is a good case that they came up with a lot of stuff that really shaped the whole world since then because of the fact that the UK became so influential, really laid the groundwork for a lot of stuff about the rights of the governed and free speech and individual rights and human rights. And then I go to the next analogy and I'm like, okay, now we're sitting here today. And someone is saying, well, instead of working on global poverty, I'm studying this kind of esoteric question about how you get an AI system to do what you wanted it to do instead of doing its own thing. And it's, you know, you could, it's not, I think it's not completely crazy to see them as analogous. Now, I don't think this is what the Enlightenment thinkers.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Transform the thinking and the culture in that country to have a better handle on human rights and more value on human rights and individual liberties and a lot of other stuff.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“It's a pretty good analogy for being thought provoking and for thinking, gosh, if you had seen the Industrial Revolution coming in advance, and this is when economic growth really reached a new level back in the 1700s and 1800s, what could you have done? And I think part of the answer is it's not that clear. And I think that is a bit of an argument that we should maybe not get too carried away today by thinking that we know exactly what we can do. But I don't think the answer is quite nothing. So I have kind of a goofy cold takes post that I never, that I never published and may never publish because I don't know. I kind of lost track of it. But it's kind of saying, well, what if you'd been sitting in that time and you had known the Industrial Revolution was coming? Are you thought it might be? You had asked yourself what you could be doing. What answer you might have given is you might have said, well, gosh, if this happens, whatever country it happens in might be just like disproportionately influential. And you know what would be great is if I could help.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Investing more in that than there is because the stakes than it is because the stakes are extremely high and I think it is just like a reasonable guess that we're living in a world that's really been incredibly transformed by the Industrial Revolution and the future could be incredibly transformed by the next thing. I just don't think this gets a lot of discussion in basically any circles. And if I got some I would feel a lot more comfortable with the whole thing. I don't think the whole world should just obsess over what the next transformative event is. But I think right now there's so little attention to it.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“What one might do to make that a world that could be better. So that's basically where I'm at on this stuff is I just think as a world, as a global civilization, we should have a really high priority on just saying we live in a weird time. Growth has been exploding, accelerating over the last blink of an eye. We really need to be kind of like nervous and vigilant about what comes next and thinking about all the things that could radically transform the world. We should make a list of all the things that might radically transform the world. We should make sure we've done everything we can to think about them, identify the ways we might be able to do something today that would actually help. And maybe after we're done doing all that and we have a lot of the world's brightest mind doing their best to think of stuff and they can't think of anymore, then we can go back to all the other things that we worry about. But I think right now the world invests so little in that kind of speculative, hey, what's the next big thing, even if it's not super productive to do so, even if there's not that much to learn, I feel the world's”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I think in general, having kind of learned about just how strange the time we live in is when you look at it in context, and here I'd just encourage people to go to my series, because I have a bunch of charts illustrating this, and it could be a little bit hard to do concisely. But I think the biggest thing I take away from that is we should just, you know, we should really look for the next big thing. I think if you'd been living 300 years ago and you'd been talking about the best way to help people, a lot of people might have been talking about, you know, various forms of helping low-income people, maybe they probably would have been talking about spreading various religious beliefs. And I think, you know, it would have seemed crazy to think that what you should be thinking about, for example, was the steam engine and how that might change the world. But I think the industrial revolution was actually just an enormous deal and probably was the right thing to be thinking about if there's any way to be thinking about it. If there was any way to be thinking about how that would change the world and what might”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“About transformative AI could come, and therefore this century could be weird, or is it about this century, could be weird, therefore, transformative AI could come? The central claim is that transformative AI could be developed in this century. And the stuff about how weird a time we live in is just a response to an objection. It's a response to a point of skepticism. It's a way of saying there's already a lot of reasons to think we live in a very weird time. And so actually this thing about AI is only a moderate quantitative update, not a complete revolution in the way you're thinking about things.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Go to the stars, fill up the galaxy with life, you know, maybe have artificial general intelligence. I just, to say it's a centuries crazy, I think it might be five hundred years. I think it might be a thousand years. I think it might be five thousand years. And a big point I make in the series is I say, well, even if it's a hundred thousand years, that's still an extremely crazy time to be in the scheme of things. If you just graphic timeline and you kind of show my view versus yours, they look exactly the same down to the pixel. And so there's already a lot of reasons to think we live in a very weird time. We're on this planet where there's no other sign of life anywhere in the galaxy. We believe that we could fill up the galaxy with life. That alone would make us among the earliest life that has ever existed or would make us the earliest life that has ever existed in the galaxy, a tiny fraction of it. So that is a lot of what the series is about. And I have sort of answered a question, but I'll do it explicitly. You ask, you know, is the series...”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“And then humanity is just a blink of an eye compared to that. You could call it 300,000, 3 million years old human civilization is a blink of eye compared to that. And we're in this really, really tiny sliver of time a couple hundred years when we've seen basically all or like just a huge amount of the technological advancement economic growth. So that's weird. I also talk about the fact that the current rate of economic growth seems high enough that we can't keep it going for that much longer. If it went for another 10,000 years, that's another blink of an eye on galactic timescales. It looks to me and we can get to this like we would run out of atoms in the galaxy wouldn't have anywhere to go. And so I think there are a lot of signs that we just live in a really strange time. One more I'll just throw in there that we can move on is that, you know, I think a lot of people who disagree with my take would say, look, I do believe eventually we will develop space colonization abilities.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Biggest source of skepticism has been just like, I don't have any specific objection. It just sounds kind of crazy and suspicious to say that we might live in one of the most significant times or the most significant time for humanity ever. And so a lot of my series is just kind of saying it is weird. It is weird to think that. But we already have a lot of evidence that we live in an extraordinarily weird time that would be on the short list of contenders.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so something we haven't covered yet, but I think it's worth throwing in now, is that a significant part of the most important Century series is kind of just making the case that even if you ignore AI, there's a lot of things that are very strange about the time that our generation lives in. The reason I spend so much effort on this is because my number one back in 2014 and before that, my number one objection to these stories about transformative AI is it's not anything about whether the specific claims about AI or economic models or alignment research makes sense. It's just this whole thing sounds crazy. And it's just suspicious. It's just suspicious if someone says to you, you know, this could be the most important century of all time for humanity. I titled the series that way because I wanted people to know that I was saying something kind of crazy and that I should have to defend it. I didn't want to be backpedaling or soft pedaling or hiding what a big claim I was making. And so I think my...”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Events go better, especially if the future events aren't crazily far away, especially if they're within the next few decades. So that is something I've changed my mind on at least to some degree.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“But I think I wrote a recent blog post entitled The Track Record of Futurists Seems dot Dot Dot Fine. Fine is how I'd put it where I don't think there's anyone who has demonstrated a real ability to predict the future with precision and know exactly what we should do. But I also don't think humans' track record of this is so bad and so devastating that we shouldn't think we are capable of at least giving it a shot. And I think if you enter into this endeavor with self-awareness about the fact that everything is less reliable than it appears and feels at first glance and you look for the few things that you would really bet on, I think it's worth doing. I think it's worth the bet. My job is to find things that we might do 10 things and have nine of them fail embarrassingly and one of them be such a big hit that it makes up for everything else. And a lot of my job is to find stuff like that. So I don't think it's totally crazy to think we could make meaningful statements about how things we do today could make these future.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, well, I don't remember exactly what kinds of things Muri was trying to figure out, and I'm not sure that I even understood what they were that well. So I definitely think it is true that it is hard to predict the future, no matter who you are, no matter how hard you think, no matter how much you've studied, I think that is true. I think parts of our kind of world or memeplex or whatever you want to call it overblow this at least a little bit. And I think I was kind of buying into that a little bit more than I could. So I think probably in 2014, I would have said something like, gosh, you know, really no one's ever done something like making smart statements about what the, you know, several decades out future could look like or making smart statements about what we could be doing today to prepare for it. Since then, I think a bunch of people have looked into this and looked for like historical examples of people kind of making long-term predictions and long-term interventions. And I don't think it's amazing.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Not so wild to imagine that we could just keep on going with these systems, make them bigger, put more work into them, but basically stay on the same path and you could get there. And if you imagine doing that, it becomes a little bit less daunting to imagine the risks that might come up and the things we could do about them. So I don't think it's necessarily the leading possibility, but it's enough to sort of start thinking concretely about the problem.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“And I am more familiar today than I was then with the research and the work people can do to make that less likely and the actions people can take to make that less likely. So that's probably more than half the answer. But another thing that would be close to half the answer is I think the world has changed. And I think that basically there's been big changes in the world of AI since then. So I think basically in 2014, that was the beginning of what's sometimes called the deep learning revolution. And since then, we've basically seen these very computation intensive but fundamentally simple AI systems and achieve a lot of progress on a lot of different unrelated tasks. And it's looking to me like not totally crazy to imagine that the current way people are developing AI systems, cutting edge AI systems could take us all the way to the kind of extremely powerful AI systems that automate roughly everything humans do to advance science technology.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Taken. I'm not seeing that with this stuff. So, what changed? I would say a good chunk of what changed is maybe like the most boring answer possible is I just kept at it. So I think I was sitting there in 2014 saying, gosh, this is really interesting, but it's all a bit overwhelming. It's all a bit crazy. I don't know how I would even think about this. I don't know how I would come up with a risk from AI that I actually believed was a risk and could do something about today. And now I've just been thinking about this for a much longer time period. And I do believe, you know, most things you could say about the far future are very unreliable and not worth taking action on. But I think there are a few things one might say about what a transition to very powerful AI systems could look like. There are some things I'm willing to say. I'm willing to say it would be bad if AI systems were poorly designed, had goals of their own, and ended up kind of running the world instead of humans. That seems bad.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“If you think that funding some work to help shape that transition, to help make sure that we don't move forward too incautiously, to make sure that we do increase the odds that we do get like a good future world instead of a bad future one, that's helping a huge number of people per dollar spend. So that's the motivation. And now you're quoting me, you know, a discussion, an argument I was having where we posted a transcript back in 2014. And at that time was that was part of my journey of getting here. So I was talking to people who are saying, Holden, you want to help a lot of people with your resources. You should be focused on this massive event that could be coming this century that very few people are paying attention to, that there might be a chance to make this go well or poorly for humanity. And I was saying, gosh, like that sure is interesting. And I did think it was interesting. That's why I was spending the time and doing the conversation. But I said, you know, when I look at global poverty and global health, I see what I can do. I see the evidence. I see the actions I can do.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, first I think I just went on for a while, but I want to come back and connect this back up to how this relates to the work I was even doing at GiveWell, why this is all kind of one theme. If we are kind of on the cusp for this century of creating these advanced AI systems, then we could be looking at a future that's very good or very bad. And I think there are decent arguments that if we move forward without caution and we develop kind of sloppily designed AI systems, they could end up with goals of their own and basically we would end up with a universe that contains very little that humans value or a galaxy that does.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“This happens. So that's the most important century hypothesis. In a nutshell, is that if we develop AI that can do all the things humans do to advance science and technology, we could very quickly reach a very futuristic world, very different from today's, could be a very stable, very large world. This is our last chance to shape it.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Extremely different from how it is today, you might think of it as if there were thousands of years of changes packed into a much shorter time period, and then if that happens, I argue that you could end up in a deeply unfamiliar future. I give one example of what that might look like using this hypothetical technology idea called digital people. That would be sort of people that live in virtual environments that are kind of simulated, but also realistic and exactly like us. And when you picture that kind of advanced world, I think there is a decent reason to think that if we did get that rate of scientific and technological advancement, we could basically hit the limits of science of technology. We could basically find most of what there is defined and end up with a civilization that expands well beyond this planet, has a lot of control over the environment, and is very stable for very long periods of time, and basically looks sort of post-human in a lot of relevant ways. And if you think that, then this is basically our last chance to shape.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Dynamic at the heart of it. So that's kind of a way of putting it that's trying to use familiar concepts from economic growth theory. Another way of putting it might just be, gosh, if we had AI systems that could do everything humans do to advance science and technology, that would be insane. You know, what if we were to take the things that humans do to create new technologies that have transformed the planet so radically and we were able to completely automate them so that every computer we have is potentially another mind working on advancing technology. So either way you think about it you could imagine the world changing incredibly quickly and incredibly dramatically and so I argue in the most important century series that it looks reasonably likely in my opinion more than 50 fifty that this century will see AI systems that can do all of the key tasks that humans do to advance science and technology that if that happens we'll see explosive progress in science and technology the world will quickly become”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“And the reason that I currently don't think that's a great thing to expect by default is that one of the steps of that feedback loop broke a couple hundred years ago. So it goes more people, more ideas, more resources, more people, more ideas, more resources, a couple hundred years ago, people stopped having more children when they had more resources. They got richer instead of more populous. And this is all discussed in the most important century page on my blog Cold Takes. And so what happens right now is that when we have more ideas and we have more resources, we don't end up with more people as a result. We don't have that same accelerating feedback loop. And if you had AI systems that could do all the things humans do to advance science and technology, meaning the AI systems could fill in that more ideas part of the loop, then you could get that feedback loop back and then you could get sort of this unbounded, heavily accelerating explosive growth in science and technology. So that's the base.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“You see acceleration, you see that it's growing a lot faster today than it ever was. And one theory of why that might be or one way of thinking about it through the lens of basic economic growth theory is that in normal circumstances, you can imagine a kind of feedback loop where you have people have ideas. And the ideas lead to greater productivity and more resources. And then when you have more resources, you can also have more people. And then those people have more ideas. So you get this feedback loop that goes people, ideas, resources, people, ideas, resources. And starting a couple hundred years ago, you run a feedback loop like that. Standard economic theory says you'll get accelerating growth. And basically, if you take the story of our economy to date and you just kind of plot it on a chart and do the kind of simplest thing you can to project it forward, you project that it will go, that our economy will reach like an infinite growth rate this century.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Call outsized return on investment, opportunities to spend some money and just get an enormous impact because you're doing something very important that is being ignored by others. And so it's through that kind of professional specialization that I've actively looked for interesting ideas that are not getting enough attention and that I encountered the effect of altruist community, which is a community of people basically built around the idea of doing as much good as you can. And so it's through that community that I encountered the idea of the most important century. It's not my idea at all. I got to it from a lot of people. And the basic idea is that if we developed the right kind of AI systems this century, and that looks reasonably likely, that could make this century the most important of all time for humanity. So the basic mechanics of why that might be or how you might think about that. So one thing is that if you look back at all of economic history, just the rate at which the world economy is.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“You know, my story is I originally co founded an organization called GiveWell that helps people decide where to give as effectively as possible. I'm no longer there, but I'm on the board and it's a website called Givewall.org that I think makes good recommendations where to give to charity to help a lot of people. And as we're working at Givewell, we met Carrie Tuna and Dustin Moscovici. Dustin is the co-founder of Facebook and Asana and started a project that became Open Philanthropy to try to help them give away their large fortune again to help as many people as possible. And so I've kind of spent my career looking for ways to do as much good as possible with a dollar or with an hour, with whatever resources you have, and especially with money. And so I've kind of developed this professional specialization in looking for ideas that are underappreciated, underrated, tremendously important because a lot of the time that's where I think you can find just kind of outsized, what you might”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“If we had AI systems that could do everything humans do to advance science and technology, that would be insane. We live in a weird time. Growth has been exploding, accelerating over the last blink of an eye. We really need to be kind of like nervous and vigilant about what comes next and thinking about all the things that could radically transform the world. If you just imagine a universe where there actually are some people who live at an especially important time, and then there's a bunch of other people who tell stories to themselves about whether they do, how would you want all those people to behave? And it's like, to me, the worst possible rule is all those people should just be like, nah, this is crazy and forget about it.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source