YouSaid · the spoken record

Holden Karnofsky

lines on the record: 131
first: 2023-01-03
most recent: 2023-01-03
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“Things that are really central to open philanthropy. What does it mean to do good? How do we handle uncertainty about how to do good? What are the most important causes? If AI might be one of the most important causes, then when might we see transformative AI? What would that mean? How big is the risk of misaligned AI? I think I need to understand those issues well enough to effectively manage people who know a lot more about them than I do. Yeah, and I'm just curious, like, what do you think about this almost important century stuff? Does this just strike you as crazy or what do you think when you read the series?”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I know much less about any given topic than the person who specializes in the topic, and I think what I try to do is I try and know enough about the topic that I can manage them effectively. And that's like a pretty general corporate best practice. And I think it just varies a lot. So I think, for example, like something like keeping our books, like keeping our finances, doing the financial audits, all that stuff, that's something where it's really easy to just judge the outputs without really knowing much about finance at all. You can just say, look, was this compliant? Did we do our audit? Did we pass the audit? Do we still have money in the bank? Like, how much money do we have in the bank? You don't need to know much about it to judge it effectively. But at any company, I mean, you may need to know a fair amount about some other topics in order to judge them very effectively. If your company is making computers or phones and design is very important, the CEO would be really bad if the CEO just had no opinions on design. It was just like, well, I'm going to let our design person decide the design. I mean, it's a central thing to the company. It matters of the company and they should know some things about it. So I do, you know.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh, I mean, I haven't dropped my, I mean, that would be a big change to drop my view that we could be in the most important century for humanity. That's still what I believe. I'm, you know, I mean, I think I've also heard from people who just think I'm like underselling the whole thing like crazy, like people who just think that I should be planning on transformative AI much sooner than what I say, than what I kind of imply in the series. So, yeah, I mean, I put out this thing most important century and I don't believe any of the critiques have been deep enough and strong enough to make me just like drop that whole thing, but it's a big picture with a lot of moving parts. And I have kind of deepened my, you know, my understanding of many of the parts.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Let me think. I mean, there's definitely been like a lot of interesting stuff. I mean, an example is I put up this post on the track record of Futurists, and then there was a post by, I think, Dan Liu recently that I haven't read yet, but this has its own analysis that the track record of futurists. And I need to compare them. Think about what I really think about how good humans have historically been about predicting the future. But he certainly has a ton of data in there that I was not aware of that, you know, that feels like a bit of a response may or may not have been prompted by it. Yeah, there's been like a lot of commentary. There's been a lot of like you wrote some critiques of some of the stuff I've written in the most important century. There have been other critiques. I mean, I don't know. I think a lot of this stuff I wrote about the biggest weak points in the most important century were based on the public criticism that was coming in. So I think it has, like, yeah, I think I have become more aware of a lot of the parts of my, parts of my thinking that are the least convincing or the most weak or the most in need of argument. And I have paid more attention to those things because of that.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“More people see where we're coming from, hopefully agree with it, and are trying to work on the things we consider important. And then to the extent that my stuff is actually just like screwed up and wrong, and I've got mistakes in there, and I've thought it all through wrong, this is also like the best way I know of to discover that. It's just, you know, I don't know how else I'm going to get people to critique it, except by putting it out there and maybe getting some attention for it. That's how I kind of think of it is it's taking views that are very important to the decisions we're making and trying to express them so that we can either get more people agreeing with us who were able to fund and support and work with or learn more about what we're getting wrong.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“No, I think it fits into my job reasonably well. I didn't want it to be open philanthropy branded. I just wanted the freedom to write about things the way I wanted and how I wanted. But I do think that like we make these high stakes decisions based on very unconventional views about the world. And I think it's good for us to be trying to make those views have contact for the rest of the world. So I think there would be something not ideal about being a large foundation, giving you large amounts of money, and we're just kind of quietly going around believing these enormously important if true things that no one else believes. And it is a way of kind of, you know, if we put the views out into the world, A, I think just like all the people seeking money from us just have a better idea of where we're coming from. Just like why is it that we're interested in funding what we're interested in funding? And I think to the extent people find the arguments compelling or even just like understand them and helps understand our thinking, I think that can help create more grantees for us. It can help, you know, cause the world to be a place where there's more good stuff for us to fund because”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“So that is a stuffed animal named Mora. And at some point, if I get enough subscribers, I will explain who all these stuffed animals are. But my wife and I basically use stuffed animal personality classification system where we will like compare someone to various stuffed animals to explain what their strengths and weaknesses are. And more as a pink polar bear who's like very creative but also very narcissistic and loves attention. So she's kind of the mascot of the blog because it's kind of this blog that's just like very crazy, very out there and is just like me writing in public. And so it just felt like her spirit”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Of at various points in my career just said, Hey, here's something that's getting very little attention and it's very important, and it's worth the sacrifice of the specialized knowledge I built up in one area to switch into this other area that I think I ought to be working on.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“You started off I don't know if I really forecast AI. I mostly distill and bring together analyses that others have done, and I manage people who work on that sort of thing. Yeah, I mean, I don't know. I think it's really good. I mean, the career advice I often give is just like, it's really good to just have something you're very focused on, that you're specializing in, that you're trying to be the best in the world at. The way that my career went is like, first off, the general theme of my career is just taking questions, especially questions about how to give effectively, where just like no one's really gotten started on this question. So even doing a pretty crappy analysis can be better than what already exists. Often what I have done in my career, what I consider myself to have kind of specialized in a sense, is I do the first cut crappy analysis of some question that has not been analyzed much and is very important. And then I build a team to do better analysis of that question. So that's been my general pattern. I think that's the most generalizable skill I've had. But I have switched around because I do think that I've kind of”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“It's not clear, like we're growing. I mean, we're bigger than we were a year ago. We'll be bigger in a year. So it's definitely not true that I'm just minimize the size of the company. I mean, we're growing. But I think we want to, you know, I think we want to watch it. I think we want to treat each hire as something that we only do because we had a really good reason to. And I think there's some companies that may have more to gain from being 10,000 people. I don't think we'll ever be 10,000 people.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I think in general, the bigger your organization is, the bigger your city is, the bigger your society is. There's more people who are who, you know, if you want everyone to be happy, there's more people you're going to have to make happy. And I think this does mean, you know, in general, by default, as a company grows, it gets less able to make a lot of disruptive, quick changes. A lot of people would use the term nimble. A lot of people in the tech world like to just use these very negative terms for big company properties and very positive terms for small company properties. So small companies are nimble and quick and practical and adaptive and dynamic and high productivity and big companies are like bureaucratic and slow and non-adaptive.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I would say I'm like medium optimistic about the impact of all these prizes. I mean, I've been part of designing some of them. I've just seen some other ones. People say, you know, hey, we'll pay you X dollars if you can give us a good critique of our, like, give well, we'll pay people to give them a good critique of their reasoning about what the best charities already give to. Open philanthropy as a prize for showing us a cause we should be looking at that or not. I don't know. I think I'm like medium optimistic. I mean, I think it will get some interest and it will get some people to pay attention who weren't otherwise. And some of those people might have good ideas. I don't think it's like the only way to solve these problems or that it will automatically solve them. And that's generally how the people designing the prizes think about them too.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“It's really TVD and unclear. I think I just think the Bayesian mindset is a cool thing to experiment with. I experiment with it a lot. I feel like it helps me sometimes. Like most things, it's good in moderation and with taste and not using it for every single thing. And maybe 10 years from now, as it gets more popular, I'll have a better sense of where the actual applied strengths and weaknesses are.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, the general population, I mean, it's practically unheard of. Not even just the name. I mean, I just the idea, this whole idea of like thinking expected value and subjective probabilities all the time. It's just like almost no one does that. I think tech founders probably do it more than like the average person.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I wrote about this idea of Bayesian mindset, which is basically like being willing to kind of put a probability on anything and use your probabilities and say your probabilities as a way of discovering why it is you think what you think. And using expected value calculations similarly, I think this is much more common among successful tech founders than it is among the general population. But there's plenty of tech founders who don't think this way at all. I don't think, you know, I say in the Bayesian mindset, I don't think it's like a super well-tested, well-proven social technology that does amazing things. I think it's more like an interesting thing to be experimenting with.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I think some people are much any given thing in open philanthropy, some people are much better at it than others. I don't think that's very surprising. I think it's true in many jobs. And I don't really know the reasons for it. It could be any combination of talent, interest, how hard someone works at it. So yeah, but I certainly think there's a lot of variance in hiring people who can do a great job at the work OpenFill does has been a lifelong challenge.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, I think no matter what position I was in in the world and however many resources I had, I would feel that my decisions were significant in some sense and affected people and were important in that way that they affect those around me. So I think it's just very natural to me to think there's a lot of different perspectives on what it means to be a good person. And rather than trying to turn them into a single unifying mathematical equation and take the expected value, which is another approach that I think is interesting. But I think the approach I do tend to prefer is to imagine the different perspectives as different people trying to get along and make a deal with each other.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“There's multiple people all living inside my head arguing about what to do, and they all are friends, they all care about each other, and they want to get along. And so they're trying to reach a deal that all of them can feel fairly good about. That is how I tend to think about dealing with different moral views. So I tend to want to do things that are like really good according to one and not too bad according to the rest and try to have the kind of different parts of myself making deals with each other so that is that relates to something I said at the beginning about not being into ends justify the means like I put a lot of effort into doing things that would be like really really good if this most important century stuff came out true but also not too catastrophic if it didn't like there's there's lines I'm not willing to cross there's behaviors I'm not willing to engage in to promote the kind of you know goals of people who worry about AI safety so it's it's a it's a moderating approach I think”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, there's something I hope to write about in the future this topic of moral uncertainty. And the basic idea is there might be a bunch of different ways about thinking about what the right thing is to do in the world. And one, you know, you might look at the world from one angle and say, well, what matters is the total sum of all the pleasure. So therefore a bigger world would be better. So therefore I should be really obsessed with like getting the world to be as big as possible. There might be another perspective that says that what really matters is suffering. We should minimize suffering. We should want the world to be small. There might be another perspective that says, you know, it doesn't matter what happens to the world. It matters how I act. What matters is that I act with integrity, that I tell the truth, things like that. And, you know, there's these interesting debates about what do you do when you think you have some sympathy for all these views. And then how do you choose an action that some perspectives would say is the best thing you've ever done and some would say is the worst thing you've ever done. And the moral parliament idea is an idea that was laid out by Nick Bostrom and an overcoming bias post, like something like a decade ago that I like, I think about it as if I'm just multiple people. I just think about it as if I'm.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, possible. I mean, I talk about this a bunch in the series, and I think I just didn't do a great job with this. Yeah, I mean, I think what I was trying to do is I was trying to use a term that you didn't have to be a moral realist to get behind. And what I was really trying to capture is like, can I think now to reduce the odds that if later I improve that I'll be horrified by my early actions? That was what I was trying to capture the concept of. I'm not sure I really did it successfully. Good.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“You know, we have some reason to think that actually just does help a bit, a really tiny bit, but I'm not confident in that at all. And I certainly don't think that society is going to have moral progress just necessarily, but I do think we've had some in the past.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, this has been definitely a point of confusion in the series and partly just something I think I didn't communicate well about and makes the series not that amazing is just that I use the term moral progress and I just used it to refer to like things getting better. I think sometimes there is such a thing as thinking more about your morality, gaining some insight and ending up in a better place as a result. I think that is a thing that is real. There are some people who believe moralities and objective truth, but I'm not one of those people. But even though I believe morality is not objective, I still think there's a meaningful notion of moral progress. There's such a thing as having more reasonable moral views than I used to. What I didn't mean to say is that moral progress has any inevitability about it. I didn't mean to say that moral progress necessarily happens just because time goes on. I don't think that. I just think it's a thing that can happen. So I do think if future Holden will probably be better at morality just because I'm really interested in the topic and I'm going to keep trying to improve it. And I think that.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Doesn't sound like the strongest version of this argument to me, to be honest. Like, I think maybe that would be fine or good. I don't know. Maybe if I was a bigger fan of, I mean, I don't know. My wife really loves these like old churches. So maybe if I had more attitude, I would be more horrified by this idea. I think that, yeah, I mean, like low-income people had a rough time in the past and them having better lives seems pretty appealing. So I don't really know if that's the best version of, I don't know if this is the best version of this argument yet.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Think it does somewhat suggest that. I mean, I am somewhat partial to that view, and that's something I may be writing in my rebuttal, but I also think it's possible to just be confused. And I think it's possible to just have lots of stuff going on in your brain. And some of it might be based on really good intentions of treating other people fairly and being good to other people. And some of it might be based on just like other weird stuff about wanting to stand up for people who look like you or help people who look like you and things like that. And so I think it's I do have some sympathy for the project of trying to say, you know, my intuitions contradict each other, but some of them are coming from a good place. Some of them are coming from a bad place. If I thought more about it, I would realize which ones are which and I want to try and do that.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“When I'm deciding whether I care about someone, it doesn't matter at all if they're away in the future, if they're way far away, if they're totally different from me, if they're not human, if I've never met them, all that matters is if they can have pain or pleasure. I think it sounds great, and I completely get why someone listening to me like, how could you ever disagree with that? But I do think there's various challenges with it, which I have not had the chance to write about yet. And I doubt I can be very convincing on this podcast yet, because I haven't thought about it. Yeah, all right.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Have a morality where you are insistent on saying that everyone counts equally in proportion of the amount of plain or pleasure they're able to have, you run into a lot of weird dilemmas that you wouldn't have to run into if you didn't have that view. So I think it's very strange, but I think it is like actually one of the more questionable parts of the view. It's kind of saying, you know,”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“So, sentientism is the idea that, well, let me back up. There's three principles that I kind of like roughly outline that you might want for a morality that is going to stand up to scrutiny or that you won't be so likely to change your mind about if you learn more and get better. So one principle is just systemization. It's like it's better to have morality based on simple general principles that you apply everywhere than have a morality that's just always you just deciding what feels right in the moment because the latter could be subject to a lot of the biases of your time and the former lets you kind of stress test the core ideas. And then two of the core ideas I propose are what I call thin utilitarianism, which is basically the greatest good for the greatest number, and sentientism, which is basically saying that someone counts or someone matters if they're able to suffer or have pleasure. And I think you just said sentientism seems reasonable to you. I think sentientism might be the weakest part of the picture to me. I think if you...”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“That even the idea that OpenAI is a negative force, I think it's highly debatable. We could talk about it all day. And then if you look at our specific grant, it's just that's even a completely different thing because a lot of that was not just about boosting them, but was about getting to be part of their early decision making. And I think that was something that there were benefits and was important. And so, yeah, I mean, my overall view is that I look back on that grant as one of the better grants we've made, not one of the worst ones. Oh, wow. Okay. But certainly we've done a lot of things that have had, you know, that have not worked out. And I think there are sometimes, surely when we've done things that have consequences we didn't intend and, you know, no philanthropist can be free of that. What we can try and do is be responsible, seriously do our homework to try to understand things beforehand, see the risks that we're able to see, and think about how to minimize them.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, you know, you can look up our $30 million grant to OpenAI. I think it was back in 2016, and we wrote about some of the thinking behind it. We wrote that, you know, part of that grant was getting a board seat for Open Philanthropy for a few years, that we could help with their governance at a crucial early time in their development. You know, I think that you could have, I mean, I think some people believe that OpenAI has been net negative for the world because of the fact that they have contributed a lot to AI advancing and to AI being sort of hyped. And they think that gives us less time to prepare for it. And I do think that all else AI advancing faster gives us less time to prepare and is a bad thing. I don't think it's the only consideration. I think OpenAI has done a number of good things too, has set some important precedents, I think is probably much more interesting in a lot of the issues I'm talking about and risks from advanced AI than the company that I would guess would exist if they didn't that would be doing similar things. I don't really accept.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Do everything we can to understand the downsides, and if after we've done everything roughly everything we can up to some reasonable diminishing returns. And once we've done that, if we believe as far as we can tell, the upsides outweigh the downsides. We're generally going to go for it. Our goal is not to avoid harm at all costs, although our goal is to kind of operate in a cooperative, high integrity way, always doing our best, always trying to anticipate the downsides. But recognize we're going to have unintended side effects sometimes. And that's life and anything you do has unintended side effects. I don't agree with the specific example you gave as an example of something that was net negative, but I don't know.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, I think in theory, what we want is we want to make grants that have more upside than downside or have expected net positive effects. I think we tend to be just, you know, in a common sense way a little bit conservative with the negative effects in that the thing we don't want to do is we don't want to like enter some field on a theory that's just like totally messed up and wrong in a way that we could have known if we just done a little bit more homework. I think that there's just like something kind of irresponsible and uncooperative about that. And so in general, like when we are making big decisions, like big dollar decisions or going into a new cause, we often will try really hard to like”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I mean, we're trying to model it, right? I mean, it's like we have, I mean, in theory, we have all these models that say, like, here's our guess at how much money is eventually going to be available, and here's our guess at how many giving opportunities will eventually be there to fund. And, you know, this is our guess that like, what's good enough to fund and what's not, and that's a very tentative guess. And a lot of it is just like really, you know, really, really imprecise stuff, but we have to have some view on it. Anyone who's spending money does. So, I mean, yeah, I do tend to assume that Sam Bakeman Fried and Dustin Mouskovitz are not and Kerry Tuna are not the last billionaires who are interested in doing as much good as possible. But it is really hard to model this stuff. And frankly, a lot of it, you know, we have various rough models we've made over the years. We also just will sometimes use our intuition and just say we fund the stuff that seems like quite good and exciting and we don't fund stuff that doesn't. And that's an input into our thinking too.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I don't know. I think even then, I think the amount of money we'd be working with would be really small by the standards of any kind of government budget. In general with philanthropy, I'm always looking for things where it's like, can we seed the creation of a field? Can we, you know, fund people to introduce new ideas, but still we're very small compared to the overall economy and the overall government. I think even multiplying everything by 10, that would still be true. Not sure exactly what we do with 10x as much money. I'm not even sure what we're going to do with the money that already exists.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Therefore, not neglected. I think the stuff we're talking about is very neglected. But if you find something that's even more neglected and more important, I mean, more power to you.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Was he just asking an interesting question? You know, was he just kind of saying, hey, this thing seems important. I'm going to use my common sense and judgment to figure out how it works, and I'm going to write about it. And I think some of this stuff gets too fancy. So I think, you know, I think today what's going on is like if I just look at the world and I say, what are the most important things that could matter for the world being a good or bad place in the future? I've looked at a lot of possibilities. I think that AI is one of the leading examples and I don't see a lot of people paying attention to it. And so that's what I want to work on. And I think a lot of the people who have done revolutionary work that we now look back on and a lot of people try to imitate, they weren't really doing all this stuff. They weren't trying to imitate the stuff that usually works and stay away from the stuff that wasn't. They were just asking interesting, important questions and working on them. And, you know, as far as myself and 80,000 hours, I mean, I just, I guess I just don't feel that we're well known enough or influential enough that our advice, that stuff we're interested in is automatic.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, a fair number of your questions, or like a fair number of dialogue I see in some of the skeptical of futurism world, it's like it feels to me like it's almost just getting unnecessarily fancy or something. I kind of just want to say, who's someone who really revolutionized the way the world thinks about stuff? Well, I don't know, like Darwin. Now, what was Darwin doing? Was Darwin saying, well, you know, I really don't want to think about this thing because, like, I don't believe humans are capable of thinking about that thing. And I don't want to think about, you know, this topic because I think it's too hard to know the future and blah, blah, blah. Was he doing all that stuff? Or was he just...”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“The Oh, I think there's just a ton of really important stuff today that not enough people are paying attention to. I just expect, I mean, that was also true 50 years ago. That was also true 100 years ago. It was probably more true 50 years ago than it is today. It was probably more true 100 years ago than 50 years ago. It's gradually the supply of amazingly important ideas that are not getting any attention. It's probably getting harder to do that. Harder doesn't mean impossible. You know, I do actually think that if people want to do something that's really new and world-changing and dramatic and revolutionary, I think maybe the worst way to do that is to kind of go into some well-established scientific field and try to revolutionize that. I think it's better to just use your common sense and ask an important question about the world that no one's working on because it isn't a scientific field, because it isn't a field of academia because it doesn't have institutions and work on that. And a lot of my blog does advocate that. So, you know, for example, AI itself is a very well-established.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I do think there's eventually limits to how much, like I think that the rate, like I've said, the atoms of the galaxy argument, but I think just a broader common sense take would be that the world over the last couple hundred years has changed incredibly dramatically. We've had just like new exciting technologies and capabilities every year. And I think just a good guess would be that that hits a wall at some point. I think that just, you know, it might be the atoms of the galaxy. It might just be something much more boring. It might just be like a gradually slowing, you know, actually what we seem to observe when we look at the numbers is that we are seeing a bit of stagnation, a bit of slowing down, and probably will keep slowing down by default. So, yeah, I think probably a good guess is that we are, the world is changing at an incredible pace. That has not been the case for most of history, and it probably won't be the case for the whole future.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Handle on what we want to do about them, then put my effort into throwing more juice and more gas behind this ongoing technological progress, which I think is a good thing. It's just a matter of priority.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I think it's something like where you could think of us. I think Toby Ord uses the analogy of humanity being like a child who's becoming an adolescent. And it's like it's great to become stronger up to a certain point. That's fun. That feels good. And then at a certain point, you're like strong enough to really hurt yourself and really get yourself in trouble. Or maybe strong enough that you don't know your own strength. I think there's a pretty good case that humanity is reaching that point. I think we're reaching the point where we could have a nuclear war or a bioweapon or AI systems that really change the world forever. And so it might have made sense 300 years ago when we were all struggling to feed ourselves to say, hey, we want more power. We want more technology. We want more abilities. Today, I think we're starting to enter the gray area. We're starting to enter the gray zone, or maybe we should slow down a little bit, be a little bit more careful. I'm not saying to literally slow down, but I'm talking about priorities. I would rather look at what I think are dramatically neglected issues that might affect all of humanity's future and at least do the best we can to have a hand.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“A lot of the way the world has gotten better over the last couple hundred years is just we've gotten richer, we've had more technological capabilities, maybe we should try and do more of that. I don't think this is like a nutty thing to think. I think this is somewhat reasonable, but I overall feel that we, you know, even a couple hundred years is not that big a percentage of the history of humanity. I wrote a series called Has Life Gotten Better that kind of asks what the whole graph of quality of life looks like over the course of humanity and there is precedent for like technological development seeming to make things worse. That's kind of what it looks like happened in the agricultural revolution. So I have some sympathy for saying, hey, this thing has been good for 200 years, let's do more of it. But I don't think it's like the tightest, most conclusive argument in the world. And I think we do have some specific reasons to believe that developing some particular new technologies, not only AI, but also potentially like bioweapons, could just be so catastrophically dangerous.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Sure. So I think we're kind of talking about maybe the progress studies crowd. Yeah, I wrote a piece about this on Cold Takes called Rowing Steering Anchoring Equity and Mutiny, where I discussed these different ways of thinking about what it means to make the world good. And I do have some sympathy for the idea that”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Prediction ever by a lot. Yeah, I think the answer is maybe. There was also some discussion in the comments about whether Morvick called big progress on AI is doing AI is doing well at vision by examining the retina. There was some debate about that. I think it's all very muddy. I don't think this is much of a knockdown argument against thinking about biological anchors. I mean, it is only one input into my thinking. I do think it looks kind of good for biological anchors that we have seen this deep learning revolution and we have seen these brute force AI training methods working really well and they didn't used to work well. And it is when AI systems started to be about the size of like insect or small animal brains or within range within a few orders of magnitude of human brains. You could call that a wild coincidence, but it doesn't feel, you know, probably these numbers are all off. They're probably all off by 10x, 100x,000x. But I mean, we're talking about very important things and trying to get our best handle and our best guess. And I think biological anchors looks fine so far. I don't know. It doesn't look amazing. Looks fine.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“It's not like this is some big, well-developed field where people have tried to do the exact same thing I'm trying to do 25 times and failed each time. This is like mostly people in academia try to advance AI systems. They don't try to predict when AI systems can do much and do what? We're not working with a lot here. We have to do the best we can, make our best guess, use our common sense, use our judgment, use the angles we've got. That's my main answer. But another answer I would give is I would just say that I don't know like more of VAT cons more was kind of the original biological anchor's person and he you know I think he predicted like artificial general intelligence like around 2020 or something and I'm kind of like on Eliaser's own views it's going to look like he was like unbelievably close like that was like maybe you know Eliaser believes we'll see it by 2030 so if that's what you and I think that's plausible so if that's what you believe then it's going to be I don't know I think we'll look back on that as like the greatest tech”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Using a I saw some discussion of this on his Facebook and I think the answer might be yes, but I mostly want to attack the premise. I mean, I mostly just want to say, like, there haven't been a lot of cases of people predicting.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“That we don't have AI systems that can do everything humans do because humans are actually doing more work in their brains than a normal AI system is doing. However, it also looks like within this century we probably will have AI systems that are that big. And if we estimate how much it would take to train them, how much it would cost to build them, that looks like it will probably be affordable this century. And then you can just talk about all the different ways you could define this and all the different ways you could quantify that and all the different assumptions you could put in. But my bottom line is like almost any way you slice it, it looks pretty, whatever, however you want to think about it, however you want to define what it means for an AI brain to be as big as a human's and what it would mean to get that brain sort of trained most angles on it suggest that it looks reasonably likely it'll happen this century. That's a data point for me. That matters. So all those are data points feeding into my view.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Cusp of transformative AI is like in some sense the world has not been trying very hard for very long. So that's an input. Another input is just like expert surveys when people just ask AI researchers when they think AI will be able to do everything humans can they tend to come out saying it's a few decades. Now that can be biased and unreliable in all kinds of ways. All these things have their problems. That's a data point. And then there's biological anchors and biological anchors is a report that kind of, you know, I would summarize it to summarize it on a podcast. It's a very complex report. There's a lot of different angles it uses. There's a lot of different questions it asks. There's a lot of different numbers. But I think you can boil it down to some fairly simple observations. You can say that in some important sense, which could be debated and analyzed, but seems true most ways you look at it. In some important sense, we've never built AI systems before that do as much computation per second as like a human brain does. So it shouldn't be surprising.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, biological anchors, I think, is a very important input into my thinking. It's not the only input. So I think it's, you know, I think my views on AI timelines are kind of a mix of just kind of A, looking at AI systems today, looking at what they did 50 years ago, looking at what they did 10 years ago, and just kind of being like, well, gosh, it sure looks plausible that these will be able to do all the things humans can do to advance science and technology pretty soon. That's one put into my thinking. Another input into my thinking is what we call the semi-informative priors analysis, which is, you know, it's a complex report because it looks from a lot of different angles, but I think you can summarize the highlights of the report as just saying most of the effort that has ever in the history of humanity gone into making AI will be in this century ever. Because the field of AI is not very old and the economy and the amount of effort invested have gone up dramatically. And I think that's a data point in favor of not being too skeptical that we could be on the”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Just a guess, but I do think that there's been a lot of progress. I mean, this is what society is, is we have a lot of progress on all kinds of intellectual fronts. And I think there has been a lot of progress on like what it looks like to make good, reasonable predictions about the future. I think that's something that's happened. So I think we should be expecting ourselves to do a bit better than people did in the past. In future, people probably do better than we do.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source
“Happening, they're just like really not convinced you couldn't have said that, definitely not convinced no one did say it. I think some people did say it. And so I think, yeah, I'm hesitant to get too defeatist just from the fact that some people were wrong about the future in the past. I think it's hard to know if there was really no way to know or if they just weren't trying very hard.”
2023-01-03 · Dwarkesh Podcast · Holden Karnofsky — History's most important century · IDENTIFIED FROM THE TRANSCRIPT · source