YouSaid · the spoken record

Carl Shulman

lines on the record: 205
first: 2023-06-26
most recent: 2023-06-26
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“Yeah, I have a blog Reflective disequilibrium. Okay. And a new site in the works. And I have an older one which you can also find just Googling Reflective Disequilibrium.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And do the right thing. And there's some evidence of progress on that front. Things like the statements and movements by Jeff Hinton are inspiring some of the engagement by political figures. Is reason for optimism relative to worse alternatives that could have been. And yes, the contrary view is present. It's all about geopolitical competition, never hold back a technological advance. And in general, I love many technological advances that people, I think, are unreasonably down on, nuclear power, genetically modified crops, yada yada, bioweapons, and AGI capable of destroying human civilization are really my two exceptions. And yeah, we've got to deal with these issues. And the path that I see to handling them successfully involve key policymakers and to some extent, and the expert communities and the public and electorate grocking the situations they're in and”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Any risk of the world ending catastrophe. I think that picture leads to a misunderstanding and it will make people think that you need less in the way of preparation, things like alignment so you can actually navigate the thing, verifiability for international agreements or things to have enough breathing room to have caution and slow down not necessarily right now. I mean, although that could be valuable, but when it's so important when you have AI that is approaching the ability to really automate AI research and things would otherwise be proceeding absurdly fast, far faster than we can handle and far faster than we should want. And so, yeah, at this point, I'm moving towards the share my model of the world, try and get people to understand.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Governments and large international coalitions of governments who can set common rules and common safety standards puts us into a much better situation. That requires a broader understanding of the strategic situation, the position they'll be in. If we try and remain quiet about the problem they're actually going to be facing, I think it can result in a lot of confusion. For example, the potential military applications of advanced AI are going to be one of the factors that is pulling political leaders to do the thing that will result in their own destruction and the overthrow of their governments. If we characterize it as, oh, things will just be a matter of chat bots and some minor things that no one cares about in exchange you avoid.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And so the opportunity to delay understanding and whatnot, there's a question of for what. And I think there were gains of building the AI alignment field, building various kinds of support and understanding for action. Those had real value and some additional delay could have given more time for that. But from where we are, at some point, I think it's absolutely essential that governments get together at least to restrict disastrous Reckless compromising of some of the safety alignment issues as we go into the intelligence explosion. And so moving the locus of the sort of collective action problem from numerous profit-oriented companies acting. Acting against one another's interests by compromising safety too.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Those seem like significant benefits. And if And so, yeah, I don't know that the attempting to have public discussion understanding has been a disaster. I have been reluctant in the past to discuss some of the aspects of intelligence explosion, things like the concrete details of AI takeover before because of concern about this sort of problem where people who only see the international relations aspects and zero sum and negative sum competition and not enough attention to the mutual destruction and sort of senseless deadweight loss from that kind of conflict. At this point, we seem close compared to what I would have thought a decade or so ago to these kinds of really advanced AI capabilities. They are pretty central in policy discussion and becoming more so.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“For the idea that these are the risks of truly apocalyptic disasters are real. I think the fact that the leaders of open AI, DeepMind, and Anthropic all make that point. They were recently all invited along with other tech CEOs to the White House to discuss AI regulation. And I think you could tell an alternative story where a larger share of the leading companies in AI are led by people who take a completely dismissive denialist view. And you see some companies that do have a stance more like that today. Yeah. And so a world where several of the leading companies are making meaningful efforts and you can do a lot to criticize could they be doing more and better and what have been the negative effects of some of the things they've done. But compared to World where even though AI would be reaching where it's going a few years later,”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And so I think there are real concerns of that type. I think it's true that AI progress has probably been accelerated by efforts like Bostrom's publication of superintelligence to try and get the world to sort of pay attention to these problems in advance and prepare. I think I disagree with Eliezer that has been on the whole bad. I think the situation is in some important ways looking a lot better than ways, alternative ways it could have been. I think it's important that you have several of the leading AI labs making at least not only significant lip service, but also some investments in things like technical alignment research, providing significant public support.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Most of what you were trying to capture could be expended on military material to fight you and you don't actually get much that is worthwhile and you paid a lot to do it that favored the defense at this level it's it's pretty challenging to net out all of the factors including all the future the future technologies I think yeah I mean the burden of interstellar attack being just like quite high compared to our conventional things seems real. But at the level of over millions of years weighing and that thing does it result in if they're aggressive conquest or not or is every star or galaxy you know approximately impregnable impregnable enough not to be worth attacking I'm not going to say I know the answer”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Send attacks at a civilization that was small but accessible. If you can quickly burn the resources that the attacker might want to acquire, if you can put stars into black holes and extract most of the usable energy before the attacker can take them over, then it would be scorched earth.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Thing is the extent to which the matter in a region can be harnessed all at once. You have a lot of mass and energy in a star, but it's only being doled out over billions of years because hydrogen, hydrogen fusion, exceedingly hard outside of a star. It's a very, very slow and difficult reaction. And if you can't turn the star into energy faster, then it's this huge resource that will be worthwhile for billions of years. And so even very inefficiently. Attacking a solar system to acquire the stuff that's there could pay off. So if it takes a thousand years of a star's output to launch an attack on another star and then you hold it for a billion years after that, then it can be the case that just like a larger surrounding attacker might be able to even very inefficiently”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“The extreme difference is that things outside, especially outside the solar system, things are very far apart. And there's a speed of light limit. And to get close to the speed of light limit, you have to use an enormous amount of energy. And so there would be that would tend to, in some ways, favor the defender because you have something that's coming in at a large fraction of the speed of light and it hits a grain of dust and it explodes. And the amount of matter you can send to another galaxy or distant star for a given amount of reaction mass and energy input is limited. So it's hard to send an amount of military material to another location because what can be present there already locally. Like it would make it harder for the attacker between stars or between galaxies. But there are a lot of other considerations.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, then that would pull in the other direction. That's the kind of values judgment and sort of social coordination problem that people would have to negotiate for and things like democracy and international relations and sovereignty would apply to help solving.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And others that go a direction of having fewer people and more natural resources per person. And they just coexist. But sort of how much of each you get sort of depends on how attached people are to things that don't work with separate policies for separate jurisdictions, things like global redistribution that's ongoing continuously versus the sort of infringements on autonomy if you're saying that a mine can't be created, even though it has a standard of living that's far better than ours because of the advanced technology of the time, because it would reduce the average per capita income by having more capital around.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And they might respond by saying, Well, something like the subdivision before, maybe there's a restriction, there's a distribution of wealth. And then when one has a child, there's a requirement that one gives them a certain minimum quantity of resources and one doesn't have the resources to give them that minimum standard of living or standard of wealth. I can't do that because of child AI welfare laws. Or you could have a system that is more accepting of diversity and preferences. And so you have some societies or some jurisdictions or families that go the route of having many”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“With this income, do it, and increase their population approximately immediately until the funds for the universal basic income from the natural resource taxation divided by the set of recipients is just barely enough to pay for the existence of one more mind. And so there's like a Malthusian element in that this income has been reduced to near the AI subsistence level or the subsistence level of whatever qualifies for the subsidy given that this all happens almost immediately. You know, people who might otherwise have enjoyed the basic income may object and say, no, no, this is no good.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Investment, or it could be something that's more like if you invested more natural resources into population, it would come at the expense of something else necessary, including military resources if you're in a competitive situation where there remains war and anarchy and there aren't secure property rights to maintain. Maintain wealth in place. If you have a situation where there's pooling of resources, for example, say you have a universal basic income that's funded by taxation of natural resources. And then it's distributed evenly to like every mind above a certain sort of scale of complexity per unit time. So each second a mind exists, it gets such and such an allocation. In that case, then, all right, well, those who replicate as much as they can afford.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Incessantly, so that could be individually resource limited. So it could be that individuals and organizations have some endowment of natural resources and they can't get one another's endowments. And so some choose to have many offspring or produce many AIs. And then the natural resources that they possess are subdivided among a greater population. in another jurisdiction or another individual may choose not to subdivide their wealth. And in that case, you have Malthusiism in the sense that within some particular jurisdiction or set of property rights, you have a population that has increased up until some limiting factor, which could be like they're literally using all of their resources, so they have nothing left for things like defense or economic”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And may do it because they're asked or programmed to or wish to gain some benefit and they can pay for their creation and pay back the resources needed to create them very, very quickly. And so, yeah, financing for that reproduction is easy. And if you have one AI system that chooses to replicate in that way or some organization or institution or society that chooses to create some AIs that are willing to be replicated, then that can expand to make use of any amount of natural resources that can support them and to do more work produce more economic value. And so you ask like, well, what limits will limit population growth given these selective pressures where if even one individual wants to Replicate a lot, they can do so”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Don't necessarily have to apply to human civilization. It's possible for there to be like collective norm setting that blocks evolution towards maximum reproduction. So right now human fertility is often sub-replacement. And if you sort of extrapolated the fertility falls that come with economic development and education, then you would think, okay, yeah, well, the total fertility rate will fall below replacement and then humanity after some number of generations will go extinct because every generation will be smaller than the previous one. Now, pretty obviously that's not going to happen. One reason is because we'll produce artificial intelligence, which can replicate at extremely rapid rate.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Nature in general is in Malthusian states. That can mean organisms that are typically struggling for food. It can mean typically struggling at a margin of how the population density rises. They kill each other more often. Contesting for that can mean frequency-dependent disease has like different ant species become more common in an area their species specific diseases move through them. And a general process is, yeah, you have some things that can replicate and expand and they do that until they can't do it anymore. And that means there's some limiting factor they can't keep up”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“He was writing stuff in the 70s and maybe even earlier, but at least in the 70s, 80s, 90s. So his book Mind Children, some of his early academic papers, fascinating, not necessarily for the methodology I've been talking about, but for exploring the substantive topics that we were discussing in the episode.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Vacla of Smail's books. I don't. I often disagree with some of his methods of synthesis, but I enjoy his books for giving Pictures of a lot of interesting, relevant facts about how the world works. I would cite So some of Joel Mokir's work on the history of the scientific revolution and how that interacted with economic growth. Sort of an example of collecting a lot of evidence, a lot of interesting, valuable assessment there. I think in the space of AI forecasting, one person I would recommend going back to is the work of Hans Morovek. And it was not always the most precise or reliable, but an incredible number of brilliant innovative ideas came out of that. And I think he was someone who really grocked a lot of the arguments for a more sort of compute-centric way of thinking about what was happening with AI very early on.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Doing things very systematically over a huge area, I can just read all of their stuff and then absorb it, and then I'm set, except there are lots of people who do wonderful works in their own fields. And some of those fields are broader than others. I think I would wind up giving a lot of recommendations of just like great particular works and particular explorations of an issue or history.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Glad to hear that One general heuristic is to find ways to hew closer to sort of Things that are rich and sort of bodies of established knowledge. And less unpunditory. I don't know how you've been navigating that so far. Learning from textbooks and the sort of the things that were the leading papers and people of past eras, I think rather than being too attentive to current news cycles is quite valuable. Yeah, I don't usually have the experience of Here is someone.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Countless academics and researchers building up local pieces of the thing. And yet people don't follow the hamming questions. What's the most important problem in your field? Why aren't you working on it? I mean, that one actually might not work because if the field boundaries are defined too narrowly, you'll leave it out. Sure. But yeah, there are important problems for the world as a whole that it's sadly not the job of a large professionalized academic field or organization to do. And hopefully that's something that can change in the future. But for my career, it's been a matter of taking low-hanging fruit of important questions that sadly people haven't invested in doing the basic analysis on.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“I think Holden was mentioning this in your previous episode with a lot of the worldview investigations work. These are questions where there is no academic field whose job it is to work on these and has norms that allow making the best efforts go at it. Often academic norms will allow only plucking off narrow pieces that might contribute to answering a big question. But the problem of actually assembling what science knows that bears on some important question that people care about the answer to it falls through the crack. There's no discipline to do that job. So you have”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, you find this extremely skewed distribution. And I found that was really a valuable benefit of doing those deep dive investigations into many things in a systematic way. Because now I can answer actually the sort of a loose agnostic. Who knows, there's all this nonsense by diving deeply.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“The media level, there were many tales of ah, here's a doomsday story. When you go further to the scientists and were there arguments for it to actually check out, it was not there. But by actually systematically looking through many of these candidates, I wound up in a different epistemic situation than someone who's just buffeted by news reports and they see article after article that is claiming something is going to destroy the world. And it turns out it's like by way of headline grabbing attempts by media to like overinterpret something that was said by some activist who was trying to overinterpret some real phenomenon. And then most of these go away. And then a few things, things like nuclear war, biological weapons, artificial intelligence, check out more strongly. And when you wait, things like what do experts in the field think, what kind of evidence can they muster?”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“There are infestations in bee populations that are causing local collapses that can then be sort of easily reversed. They just breed some more or do some other things to treat this. And even if all the honeybees were extinguished immediately, the plants that they pollinate actually don't account for much of human nutrition. You could swap the arable land with others and there would be other ways to pollinate and support the things.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Of threats of doom, search for previous literature of people who have done discussions, and then, yeah, have a big spreadsheet of what the candidates are. And some other colleagues have done work of this sort as well. And just go through each of them, see how they check out. And it turned out doing that kind of exercise found that actually the distribution of candidates for risks of global catastrophe, it was very skewed. There were a lot of things that have been mentioned in the media as like a potential doomsday story. So things like, oh, something is happening to the bees. Well, that be the end of humanity. And this gets to the media, but if you track it through, well, okay, no, there's”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Computer science, physics. What are the doom stories or like candidates for big things associated with each of these fields? Go through the industries that the US Economic Statistics Agencies recognize and say for each of these industries, is there something associated with them? Go through all of the lists that people”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“So, I'm very interested in finding things like taxonomies of the world where I can go systematically through all of the possibilities. So for example, my work with open philanthropy and previously on global catastrophic risks, I wanted to make sure I'm not missing Any big thing, anything that could be the biggest thing I wound up mostly focused on AI, but there have been other things that have been raised as candidates. And people sometimes say, I think falsely, that, oh, yeah, this is just another Doomjay story. You know, there must be hundreds, hundreds of those. And so I would do things like go through all of the different major scientific fields from anthropology to biology, chemistry.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Someone mentions the diminishing returns. I immediately ask, hmm, okay, so you have 2x mentioned processes. What's the ratio between the Dublin you get on the output versus the input and find, oh yeah, actually, it's interesting. For computing and information technology and AI software, it's well on the one side. There are other technologies that are closer to neutral. And so whenever I can go from here's a vague qualitative consideration in one direction, here's a vague qualitative consideration in the other direction. I try and find some data, do some simple Fermi calculations, back of the envelope calculations, and see like, can I get a consistent picture of the world being one way, the world being another, also compared to some, I try to be exhaustive more.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“In terms of what is a day in the life, how do I go about it? So, one is just keeping abreast of literatures on a lot of these topics, reading books and academic works on them, doing my approach compared to some other people in forecasting and assessing some of these things. I try to obtain and rely on more any data that I can find that is relevant. I try early and often to find factual information that bears on some of the questions I've got, especially in a quantitative fashion. Do the basic arithmetic and consistency checks and checksums on a hypothesis about the world do that early and often. And I find that's quite fruitful and that people don't do it enough. But so things like with the economic growth.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so I'd say you're somewhat getting the benefit of the fact that I've done fewer podcasts. And so I have a backlog of things that have not shown up in publications yet. But yes, also, I've had a very weird professional career that has involved a much, much higher proportion than is normal trying to build more comprehensive models of the world. And so that included being more of a generalist trying to get on understanding of many issues and many problems that had not yet been widely addressed, but do a first pass and a second pass dive into them. And just having spent years of my life working on that, some of it accumulates.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, a general issue with sort of tracking that kind of thing is new companies come in. Anthropic did not exist. Any number of things. It's a personal portfolio. I do not invest in any AI labs for conflict of interest reasons. I have invested in the broader industry. I don't think that the The conflict issues are very significant because they're enormous companies and their cost of capital is not particularly affected by marginal investment and I'm not really in a I have less concern that I might find myself in a conflict of interest situation there.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Coming into the topic in the 2000s, I thought, yes, the strong case, even an old case, that AI will eventually be. Thing in the world, it's kind of crazy that the investment in it is so small. And over the last 10 years, we've seen the tech industry and academia sort of realize, yeah, they were wildly underinvesting in just throwing compute and effort into these AI models, and particularly letting the neural network connectious paradigm kind of languish in an AI winter. So, yeah, I expect that process to continue as it's done over several orders of magnitude of scale up. And I expect at the later end of that scale up, which the market is partially already pricing in, it's going to go further than the market expects.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And then their companies working on the software. So the big tech giants and also companies like OpenAI and DeepMind. And in general, the portfolio picking at those has done well. It's done better than the market because as everyone can see, there's been an AI boom. But it's obviously far short of what you would get if you predicted this is going to go to be like on the scale of the global economy and the global economy is going to be skyrocketing into the stratosphere within 10 years. If that were the case, then collectively these AI companies should be worth a large fraction of the global portfolio. And so I embrace the criticism that this is indeed contrary to the efficient market hypothesis. I think it's a true hypothesis that the market is in the course of updating on in the same way that”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Be quite in demand, but what that would happen before you have so much investor cash making purchases and sales on this basis, you would first see it in things like the valuations of the AI companies, evaluations of AI chipmakers. And so far there have been effects. So some years ago in the 2010s, I did some analysis with other people of if this kind of picture happens, then which are the firms and parts of the economy that would benefit? And so there's the makers of chip equipment, companies like ASML, there's the fabs like TSMC, there's chip designers like NVIDIA, or they component of Google that does things like design the TPU.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so I think at some point, so in the case we were talking about where There are visible, so this intelligence explosion happening in software to the extent that investors are noticing that, yeah, they should be willing to lend money or make equity investments in these firms or demanding extremely high interest rates. Because if it's possible to turn capital into twice as much capital in a relatively short period, and then more shortly after that, then yeah, you should demand a much higher return and competition if there is, assuming there is competition among companies or coalitions for resources, whether that's investment or ownership of cloud compute, is this cloud compute made available to a particular AI development effort?”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Working on AI largely, which is one reason for the report from Open Philanthropy by Tom Davidson building on these models and putting that out for review, discussion, engagement, and communication on these ideas. So part of it is. I want to raise these issues. It's one reason I came on the podcast. And then they have the opportunity to actually examine the arguments and evidence and engage with it. Do predict that over time these things will be more adopted as AI developments become more clear? Obviously, that's a coherence condition of believing the things to be true, if you think that society can see things when the questions are resolved, which seems likely.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“That these views are like contrarian, not widely held. In particular, the sort of detailed models that I've been working with are not something that most people or almost anyone is examining these problems through. You do find parts of similar analyses people in AI labs. There's been other work. I mentioned Moravec and Kurtzweil earlier. There also have been a number of papers doing various kinds of economic modeling. So standard economic growth models, when you input AI related parameters commonly predict explosive growth. And so there's a divide between what the models say, and especially what the models say with these empirical values derived from the actual field of A, that link up had not been done even by the economists.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh, also on the topic of takeoff and the AI expert survey, I think the general argument for intelligence explosion, I think commanded majority support, but not a large majority. So you can say I'm closer on that front. And then, of course, there's at the beginning I mentioned these greats of computing. The founders of computing like Alan Turing, Van Neumann, and then today you have people like Jeff Hinton saying these things or the people at OpenAI and DeepMind are making noises suggesting timelines in line with what we've discussed and saying serious risk apocalyptic outcomes from them. So there's some other sources of evidence there. But I do acknowledge, and it's important to say and engage with and see what it means.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so one, of course, there's some of those components was the metaculous AI timelines are relatively short. There's also, of course, the surveys of AI experts conducted at some of the ML conferences, which have definitely longer times to AI, more several decades in the future. Although you can ask the questions in ways that elicit very different answers and show most of the respondents are not thinking super hard about their answers. It looks like now close in the recent AI surveys, close to half were putting around 10% risk of an outcome from AI, close to as bad as human extinction. And then another large chunk, like 5%. That was the median. So I'd say compared to the typical AI expert, I am estimating a higher risk.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, some of the metaculous AGI questions actually are kind of shockingly soon. For AGI. There's a much larger differentiator there on the market, on the meticulous forecasts of AI disaster and doom. Like a few percent or less rather than like 20.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And institutions that will very stably maintain in variance, like no WMD war allowed. And so like a treaty to ban genocide, weapons of mass destruction war like that would be the kind of thing that becomes much more attractive if the alternative is not well. Maybe that will happen in 50 years. Maybe it'll happen in 100 years if it's well. Maybe it'll happen this year.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Bad to have the future compressed and there be such a rate of catastrophic outcomes. So when we're speeding up and compressing the future, that gives us in the short term, even like human societies discount the future a lot, don't pay attention to long-term problems. But the flip side to the scary parts of compressing a lot of the future, a lot of technological innovation, a lot of social change, is it brings, what would otherwise be long-term issues into the short term where people are better at actually attending to them? And so people facing this problem of, well, will there be a violent expropriation or a civil war or a nuclear war in the next year because everything has been sped up by a thousand fold? And if their desire to avoid that is reason for them to set up systems.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“And if any of them causes trouble, then we have to face quite a lot of trouble period. There's also this issue of if there's occasional wars or conflicts measured in subjective time, then if a few years, a thousand years or a million years of subjective time for these very fast minds that are operating in a much higher speed than humans, you don't want to have a situation where every thousand years there's a war or an expropriation of the humans from AI society. Therefore, we expect that within a year we'll be dead. That would be pretty...”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, one thing might be that going at the maximal speed and doing things in months rather than even a few years if you have the chance to slow things, losing a year or two seems worth it to have things a bit better managed than that. But I think the big thing is that it condenses a lot of issues that we might otherwise have thought would be over decades and centuries to happen in a very short period of time. And so that's scary because say if any of these the technologies we might have developed with another few hundred years of human research, if any of them are really dangerous, so scary bioweapon things, maybe other dangerous WMD, they hit us all very quickly.”
2023-06-26 · Dwarkesh Podcast · Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future · IDENTIFIED FROM THE TRANSCRIPT · source