YouSaid · the spoken record

Philip E. Tetlock

lines on the record: 78
first: 2020-04-22
most recent: 2020-04-22
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“It's actually not too difficult in that particular case. It really depends on the type of accountability. Tournaments create a very stark monistic type of accountability in which one thing and only one thing matters, and that is accuracy. You get no points for playing to your, for being an ideological cheerleader and pumping up the probabilities of things that your team wants to be true or downplaying the probabilities of the things your team doesn't want to be true. You take a reputational hit. So the incentives are very unusually tightly aligned to favor accuracy. That's extremely unusual in the social world. Most forms of accountability occur in organizational settings in which there are lots of distortions at work. And the rational political response for a decision maker located in most accountability matrices and organizations is to engage in some mixture or strategic attitude shifting toward the views of important others or, as you put it, evasion, profound”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“I think humans have been repeatedly humbled in competitions against simple statistical algorithms. Going back to Paul Meale, his famous little book on clinical versus actuarial approaches to predicting in medicine and psychiatry, I would say be humble”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“Indeed. Well, simple extrapolation algorithms are historically hard to beat. And if you, you know, we just were running a COVID-19 mini forecasting tournament right now and they're proving to be hard to beat. The skill, of course, is when to alter the trend, whether to accelerate it or to decelerate it or to change direction.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“I'm not sure there's a definite trend one way or the other. You hear a lot of talk, a lot of claims that there are, but if you look back on the 20th century, there certainly were lots of major pockets of unpredictability. It's not clear.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“Very domain specific, it sounds like a great idea. Who could be against it? But the devil lurks in the details and it doesn't deliver as automatically as you might hope. It's trench warfare here.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“Quite there yet, but they're StarCrafter, whatever the next thing that Dennis Osabus is going to conquer. But no, I don't see evidence that those approaches work in the domains that we study with the intelligence community.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, ARPA just ran a forecasting tournament called Hybrid Forecasting Competition in which they pitted algorithmic brooches and human approaches and hybrid approaches against each other. And I should let IRPA speak for itself about how well its programs work or don't work. I don't think there's a lot of evidence to support the claim that machine intelligence is well equipped to take on the sorts of problems that the intelligence community wanted to have answered when it runs the forecasting tournaments has been running with our research team, things like the Syrian civil war, Russia, Ukraine, settlement on Mars. These are events for which base rates are elusive. It's not like you're screening credit card applicants for visa, machine intelligence is going to dominate human intelligence totally. Machine intelligence dominates humans and Go and chess. It may now dominate humans and poker. I don't know that we're with the state of the art is.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“Think a lot of good executives have the intuition that you get more out of a team of forecasters or problem solvers if you elicit independent judgments initially that are uncontaminated by conformity pressure and then you create an environment which ideas can be freely critiqued before lifting the veil of anonymity and letting people see who's taking which positions.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“That's a matter of managerial skill, and I would say, you know, going back to my one of my old dissertation advisors at Yale 40 plus years ago, Irv Janice on Groupthink, I would pick a leader who knows how to shut up and not reveal opinions at the beginning of the meeting and knows how to listen.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“I think you'd want an interdisciplinary team. I mean, diversity is one of these words that's been reduced to a cliche, but I think we have found in our work that cognitive diversity helps. And it helps in certain quite well-defined ways. If you want to create a composite that outpredicts the vast majority of the super forecasters, a good way to do it is not only to take the most recent forecasts. But it's also to extremize that forecast to the degree that people who normally disagree agree with each other. And when you have convergence among diverse observers, that's a signal that the weighted average composite is probably too conservative and you should extremize.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, is it a mistake if you're a public health expert who feels that one mistake is much worse than the other? It's much better to overestimate the threat of the virus than to underestimate it because you have to influence public opinion and public behavior. There you're not forecasting. You're engaged in manipulation, social influence. So this comes back to your original question about what do we want from our forecasters. And accuracy is only one of the things we want from them. We look to forecasters for a lot of things to inspire confidence, to inspire fear, and so forth.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, I would know who has arguments to take the more integratively complex forms on the one hand, on the other hand, and then synthesis, whether you want to consider those arguments smarter or not is another matter. But yes, I suppose that's fair. I mean, I've always resonated a little more to the foxes and to the hedgehogs, but I think when you look at the great achievements in science, they often come from hedgehogs.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“I guess I have a little bit of attention deficit disorder, and my attention tends to Rome, so I've not just worked on forecasting tournaments. I mean, I've been fairly persistent in pursuing this topic since the mid-1980s, you know, back, even before Gorbachev became general party secretary, I was doing a little bit of this. But I've been doing a lot of other things as well on the side. So my attention tends to roam. I'm interested in taboo trade-offs. I'm interested in accountability. There are various things I've studied that don't quite fall in this verb.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, they have a skill set, and most in the most recent tournaments we've been working on with them, that this becomes even more evident that their willingness to delve into the details of a really pretty obscure problem for very minimal compensation. It's quite extraordinary. They are intrinsically cognitively motivated in a way that is quite remarkable. How am I different from that?”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“I finished in the middle of the super forecaster pack. So that doesn't mean I'm a super forecaster, it just means that when I tried to make forecast better than the composite, I degraded the accuracy significantly.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“I don't think so. I think I have the patience or the temperament for doing it. I did give it a try in the second year of the first set of forecasting tournaments back in 2012. And I monitored the aggregates. We had an aggregation algorithm that was performing very well at the time and it was outperforming 99.8% of the forecasters from whom the composite was derived. So if I simply had predicted what the composite said at each point in time in that tournament, I would have been a super, super forecaster. I would have been better than 99.8% of the super forecasters. So even though I knew that it was unlikely that I could outperform the composite, I did research some questions where I thought the composite was excessively aggressive and I tried to second guess it. And the net result of my efforts, instead of finishing in the top 0.02% or whatever, I think”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“To 0.65, for example. If on average that doesn't improve their accuracy, we would conclude that they can't achieve that level of granularity. Our best statistical estimates are forecasters for the types of questions the intelligence community often poses. Can distinguish between 10 and 15 degrees of uncertainty, which is considerably more than the seven they think they can now, a lot more than the five they thought they used to be able to distinguish. But how useful is it to be able to distinguish varying degrees of uncertainty is going to hinge on the kind of game you're playing? If it's poker, you might well be on someone who's very adept at distinguishing things to say three decimal points.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“The optimal level of granularity for different categories of forecasts, I think for the kinds of things we were looking at in the IARPA original forecasting tournaments, the geopolitical events like how long the Syrian civil war would last or what would Russia do in eastern Ukraine or things of that sort. Yes, it would be absurd to go to three or four decimal points. Originally, the National Intelligence Council, which synthesizes a lot of intelligence analysis, They only distinguished five degrees of uncertainty and they didn't put numbers on it. More recently they've moved to seven degrees of uncertainty and they do put numerical ranges on. So somewhat likely represents a certain probability range. Now in our work we've explored how granular it was the best forecasters are doing various rounding experiments where we round their forecast off to the nearest tent that kind of thing if it's pseudo-precision if when they adjust when they move from point”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“Term best is a term I'm just not comfortable with. I rather doubt that they're at the optimal forecasting frontier at the moment, but they're often probably fairly close. And is there room to incentivize people to predict the market? Well, that's one of those paradoxes that economists have written about, right?”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“It's an interesting experiment. And the person, okay, my project manager from the first set of forecasting tournaments, Terry Murray, founded a company, Good Judgment Incorporated, which does things like that. So that's a proprietary venture. And you probably want to talk to Terry about how successful or not successful they've been in doing that. I think they've had some success. I think it's extremely hard to do that. It's non-trivial. I think there's a good deal of similarity in the cognitive ability, cognitive style profiles of super forecasters and the kinds of people you see on the staffs of Goldman Sachs. And it would be a tight race.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“Not perform quite as well as forecasting journalists. When they fall short, it's hardly a decisive rebuke of the market mechanism for eliciting forecasts. There are lots of very powerful institutional actors like Goldman Sachs and so forth that are continually trying to do exactly what you describe. It's an ongoing process. You're an economist. I don't need to tell you that sure.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh boy, well, you know, we play with prediction markets and the work with the intelligence community. Going all the way back to Admiral Poindexter in the original DARPA effort to launch prediction markets inside the intelligence community. I think your colleague Robin Hanson was involved with some of that work around 2,000 in that range. We've also been working with the prediction markets in parallel with forecasting tournaments and there are pros and cons to each method of eliciting judgments. But the IC, the intelligence community, doesn't let us make those markets deep and liquid the way they are on Wall Street. People are essentially competing for reputational points the way they are in forecasting tournaments, the monetary prices are either small or non-existent. So most economists, I think, would not consider that to be a very robust test of the efficacy of prediction markets. And when prediction markets fall short, and they do typically”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“You could certainly have induced the epidemiologists who are worried about the horseshoe bats in central China or other possible sources of zoonotic viruses. You could certainly have induced them to pump up their probabilities. And you would, of course, start to run into a problem of crying wolf if they'd been saying there's a 30, 40, 50 percent chance of a viral leap into human beings each year and it didn't happen, didn't happen, didn't happen, people would you get a crying wolf effect, right?”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“Right. So you hear aversion to that argument, I suppose, on Wall Street as well. Sure. There's a good deal of truth to it. I certainly have been guilty of overconfidence at many junctures in my career thinking I'm going to be able to take on things that looked impossible and often turned out to be impossible. Most projects that most scientists embark on, I think, don't succeed. It does take a certain amount of quasi-irrational persistence.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“But you have lots of people competing in the marketplace of ideas for attention, and that's a hard competition for scientists to beat.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, the signal to noise ratio isn't going to be great, I can assure you of that because there are plenty of people who are naturally more entertaining than epidemiologists.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, in some sense, we already did have that portfolio. It was a mainstream position among epidemiologists for the last 20 years or so that you had a recipe for a disaster. David Epstein, the guy who recently wrote Range, a very interesting guy, you may have had him on your show, I don't know, but he recently quoted himself from a 2007 newsletter that he wrote. And he said something like, the presence of a large reservoir of SARS-like viruses among horseshoe bats combined with a culture of eating exotic meats is a time bomb. And that was in microbiology books. In the first decade of the 21st century, it was common knowledge. And indeed, even before SARS won, even before the first SARS outbreak, but even before SARS-1, epidemiologists were acutely aware of this. So it's not as though... We didn't have it in our portfolio. We did.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source
“I think we want a lot of things from our forecasters, and accuracy is often not the first thing. I think that we look to forecasters for ideological reassurance. We look to forecasters for entertainment, and we look to forecasters for minimizing regret functions of various sorts. So that we would really regret not having anticipated X, Y, or Z. So we want to pump up the probabilities of those things.”
2020-04-22 · Conversations with Tyler · Philip E. Tetlock on Forecasting and Foraging as a Fox · IDENTIFIED FROM THE TRANSCRIPT · source