YouSaid · the spoken record

Philip Tetlock

lines on the record: 81
first: 2016-03-24
most recent: 2016-03-24
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“Well, I think because there's a lot of motivated reasoning going on, as we noted earlier, there's this tendency to do a lot of vague verbiage forecasting to paint a dramatic scenario and then hold it together with some very weak verbs like this might happen or could happen or the distinct possibility of this happening. So there's this interesting tendency that the pundits have of engaging our attention with a vivid scenario, disintegration of the Saudi regime or Sino-Japanese wars, something Tom Clancy-ish on that kind of scale. But to stitch it all together with weasel word terms that allow them to retreat later on. Bloomberg is a major purchaser of expertise.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“And then he paused, said, Well, maybe it's more like 55, 45, 45, 55. Or indeed 52, 48, 48, 52. How granular can you get in poker? Well, poker is a game with repeated play, quick, clear feedback. It's possible to get more granular on poker than it is about the location of terrorists or about whether countries are going to leave the Eurozone. But it's an open question of how granular you can get. And you need to grapple with this thing. The distinction between precision and pseudo-precision. And that's one of the things super forecasters are just very thoughtful people who push the frontiers of knowledge as far as they can. And that means sometimes pushing them a little too far, in which case they retreat.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Right. And the chief risk officer is Aaron Brown. And when we talked with Aaron, he's also a really serious poker player. And well, you can tell the difference in a world-class poker player and a talented amateur on the basis that the world-class player knows the difference between a 60-40 bit and a 40-60 bit.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“I think to ask the question is to answer it. He would have seen opportunity for being much more granular in making bets about sports than he would in making estimates about the likelihood of a particular terrorist being in a particular location. It turns out that for many categories of problems where we think it's impossible to be more granular, it is possible. And that's one of the things Super Forecasters have learned that there's a difference between 50 and 75%. Sometimes they can even make distinctions between 50 and 60%. Now, we quote the chief risk officer of AQR, the hedge fund, Aaron Brahm.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“He was getting, he would have said, looks like about a 75% probability. But instead, he said something interesting. He said, well, look, look guys, this is a coin flip. It's a 50-50 thing. Now, the president, I think, is a very intelligent person, and I think he's capable of being very granular in his assessments of uncertainty. And if you doubt it, think about the following thought experiment, which is appropriate given that we're approaching March Madness. He follows March Madness and basketball. He's a basketball fan. Imagine he'd been sitting around with friends waiting for Duke to play some team in March in the March Madness tournament. And they offered him exactly the same probabilities about whether Duke would win the game, somewhere between 35 and 95 with a center of gravity of opinion around 75. Would he have said, hmm, sounds like a 50-50 thing, or would he have said, hmm, sounds like about three to one Duke?”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“You want to just pick a one at random? Sure Okay, well, one of them has to do with granularity. And it's actually grounded in a story that about President Obama and how he reacted to his advisors who were offering him somewhat conflicting probabilities on how likely it was that Osama bin Laden was residing in a particular compound in Abadabad, Pakistan. As we all know now, he was indeed residing there and the president did authorize a Navy SEAL mission and that resulted in Osama bin Laden's death. Now, when the president was confronted by these probability estimates from really smart people at the top of the intelligence apparatus, ranging from about 35 or 40 percent up to about 95%, the president's reaction was an interesting one. If you had computed the average of the median estimate of the advice”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, and I'll just add one other thing to that. They weren't just outperforming prediction markets in the public sphere. They were also outperforming intelligence analysts who were working behind a veil of classified information. And it's just a remarkable thing. And I think U.S. intelligence community, which is much maligned for many things, deserves some credit for its willingness to sponsor a forecasting tournament that has the potential to be embarrassing for a government bureaucracy. How often do you see a government bureaucracy spend money, a lot of money, on a project that has the potential to yield results that are fairly embarrassing?”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“rather than allowing them to contaminate each other and you get conformity and you get groupthink and you get to get kind of a blur rather than a number of distinct points of view and then you can combine them statistically somehow so we were the only competitor in the forecasting tournament sponsored by the US intelligence community that used teams but we hedged our bets we weren't sure that teams would work we ran an experiment and we randomly assigned people to teams and we randomly and had other people work as individuals and we were truly surprised that the teams functioned as well as they did and it's an interesting question of why our teams were so dynamic and open minded relative to many teams you see in actual organizations”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, the teams of Super Forecasters truly astonished us because the statisticians were telling us the right thing to do here was to have the each individual top performer make judgments completely independently of the others.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Infer the strength of the first atomic blast by dropping little pieces of paper when the winds came in by estimating how far the winds blew. I think he was off by about 40 or 50 percent. But for Fermi estimates, that's not too bad. It's a lot better than simply shrugging your shoulders and saying, I have no idea. I say, oh, a 10 kiloton blast as opposed to 20.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Answer. Well, you're making a lot of guesstimates. And it's not just the breaking down and trying to get the answer. What you're doing in the process is you're revealing sources of ignorance and your colleagues on your team, for example, because our forecasters often work together on teams, your colleagues can help you correct your errors. So what's the population of Chicago? Well, I don't really know. Is it between 2.5 and 4 million? Maybe, you know, some I would guess about 3 million. Somewhere in the middle there. What proportion of the population would have a piano and so forth. So you would break it down and you'd try to find eventually how many people could conceivably make a living working as piano tuners given the number of people who own pianos in Chicago and their willingness to pay for the services of piano tuners. So Fermi did this. One legend is that he tried to”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Interestingly, you know, some of the high tech firms in Silicon Valley were quite fond of asking off the out-of-left field questions of this sort because they thought it was a great way of testing how well people think on their feet. In the book we call these Fermi questions, named after the great Italian-American physicist Enrico Fermi, who developed the first nuclear reactor. It was a key part of the Manhattan Project, developing the bomb. And Fermi was fond of posing these oddball questions to his students. What he wanted the students to do was to take a seemingly intractable problem and break it down into parts or components that were more tractable. So you might not have any idea how many, well no one has any idea initially on how many piano tuners there might be in Chicago, but you make guesstimates about the population of Chicago. So walk us through that.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“We need to be very clear about the functions that are being served. Some of these big ideas are very useful lenses for viewing the world at particular moments in history in conjunction with other ideas. So I'm not saying that the intellectual apparatus is useless. I'm saying that what's really dangerous is when you have a smart person who runs too far with a big idea and fails to see that the complexity of the world puts a lot of brakes on it. So one of our rules of thumb for distinguishing better forecasters and worse forecasters on the media is the ratio of the number of times they say however versus moreover. So if you have a high however over ratio, that means you're a fox. That means you're boring. That means you're probably going to be kicked off the show. But more accurate, more likely to be more accurate. But that's right. You're going to have better Briar score.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, I would just suggest that people be better off if they were more honest about the functions that are served by consuming different types of information. So if you said to yourself, look, I want to be entertained. I want to see somebody who's saying outrageous things, and I'm going to be, but I'm not going to base my probability judgments on them. I just want to be entertained by these amazing stories this person is going to tell about how the Saudi regime is going to disintegrate and how we're on the verge of a World War III. This is really entertaining stuff as opposed to listening to this much more tentative, nuanced fox-like forecaster who's saying on the one hand, on the other hand, drones on and on, about why there's really only about a 22% probability of the Saudi regime changing in the next 24 months.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Predictions, their overall batting average is pretty bad. We also found that the hedgehogs tended to be more prominent. They're more attractive to the media. The media like the kinds of soundbites that hedgehogs can deliver. And we found that the hedgehogs also tended to have at least a few home runs. Matt, if you're making a lot of pretty extreme predictions on a wide range of subjects, at least a few of them are almost by chance going to be accurate. Whereas the foxes are making more moderate probability judgments and they have less claim on home runs. So you get on somewhat ironic situation that the worst forecasters have the greatest media prominence.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“The work on political judgment because it captures rather well different styles of thinking. You could be, for example, a hedgehog of many different political sorts. You could be a free market hedgehog, or you could be a Marxist hedgehog. You could be an environmentalist doomster hedgehog, or you could be a boomster utopian kind of techno-utopian sort of hedgehog. They're going to find a cost-effective substitute for whatever we're running out of. So there are many different forms of hedgehog, left, right, pessimistic, optimistic. And we identified many of them in the early work, and we track their accuracy, and we compared their accuracy to that of Moore Fox-like forecasters. And we found a couple of things. One is that the hedgehogs tend to have pretty bad batting average. When you look at all their...”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“So, Isaiah Berlin was a British scholar, political historian, philosopher, who took a quote from the Greek warrior poet, Archillochus, from 2,500 years ago, and he built a really interesting essay around it. And the quote, one of the few surviving fragments of this man's work was that the fox knows many things, but the hedgehog knows one big thing. And he intended that to capture different styles of thinking. He thought that some thinkers were much closer to being foxes. Shakespeare, I think, was one of his classic examples of a fox who could just very multifaceted view of human nature. And other writers, he thought, could be pigeonholed better as hedgehogs. Now, we use this fox hedgehog distinction in”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, here's the thing, Barry. They don't even think that they're wrong. If I say there was a distinct possibility that Putin is going to invade Estonia this coming year and he doesn't do it, I'm going to interpret distinct possibilities having met a very low probability. And if he does do it, I'm going to interpret distinct possibilities having met a very high probability. We tend to be somewhat self-serving in our own mental calculus. We say, well, you know, we interpret, we give ourselves the benefit of the doubt on how we interpret distinct possibility. Just did we give the benefit of the doubt to political pundits who favor our political party?”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Some waffle words like this could happen or this might happen, or there's a distinct possibility this will happen. Now, distinct possibility is one of those wonderful phrases, because if it happens, I can say, hey, I told you there was a distinct possibility. If it doesn't happen, I can say, hey, I just said it was possible. So I'm covered either way. And that helps to explain why pundits and indeed why traditionally people in the US intelligence community as well have relied so heavily on vague verbiage forecasting because they need to be saying something that sounds informative, but they need strategy for preserving their long-term credibility at the same time.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, if you're a pundit, you're playing a complicated game. If I'm a pundit on your show or in anyone's show, I need to make it sound as though I know what I'm talking about. I need to make it sound as though I'm telling the listeners something they didn't know before. I also need to preserve my long-term credibility, which means I have to have some escape clauses. So if the claims I make about the future turn out to be wrong, I need some way of walking away from it. So when you have this problem of a career as a pundit, you need to be saying something surprising, but you also need to preserve your long-term credibility. That's a real dilemma the pundit is in. So the typical way pundits cope with this is by saying something dramatic, like Canada will disintegrate or the Eurozone will disintegrate, or Putin will reinvade the Ukraine, but build in.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Interestingly, very little work had been done. There was a little bit of work assessing the accuracy. Well, there was quite a bit of work assessing the work of weather forecasters. There was some work assessing the accuracy of expert bridge players. And there was some work assessing the accuracy of economists, the Federal Reserve in Philadelphia and elsewhere had been doing some studies along those lines. But as for assessing the accuracy of political pundits, at the time my book came out, I think there was extremely little work. On that subject”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“All three of them. And the interesting question is, do those things give us much guidance on how accurate what the person is saying? So we're hoping that they do. We're hoping to say, well, this person must know what he or she is talking about by virtue of the fact that this person has done X, Y, or Z. But the relationship between having done X, Y, or Z and accuracy is unknown. And the more honest we are about our ignorance, the more honest we are about when we're using proxy cues for judging how credible a source of advice is, the better off we're going to be in the long term. And by the way, that applies to me too.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, there's the big question that we want the answer to, and then there are all these proxy cues that we kind of latch on to in the hope that those proxy cues will get us closer to the answer. So the big question we want the answer to, say, is whether the U.S. economy is going into recession next year, whether the Dow is going to be over $20,000 or under $15,000. There are some big questions that people in the financial community or political community want answers to. And there are various people who pass through our lives, pass through your radio station, pass in the op-ed pages of newspapers, on television, and so forth, who offer opinions on these things. And they come with various types of credentials. You might say so-and-so is the muckety muck professor blubbedly blump. Or you might say that so-and-so won a Nobel Prize, or you might say that so-and-so is worth $10 billion. There are a lot of things you could say about that.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Right. And it was a lunchtime conversation about 30 years ago in which he said rather casually that he thought that the experts I was interviewing from my early work on expert political judgment would have a hard time doing better than an attentive reader of the New York Times, which is kind of a fancy way of saying more or less what you just said. Now he didn't know that as a fact. He was offering that as an hypothesis. And I think the right way to look at this is it is an hypothesis we can be continually testing. It's not always going to be the case that experts are going to fall short, but they're going to fall short much more often than we would expect.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“It turns out that you reach the point of diminishing marginal returns for knowledge quite quickly in a lot of the domains we care the most about. Now, what does that mean? When I started off doing research on political judgment, one of the greatest psychologists on the planet advised me, Daniel Kahneman.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Make extremely granular distinctions among degrees of uncertainty. Human beings can't make that many distinctions among degrees of uncertainty in most environments. I mean, there are some environments where we can pull out a calculator and do it. You can do it with card tricks and so forth, or poker. But there are real limits on how granular you can become when it comes to whether there's going to be a country leaving the Eurozone in the next year or whether there's going to be a violent Sino-Japanese clash in the East China Sea or things of that sort. How many degrees can you distinguish there? Is it just yes or no? Is it yes, maybe, no? Or can you make finer distinctions than that?”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“That's right, the probability scale in principle, if you want to look at it the way a mathematician would, is infinitely divisible. There's an infinite number of points between 0 and 1.0. Now, obviously, people can't distinguish that many levels of uncertainty. Now, when IBM's Watson was playing in the Jeopardy competition and beat the best human players, you might have noticed that under its answers, occasionally there would be this little Bayesian probability estimate of how confident Watson was in his answer by 0.87362 or something. These types of forms of artificial intelligence do try to”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“The Briar score. Well, the Briar score was originally developed by some statisticians working with meteorologists who wanted to keep score. And this goes all the way back to 1950. And it's a very simple idea. You want to minimize the gaps between probability and reality. So you code reality as either zero or one, depending on whether the event occurred or didn't occur. And you have probability judgments that range from zero to one. You get a really good Breyer score. If you assign probabilities very close to zero to things that don't happen and probabilities very close to 1.0 to things that do happen. So it's your ability to be justifiably decisive to make appropriately extreme judgments as the circumstances dictate, but to avoid driving off a cliff in the pursuit of that objective by making extremely overconfident judgments and saying 90% of things that don't happen and 10% of things.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, there are a couple of theories about that. One is that the problem lies in the experts and how the experts think, and they could do better if they had used better analytical tools and were more self-critical and creative and thoughtful. And the other theory is that we just live in a radically unpredictable world in which virtually nobody can do appreciably better than chance over extended periods of time. It's a world roiled by black swanish, dark gray swanish events, and buyer beware.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“I think there's a great deal of superstitious reasoning about the impact that leaders have on organizational performance and also the impact that presidents have on the economy. The number of levers that presidents have to influence economic outcomes is quite limited. But there is this deep intuition we have that leaders should be accountable for what happens on their watch, like the captain of a ship. It really doesn't matter whether a storm suddenly emerges out of nowhere, you want to hold somebody accountable. And that is also deeply wired into us. Someone has to be accountable for it.”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, it would be deeply dissonant for people to come to the conclusion that it doesn't matter who is elected president. It doesn't matter whether we raise taxes or cut them, whether we help the Ukraine or don't help the Ukraine or intervene in Syria or don't intervene in Syria or sign free trade pacts or don't do that, we could just toss a coin because nobody is on either side of the political spectrum has any demonstrated ability to assign good probabilities to the consequences of those policy options. To say that sounds nihilistic, it's dissonant. People just don't want to live that way. People don't want to live that way. They want structure. And for the same reason that people have turned to psychics and witch doctors and all sorts of things over the past, we seek guidance, predictive guidance, and we seek, there is a demand for it even when the supply of good forecasting is extremely...”
2016-03-24 · Masters in Business · Barry Ritholtz's Masters in Business: Philip Tetlock Interview · IDENTIFIED FROM THE TRANSCRIPT · source