YouSaid · the spoken record

Jon McAuliffe

lines on the record: 93
first: 2023-09-08
most recent: 2023-09-08
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“It is, yeah. I think you're talking about prediction problems ultimately. So in recommender systems, you can think of the question as being, well, if I had to predict what thing I could show a person that would be most likely to change their behavior and cause them to buy it, that's a kind of prediction problem that motivates recommendations. In biotechnology, very often we are trying to make predictions about whether someone Condition a disease based on lots of information we can gather from high throughput diagnostic techniques. These days, the keyword in biology and in medicine and biotechnology is high throughput. You're running analyses on an individual that are producing hundreds of thousands of numbers. And you want to be able to take all of that kind of wealth of data and turn it into diagnostic information.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“So Occam's razor. The philosophical idea that you should choose the simplest explanation that fits the facts. the simplest approach, even though you wish you could choose it, is not the most accurate approach. If you care about predictive accuracy, if you're putting predictive accuracy first, then you have to embrace a certain amount of complexity and lack of interpretability.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Complains about that because the results are astounding. The thing that you get is incredible. And so that is by analogy the way that we reason about running systematic investment strategies. At the end of the day, predictive accuracy is what creates returns for investors. Being able to give complete descriptions of exactly how the predictions arise does not in itself create returns for investors. Now, I'm not against interpretability and simplicity. All else equal, I love interpretability and simplicity. But all else is not equal. If you want the most accurate predictions, you are going to have to sacrifice some amount of simplicity. In fact, this truth is so widespread that Leo gave it a name in his paper. He called it Occam's dilemma.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Or have we shifted quite a bit? We shifted quite a bit and different arenas of prediction problems have different mixes these days. But even in finance, I would say it's probably more like 50-50. Really? That much. That's amazing. I think, you know, and if you logical extreme is natural language modeling, which was done for decades and decades in the model-based approach where you kind of reasoned about linguistic characteristics of how people kind of do dialogue and those models had some parameters and you fit them with data. And then instead you have, as we said, a database of a trillion words and a tool with 175 billion parameters and you run that and there is no hope of completely understanding what is going on inside of GPT-3. But nobody”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Reason about how it must work, make theories. In our case, these would be sort of econometric theories, financial economics theories. And then those theories have knobs, not many, and you use data to set the knobs, but otherwise you believe the model, right? And he contrasts that with the machine learning school of thought, which is also has the idea of nature's box, the inputs go in, the thing you wish you knew comes out. But in machine learning, you don't try to open the box. You just try to approximate what the box is doing. And your measure of success is predicted accuracy and is only predictive accuracy. If you build a gadget and that gadget produces predictions that are really accurate, they turn out to look like the thing that nature produces, then that is success, right? And at the time he wrote the paper, his assessment was 98% of”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“If you like, yeah. And so he identified two schools of thought about solving prediction problems. And one school. Is sort of model based. The idea is there's some stuff you're going to get to observe stock characteristics, let's say. There's a thing you wish you knew, future price change, let's say, and there's a box in nature that turns those inputs into the output, right? And in the model-based school of thought, you try to open that box.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so this paper was written about 20 years ago. Leo Bryman was one of the great probabilists and statisticians of his generation, Berkeley professor, need I say. And Leo had been a practitioner in statistical consulting actually for quite some time in between a UCLA tenured job and returning to academia at Berkeley. And he learned a lot in that time about actually solving prediction problems and instead of hypothetically solving them in sort of the academic context. And so all of his insights about the difference really culminated in this paper from 2000 that he wrote.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, the short answer is we spend a huge amount of energy on recruiting and identifying the sort of premier people in the field of machine learning, both academic and practitioners. And we exhibit a lot of patience. We wait a really long time to be able to find the people who are kind of really the best. matters enormously to us both from the standpoint of the success of the firm and also because it's something”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“And so you take that data and you run it through each of these prediction rules that's frozen that you built. And now it is not the case at all that the most complex rules look the best. Instead, you'll see a kind of U-shaped behavior where the very simple rules are too simple. They've missed signal. They left signal on the table. The two complex rules are also doing badly because they've captured all the signal, but also lots of noise. And then somewhere in the middle is a sweet spot where you've struck the right trade-off between how much expressive power the prediction rule has and how good a job it is doing of avoiding the mistaking of noise for signal.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Certainly, I suspect it must be overfitting if it's doing that well, right? Okay, so now you freeze all those prediction rules. You're not allowed to change them in any way anymore. And now you unlock the drawer and you pull out all that data that you've never looked at. You can't overfit data that you never fit.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“On this half, you get to go hog wild. You build every kind of prediction rule, simple rules, enormously complicated rules, everything in between, right? And now you can check how accurate all of these prediction rules that you've built are on the data that they have been looking at. And the answer will always be the same. The most complex rules will look the best. Of course, they have the most expressive power, so naturally they do the best job of describing what you showed them. The big problem is that what you showed them is a mix of signal and noise, and there's no way you can tell to what extent a complex rule has found the signal versus the noise. All you know is that it's perfectly described the data you showed it.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, that is, you know, if you like the million dollar question in statistical prediction And you might find it surprising that relatively straightforward ideas go a long way here. And so let me just describe a little scenario of how you can deal with this, all right? We agree we have this big historical data set. One thing you could do is just start analyzing the heck out of that data set and find a complicated prediction rule. But you've already started doing it wrong. The first thing you do before you even look at the data is you randomly pick out half of the data and you lock it in a drawer. And that leads you with the other half of the data that you haven't locked away.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“And the other is that it's hard to reason about what's going on under the hood. When you have very simple prediction rules, you can sort of summarize everything that they're doing in a sentence, right? You can look inside them and get a complete understanding of how they behave. And that's not possible with high complexity prediction rules.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Price action, it's financials, analyst information, and then what did its price do in the subsequent 24 hours or the subsequent 15 minutes or whatever? Okay. And so when you talk about the amount of complexity that a prediction rule has, that means how well is it able to capture the relationship between the things that you can show it when you ask it for a prediction and what actually happens to the price? And naturally, you kind of want to use high complexity rules because they have a lot of approximating power. They do a good job of describing anything that's going on. But there are two disadvantages to high complexity. One is it needs a lot of data. Otherwise, it gets fooled into thinking that randomness is actually signaled.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“And it hands you a guess about how that stock's price is going to change over some future period of time. And so there is one most important question about prediction rules, which is how complex are they? How much complexity do they have? Complexity is a colloquial term. It's, you know, unfortunately another example of a place where things can be vague or ambiguous because a general purpose word has been borrowed in a technical setting. But when you use the word complexity in statistical prediction, there's a very specific meaning. It means how much expressive power does this prediction rule have, how good a job can it do of approximating what's going on in the data you show it. Remember, we have these giant historical data sets, and every entry in the data set looks like this. What was going on with the stock at a moment in a certain moment in time?”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“I think It's kind of the same question as asking what do we mean when we say we use machine learning or that our principles are machine learning principles. And so how does that make us different than the kind of standard approach in quantitative trading? And the answer to the question really comes back to this idea we mentioned a little while ago of how powerful the tools are that you're using to form predictions. In our business, the thing that we build is called a prediction rule. That's our widget. And what a prediction rule does is it takes in a bunch of input, a bunch of information about a stock at a moment in time.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh no, we apply it to equities, to credit, to corporate bonds, and we trade futures contracts. And in the fullness of time, we hope that we will be trading every security in the world.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“In that So, we don't hold portfolios that are exposed to those things. So it's really a business decision on our part. We are working with institutional investors who already have as much exposure as they want to things like the market or to well-recognized econometric risk factors like value. And so they don't need our help to be exposed to those things. They are very well equipped to handle that part of their investment process. What we're trying to provide is the most diversification possible. So we want to give them a new return stream, which has good and stable return.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so as you're saying, we begin with a very large historical data set of prices and volumes, market data of that kind, but importantly all kinds of other information about securities. So financial statement data, textual data, analyst data.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“There's lots of human diagnostic supervision, right? So there are people who are watching screens full of instrumentation and telemetry about what the systems are doing, but those people are not taking any actions unless there's a problem. And then they do.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Sure. So we run trading strategies, investment strategies that are fully automated. So we call them fully systematic. And that means that we have software systems that run every day during market hours and they take in information about the characteristics of the securities we're trading. Think of stocks. And then they make predictions of how the prices of each security is going to change over time. And then they decide on changes in our inventory, changes in held positions based on those predictions. And then those desired changes are sent into an execution system which automatically carries them out.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“And you lay all those genomes on top of each other, and then you look for places where all of the genomes agree, right? There hasn't been variation that's happening through mutations. And why hasn't there been? Well, the biggest force that throws out variation is natural selection. If you get a mutation in a part of your genome that really matters, then you're kind of unfit and you won't have progeny, and that'll get stamped out. Natural selection is this very strong force that's causing DNA not to change. And so when you make these primate alignments, you can really leverage that fact and look for conservation and use that as a big signal that something is functional.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“And so that's the problem I worked on. And a really important insight is that you can take advantage of the idea of natural selection and the idea of evolution to help you. And the way you do that is you have the human genome, you sequence a bunch of primate genomes nearby relatives of the human.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“But you don't want to just know the four letters. They're kind of a code. And some parts of the DNA represent useful stuff that is being turned by your cell into proteins and et cetera. And other parts of the DNA don't appear to have any function at all. And it's really important to know which is which as a biology researcher. And so it's for a long time before high throughput sequencing biologists would be in the lab and they would very laboriously look at very tiny segments of DNA and establish what their function was. But now we have the whole human genome sitting on disk and we would like to be able to just run an analysis on it and have the computer spit out everything that is functional and not functional, right?”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah. So we're back to genomes. Actually, this was around the time when I was in my first year of my PhD program, is when the human genome was published in Nature. So it was kind of really the beginning of the explosion of work on kind of high throughput large-scale genetics.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, you know, to talk about the ad placement, so the people who are supplying these auctions, they have a problem, which is how much to bid. And so, how would you decide how much to bid? Well, you want to know basically the probability that somebody is going to click on your ad, right? And then you would multiply that by how much money you make eventually if they click. And that's kind of an expectation of how much money you'll make. And so then you gear your bid price to make sure that it's going to be profitable for you. And then so really you have to make a decision about what this click-through rate is going to be. You have to predict the click-through probability. So I was going to say this sounds...”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“And a whole bunch of companies running software bid electronically to place their ads at the top of your search results. And the more or less the results that are shown on the page are in order of how much they bid. It's not quite true, but you could think of it as true.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Sure, the revenue engine that drives Google is search keyword ads, right? So every time you do a search at the top, you see add add. And so how do those ads get there? Well, actually it's... Surprising maybe if you don't know about it, but every single time you type in a search term on Google and hit return, a very fast auction takes place.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Lot faster. In fact, even the technology I worked on in 2005, 2004 is multiple generations old and not really what's used anymore”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Okay, so I can safely say your genome is unique in the world. There's no one else who has exactly your genome. On the other hand, if you were to lay your genome in mine alongside each other lined up, they would be 99.9% identical, about one position in a thousand is different. But those differences are what cause you to be you and me to be me. So they're obviously of intense kind of scientific and applied interest. And so it's very important to be able to take a sort of a sample of your DNA And quickly produce a profile of all the places that have variability, what your particular values are. And that problem is the genotyping problem.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so that was work I did as a summer research intern during my PhD. And that work was about what's called the problem is called genotype calling.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“I wouldn't call the system dumb and I wouldn't call it smart. Those are not characteristics of these systems. But it's complex.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“So Yeah, I think a critic would say that artificial intelligence is a complete misnomer. There's sort of nothing remotely intelligent in the colloquial sense about these systems. And then a common defense in AI research is that artificial intelligence is a moving target. As soon as you build a system that does something quasi-magical that was the old yardstick of intelligence, then the goalposts get moved by the people who are supplying the evaluations. And I guess I would sit somewhere in between. I think the language is unfortunate because it's so easily misconstrued.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“If you like. But the thing about Predicting the next word in a sentence is whether the sequence of words that's being output is leading to something that is true or false is irrelevant. The only thing that it is trained to do is make highly accurate predictions of next words.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, you know, underlyingly, there's this tool GPT-3. That's really the engine that powers ChatGPT. And that tool It has one goal, it's a simple goal. You show at the beginning of a sentence and it predicts the next word in the sentence. And that's all it is trained to do. I mean, it really is actually that simple.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“The answer it turns out is that it's a question of scale that wasn't at all obvious before GPT-3 and ChatGPT, but it just turned out that when you have, for example, GPT is built from a database of sentences in English, it's got a trillion. Words in it, that database. And when you take a trillion words and you use it to fit a model that has 175 billion parameters, there is apparently a kind of transition where things become, frankly astounding. I think I don't think that anybody who isn't astounded is telling the truth.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, that's a great question because the two are inextricably linked. The way that you make algorithms great is by making them more powerful, more expressive, able to describe lots of different kinds of patterns and relationships. But those kinds of approaches need huge amounts of data in order to correctly sort out what signal and what's noise. The more expressive a tool like that is, like a recommender system, the more prone it is to mistake one-time noise for persistent signal. And that is a recurring theme in statistical prediction. It is really the central problem in statistical prediction. So you have it in recommender systems, you have it in predicting price actions.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“I would say I worked on it. Okay. It existed. It was in place when I got there and sort of the things that are familiar about the recommendation engine had already been built by my manager and his colleagues. But I did research on improvements and different ways of forming recommendations. It was funny because at the time the entire database of purchase history for all of Amazon fit in one 20 gigabyte file. On a disc, so I could just load it on my computer and run now.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, it made me really interested and excited about using statistical thinking and data analysis to sort of understand the dynamics of securities prices. Machine learning did not play really a role at that time. I think not at DEShaw, but probably nowhere. It was too immature a field in the 90s. But I had already been curious and interested in using these kinds of statistical tools in trading and in investing when I was finishing college. And then at DE Shaw, I had brilliant colleagues and we were working on hard problems. So I really got a lot out of it.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I actually spent time in DESHA in between my undergrad and my PhD program. So it was after Harvard that I went to Shaw.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“It did frustrate me, yeah. And so I stayed home over winter. I stayed, excuse me, I didn't go home. I stayed at college over winter break to try to sort out what the heck I was going to do because I could see that it wasn't, my plan was in disarray. And I'd always been interested in computers, had played around with computers, never done anything very serious. But I thought I might as well give it a shot. And so in the spring semester, I took my first computer science course. When you write software,”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, it took about one semester for me to realize that none of the questions that were being asked in my classes had definitive and correct answers.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, it was a winding path, actually. I was very interested in international relations and foreign languages when I was finishing high school. In fact, I spent the last year of high school as an exchange student in Germany. And so when I got to college, I was expecting to major in government and go on to maybe work in the foreign service, something like that.”
2023-09-08 · Masters in Business · Jon McAuliffe on Innovation and Statistical Methods · IDENTIFIED FROM THE TRANSCRIPT · source