YouSaid · the spoken record

David Ferrucci

lines on the record: 179
first: 2019-10-11
most recent: 2019-10-11
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“Machine is not conscious not having the same richness of emotional reactions and understanding that it doesn't really share the understanding, but is essentially just moving its eyebrow or drooping its eyes or making them big or whatever it's doing, just getting the emotional response. Well, you still feel it. Interesting. I think you probably would for a while. And then when it becomes more important that there's a deeper shared understanding, it may run flat. But I don't know.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, there was some interest in doing that, and that's another thing I didn't want to do because I didn't want to distract from the actual scientific task. You're absolutely right. I mean, humans do anthropomorphize. And without necessarily a lot of work, I mean, you just put some eyes in a couple of eyebrow movements and you're getting humans to react emotionally. And I think you can do that. So I didn't mean to suggest that that connection. Cannot be mimicked. I think that connection can be mimicked and can get you to. Can produce that emotional response. I just wonder, though. You're told what's really going on, if you know that.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“You to share that emotion with them. And then that's when it becomes compelling. So they're communicating at a whole different level. They're just not communicating the artifact. They're communicating their emotional response to the artifact. And then you feel like, oh, wow, I can relate to that person. I can connect to that. I can connect to that person. So I think humor has that aspect as well”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Do I feel differently when I know it's a robot and when I know, when I imagine that the robot is not conscious the way I'm conscious, when I imagine the robot does not actually have the experiences that I experience, do I find it funny? Because it's not as related. I don't imagine that the person's relating it to it the way I relate to it. I think this also, you see this in the arts and in entertainment where sometimes you have savants who are remarkable at a thing, whether it's sculpture, it's music or whatever. But the people who get the most attention are the people who can evoke a similar emotional response, who can get you to emote, right, about the way they are. In other words, who can basically make the connection from the artifact, from the music or the painting of the sculpture to the emotion.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“About that more formally. I think again, I think you do a combination of both, and I'm always a big proponent of that. I think robust architectures and approaches are always a little bit combination of us reflecting and being creative about how things are structured and how to formalize them, and then taking advantage of large data and doing learning and figuring out how to combine these two approaches. I think there's another aspect to humor though, which goes to the idea that I feel like I can relate to the person telling the story And I think that's an interesting theme in the whole AI theme, which is”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I think there's a couple things going on there. So I sort of feel like, and I might be too optimistic this way, but I think that there are, we did a little bit about with this with puns in Jeopardy. We literally sat down and said, how do puns work? And it's like wordplay. And you could formalize these things. So I think there's a lot aspects of humor that you could formalize. You could also learn humor. You could just say, what do people laugh at? And if you have enough, again, if you have enough data to represent the phenomenon, you might be able to weigh the features and figure out what humans find funny and what they don't find funny. The machine might not be able to explain why the human unless we sit back and”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“We could bootstrap it. In other words, we can be creative. And we could say, what do we think? What do we think the structure of a good dialogue is that does this well? And we can start to create that. If we can create that more programmatically, at least to get this process started, and I can create a tool that now engages humans effectively, I could start both generating data. I could start the human learning process, and I can update my machine. But I could also start the automatic learning process as well. But I have to understand what features to even learn over. So I have to bootstrap the process a little bit first. And that's a creative design task that I could then use as input into a more automatic learning task.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I have to be responsive and I have to be opportunistic with regard to what is the human saying. So I'm goal oriented and saying I want to solve the problem. I want to acquire the knowledge necessary. But I also have to be opportunistic and responsive to what the human is saying. So I think that it's not clear that we could just train on the body of data to do this.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So, I think any problem like this, where you don't have enough data to represent the phenomenon you want to learn, in other words, if you have enough data, you could potentially learn the pattern. In an example like this, it's hard to do. This is sort of a human sort of thing to do. What recently came out of IBM was the debater projects interesting, right? Because now you do have these structured dialogues, these debate things where they did use machine learning techniques to generate these debates. Dialogues are a little bit tougher, in my opinion, than generating a structured argument where you have lots of other structured arguments like this. You could potentially annotate that data and you could say this is a good response, this bad response in a particular domain.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So, I think to do this, you kind of have to be creative in the following sense. If I were to do this as purely a machine learning approach, and someone said, learn how to have a good, fluent, structured knowledge acquisition conversation. Go out and say, okay, I have to collect a bunch of data of people doing that, people reasoning while having a good structured conversation that both acquires knowledge efficiently as well as produces answers and explanations as part of the process. And you struggle. I don't know”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Not distracting. So there has to be a model of, in other words, the machine has to have a model of how humans think through things and discuss them.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Produce a machine whose goal is to help you think and help you reason about your answers and explain why. So instead of like talking to your friend down the street about having a small talk conversation with your friend down the street, this is more about like you would be communicating to the computer on Star Trek where like what do you want to think about? Like what do you want to reason about? I'm going to tell you the information I have. I'm going to have to summarize it. I'm going to ask you questions. You're going to answer those questions. I'm going to go back and forth with you. I'm going to figure out what your mental model is. I'm going to now relate that to the information I have and present it to you in a way that you can understand that we could ask follow-up questions. So it's that type of dialogue that you want to construct. It's more structured. It's more goal-oriented, but it needs to be fluid. In other words, it can't, it has to be engaging and fluid. It has to be productive.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so dialogue is important in a number of different ways. I mean, it's a challenge. So, first of all, when I think about the machine that understands language and ultimately can reason in an objective way that can take the information that it perceives through language or other means and connect it back to these frameworks reason and explain itself. System ultimately needs to be able to talk to humans, right? Needs to be able to interact with humans. So, in some sense, needs to dialogue. That doesn't mean that sometimes people talk about dialogue and they think, How do humans talk to each other in a casual conversation? And you could mimic casual conversations. Not trying to mimic casual conversations, we're really trying to”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, this goes back to the interpretation of what we were talking about before Jeopardy, the system's not trying to interpret the question, and it's not interpreting the content, it's reasoning. And with regard to any particular framework, I mean, it is parsing it and parsing the content and using grammatical cues and stuff like that. So if you think of grammar as a human framework, in some sense it has that. But when you get into the richer semantic frameworks, what do people? How do they think? What motivates them? What are the events that are occurring and why are they occurring and what causes what else to happen, where are things in time and space? And like when you start thinking about how humans formulate and structure the knowledge that they acquire in their head and wasn't doing it any of that.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Compatible in the sense they could communicate with one another and they can reason with this shared understanding. So how they think about things and how they build answers, how they build explanations becomes a very important question to consider.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“They're doing shallow analysis. So they are. Very quickly analyzing the question and coming up with some key vectors or cues, if you will. And they're taking those cues and they're very quickly going through their library of stuff, not deeply reasoning about what's going on. And then sort of like lots of different, like what we call these scores would kind of score that in a very shallow way and then say, oh, boom, you know, that's what it is. And so it's interesting as we reflected on that. So we may be doing something that's not too far off from the way humans do it, but we certainly didn't approach it by saying, you know, how would a human do this? Now in elemental cognition, like the project I'm leading now, we ask those questions all the time because ultimately we're trying to do something that is to make the intelligence of the machine and the intelligence of the human very compatible.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Right. So I was a little bit surprised actually to discover over time, as this would come up from time to time and we'd reflect on it and talking to Ken Jennings a little bit and hearing Ken Jennings talk about how he answered questions, that it might have been closer to the way humans answer questions than I might have imagined previously.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I need to build in the context of this project an NLU and in building an AI that understands how it needs to ultimately communicate with humans, I very much care. So it wasn't that I didn't care. In general, in fact, as an AI scientist, I care a lot about that. But I'm also a practical engineer, and I committed to getting this thing done. And I wasn't going to get distracted. I had to kind of say, look, if I'm going to get this done, I'm going to chart this path. And this path says we're going to engineer a machine that's going to get this thing done. And we know what search an NLP can do. We have to build on that foundation. If I come in and take a different approach and start wondering about how the human mind might or might not do this, I'm not going to get there from here. The time fr”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so that actually came up very early on in the project, also. In fact, I had people who wanted to be on the project who were. Early on, who sort of approached me once I committed to do it Wanted to think about how humans do it, and they were, you know, from a cognition perspective, like human cognition and how that should play. And I would not take them on the project because another Assumption or another stake I put in the ground was I don't really care how humans do this.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“It was a success. It was a success for our goal. Our goal was to build the most advanced open domain question answering system. We went back to the old problems that we used to try to solve, and we did dramatically better on all of them, as well as we be jeopardy. So we wanted Jeopardy. So it was a success. I worry that the world would not understand it as a success because it came down to only one game. And I knew statistically speaking, this can be a huge technical success and we could still lose that one game. And that's a whole other theme of this, of the journey. But it was a success. It was not a success in natural language understanding, but that was not the goal.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Was critical because that way you can divide and conquer so you can say, okay, you work on your candidate generator or you work on this approach to answer scoring, you work on this approach to type scoring, you work on this approach to passage search or to passage selection and so forth. But when you just plug it in and we had enough training data to say now we can train and figure out how do we weigh all the scores relative to each other based on predicting the outcome which is right or wrong on jeopardy. And we had enough training data to do that. So this enabled people to work independently and to let the machine learning do the integration.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, I think it was a gradual process, but one of the things that I think gave people confidence that we can get there was that as we follow this procedure of Different ideas, build different components, plug them into the architecture, run the system, see how we do, do the error analysis, start off new research projects to improve things. And the very important idea that the individual component Not have to deeply understand everything that was going on with every other component. And this is where we leveraged machine learning in a very important way. So while individual components could be statistically driven machine learning components, some of them were heuristic, some of them were machine learning components. The system has a whole combined all the scores using machine learning.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“That's right. And they would do research on their component And they would say things like, Know, I'm going to improve this as a candidate generator, or I'm going to improve this as a question score, or as a passage scorer, I'm going to improve this, or as a parser, and I can improve it by 2% on its component metric, like a better parse or a better candidate or a better type estimation, whatever it is. And then I would say, I need to understand how the improvement on that component metric is going to affect the end-to-end performance. If you can't estimate that and can't do experiments that demonstrate that, it doesn't get in.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“End to end. So I was always driving end to end performance. It was a very interesting. Very interesting engineering approach and ultimately scientific and research approach, always driving end to end. Now, that's not to say we wouldn't make hypotheses that individual component performance was related in some way to end-to-end performance. Of course we would because people would have to Build individual components, but ultimately to get your component integrated into the system, you had to show impact on end to end performance, question answering performance.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“We actually looked at the raw scores as well, standardized scores, because humans are not involved in this. Humans are not involved.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Think of it as a, if you want to think of it, what you're doing, if you want to think about what a human would be doing, human would be looking at a possible answer. They'd be reading the Emily Dickson. They'd be reading the passage in which that occurred. They'd be looking at the question and they'd be making a decision of how likely it is that Emily Dickinson, given this evidence in this passage, is the right answer to that question.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“You're going to use all the data that built up. You're going to use the question analysis. You can use how the query was generated. You're going to use the passage itself. And you're going to use the candidate answer that was generated. And you're going to score that. So now we have a group of researchers coming up with scorers. There are hundreds of different scorers. So now you're getting a fan out of it again from however many candidate answers you have to all the different scorers. So if you have a 200 different scorers and you have a thousand candidates, now you have 200,000 scores. And so now you've got to figure out, you know, how do I now rank these answers based on the scores that came back? And I want to rank them based on the likelihood that there are correct answers to the question. So every scorer was its own research project.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So, right, speed and accuracy for the most part were separated. We handle that sort of in separate ways, like I focus purely on accuracy, end-to-end accuracy. Are we ultimately getting more questions and producing more accurate confidences? And then a whole nother team that was constantly analyzing the workflow to find the bottlenecks and then figuring out how to both parallelize and drive the algorithm speed. But anyway, so now think of it. Like, again, you have this big fan out now, right? Because you have multiple queries, now you have thousands of candidate answers. For each candidate answer, you're going to score it.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“And so you had candidate answer called Candidate Answers Generators, a whole bunch of those. So for every one of these components, the team was constantly doing research coming up, better ways to generate search queries from the questions, better ways to analyze the question, better ways to generate candidates.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“And they would come back with passages. So these were passage search algorithms. They would come back with passages. And so now let's say you had a thousand passages. Now for each passage, you'd parallelize again. So you went out and you parallelized the search. Each search would now come back with a whole bunch of passages. Maybe you had a total of 1,000 or 5,000 whatever passages. For each passage now, you'd go and figure out whether or not there was a candidate, we'd call a candidate answer in there. So you had a whole bunch of other algorithms that would find candidate answers.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Question analysis had to finish and had to finish fast. So we do the question analysis because then from the question analysis, we would now produce searches. So we had built using open source search engines, we modified them, but we had a number of different search engines we would use that had different characteristics. We went in there and engineered and modified those search engines ultimately to now take our question analysis, produce multiple queries based on different interpretations of the question, and fire out a whole bunch of searches in parallel”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Right, so therein lies the Watson architecture, right? So we would take the question, we would analyze the question, so which means that we would parse it and interpret a bunch of different ways. We'd try to figure out what is it asking about so we would come, we had multiple strategies to kind of determine what was it asking for that might be represented as a simple string, a character string, or something we would connect back to different semantic types that were from existing resources. So anyway, the bottom line is we would do a bunch of analysis in the question.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Shared memory, right? And all this data was pre-analyzed and put into a very fast indexing structure that was all in memory. And then we took that question. Would analyze the question so all the content was now pre-analyzed? So if I went and tried to find a piece of content, it would come back with all the metadata that we had pre-computed.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Used IBM, we used IBM hardware. We had something like, I forget exactly, but close to 3,000 cores completely connected. So you had a switch where, you know, every CPU was connected to every other shared.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“We then took all that content. We pre-analyzed the crap out of it, meaning we parsed it, broke it down into all its individual words, and then we did semantic and semantic parses on it, had computer algorithms that annotated it. We indexed that in a very rich and very fast index. So we have a relatively huge amount of, let's say the equivalent of for the sake of argument, two to five million bucks. We've now analyzed all that blowing up in size even more because now we have all this metadata. And we then we richly indexed all of that. And by the way, in a giant in-memory cache, so Watson did not go to disk.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“I get, but then we would take that stuff and we would go out and we would expand. In other words, we'd go find other content. Wasn't in the core resources and expanded the amount of content grew it by an order of magnitude, but still, again, from a web scale perspective, this is a very small amount of content”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Different types of semantic resources like WordNet and other types of semantic resources like that, as well as some web crawls. In other words, where we went out and took that content and then expanded it based on producing statistical seeds using those seeds for other searches, and then expanding that. So using these expansion techniques, we went out and had found enough content and we're like, okay, this is good. And even up until the end, we had a threat of resources always trying to figure out what content could we efficiently include.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So now forgetting about the huge volumes that are on the web, right? So now we have to figure out we did a lot of source research. In other words, what body of knowledge is going to be small enough but broad enough to answer jeopardy. And we ultimately did find the body of knowledge that did that. I mean, it included Wikipedia and a bunch of other stuff.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Circle, you have to be up over 70, and you have to do it really quick, and you have to do it really quickly. But now the problem is, well, even if I had somewhere in the top 10 documents, how do I figure out where in the top 10 documents that answer is, and how do I compute a confidence of all the possible candidates so it's not like I go in knowing the right answer and have to pick it? I don't know the right answer. I have a bunch of documents somewhere in there is the right answer. How do I, as a machine, go out and figure out which one's right? And then how do I score it? So, and now how do I deal with the fact that I can't actually go out to the web?”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“So, first of all, that's not even good enough to play Jeopardy. In other words, even if you could pull the, even if you could perfectly pull the answer out of the top 20 documents, top 10 documents, whatever it was, which we didn't know how to do. But even if you could do that, and you knew it was right, we've had enough confidence in it, right? You'd have to pull out the right answer, you'd have to have confidence it was the right answer. And then you'd have to do that fast enough to now go buzz in. you'd still only get 65 of them right, which doesn't even put you in the winner circle.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“We want a self contained device. If device is as big as a room, fine, it's as big as a room, but we want a self-contained advice. You're not going out the internet. You don't have a lifeline to anything. So it would have to kind of fit in a shoebox, if you will, or at least a size of a few refrigerators, whatever it might be. But also you couldn't just get out there. You couldn't go off network, right? To kind of go. So there was that limitation. But then we did, but the basic thing was go do web search. The problem was even when we went and did a web search, I don't remember exactly the numbers, but someone in the order of 65% of the time, the answer would be somewhere, you know, in the top 10 or 20 documents.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Look, web searches come a long way even since then. But at the time, first of all, I mean, there are a couple of other constraints around the problem, which is interesting. So you couldn't go out to the web. You couldn't.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“And I happen to be, and in this case, I happen to be right, but we didn't know. You kind of have to put a stake on it. I said, how are you going to run the project?”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Said, we're going to need Maxwell's equations for question answering. And I said, if we need some fundamental formula that breaks new ground in how we understand language, we're screwed Not going to get there from here. Like, I am not counting my assumption is I'm not counting on some brand new invention. What I'm counting on is the ability to take everything it has done before, to figure out an architecture on how to integrate it well and then see where it breaks and make the necessary advances we need to make until this thing works.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Could just like whatever, like, just figure out what works because I want to be able to go back to the academic and science community and say, here's what we tried, here's what worked, here's what didn't work. I don't want to go in and say, oh, I only have one technology. I have a hammer. I'm only going to use this. I'm going to do whatever it takes. I'm like, I'm going to think out of the box and do whatever it takes. And I also, there was another thing I believed. I believe that the fundamental NLP technologies and machine learning technologies would be adequate. And this was an issue of how do we enhance them? How do we integrate them? How do we advance them? So I had one researcher and came to me who had been working on question answering with me for a very long time.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, no, I was committed to saying, look, we're going to solving the open domain question answering problem. We're using Jeopardy as a driver for that.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“And I knew how hard the language understanding problem was. I said, we're not going to actually understand language to solve this problem. We are not going to interpret the question and the domain of knowledge that the question refers to in reason over that to answer these questions. Obviously, we're not going to be doing that. At the same time, simple search wasn't good enough to confidently answer with a single correct answer”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Meaning, you know, it just went on and on. And so even if you focused on trying to encode the types at the very top, like there's five that were the most, let's say five of the most frequent, you still cover a very small percentage of the data. So you couldn't take that approach of saying I'm just going to try to collect facts about these five or ten types or 20 types or 50 types or whatever. That was like one of the first things like what do you do about that? And so we came up with an approach toward that. And the approach looked promising. And we continue to improve our ability to handle that problem throughout the project. The other issue was that right from the outside, I said we're not going to, I committed to doing this in three to five years. So we did it in four. So I got lucky. But one of the things that that putting that stake in the ground.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, there were lots of hard aspects to it. I mean, one of the reasons why prior approach is that we had worked on in the past failed was because of because the questions were difficult to interpret, like what are you even asking for, right? Very often like if the question was very direct, like what city or what even then it could be tricky. But what city or what Person often when it would name it very clearly, you would know that. And if there was just a small set of them, in other words, we're going to ask about these five types. Like it's going to be an answer. And the answer will be a city in this state or a city in this country. The answer will be a person of this type, right? Like an actor or whatever it is. But it turns out that in jeopardy there were like tens of thousands of these things. And it was a very, very long tale.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source
“This is an act of research here. We've been in this for years. Why wouldn't we take this grand challenge and push it as hard as we can? At the very least, we'd be able to come out and say, here's why this problem is way hard. Here's what we tried and here's how we failed. So I was very driven as a scientist from that perspective. And then I also argued, based on what we did a feasibility study, of why I thought it was hard but possible. And I showed examples of where it succeeded, where it failed, why it failed, and sort of a high-level architectural approach for why we should do it. But for the most part, at that point, the execs really were just looking for someone crazy enough to say yes, because for several years at that point, everyone had said no. I'm not willing to risk my reputation and my career, you know, on this thing.”
2019-10-11 · Lex Fridman Podcast · David Ferrucci: IBM Watson, Jeopardy & Deep Conversations with AI · IDENTIFIED FROM THE TRANSCRIPT · source