YouSaid · the spoken record

Rajat Monga

lines on the record
75
first
2019-06-03
most recent
2019-06-03
sittings or episodes
1
sources
podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

  1. “Yeah, so Francois started the Keras project before he was at Google. And the first thing was Theano. I don't remember if that was Afro TensorFlow was created or way before. And then at some point TensorFlow started becoming popular. There were enough similarities that he decided to, okay, create this interface and put TensorFlow as a back end. I believe that might still have been before he joined Google. So we weren't really talking about that. He decided on his own and thought that was interesting and relevant to the community. In fact, I didn't find out about him being at Google until a few months after he was here. He was working on some research ideas and doing kerosene as nights and weekends project and stuff.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  2. “Yes, I think with TensorFlow data sets that we just released That's definitely come up where people want these data sets. Can we organize them and can we make that easier? So that's definitely one important thing. The other related thing I would say is I often tell people, you know what? Don't think of the most fanciest thing that the newest model that you see Make something very basic work, and then you can improve it. There's just lots of things you can do with it.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  3. “Yeah, yeah, I get all kinds of questions there from Okay, what can I need to make this work right Do we really need deep learning? I mean, all these things I already used this linear model. Why would this help? I don't have enough data, let's say, you know. Or I want to use machine learning, but I have no clue where to start. So it varies all the way to the experts who I expert very specific things. So it's interesting.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  4. “That's correct. That's right. And then I think the other pieces that they want, again with 2.0 that the developer summit, we put together is the whole TensorFlow extended piece, which is the entire pipeline. They care about stability across doing their entire thing. They want simplicity across the entire thing. I don't need to just train a model. I need to do that every day again, over and over again.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  5. “What enterprises warn is that is part of it, but that's not the big thing. Enterprises really have data that they want to make predictions on. This is often what they used to do with the people who are doing ML was just regression model, just linear regression, logistic regression, linear models, or maybe gradient boosted trees and so on. Some of them still benefit from deep learning, but they weren't that that's the bread and butter, like the structured data and so on. So depending on the audience you look at, it's a little bit different there.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  6. “So I would say for the hobbyist perspective, that's the most common case, right? In fact, the apps on phones and stuff that you'll see, the early ones, that's the most common case. I would say there are a couple of reasons for that. One is that everybody talks about that. It looks great on slides. That's a visual presentation, yeah. Exactly”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  7. “And so there is value in providing that with that kind of stability. And in making it really simple, because that allows a lot more people to access it. And then there's the research crowd which wants, okay, they want to do these crazy things exactly like you're saying, right? Not just deep learning in the straight up models that used to be there. They want RNNs and even RNNs are maybe old. They are transformers now. And now it needs to combine with RL and GANS and so on. So there's definitely that area that the boundary that's shifting and pushing the state of the art. But I think there's more and more of the past that's much more stable and even stuff that was two, three years old is very, very usable by lots of people. So that makes it a lot easier.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  8. “It's, I think, in the midst of it, it's often easy to forget what an enterprise wants and what some of the people on that side want. There are still people running models that are three years old, four years old. So inception is still used by tons of people. Even ResNet 50 is, what, couple of years old now or more, but there are tons of people who use that and they're fine. They don't need the last couple of bits of performance or quality. They want some stability and things that just work.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  9. “The next few months, as we looked at how do we stabilize things, as we look at not just researchers, now we want stability, people want to deploy things. That's how we started planning for 1.0. And there are certain needs for that perspective. And so again, documentation comes up, designs, more kinds of things to put that together. So that was exciting to get that to a stage where more and more enterprises wanted to buy in and really get behind that. I think post 1.0 and with the next few releases, their enterprise adoption also started to take off. I would say between the initial release and 100, it was, okay, researchers, of course, then a lot of hobbies and early interest people excited about this who started to get on board. And then over the 1.x thing, lots of enterprises.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  10. “It's interesting. I think we've gone through a few things there. When we started out, when we first came out, people loved the documentation we have because it was just a huge step up from everything else because all of those were academic projects, people doing, don't think about documentation. I think what that changed was instead of deep learning being a research thing, Some people who were just developers could now suddenly And that, I think, really changed how things started to scale up in some ways and pushed down it.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  11. “So I think we did see a need for this. Lot from the research perspective and like early days of deep learning in some ways. 41 million now. I don't think I imagine this number. Then it seemed like there's a potential future where lots more people would be doing this and how do we enable that? I would say this kind of growth. Probably started seeing somewhat after the open sourcing where it was like, okay, you know, deep learning is actually growing way faster for a lot of different reasons. And we are in just the right place to push on that and leverage that and deliver on lots of things that people want.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  12. “So, I think where it came from was our disbelief, had a graph thing as well, a much more, it wasn't a general graph, it was more like a straight line. More like what you might think of cafe, I guess, in that sense. But the graph was, and we always cared about the production stuff, like even with disbelief, you were deploying a whole bunch of stuff in production. So graph did come from that when we thought of, okay, should we do that in Python and we experimented with some ideas where it looked a lot simpler to use? Not having a graph meant, okay, how do you deploy now? So that was probably what tilted the balance for us and eventually we ended up with a graph.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  13. “A few of them were just in terms of research was moving so fast, we wanted the flexibility, we want the hardware was changing fast, we expected to change that so that those probably were two things. And yeah, I think the flexibility in terms of being able to express all kinds of crazy things was definitely a big one then.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  14. “Totally. So when we build this belief, and some of that was in parallel with some of these libraries coming up, I mean Theano itself is older We were building this belief focused on our internal thing because our systems are very different. By the time we got to this, we looked at a number of libraries that were out there, Thiano. There were folks in the group who had experience with Torch, with Lua. There were folks here who had seen actually Yang Ching was here as well. There's other libraries. I think we looked at a number of things, might even have looked at Chainair back then. I'm trying to remember if it was there. In fact, we did discuss ideas around, okay, should we have a graph or not? And so putting all these together was definitely there were key decisions that we wanted. We had seen limitations in our prior disbelief things.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  15. “That's correct. We already had a couple of products that were doing that by then. In those cases, we had basically customized handcrafted code or some internal libraries that we're using.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  16. “At that point, by that point, we had seen a few lots of different use cases at Google. So there were things like, okay, yes, we want to run at large scale in the data center. Yes, we need to support different kind of hardware. We had GPUs at that point. We had our first TPU at that point was about to come out roughly around that time. So the design sort of included those. We had started to push on mobile, so we were running models on mobile. At that point, people were customizing code. So we wanted to make sure TensorFlow could support that as well so that that sort of became part of that overall design.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  17. “Yes, yes. So looking back, we were building TensorFlow. I guess we open sourced it in 2015, November 2015. We started on it in summer of 2014, I guess. Somewhere like three to six late 2014 by then we had decided that okay there's a high likelihood we'll open source it so we started thinking about that and making sure we're heading down that path”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  18. “So, TensorFlow itself is open and you can use it anywhere, right? And we want to make sure that continues to be the case. Google Cloud, we do make sure that there's lots of integrations with everything else.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  19. “Turns out we have Google Cloud and we are now not really providing RTEC, but we are saying, okay, we have Big Table, which was the original thing. We are going to now provide HPEC APIs on top of that, which isn't as good, but that's what everybody's used to. So there's like, can we make something that is better and really just provide? Helps the community in lots of ways, but also helps push a good standard forward.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  20. “The other one was from a software perspective. Google had done lots of software that we used internally. And we published papers. Often there was an open source project that came out of that that somebody else picked up that paper and implemented and they were very successful. Back then it was like okay, there's Hadoop which has come off of tech that we built. We know the tech we've built is way better for a number of different reasons. We've invested a lot of effort in that.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  21. “I would say, I think, so the initial idea came from Jeff, who was a big proponent of this. I think it came off of two big things. One was research-wise. We were a research group. We were putting all our research out there. We were building on others' research and we wanted to push the state-of-the-art forward. And part of that was to share the research. That's how I think deep learning and machine learning has really grown so fast. The next step was okay now would software help with that and it seemed like they were existing a few libraries out there Tiano being one torch being another and a few others but they were all done by academia and so the level was significantly different”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  22. “So by 2014, we were looking at okay, this is a big thing, it's going to grow and not just internally, externally as well. Yes, maybe Google's ahead of where everybody is, but there's a lot to do, so a lot of this started to make sense and come together.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  23. “Yeah, I think that was interesting. If I think back to 2012 or 2011, and first was, can we scale it in the year or so we had started scaling it to hundreds and thousands of machines? In fact, we had some runs even going to 10,000 machines. And all of those shows great promise. In terms of machine learning at Google, the good thing was Google's been doing machine learning for a long time. Deep learning was new, but as we scale this up, we showed that yes, that was possible and it was going to impact lots of things. We started seeing real products wanting to use this. Again, speech was the first. There were image things that photos came out of in many other products as well. So that was exciting. As we went into with that a couple of years, externally also academia started to, you know, there was lots of push on, okay, deep learning's interesting, we should be doing more and so on.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  24. “I think the two early wins where one was peach that we collaborated very closely with the speech research team who is also getting interested in this and the other one was on images where the cat paper as we call it that was covered by a lot of folks.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source

  25. “It was interesting back then when I started, or when you were even just talking about it. The idea of deep learning was interesting and intriguing in some ways. It hadn't yet taken off, but it held some promise that it had shown some very promising and early results. I think the idea where Andrew and Jeff had started was what if we can take this, what people are doing in research and scale it to what Google has in terms of the compute power. And also put that kind of data together. What does it mean? And so far the results had been if you scale the compute, scale the data, it does better and would that work. And so that was the first error or two. Can we prove that out? And this belief when we started the first year, we got some early wins, which is always great.”

    2019-06-03 · Lex Fridman Podcast · Rajat Monga: TensorFlow · IDENTIFIED FROM THE TRANSCRIPT · source