YouSaid · the spoken record

Boris Sofman

lines on the record: 198
first: 2021-11-16
most recent: 2021-11-16
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“Where you could argue a negative on any new technology, but you start to kind of see that if there is a big demand for something like this, in almost all cases, it's an enabling factor that's going to kind of propagate through society. And particularly as life expectancies get longer and so forth, like there's just a lot more need for a greater percentage of the population to kind of just be serviced with a higher level of efficiency because otherwise we're going to have a really hard time kind of scaling to what's ahead in the next 50 years.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Change the way we then can integrate with warehousing, with shipping ports. You can start to think about greater automation through the whole kind of stack and how that supply chain, the ripples become much more agile versus like very grindy the way they are today where just the adaptation is like very tough and there's a lot of constraints that we have. I think it'll be great for the environment. It'll be great for safety where like probably about 95% of accidents today statistically are due to just attention or things that are preventable with the strengths of automation. Yeah and it'll be one of those things where like industries will shift but the net creation is going to be massively positive and then we just have to be thoughtful about the negative implications that will happen in local places and adjust for those. But I'm an optimist in general for the technology.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“You can imagine city where people live versus work becoming more distributed because the pain of commuting becomes different, just easier. And there's a lot of options that open up. The way out of cities themselves and how you think about car storage and parking obviously just enables a completely different type of experience in urban environments. I think there was like a statistic that something like 30% of the traffic in cities during rush hour is caused by pursuit of parking or some really high stats. So those obviously kind of open up a lot of options. Flexibility on goods will enable new industries and businesses that never existed before because now the efficiency becomes more palatable. Good delivery timing consistency and flexibility is going to change the way we distribute the logistics network.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, in other countries too. You look back decades from now and it might be one of those things that just feels so natural and then it becomes almost like a kind of interesting kind of oddity that we had none of it like kind of decades earlier. And it'll take a long time to grow and scale. Very different challenges appear at every stage. But over time, like this is one of the most enabling technologies that we have in the world today. It'll feel like how was the world before the internet? How was the world before mobile phones? Like it's going to have that sort of a feeling to it on both sides.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Thoughtful, kind of movement and tiptoeing and like kind of like a push to society realizes how wonderful of an enabler this could become and it becomes more of a pull. And hard to know exactly how that'll play out. But at the end of the day, both the goods transportation and the people transportation side of it has that property where it's not easy. There's a lot of open questions and challenges to navigate and there's obviously the technical problems to solve as a kind of prerequisite. They have such an opportunity that is on a scale that very few industries in the last 20, 30 years have even had a chance to tackle that I maybe were pleasantly surprised by how much that tipping point like in a very short amount of time actually turns into a societal pull to kind of embrace the benefits of this.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“And it's like, it's an open question on how this plays out. I mean, maybe we're pleasantly surprised, and it just people just realize that this is such an enabler of life and efficiency and cost and everything that there's a pull. At some point, I should fully believe that this will go from a”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Education of society and regulators and everything else, where it's multidimensional and it's not a purely logical argument. But ironically, the logic can actually help with the emotions. And just like any technology, there's early adopters and then there's kind of like a curve that happens after it.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“That's right, and there is a point where, like, you can imagine a scenario where Waymo has a system that is even when it's kind of beyond human relative safety and provably statistically will save lives, there is a thoughtful navigation of that fact versus just kind of society readiness and perception.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Many accidents are not even under due to you, right? Obviously, so there's a big difference though. That's not a personal decision. You're also impacting obviously kind of the rest of the road. And we're facilitating it, right? And so there's a higher kind of ethical and moral bar, which obviously then translates into as a society and from a regulatory standpoint, kind of like what comes out of it where it's hard for us to ever see this even being debate in the sense that like you have to be beyond reproach from a safety standpoint because if you're wrong about this, you could set the entire field back a decade, right?”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“But at the same time, right? Like, we've gotten there where you think of surgery, right? Like you have surgery, there's always a risk, but like it's really, really bounded. You know that there's an accident, right, when you go out and drive your car today, right? And you know what the fatality rate in the US is per year. We're not banning driving because there was a car accident. But the bar for us is way higher and we hold ourselves very serious to it where you have to not only be better than a human, but you probably have to like at scale be far better than a human by a big margin and you have to be able to like really, really thought.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, like the team definitely noticed that once you go driverless we were driverless in Phoenix and you continue to iterate, your iteration pace slows down because your fear of regression forces so much more rigor that obviously you have to find a compromise on like, okay, well, how often do we release driverless builds? Because every time you release a driverless build, you have to go through this like validation process, which is very expensive and so forth. So it is interesting. It's like one of the hardest things. There's no other industry where you wouldn't release products way, way quicker when you start to kind of provide even portions of the value that you provide. Healthcare maybe is the other one.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“How much of the technology foundation of that space can go and have tremendous, just transformative impacts on other problem areas and other spaces that have subsets of these same problems? Like it's just incredible to think about that.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“All the subtle cues, like even little things your interaction with a pedestrian where you look at each other and just go, okay, go, right? Like that's hard to do without a human driver, right? And you're missing that dimension. How do you communicate that? So there's like really, really interesting kind of like elements here. Now, here's what's beautiful. Can you imagine that when autonomous driving is solved?”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“That's incredible, and they're actually helping kind of push it forward. And that is valuable, by the way, where even for us, a decent percentage of our data is human driving. We intentionally have humans drive higher percentages than you might expect because that creates some of the best signals to train the autonomy. And so that is on its own value.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“When you think about the safety bar and what it takes to actually go full driverless, not like incredible assistance driverless, but full driverless, that bar gets crazy high. And not only do you have to solve it on the behavioral side, but now you have to push computer vision beyond arguably where it's ever been pushed. And so you now on top of the broader AV challenge, you have a really hard perception challenge as well.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Even beyond the perception side, some of the hardest elements of the problem are on behavioral side and decision making and the long tail safety case. If you are adding risk and complexity on the input side from perception, you're now making a really, really hard problem, like which is on its own is still almost insurmountably hard, even harder. And so the question is just how much? And this is where you can easily get into a little bit of a kind of a trap where similar to how you how do you evaluate how good an AV company's product is, like you go and you do a trial kind of a test run with them, a demo run, which they've kind of optimized like crazy and so forth. And like, and it feels good. Do you put any weight in that, right? You know that that gap is kind of like, you know, pretty large still. Same thing on the perception case. Like the long tail of computer vision is really, really hard. And there's a lot of ways that that can come up. And even if it doesn't happen that often at all.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“It's a risk. It's a big. So there's no argument that it's not a risk, right? And it's already such a hard problem. And so much of that problem, by the way, is”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“And so then the question becomes can you use it in the right way and do you have the right software systems and hardware systems in order to solve the problem? You are right that in the long term there's no reason to believe that pure camera systems can't solve the problem that humans obviously are solving with vision systems.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“That's right. And what you mentioned, like if I were to make the opposite argument, like, what puts Tesla in the strongest position, it's data. That is their superpower where they have an access to real world data effectively with a safety driver and they found a way to get paid by safety drivers versus paid for safety drivers. It's brilliant. But, you know, all joking aside, one, it is incredible that they've built a business that's incredibly successful that can now be a foundation and bootstrap kind of like really aggressive investment in autonomy space. If you can do it, that's always an incredible kind of advantage. And then the data aspect of it, it is a giant amount of data if you can use it the right way to then solve the problem. But the ability to collect and filter through the things that matter at real world scale at like a large distribution, that is huge. Like it's a big advantage.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Meant to be safer and help a human, you could do that with far less sensors, far less complexity, and provide value very quickly, arguably with what we already have today, just packaged up in a good product. But you would take a huge risk in having a gap from even the compute and sensors, not to mention the software, to then jump from that system to an L4 system. So it's a huge risk, basically.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Another place and you grow it this way. And just to give you an example, we fundamentally changed our hardware and our software stack almost entirely from what when Driverless and Phoenix to what is the current generation of the system on both sides because the things that got us to driverless, even though it got to driverless at way beyond human relative safety, it is fundamentally not well set up to scale in an exponential fashion without getting into huge kind of scaling pains. And so those learnings, you just can't shortcut. And so that's an advantage. And so there's a lot of open challenges to kind of get through technical organizational, like how do you solve problems that are increasingly broad and complex like this, work on multiple products. But there's a few in that, okay, like balls in our court, there's a head start there. Now we got to go and solve it. And I think that focus on L4, it's a fundamentally different problem. If you think about it like, let's say we were designing an L2 truck that was”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“But fundamentally, there's a big, big jump every order of magnitude that you drive in numbers of miles in what you earn. And the gap from really kind of like decent progress for L2 and so forth to what it takes to actually go out for. And at the end of the day, there's a feeling that Waymo has, there's a long way to go. Nobody's won, but there's a lot of advantages. all of these buckets where It's the only company that has shipped a fully driverless service we can go and you can use it and it's at a decently sizable scale and those learnings can feed forward into how to solve the more general problem”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, for sure. So maybe I'll start with Waymo. And you're right, both incredible companies and just a gigantic respect to everything Tesla's accomplished and how they pushed the field forward as well. So on the Waymo side, there is a fundamental advantage in the fact that it is focused and geared towards L4 from the very beginning. We've customized the sensor suite for it, the hardware, the compute, the infrastructure, the tech stack, and all of the investment inside the company. That's deceptively important because there's like a giant spectrum of problems you have to solve in order to really do this from infrastructure to hardware to autonomy stack to the safety framework. And that's an advantage because there's a reason why it's the fifth generation hardware and why all of those learnings went into the Dimore program become such an advantage because you learn a lot as you drive and you optimize for the best information you have.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Even though you'll always be surprised by things you'll encounter, you feel good about your ability to generalize from what you've learned.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Hardest stuff, yeah. You know, it's like go. Like, you'll never outfully itemize all the world states that you'll expand. And so you have to come up with different approaches. And this is where you start hitting the struggles of ML, where ML is fantastic at optimizing the average case. It's a really unique craft to think about how you deal with the worst case, which is what we care about in the AV space when using an ML system on something that occurs super infrequently. So you don't care about the worst case really on ads because if you miss a few, it's not a big deal, but you do care about it on the driving side. And so typically you'll never fully enumerate the world. And so you have to take a step back and abstract away what are the signals that you care about and the properties of a driver that correlate to defensive driving and avoiding nasty situations that”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“animal foreign objects on a road that pop out on short notice, a mechanical failures, sensor braking, tire popped, weird behaviors by other vehicles like a heartbreak, something reckless that they've done, fouling of sensors like bugs or birds or something. But yeah, like you have these kind of like extreme conditions where like you have a nasty construction zone where everything shuts down and you have to like, you know, get pulled to the other side of the freeway with a temporary lane that, right? Those are sort of conditions where we do that to ourselves, right? We itemize everything that could possibly happen to give you a starting point to how to think about What you need to develop, and at the end of the day, there's no substitute for real miles”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, and the hard part is how do you know you didn't get worse in something that you just changed? And so in a lot of ways, like the Turing test starts to fail pretty quickly because you start to feel driverless quality pretty early in that curve. If you think about it, right, like in most kind of really good AV demos, maybe you'll sit there for 30 minutes, right? So you've driven 15 miles or something like that.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Huge amount of kind of structured approaches in order to validate it. And then by thoroughness, you can make a strong argument that you're ready to go. This is actually a harder problem in a lot of ways, though, because you can think of a space shuttle as getting to a fixed point and then you kind of like, or an airplane and you like freeze the software and then you like prove it and you're good to go. Here you have to get to a driverless quality bar but then continue to aggressively change the software even while you're driverless.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Kind of simulate those scenarios and show that you can handle them and metrics that are correlated with what you care about, but you can measure much more quickly and get to a right answer. And that's what makes it pretty hard. And in the end, you end up borrowing a lot of properties from aerospace and like space shuttles and so forth where you don't get the chance to launch it a million times just to say you're ready because it's too expensive to fail. And so you go through.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“The hardest part is behavioral, where you have just infinite situations that could in theory happen and you want to figure out the combinations of approaches that can work there. You can probably pass the Turing test pretty quickly, even if you're not completely ready for driverless, because the events that are really kind of like hard will not happen that often just to give you a perspective A human has a serious accident on a freeway, like a truck driver on a freeway has, there's a serious event happens once every 1.3 million miles and something that actually has really serious injury is 28 million miles. And so those are really rare. And so you could have a driver that looks like it's ready to go, but you have no signal on what happens there. And so that's where you start to get creative on combinations of sampling and statistical arguments, focused structured arguments where you can.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“It ends up being different portfolio of approaches. There's easy things like, are you compliant with all these fundamental rules of the road? Like you never drive above the speed limit. That's actually pretty easy. You can fundamentally prove that it's either impossible to violate that rule or that in these like you can itemize the scenarios where that comes up and you can do a test and show that you pass that test and therefore you can handle that scenario. And so those are like traditional structured testing kind of system engineering approaches where you can just fault rates is another example where when something fails, how do you deal with it? You're not going to drive and randomly wait for it to fail. You're going to force a failure and make sure that you can handle it in closed courses and simulation or on the road and run through all the permutations of failures, which you can oftentimes for some parts of system itemize like hardware.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“But yeah, how do you actually know you're ready? Basically, like, and how do you know it's good enough? And by the way, this is the reason why the safety framework for the car side, because one, it sets the bar so nobody cuts below it and does something bad for the field that causes an accident. Two, it's to start the conversation on framing what does this need to look like. Same thing will end up doing for the trucking side.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Of scenarios and severities to make sure that you're better than a human bar by a good amount. But that's not actually the most useful for development. For development, it's much more kind of analog metrics that... Are part of the art of finding what are the properties of driving that give you a way quicker signal that's more sensitive than a collision that can correlate to the quality you care about and push the feedback loop to all of your development. A lot of these are, for example, comparisons to human drivers, like manual drivers. How do you do relative to human driver in various dimensions of various circumstances?”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“So, in the end, you care about safety. Like, that's in the end, what keeps you. Can be deceptively long, even when that demo looks like it's driver's quality. And the difference is that the thing that keeps you from going driverless is not the stuff you encounter on a demo. It's the stuff that you encounter once in 100,000 miles or 500,000 miles. And so that is at the root of what is most challenging about going driverless because any issue you encounter, you can go and fix it. But how do you know you didn't create five other issues that you haven't encountered yet? Those learnings, those were painful learnings in Waymo's history that Waymo went through and led to us then finally being able to go driverless in Phoenix. And now we're at the heart of how we develop. Evaluation is simultaneously evaluating final kind of end safety of how ready are you to go driverless, which may be as direct as what is your collision human relative kind of collision rate for all these times?”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Can simulate it forward and you can even start to do really interesting things where you add virtual agents to create harder environments. You can fuzz the locations of physical agents. You can muck with the scene and stress test the scenario from a whole bunch of different dimensions. And effectively, you're trying to more efficiently sample this infinite dimensional space, but try to encounter the problems as fast as possible. Because what most people don't realize is the hardest problem in autonomous driving is actually the evaluation problem in many ways, not the actual autonomy problem. And so if you could in theory evaluate perfectly and instantaneously, you can solve that problem in a really fast feedback loop quite well. But the hardest part is being really smart about this suite of approaches on how can you get an accurate signal on how well you're doing as quickly as possible in a way that correlates to physical driving.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Is the least efficient form of testing and as expensive, it's time consuming. So grabbing a large scale batch of historical data and simulating it to get a signal of over these last or just random sample of 100,000 miles, how has this metric changed versus where we are today? You can do that far more efficiently in simulation than just driving with that new system on board, right? And then you go all the way to the validation phase where to actually see your human relative safety of like how well you performing on the car side or the trucky side relative to a human a lot of that safety case is actually driven by taking all of the physical operational driving which probably includes a lot of interventions where like where the operate the driver took over just in case and then you simulate those forward and see if would anything have happened and in most cases the answer is no but you”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Across the board, yeah. So you think of, for example, well, if we've driven over 20 million miles, that's over 20 billion miles in simulation. Now, how do you use simulation? It's multi-purpose. So you use it for basic developments. So you want to do make sure you have regression prevention and protection of everything you're doing, right? That's an easy one. When you encounter something interesting in the world, let's say there was an issue with how the vehicle behaved versus an ideal human. You can play that back in simulation and start augmenting your system and seeing how you would have reacted to that scenario with this improvement or this new area. You can create scenarios that become part of your regression set after that point, right? Then you start getting into like really, really kind of hill climbing where you say, hey, I need to improve this system. I have these metrics that are really correlated with final performance. How do I know how well I'm doing? The actual physical drive.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Really interesting kind of technical challenges that push some of the research that enables these new suites of approaches.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“It's all still relevant. And then just the fundamentals of how you detect the car, does it really change that much, whether you're detecting it from a car or a truck? The fundamentals of how a person will walk around your vehicle, it'll change a little bit, but the basics, like there's a lot of signal in there that as a starting point to a network can actually be very valuable. Now we do have some very unique challenges where there's a sparsity of events on a freeway. The frequency of events happening on a freeway, whether it's interesting objects in the road or incidents or even like from a human benchmark, like how often does a human have an accident on a freeway is far more sparse than on a surface street. And so that leads to really interesting data problems where you can't just drive infinitely to encounter all the different permutations of things you might encounter. And so there you get into interesting tools like structure testing and data collection, data augmentation, and so forth. And so there's”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“And a lot of that 85% comes from surface streets because we just had so much of it and it was really valuable. And so we're adding in more and more, particularly in the areas where we need more data. But you get a huge boost out of the gate.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, it's like you can think of despite the different domain and different numbers of sensors and position of sensors, there's a lot of signals that carry over across driving. And so it's almost like pre-training and getting a big boost out of the gate where you can reduce the amount of data you need by a lot. And it goes both ways, actually. And so we're increasingly thinking about our data strategy on how we leverage both of these. So you think about how other agents react to a truck. Yeah, it's a little bit different, but the fundamentals are actually like what will other vehicles in a road do. There's a lot of carryover that's possible. And in fact, just to give you an example, we're constantly kind of like adding more data from the trucking side. But as of right now, when we think of our one of our models, behavior prediction for other agents on the road, like vehicles, 85% of that data comes from cars.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“So we reused all the same infrastructure, so labeling workflows, ML workflows, everything. So that actually carries over quite well. We heavily reuse the data even where almost every model that we have on a truck, we started with the latest car model.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“It's massive and it's increasing over time. If you go back to the grand challenge days and the early days of kind of AV development, there was ML, but it was not in the mass scale data style of ML. It was like learning models, but in a more structured kind of way.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“As possible to the original stack and be much more fluid about the strengths and weaknesses where your camera is much more susceptible to like kind of fouling on the actual lens from rain or random stuff. Whereas you might be a little bit more resilient in other sensors. And so there's an element of logic that always happens late in the game, but that fusion early on actually, especially as you move towards ML and large-scale data-driven approaches, just maximizes your ability to pull out the best signal you can out of each modality before you start making constraining decisions that end up being hard to unwind late in the stack.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so people refer to this as early fusion or late fusion. So late fusion might be that you have the camera pipeline, the LIDAR pipeline, and then you fuse them and like when it gets to final semantics and classification and tracking, you kind of fuse them together and figure out which one's best. There's more and more evidence that early fusion is important. And that is because weight fusion does not allow you to pick up on the complementary strengths and weaknesses of the sensors. Weather's a great example where if you do early fusion, you have an incredibly hard problem for any single sensor in rain to solve that problem because you have reflections from the LIDAR. kind of noise from the camera, blah, blah, blah, right? But the combination of all of them can help you filter and help you get to the real signal that then gets you as close.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“That might be present in the union of all of these and leave it to the system as much as possible to start to really identify how to extract that. And then there's places we have to intervene and actually include more. But no single sensor is in a great position to like really solve this problem and end without a huge extra challenge.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“The difference between being on the shoulder and being in your lane, and so you have interesting challenges there that you have to solve, which have a bunch of approaches to come into it. Radar is interesting because it also has longer range than LIDAR and it gives you speed information. So it becomes very, very useful for dynamic information of traffic flow, vehicle motions, animals, pedestrians, like just things that might be useful signals. And it helps with weather conditions where radar actually penetrates weather conditions in a better way than other sensors. And so it's just kind of interesting where we've kind of started to converge towards not thinking about a problem as a LIDAR problem or a camera problem or a radar problem, but it's a fusion problem where these are all like large-scale ML problems where you put data into the system. And in many cases, you just look for the signals.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Now LIDAR is the first sensor to drop off in terms of range, and RS has a really good range, but at the end of the day, it drops off. And so particularly for trucks, on top of the general redundancy that you want for near-range and complements through cameras and radar for occlusions and for complementary information and so forth, when you get to long range, you have to be radar and camera primary because your YDAR data will fundamentally drop off after a period of time and you have to be able to see kind of objects further out. Now cameras have the incredible range where you get a high density high resolution camera. You can get data well past a kilometer and it's like really potentially a huge value. Now the signal drops off, the noise is higher detecting is harder, classifying as harder. And one that you may not think about localizing is harder because you can be off by like two meters in where something's located a kilometer away.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, daytime, nighttime. It's about 3D physical existence, in effect. Like you're seeing beams of light physically bounce off of something and come back. And so whatever the conditional conditions are, like the shape of a human sense of reading from a human or from a car or from an animal, like you have a reliability there, which ends up being valuable for kind of like the long tail of challenges.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah. When you have a camera, the position of the sun, the time of the day, various of the properties can have a big impact, whether there's glare, the field of view, things like that.”
2021-11-16 · Lex Fridman Podcast · #241 – Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics · IDENTIFIED FROM THE TRANSCRIPT · source