YouSaid · the spoken record
Ramesh Johari
- lines on the record
- 97
- first
- 2023-11-09
- most recent
- 2023-11-09
- sittings or episodes
- 1
- sources
- podcast
Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections
“The things I teach, the things I do research on, they're all connected to that theme. And so, yeah, that's where I'm pretty excited. I do work with companies regularly. And so if there's interesting opportunities that kind of fall in the sphere of stuff we've discussed on the podcast, always, always happy to listen.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“I think the easiest way of someone's interested more on the industrial side is probably LinkedIn, you send me a message or connect there. I'm also because I'm an academic, I have my own standpoint webpage and it's pretty easy to figure out how to find me there as well. And how can listeners help me? I mean, I kind of feel that most important thing that someone listening to this could do is take forward some of the messages that came out in terms of what it means to be data literate. And I think there's a lot you can do to educate yourself there. You know, maybe one final thought I'll share is that in the same way that AI generates a lot of ideas, AI also generates a lot of pros. And in data science, that can actually be deadly because you're getting more explanations that sometimes maybe are extraneous. I think what the world needs is data literacy on the part of people interacting with these tools and with each other. So that's the thing I care most about.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Those things combined to create a culture and an environment where you don't credential everybody, I think that means a lot. I think that's something that I haven't found elsewhere. And if people wanted to know something about what Stanford's like on the inside, I think that's one aspect of it that probably isn't discussed very much. And that's part of what makes it really fun to be here.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“One of the things that I found very surprising when I came here is just how that never happened at any level. Grad students tell me this all the time. Go talk to someone across campus and just launch right into a conversation about how your ex meets my Y and we have something we can do together. As a faculty member, it happens all the time that I just had a conversation a couple days ago with someone about effectively a marketplace of experiment designs for nanofabrication here, right? Which was like totally out of left field for things I do and yet seamless, right? Our conversation was about the substance rather than the credentialing. I really think part of the reason for that is that Stanford is sort of unique in that it doesn't have a weakness across the board. We have strong professional schools, law, business medicine, strong engineering schools, strong humanities and social sciences. And then that and the weather is what I usually tell people honestly, which matters a lot. People are willing to walk anywhere. I think”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I mean, we've had a rough ride, as everybody probably knows. There's Stanford's been in the news for a lot of not so great reasons, I think, over the last five years especially. So, you know, I don't know if this is the right kind of surprise, but I think one thing that I find really energizing at Stanford is people have never asked me for credentialing here. And what I mean by that is that I came from kind of a bunch of other good schools and, you know, obviously I've spent time in industry with a lot of great companies. A kind of cultural dynamic that can often develop is well, before I'm going to talk to you, I want to know something about why you're worst talking to. Give me your credentials, right? Like, oh, you know, where are you a grad student or like where are you a professor? Like, tell me about yourself first.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“With my students, but also I think with people I interact with in industry. I think slowing down is actually more of a virtue that it's given credit for.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Because I think what I've found has been happening is just we're so convinced that speed is the way you're going to find the right answer that I just don't think we slow down to develop meaningful mental models of the things we're doing. That's certainly true in the research projects I work on. It's consistently true. And when I talk to people in business and I ask them about my mental model, I just mean if you're running a marketplace. What is your model of what people care about? What makes people stay versus leave? What makes matches work versus not work? All those things shape a roadmap in your mind. And I think a lot of roadmap being, a lot of execution, paper writing and academia has all just become far more fast-paced at the expense of kind of deeper thinking about these kind of structural features of the thing you're building. And so, yeah.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“A lot of my work involves talking to students of all stripes, and I guess these students go on to be data scientists, go on to be founders, and a lot of them go in the tech industry. So maybe in that sense the advice is relevant. My main thing I tell people is slow down.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“I also really like cycling, and I'm not ashamed to admit that I think that e bikes are like the greatest thing for cycling. Admittedly, I'm like late 40s. So maybe I'm the right target demographic too. But yeah, I love my e-road bike. It's like great because it's not like one of those with a throttle. You have to work, but it kicks in just when you're on like your sixth hill and you don't want to go up the last hill anymore on the way home. So yeah, that's amazing. I think that just like transformative for people that like cycling but have busy lives. And I think another one that my son who's 10 roped me into actually is we were like in Santa Cruz browsing at a kitchenware shop of all places and he saw an outdoor pizza and like a tiny portable one and he insists he just did research for like two weeks and insisted we get one. So he gotten over the summer and after we got it he refused to eat pizza out anymore as a 10-year-old. So, you know, that's like a, maybe that's the best thing I could say about like the quality.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah. Like, normally there's a coding question, right? I should say, I would never ask a coding question post November 2022 after we got AI to help us code. I think it's a superpower.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Startup founder tend to be better at this than most people, obviously. But another reason I like it is because you'll find in that conversation that their vision expands a little bit of additional spheres that are touched or impacted by what they're thinking about doing. So on both sides, it's kind of a revealing question, I think. So I find it important for my line of work, but my hunch is that might be useful for some of your listeners too.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“I interview people probably that are a little bit different than most of your podcast listeners. That said, there's one question I like to ask a lot, and that's if you imagine often in our interviews in academia, whether it's grad students or faculty, well, ask people about their plans. And what I'd like to ask people is, okay, now imagine everything works out, all the challenges you're facing work out, all your plans work out, everything hits the top end of your vision for what this could be. Do you imagine is the impact of having done that? Like, who's being impacted by that? Why is that a big deal that happened? And I find that's a really valuable question to ask because, first of all, many people haven't thought about that. We're so short term focused. We don't even think, boy, if everything worked out, what would be the big thing?”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“I am a climber, and one movie that I really liked was The Elthanist. I know a lot of people have seen Free Solo, but for anyone that kind of likes that genre, I would recommend they watch the Alphinist. I think climbing is an interesting sport because it has a very much like a psychological aspect of it. And I think that movie is pretty good at this sort of meta level where you kind of reflect a little bit on what does it mean to make a movie about people who are obviously putting themselves into such risky situations. So I really enjoyed that. On TV, we've been watching Only Murgers in the building, but I'm like enough episodes behind right now that I probably won't say anything more because I'm trying to avoid any spoilers and I'm sure there's people out there trying to do the same. So great show though on Hulu.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Recognize it doesn't matter what you do, you're always going to have too much to do. There's no point in stressing out about having too much to do. And just that small shift of mindset then puts a lot more attention on the usual thing people worry about, which is where do I want to prioritize my time? So he has a great way of writing about it, some concrete rules of thumb to help manage that way of thinking. And yeah, I think it's a great book”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“He put such emphasis on driving evidence and understanding of your processes that generate data. And I find often data scientists don't even look at examples. So like at ODES, it meant are you looking at actual jobs and what's actually going on in your product before you're trying to do data science on it? So I think that's like a Friedman insight, Friedman mantra. And so writing is great. The last one I was going to mention has nothing to do with data science or anything. It's called 4,000 weeks by Oliver Berkman. I'm not like a huge like self-help type person, but I really like this book a lot. It's a little bit, I think it's a little bit stoic in its approach, like stoic philosophy, but it's the basic point is you're only on Earth somewhere in the neighborhood of 4,000 weeks. And my wife and I have this term we called infinite Q, which is like no matter what you think you get done on a given day, more stuff's going to just keep coming in. And he basically says that recognizing that is liberating. Because once you recognize,”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“It comes to books. I have one I love that I start with always, which is how to live with statistics. It's a tiny book for Daryl Huff from 1954, which is just for anyone that likes data at any level. It's like such a fun read. It's like, it's a great book. The second thing I recommend to people, and actually this is true even for people who are not expert, is David Friedman was a statistician at Berkeley who passed away in the 2000s, early 2000s. writing was fantastic in getting us to think hard about process. He was especially fond of what he called shoe leather statistics, where you really got your, you know, you rolled your sleeves up. You got on the ground, boots on the ground, really getting in there, really trying to understand your data. His writing is fantastic. His explanations are fantastic. He has a few different books at different levels. I think people would love reading. Most importantly, what I like about it is...”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“But what I've seen so far, and again, you know, who knows by 2024, I might have a totally different answer for you. I don't think so. But at the moment, what I see is that humans have actually become far more important to the productive data science loop, not far less.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“And I really think, actually, what that does is puts more pressure on the human, not less. I think it's becomes more important for humans to be in the loop in interacting with these tools to drive the funneling down process of identifying what matters at all levels. That ranges from you're carrying out a data scientific analysis. And now because you've got these tools, you can hypothesize 10 explanations, maybe 100 explanations. Which of those are you going to focus attention on? What are you going to tell other people to focus their attention on? Two, you're running experiments used to have 10 creatives you're testing for a marketing campaign. Now you got a thousand creatives you're testing for that marketing campaign. Maybe that completely changes the game of what it means to run an experiment. What are you actually looking for now? How do you evaluate that you found something that was good enough? And I think these questions are not getting enough attention. I think people are looking for the automated tool that really cuts the human out.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“I believe very strongly actually is what AI has done for us is it's massively expanded the frontier of things we could think about our problem, hypotheses we could have, maybe things we could test. It's just an astronomical explosion of explanations and ideas and principles.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Think one of the high level points I would make, and like you said, there's an entire podcast in this topic, is that I think people want to imagine LLMs and AI-driven data science automating out large parts of what it means to do data science in industry. And I think that's probably the wrong perspective. In some mundane sense, that's true. It's easier for me to code than it used to be before. Easier for me to develop visualizations that used to be. I can make dashboards faster. programmatically I think it's true in some basic sense but what I believe pretty strongly and I teach data science here and my students are asked to use LLMs and generative AI on a weekly basis on all their assignments so I've got like an up close and personal beat on this”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“There's a great concept in the ratings, in the literature on rating systems called the sound of silence, which is this idea that there's a lot of information in ratings that are not left. So Steve Tedelis, who's a professor at Berkeley, he had a really nice”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“I'll say general actually, I think rating systems are understudied, which to me is astonishing because the biggest change from those agoras and Trajan's market elements of those kinds of markets, to me, the biggest change is that we get to see what happened with our matches. So as a data scientist working on marketplaces, I feel like it's incredible that more of us don't spend our time thinking about what we're learning from the matches and what these rating systems are telling us and what the impact of that is on who wins and who loses in these markets. Kind of thinking about like the social implications of these things. So that's something I'm pretty passionate about.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“And some platforms do things like maybe they won't show you ratings until you've accumulated a few. But in the end, this kind of distributional fairness aspect of averaging is pretty significant. And one of the recent papers that we've written is trying to get platforms to think a little bit about that. There's ways to address that. Interestingly, through the same concept of a prior. And the prior basically says, hey, if someone comes into the marketplace and instead of averaging them, I average them together with a prior belief, then maybe what that prior belief does is it says, yeah, you got one negative rating, but maybe you got a little bit unlucky. And maybe my prior belief is something which actually pulls your rating up a little bit and allows me to still have you alongside others in the marketplace that give you a chance at getting work, you know, getting rides, et cetera. So I believe pretty strongly in this kind of like distributional fairness element of designing rating systems. I think it's been understudied.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“If you're new and you break into that market and your first review is negative, you might be completely screwed. In fact, there are some early work on eBay that showed that if your first rating is negative, that could actually immediately cause like an 8% hit on your immediate expected revenue, say nothing of long-term consequences. Subsequent work has found that that's a significant indicator of potential exit from the platform just because now it's very hard to find work.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Feels very natural, right? Like Lenny's got five ratings. Let me average them. And that actually has some pretty important distributional consequences for the marketplace, distributional in the sense of who wins, who loses. And that's because if you're averaging and you're really established on a platform, think of a restaurant on Yelp with 10,000 reviews, it's irrelevant what the next review is. Doesn't matter. Nothing's moving at that point.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“And Airbnb had something like this in place where they would actually ask you to compare or ask you questions about expectations. I find that that's really valuable because it's easier for people to say that was good but didn't exceed my expectations when that was good, but definitely not better than this amazing stay I had like two months ago than it is to say, well, you know, I'm going to ding this person and give them four stars. So that's one issue. And I think another thing I want to point out for any marketplace founder is that something you want to be really careful about is the concept of averaging and what are the implications of averaging. And that's because a default for many marketplaces is to just average the ratings that people get.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“And there's a lot of reasons for this. But one of them is just that there's a reciprocity issue, which is it's effectively, from your perspective, it's kind of costless if someone says to you, hey, please leave me a nice rating. And if you're seeing them or you're interacting with them, most people don't want to be mean. So that happens. But there's another aspect of it, which is norming. As the ratings in the marketplace go up, they get normed, right? So that now you're in a condition, you're like, oh, a four-star rating, I'm really screwing this person over. Whereas maybe when the marketplace started, you didn't think that. So definitely one thing that we worked on in our research was to think about renorming the meaning of some of these labels. And renorming could mean something like rather than the star ratings just being, you know, poorer to excellent. The top rating is actually exceeded expectations, right? You could go one step further and you could say, how did this compare to this experience you had in the past that you rated really highly?”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“eBay and Amazon first started thinking about how to do rating systems ages ago. And part of the reason we haven't nailed it is because there's a lot of dynamics in play that lead to what's called rating inflation, where if you look at ratings over time in a marketplace, one of my colleagues, John Horton, who was a professor at MIT and has worked very closely with Upwork, we work together when I was at ODESC. He was the staff economist there. He's written a couple of really nice papers with this empirical phenomenon that over time you see the median rating inflating, let's say, on marketplaces like ODASC, like Uber, like any of these, right?”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh man, that's a tough one. I think I'll answer the second part first. I don't feel like anyone's really nailed this. I think there's a lot of innovation that's happened, but I think fundamentally we're still playing with the same kind of tools that we had when.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Of all stripes. Not everybody comes from a data science or experimentation background. And this idea that learning is costly is not natural, actually. It's not natural as you sound as a matter of human nature. It's certainly not natural as a matter of running a business.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“If we use that language, we're implicitly saying that we wasted time when we ran an AB test on loser. If I reword you for shipping winners, then what I'm really telling you is all the time that you spend testing out failures was wasted time. And I think, of course, yeah, like you don't want to keep data scientists around who regularly are just generating failures. That's not my point. But my point is there's a disconnect there. On one hand, we can all look at the story of this marketing manager and chuckle at it, right? And yet every day we're instantiating language and processes that are reinforcing that same theme, which is essentially trying to say to you, if you're wasting samples on things that don't ultimately end up being a winner, then that is the act of doing so is a failure. So I really feel, you know, that idea that you have to pay to learn is, again, it's a cultural thing, but it's also an education issue for businesses that are populated by people.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“You have to put yourself in the frame of reference when you didn't have the answer. And at that moment, what you're essentially saying to yourself is that it's worth paying to learn the answer. I think it sounds obvious the way we're saying it now, or this anecdote of the marketing manager and the holdout sounds obvious, right? What's culturally not baked in, I think, is that idea. And the reason I say it's not culturally baked in, by the way, is because of”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Unless I've done that on my own, right? Now, why is that so powerful? I think what I find so interesting about experiments is that when you don't know something, it seems not even a question that you would allocate some of your samples to all options, right? Treatment and control. Like I have two different ways of doing something. I don't know which one's better. So of course I'll give some samples to each of them. After the fact, you're like, oh, treatment was better. What the heck were we thinking? Why'd we give all those samples to control, right? That doesn't make any sense now. There's this great Seinfeld clip where, you know, they get the bill at the, he mentions getting a bill at the end of a large, luxurious meal, and people stare at the bill. We're not hungry now. Why'd we order all this food? Right. So it's the same thing. You know treatments better now. Why'd you waste all those samples on control? And I think that is such a powerful observation that”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, exactly. What's called a holdout group in experimentation? And the important thing about this holdup was it wasn't authorized. That's not the way things are supposed to work. They've got their ad spend, allocate out your ad spend. Great. So at the end of the year, they look to the whole out there like, wow, that cost us like a couple million dollars or something in that range. And it's like not a trivial amount of money. Like, what's the deal? What were you thinking, basically? And of course, the answer was, well, I get that I cost you that much. But number one, now you know what my team's worth And number two, you would never have had that answer”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Let me start with an anecdote that I just absolutely love this anecdote. I use it every year in class. So, yeah, I was talking to a real estate platform and they. They had a marketing data science manager who's basically responsible as many marketing managers are for allocation of ad spend across different channels. And what they discovered had happened at the end of the year is in one hand, the team had done great. But the manager had held out some subset of arriving visitors, not shown them any of the innovations they were making.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“This experiment, what's it telling me about the future? And that falls broadly under the category of what's called Bayesian AB testing. So that's one of the things I think can help culturally, weirdly. It's like a super technical thing, but I think it can help culturally because what it's doing is it's now rewarding people for contributing information to that prior. And I think it then becomes possible to say, oh, like your experiment that failed actually moved our prior. And that's an important thing because by doing so, you're now altering kind of how we're going to think about this flow or this pricing plan in all future experiments, right? So there's like an information positive externality, positive network effect that's generated for the rest of your business. If I can somehow encode what you learned into the analysis of future experiments. So this is one thing there's a strong connection between the culture and incentives of A-B testing and the ability to actually incorporate past learning.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“A funny thing about experiments is that we throw past learning away effectively. And this is just an artifact of how we analyze experiments, the methods used, the statistical methods used typically, p-values, confidence intervals. These fall into branch of statistics known as frequentist statistics. And the idea behind frequentist statistics without being overly technical is just I let that data speak for itself. There's no beliefs brought to the table about where that data came from. But if you think about this in like a company in AB testing a company, it's like a weird thing, right? Because I might have run a thousand AB tests in the past on this exact same button or call to action or color. And now I'm going to completely ignore that and focus only on this. So there's ways to take the past into account to build what's called a prior belief before I run an experiment and now take the data from the experiment, connect it with the prior to come up with a conclusion of like, okay, in light of the past plus”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Citing the norms that learning is part of the discourse, and it's expected, actually, I think is important. But the other thing I would say that's maybe a little bit more about kind of programmatically like what could a team, a data science platform team do.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“And the strategic aspects of this has changed. So, the cultural aspect there is, I think it's partly incumbent on the leaders. To expect something more of their data scientists. And what I mean by expect more is that you expect them to do more than deliver narrowly defined statistically rigorous results to you in their reports. You're actually expecting them to talk also about what they're learning about the business and the process. So where that's headed is this concept of being hypothesis driven, which is like the technical phrase, what does that mean again in a more lay sense? What it means is tests aren't going to be defined only in terms of winners and losers that each test should also say something about what will we learn about a business flow, a funnel, preferences of the guests. Preferences of the hosts, right? What will we learn about their demand elasticity if we're changing prices around these kinds of things? So it's possible to articulate in an experiment doc, a launch document, what are the hypotheses that are being tested? So that's one thing I would say. It's like just culturally.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Interestingly, it's actually sort of an active area of research for me now. What I mean by active area of research is I care a lot about the incentives. That we create for data science through how we set up reward mechanisms. So there's a couple things I think that could be helpful that are maybe a little bit less about, like maybe I'm not going to directly answer the question you ask because I think that's a hard one, right? I think I recognize that measurement on impact is critical. Well, let me answer that actually for the most obvious way first. I think there's a cultural issue here that's really critical. You know, one of the things I often find is that my PhD students, our PhD students here, often go off and get great data scientist jobs. And in one sense, they're doing amazing stuff. They apply really technically sophisticated methods. But when I look at kind of the problems they're working on, they're often more at the margins of the business than they should be. And it's a cultural thing. It's basically because if you're measured narrowly on impact and that's all anyone sees around you, then it's very hard to engage with the creative aspect of business change.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Which is, you know, you have to recognize when you run marketplaces that many of the changes that are most consequential create winners and losers. And rolling with those changes is about recognizing whether the winners you've created are more important to your business view than the losers you've created in the process. And it's a hard reality because nobody likes to articulate the idea that a feature change is hurting some of the people in your marketplace. But because of this fundamental constraint baked into how marketplaces work, many of the things that we would choose to do and the reallocation they create can't necessarily create observed pie expanding wins in the short run. You're often making bets that that's where you're headed partly through the reallocation that you're doing right now. And so I think that's what's interesting about Superhost to me is that partly points to thinking about what's the objective you would have defined the metric you would have defined in the short run that captures this idea of a trade-off.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Sometimes you get lucky and you really expand the pie for everybody. But I think Serbas Salan, who was a CFO at Upwork that I got to know there and then went to Thumbtack later, he had this line when he came to visit our class that I love.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Oh, we got to do something about this. So, what we decided to do is build some custom bespoke features that were really going to direct them to more experienced folks on the other side of the market. Good. And then, yeah, lo and behold, you know, pretty soon those metrics start to look better. But then we're looking at we're like, wait a second, now, you know, the existing folks on the other side are having a worse experience. So you kind of whiplash around. You're like, oh, wait a second. We better do something about that. So we take them, we try to match them up with more experienced folks. And now suddenly month after that, you're like, you know, wait a second. And your metrics just keep moving around. And that's because the whack-a-mole game here is ultimately a lot of marketplace management is moving attention and inventory around.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“So, one of the reasons the thing you said happens is because marketplaces are a little bit like a game of whack-a-mole. And what I mean by that is like so narrowly in the context of Superhost, because you're redirecting attention to some hosts at the expense of others. It's not even obvious if bookings can really go up. Maybe you get lucky and maybe you get a bunch more bookings. One reason you probably wouldn't expect that in the first place is there's only a limited number of super hosts. How many more bookings are they going to be absorbing because of all this extra attention? And you're taking attention away from other people without doing any data analysis, my prior would have been that booking should probably go down, right? And like one example that I came across with one of the companies I worked with that I love is we were working together over a period of time. And in a month, we looked at some of the data and it suggested that our new supply side was having a pretty bad experience.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Can't measure, right? But maybe you've got a leadership team with different beliefs about what they think the retention value of super host is going to be. They might be all over the place. You can process your experiment results in the context of these competing beliefs. It's almost like a prediction market kind of a thing and start asking, well, okay, like if this is what we believe about our business, this is what the data is telling us of the experiment, let's put those two together and ask, is this enough for us to make the bet that we're still going to go with it, even though maybe that short-term test you ran was flat?”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“I think that's one side of it. The other thing is you can't always measure everything that's important that's needed to really develop like a full sense. So with Superhost, right? One of the things that's hard to measure is the long-term impact of superhost. Because, I mean, the short run, superhost causes a rebalancing of inventory. There's going to be winners and losers. Part of superhost is actually about retaining hosts that get the badge over a longer period of time, recognizing that hypothesis actually says something about maybe how long the experiment needs to be run or what kinds of data analyses need to be done. And in the end, if you can't do that, you can't run it long enough or you can't do that data analysis due to sparsity of data or lack of data to address the question, it matters what you bring to the table, right? What are your beliefs about that? So what I like to tell people to do there is I like to push people to be what's called quantified rather than data driven, which is, okay, fine.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Without necessarily going into the weeds on the data science of Superhost. I think there's a lot wrapped up in what you said. I guess another thing I'll say is that I'm a big believer that you don't throw your understanding of the business out the window when you process experiment results. And it's partly, I guess, what I mean by this is data science is really about accumulation of evidence. It's never about one finding in isolation. It's another kind of trap, I think, is to sometimes say, well, I hit statsig on my AB test, green light, it's all go. And, you know, I think, you know, you had Ronnie Cojave on your show and he made a similar point that there are different levels of evidence. And it's just having an outlier AB test that goes against everything you believe about your business doesn't mean that you somehow have controverted all your knowledge.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“A cultural thing. It's about saying that we're allowing that to be part of our social contract with our data scientists or actually our employee contract with our data scientists that not everything is just about how many launches you had and how many wins there were. It's okay to say that's how I want to use experimentation, but if you're going to use it that way, then I would say don't be a we experiment everything business. Because then I think you need some other way to deal with these big changes that teach the whole company a lot, but maybe can't fall into the incentives you've created for your data scientists.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Through the badges. And you really want to think not in terms of winning and losing, but learning. So learning is a win. And I feel that that's a cultural thing fundamentally. It's very hard to somehow attach dollars and cents at the top to data scientists running experiments that fail but learn. And ultimately, I think getting into that space where you experiment more, meaning you don't run all your experiments for quite as long and you accept the willingness to try experiments that are into the tails where you might fail bigger.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source
“Nevertheless, if I was being rigorous about what hypotheses that's testing about my business, I'm potentially learning a lot, right? So a great example of this kind of thing is that there's an important feature of marketplaces is badging, right? So sometimes it's really important to have badges on your kind of top-rated profiles or whatever when people are searching. And without going too far into the details, a common kind of finding with badges is that badges you think are going to be great actually turn out to be terrible. And one reason they're terrible is they focus too much attention on the badged folks and pull too much attention away from the unbadged folks, right? And if we judge that only in terms of winners and losers, you throw the baby out with the bathwater. You're like, oh, well, that badging idea was terrible. So ditch that with no badges. But that's not what it's telling you. It's teaching you something about how inventory is being reallocated, how attention is being redirected.”
2023-11-09 · Lenny's Podcast · Marketplace lessons from Uber, Airbnb, Bumble, and more | Ramesh Johari (Stanford professor, startup advisor) · IDENTIFIED FROM THE TRANSCRIPT · source