YouSaid · the spoken record

Judd Antin

lines on the record
84
first
2024-01-04
most recent
2024-01-04
sittings or episodes
1
sources
podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

  1. “Yeah, thanks for asking those questions. So people can find me at judanton.com. That's the best way to find out what I'm up to. These days I'm a consultant. I help people with UX strategy, org design, and crisis management. Somehow I love dealing with other people's dumpster fires, and I found that I'm constitutionally good at it somehow. So Juddanson.com is the place to find that. I also write. I write a medium post that you can find at one bigthought.com. And you'll find a lot of the topics we talked about today, including the original post that started this at OneBigTought.com. If there's one thing I could ask your listeners to do is to get next to your researcher, you know, like I just think if you build those relationships and involve a researcher and insights person early and often, beautiful things will happen for you and for the business. So that's the thing everyone can do for me.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  2. “I know you were because we talked about it. And so the motto is comes from Stoicism, which is basically focus on the things you can control and ignore the rest. And a lot of people think of this, a lot of people think of this as the serenity prayer or the serenity saying. That was the 20th century invention, but Epictetus was writing about this BC. And I think about it all the time. So much of the stress and pain and worry that we have in life comes from things we can't control. So I try to let those things go.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  3. “Yeah. So, this is going to seem like pandering, Lenny, but I don't know if you remember a conversation that you and I had. It must have been eight years ago. I remember where we were sitting. And it was about Stoicism. Do you remember this? Anyway, we had this conversation.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  4. “This is a really weird one, but my whole family started indoor rock climbing recently. And there's a challenge you have when you top rope, which is that you're looking up all the time. So they make these glasses, which are called belay glasses, and they have an angled mirror embedded in the lens so that you can look straight ahead. And the view you see is up towards the person who you're belaying. And I just thought that product is like so perfect for that. That's a niche problem and there isn't a better way to solve it.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  5. “Of a topic that you had to explain lately that was the most complex, and then explain it to me like I'm five. And there are a lot of ways to vary that question, but the reason I like it is because I think, and I've asked this question to VP and C-suite candidates in like multiple disciplines. And sometimes it's related to a conversation. Like I might ask them to explain something complicated about quantum computing or music theory, or it could be a complex business decision. But I want to see if somebody can break a complex problem down in a really simple way and give me an intuition for it in a short amount of time. I think that is a differentiator between good and great for many people.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  6. “We recently watched The Last of Us, and it blew our mind. I watched it after I played the video game after Long Last. If you are a person who plays video games and you haven't played The Last of Us, play it. If you don't know the show is based on the video game, not the other way around.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  7. “I recently read a book by a business book by Barbara Kellerman called Bad Leadership. And what I love about it is that we spend a lot of time talking about good leaders and she really dives into like the worst leaders and what makes them bad leaders in a way that I think is really valuable for everybody. I'd also recommend, I read a lot of fiction. So two recommendations there. One, recent Pulitzer Prize winner Demon Copperhead by Barbara King Solver. It's like an outstanding read that also is like really sad and moving and illustrative, especially when you want to understand rural poverty. And then completely other side of the fiction spectrum, if you're interested in science fiction, which I am, read the murder bot diaries. It's like about a sarcastic robot, a sarcastic killer robot and who doesn't love them.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  8. “Necessarily recognize like some things with a product, some problems with a product, you need a pulse to recognize. And like most good PMs that I know have But a lot of them require context of use, priorities, constraints that you just don't have. And you can't imagine purely on the basis of your own usage. So what I think that means is that you should definitely dog food your own product. Doing product walkthroughs to identify lists of potential issues is a great thing to do. Prioritizing that list, figuring out which ones are more or less a problem and for whom is an area where you should be extremely wary of relying on your own sort of opinion, expertise, or intuition when you are dog fooding your own product.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  9. “Yeah, I want to add one thing, if I could, because this has come up on your podcast a few times recently, which is about the idea of people doing their own product walkthroughs. So should a PM just rely on their own dog fooding of the product and their own walkthrough to figure out how to fix it? And a couple of times recently this has come up, like this, and I think the consensus seems to be sort of yes. This is a good thing. And I have a contrarian opinion there too, which is that I think it is really important for everyone to dog food their own products. The problem is related to relying on your intuition about those products, which is the thing most PMs have trouble with is realizing you are nothing like the user. You are nothing like them in ways that will bias the way you think about what's good and bad in your product, in ways that you can't”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  10. “Woover Oh, how satisfied are you with your experience with Airbnb? Or it could be some version of that, which is like overall how satisfied are you with your experience with customer service when you have a problem? So there could be a more specific version of that question. But those questions have better properties. And, you know, a lot of people say, well, hey, everybody's using NPS, right? So at least it gives me a benchmark because I can compare my NPS to industry NPS. The problem with that is the research shows that NPS is idiosyncratic. So it goes up and downs in ways that we don't understand and there's a lot of inconsistency in how it's asked that creates variations in the data, which means it's not apples to apples. So you can't even compare your MPS meaningfully to somebody else's.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  11. “Not a person who goes around recommending operating systems. The question is fundamentally flawed. The argument is that's the question is a good indicator of loyalty, but there's a really simple solution. Customer satisfaction, a simple CSAT metric is better. It has better data properties. It is more precise. It is more correlated to business outcomes. I wanted to prove this. This is something that survey scientists know and marketers don't want you to know. And so we did the work with Mike Murakami, who led Survey Science at Airbnb, and he's still there. Great researcher. And we basically redid all that work to find out if all that stuff was true just for Airbnb. And it is. It's simple. Don't ask NPS. Ask customer satisfaction.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  12. “Especially on mobile, if you're taking this survey, what percentage of those options are below the fold? And so from a survey perspective, it's really bad. There's also this intuition, which is like, are you a person who will let recommend, how likely are you to recommend Windows 11 to your friends and family?”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  13. “I do have a strong opinion about MPS. I like to say NPS is the best example of the marketing industry marketing itself. And I think I'm going to, the problem is this threatens many people's livelihoods because there's an entire industry of consultants and software providers that want you to believe NPS is a useful and accurate metric. The problem is, so the consensus in the survey science community is that NPS makes all the mistakes. So it's a garbage in, garbage out problem. So the likelihood to recommend question is bad for a whole variety of reasons. So it's bad because it's a 0 to 11 scale. It's bad because it's usually unlabeled. So we label the polls, but that's not the gold standard for research. It's bad because it's 11 items, right? And there's a couple problems with that. Number one, we find that precision goes down after five items on average, maybe seven. Number two is.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  14. “You're a researcher, it's not hopeless. Actually, the discipline, the future is so bright and we can help it along by continuing to develop these different skills as companies build a model that's more inclusive.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  15. “Can set this up in a different way, which responds to the current environment in a way which will drive a huge amount of impact. Now, that takes companies making the right choices. It also takes researchers owning up and developing skills, right, pushing back, understanding what research can have the most value, developing the skills and the knowledge and language around the business, like becoming more influential, being excellent communicators. Like it's one of the things I would evaluate the most in hiring especially research leaders because I needed them to show and teach by example is like, can you, like it isn't just rigorous research, it's like if a tree fell in the forest and no one was there to hear it, you know, you need to communicate it effectively and you need to do it in a way that's appropriate to the audience because if I'm if I'm talking to you, Lenny, it's different than I'm talking to Brian Chesky, you know, at Airbnb. And so I got to be able to give that presentation effectively and get right to the heart of it and speak the right language. And so”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  16. “Yeah, it's tough to be operating in a broken system. And so I feel that response where you're like you feel kind of powerless. But I think that's not likely to lead us past this moment to the next evolution of research. So that's kind of where it's like. I don't blame any researcher at all for being in the spot there. And it's been a tough go, right? However, crying about our lot is like not going to get us anywhere. So I think the point of the article for me, and this is advice I give companies all the time when I do consulting with them, is like, hey,”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  17. “And the fact of the matter is, like, if you have an insights person with you who has that Swiss army knife of tools, like you're not in it alone. You don't have to guess. Like ultimately it will still come down to a tough decision that you and founders have to make, but you can have evidence that bears on that decision, which you wouldn't be able to get any other way.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  18. “I am reluctant to overgeneralize, but I would say, and I would say, I know many founders who are in startup mode are like, I know what I need to build. The problem is that I need people who can help me execute. And I think that's right. And so everything's a trade-off. But remember, imagine that you could have that Swiss army knife at your disposal. Maybe you've got like an MVP out the door and you're looking to make your first major iteration or like many startups you need to pivot.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  19. “It's just these 20 companies, and each is in their own space. So I'm not going to overgeneralize, but there is a researcher can drive incredible value no matter what stage a company is at because a good researcher makes you go faster, not slower, and they drive impact because they answer questions which are impossible to answer in any other way. That's true if you're a startup. It's true if you're a late stage company. Now, if it's your first 10 employees, you know, like one researcher is going to go a long way. As you grow, making sure that you're matching up researchers so that they have strong partners in the key parts of the business is the best way to figure out if you have enough.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  20. “Do you remember how many researchers are anywhere on that list? I'll give you a hint Between zero and two. It's one. There is one researcher on that list anywhere Anywhere. And that's messed up to me. Now, look”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  21. “More headcount was not arguing abstractly for why researchers important, but by asking partners who wish they had it. Do the arguing for me. And so there isn't a, you're right, there isn't a clean answer for, hey, this is the right ratio because it really depends on the nature of the product. Is it an early stage product? Is it a late stage product? Are we talking about a startup or a late stage company? But I would argue there's always room for a researcher. Lenny, I'll tell you, and I use this in a keynote talk I gave lately. You published recently a list of, I think it was about 20 B2B companies and their first 10 employees. Remember doing that”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  22. “So, this is a thing I've thought a lot about, especially in my role as the head of the design studio. That was like my fundamental question. It's like, you have all these writers, designers, researchers. Like, how do you structure them? How many and where and who works on what? And the organizing principle for me was always relationships. You know you have enough when the people who need to have a constant research partner have them. And I would much rather create pain in that situation than spread someone too thinly. So my advice was always like, don't try to create a researcher to cover this entire product space. Pear a researcher up with somebody who's going to involve them in a consistent, engaged process and let them go to work and see the impact they're going to have, but protect their time. And then other people are like, wait a second, that person's doing great work. I want some of that. And creating that pain for them, like because it's the pain of loss, like is the number one way to grow headcount. That's how I always approached getting.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  23. “Yeah, that's so interesting. Germany and France were always our like bellwethers for what the rest of the world would be thinking with private data privacy concerns.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  24. “Allowed them to burst their own bubble and realize, okay, we can't think so naively about hides as a signal anymore. And we came up with a better solution.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  25. “And the people in the back room were starting to chatter, and they're like, wait, what? What is happening right now? And like good research that this person was, like they let it continue and they're like, oh, can you tell me what you're thinking right now? Come to find out that she was like, well, I hate that story because I'd seen it already. Model she was going for was inbox zero, which was like sad because it was infinitely scrolling she would never get there. And the reason I like that story is because The people in the back room had their minds blown. It was not that we assumed that was common behavior. Like this person could have been unique, but it was enough because those people were there experiencing the research that N of 1”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  26. “But they did. They were amazing at it. And it was one of those rooms with the glass. And I insisted that the engine directors, the product people, and they were willing, whatever. So everybody's behind the glass and I'm there with them. And the excellent researchers in the room, and they come in and we're just doing a traditional think aloud study. And so they, hey, can you open up your Facebook app? We would just love to see you, you know, what your experience is like. And so they opened up Facebook and we're looking and they look at the first story and they hide it. And they go to the second story and they hide it. And this went on for a while. And she's definitely using Facebook. But every time she'd finished with a story, she'd hide it.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  27. “And so then, what we did was we said we call these superhiders. We called them superhiders. And so we said, let's find superhiders around the office, and we'll get a superhider and we'll do like a really traditional user interview. We just want to see. So literally the first person who walked in, I remember because this was a person who had those fingernails that are so long, you don't know how they can do cup touchscreens.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  28. “And so there was kind of a team of engineers that thought one thing that you could do on Facebook is hide a post, right? So they were like, this is easy. Let's look at the posts that are hidden the most and use that as the signal of what's a good post on Facebook. Seems kind of reasonable, right? And like something tripped me on this one. And so I was like, so I did two things. So the first thing I did is I looked at the distribution of hiding by user and found out that it's power law distributed like everything on the internet. There are a few people on Facebook who hide a ton and then most people don't hide at all.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  29. “Can I tell you a quick story about behind the mirror? So, this is back from when I was at Facebook. And it was the high times. There was like 2012, 13. And NewsFeed is like really taking off. Ads are going to news feed. And I was working, I was a leader of a team that was working, among other things, on how to address post-quality.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  30. “Think that is in many ways the answer making sure that they are creating a process for the product for their products that integrates user researchers and insights from beginning to end. Also being willing to partner with the research on the ruthless prioritization, I used to say that a full plate for a researcher was probably three things, two big projects and a small project, right? Like a side project. More than that, your researcher is probably not doing a very good job. And a project may take 48 hours. That's okay. But so they need your help to prioritize. They need you to participate. PMs will take the time to be with researchers, to go into the field, even to travel. Did you ever do that, Len”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  31. “So, if the researcher is not there from beginning to end, if there are not strong relationships between product people and design people at every level, engineering people at every level, and somebody who's their partner, their insights partner, we're going to fall back into this problem where we're just a service discipline. We're not extracting the maximum value. It comes too late. We don't know what questions to ask. We're ignorant about what research can do. And so creating that integrated lean process where a researcher is arm in arm from the beginning is the most important advice I'd give.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  32. “Scientists from the product specialist, from the customer service people, and the voice of the customer, whatever, all that stuff. But they're all coming over the side and we don't know what to make of it, right? It's like too much. And that as much as anything is an argument for companies to stop siloing research disciplines. So when I joined Airbnb, I set out to create an integrated insights function where it's like, let's do UX research. Let's talk about the market and competitors when we have to. Let's integrate smartly with data science functions. Let's integrate all the stuff we're getting from customer service feedback. We brought over what was then the NPS program and sort of said like, hey, like if we're getting customer feedback there, let's all just like use it all to fuel this one insights machine. So that's the first piece of advice I'd give companies. And the second one without being a broken record is to think differently about the broken cycle. So integrate researchers into a unified lean process.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  33. “One is it's a little bit of a side, but not really. Like, one thing I learned through the responses to the article was everybody came out of the woodworks from the variety of Insights disciplines that are out there. Because I come from a tradition of user experience research or user research, but there are many insights disciplines in many industries. And they all wanted to claim one type of research or another and say like, oh, well, we over here in consumer insights or market research have been doing that well for years. And there are many insights disciplines and generally I think creating silos is stupid. Actually, I'm curious what you think because here's the number one thing I heard when I joined Airbnb. And you were there is I did a quick listening tour where I talked to a bunch of product people. And they all kind of said the same thing. They were like, listen, we have all these different people throwing insights over the transom. And it's great. We want to hear from the data.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  34. “Develop diverse research skills, remembering the five or five and a half tool list that I mentioned earlier, really go deep on that business knowledge. So speaking the language of product and business and metrics and understanding exactly how to use your insights like a scalpel, building those strong relationships, which is not a thing that researchers can do by themselves. It requires two-way engagements and also in a way which allows researchers to do fewer things better. So most researchers that I know are working on teams where they're like, I'm the only researcher. And there's like, I have seven pms and 20 designers. And like I'm trying to do 10 projects and no one's going to do a good job that way. So researchers have to learn with their partners about how to say no and focus on the most important things. But that's like only half of it, right? That's the research side. I have like two thoughts about what companies should be doing. The first”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  35. “And I want to help, I believe so much in this discipline of research and insights. And I think there's, you know, when I said the user, the UX research discipline of the last 15 years is dying. I didn't mean that I think research is dying. From it. I think that there's a version of it which we're now moving past and into a new version. We're going through an evolution as many do. And so the question for me is like, how can researchers and the companies and the other people with whom they work create a new version, a different version in evolution, which is hugely impactful for the business? And so the advice I'd give to researchers about that is”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  36. “You for asking. I think I said to you earlier, and I feel some pressure as maybe the first conversation that you've had specifically about research on this podcast.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  37. “The multimillion dollar button I told you about. In a lot of the bigger picture questions, especially the macro ones and maybe also the really pointed middle range ones, the point isn't really, this is exactly what we should do, and this is exactly what we should build. It is let us develop a framework which is based on actual evidence. And then together as a team figure out how we want to experiment our way to a successful product.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  38. “Yeah, I mean, there's a double edged sword we talk about a lot in the research community, which is about making recommendations, recommendations for design, right? So the best research doesn't like leave it at that, right? It tells you, and it's like the what, the so what, and then the then what. But the problem with that is some researchers go too far in the other direction where they're like, we ran the study, it yielded these insights, and therefore this is what we should build. And everyone else on the team is like, whoa, whoa, whoa. Right. Glad to hear your thoughts on the matter, but like, there's a lot going on here. Maybe we should talk about it. And that makes perfect sense. That's like a failure of communication. And I think that it speaks to the thing that Patrick is saying is like good research can sometimes tell us exactly what the problem is and exactly how to fix it. An example of that is this.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  39. “And he talks about this other than one of those cognitive biases called the narrative fallacy, which is the idea that like people love to make convenient, simple stories about the past. Like if I asked you about your career, Lenny and how you got to be this amazing podcast host, you'd be like, well, let me tell you about this series of events. And like, we do that. You know, it's like part of how we make sense of our lives and the information around us. It would probably be a lie. You know, like in the sense that we all kind of twist the evidence we have to fit the narrative we want to be true because it's simple and lovely and makes us happy.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  40. “And it's about hindsight bias. He kind of like makes the argument that we rely too much on intuition heuristics and pattern matching in a way that sort of is inappropriate to our experience. And it's like it leads us astray. It's kind of like a form of self-gaslighting. And it happens because we end up sort of selectively remembering things and then constructing narratives around them in a way which makes us feel like we already knew that when we infected not.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  41. “But the reason that makes researchers so angry is because that's not research. That's not what researchers do. A researcher who's going to ask customers what they want is a bad researcher. You need a different researcher. Like that's not, I've never done that in my career. No one on my team has ever run a study that's like that. So that just makes researchers mad. And then the last one is. We knew this already. That was obvious. And I think a lot about this book, which I would recommend to your listeners. So the author is a sociologist at UPenn named Duncan Watts. And the title is, Everything is Obvious, if you already know the answer.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  42. “Is it vagable now? I know. What is reality? Geez, can we? Well, let's just okay. Maybe he said that. Like he certainly believed that. That's what the historians say.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  43. “So, one of them is like a simple one, which is like everyone loves to quote that, it turns out totally apocryphal Henry Ford quote about if I'd asked my users, it turns out. The best of our knowledge. He did not say that. Yeah, I know. Isn't that sad?”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  44. “No, I think they're both right. A-B testing is as close as we can get to making causal claims about products, right? Research is usually not oriented towards making causal claims, or it should not be. But those causal claims rarely tell you how and why things happen. And if you want to develop, if you want to not make that mistake again in the future, you need to know how and why. If you want to build a better product in a way that doesn't just answer this narrow question that an A-B test answered, you need to know how and why. And so you kind of need both, like beautiful partnerships between data scientists and research and insights people are, I think what we're going to see in that next evolution. And if you set that up, that sort of virtuous cycle up, if you set up the engagement where those people are involved from the beginning, you don't make those mistakes. You get the causal relationship, which is valuable for one reason, and the housing whys, which are valuable for other reasons.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  45. “Into something that's not garbage and avoid the bias that can come from you just reaching out to your cousin's family and then doing whatever they thought you should do to the product. So that's the second trope. The third one is AB test everything, right? And like A-B tests are great, but one of my most painful things to do is to sit in a room full of PMs and data scientists who have just seen the results of an experiment that flipped a stat sig. And then they're like, cool, I was significantly down over this course of time for these users. And then they just start speculating about why that is because the A-B test rarely tells you why it changed in the way it did. And then this endless flywheel of AB testing goes. And I'm like, hey, you don't have to guess. I know somebody who can get you an answer to the, or at least evidence that addresses the question of why did we see the test result we did in a very short amount of time. Or you could use your customers as guinea pigs and throw more experiments at them over and over and spend a long time on it and get an”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  46. “Okay, so research is too slow is the first one. The second one, I can do my own research. Why do I need researchers? And like, that's true. As product people, I hope. I hope you are engaging with customers and listening well, but no offense, garbage in, garbage out. The thing is anyone can talk to a user. That does not constitute research or insights work. One user can be powerful, but one user can be idiosyncratic. And a researcher knows how to get to the heart of that really quick. They know how to take that conversation and understand and situate it in a way which means like, sure, like democratize research. That's happening. There are tools out there that will let anybody get customer feedback, voice a customer type stuff. But a researcher is there to help you turn garbage into...”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  47. “Exactly. The micro research should be much more common. A lot of research. I think we need to get back there as an industry and be like, when you make a product easier to use, when you discover problems with functionality, business metrics we care about will go up. I've seen it happen, right? But that's not just work for interns and newgrads. That's for sure. And then the planning process, like absolutely, if we're integrated from beginning to end, we can help. And, you know, the thing about that middle range, I think you're right. Like that's the stuff that makes the sort of stereotype that researchers is slow. And a lot of times it's also because it's just not pointed enough. The researcher can also say in that moment, I have studied the business plan. I know exactly where, like, I've seen the metrics trend. I have an idea about exactly where that's going to go. We still need to do that middle range research. The question is valuable, but it's now very pointed and that the time is worth it.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  48. “The conversion, like we added like 1%, which is really, really hard to do, so that's a quick example of how that type of quick research can drive a huge amount of business value”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  49. “There's a famous story which I'll only tell in the abstract because I don't want to add anything, but it's like we call it the multimillion dollar button. And basically we did research which revealed that people weren't going down the purchase funnel because they were afraid the calls to actions on the button was making them afraid that it would initiate a purchase when really it was just taking the next step, right? We changed the text on the button with help from our amazing content design or UX writing team. We basically changed seven characters and made Airbnb millions of dollars, right? Because what we found out, it was really simple. It was just like, hey, this button feels scary. The CTA on the button feels scary. So that's a great example of how micro, and that happened in like 48 hours. We would discover that insight or overnight, basically. And we were like, maybe we should test some other CTAs. We did the convert.”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source

  50. “Yeah, research can go super fast. I think especially so the macro level research, I hope what it is is tied to things like annual planning processes. We did a thing at Airbnb several years that we called it was like Insights 2019, Insights 2020. They were concept card projects and we spent quite a long time synthesizing the entire year's worth of insights from every place we could get them and then developing with designers and engineers like a concept car for five years in the future. So that's a long process. But the micro level, there's so much business value to be derived there. So much business value. And it can go so fast, Lenny, right? It can go so fast. You can have, you know, you can have results in 48 hours on these things. Like we did a thing at Airbnb”

    2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source