YouSaid · the spoken record

Judd Antin

lines on the record: 84
first: 2024-01-04
most recent: 2024-01-04
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“This is bullshit. A great research team can do research in a day, a week, or a month. It just depends on what you want to get out of it. How much detail do you need? How many people do we need to talk to? What is the depth or breadth do we need to go to seven different countries to talk about our constituencies in Latin America? Well, that's not going to happen overnight. But we don't often need that. The other way to look at that is that is it slower to get it wrong and fix it than to take a hot second to do the work to get it right the first time? You know, so that's BS. Like research, good research doesn't slow us down. It speeds us up.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“I'm wary of overgeneralizing, but I can tell you that from my experience and from what I hear, the product research or product insights relationship is one of the most challenged. And I think it comes from The fact that fundamentally many researchers are just not included in the process that PMs are running, and then actually I kind of thought I did some asking around before this podcast. And so I thought there are some tropes that researchers have about PMs that are worth PMs knowing, right? Like just like four or five of them, like the things that researchers know PMs say, which drive us nuts because they're not true. So the first one is that research just slows us down. Research is too slow.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“This is the structural solution I'm talking about, Lenny is like, I never asked for research teams to have their own separate OKRs. I said two things. Number one, What's the teams? Okay. Like, shouldn't the PMs, the engineers, the designers, and the research? Everybody should have the same set of metrics for success, right? Because either we're doing it together or we're not. And then I said, my metric for success is when they won't have that meeting without you. That's my metric for success. If they cannot have that decision making meeting without the researcher there, that means you've developed influence, strong trusting relationships. You're an active participant in the process, not just sort of somebody who provides input into someone else's process. And that is when researchers can have huge impact.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“And intuition to bear and have an open, like direct end kind conversation in which we might disagree. You know who's great at that? Researchers.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“That's so appropriate, Lenny. So the secret is not that sexy, it's system two, right? So you engage that slow, methodical process in which you do analytic thinking as a means of checking your gut. Slow in the grand scheme of things, right? Slow meaning not as foot second decision, not like months of analysis. That's not what I mean. The other thing you can do, and there's really great research on this, is you bring in the wisdom of the crowd, right? The wisdom of the crowd is a phrase a lot of people are familiar with and it works in a specific situation. The wisdom of the crowd works when the people involved with the decision are bringing diverse sources of information and judgment to the table. Obviously, if everybody has the same sources of information, then it doesn't matter how many people are out there. So if you want to check your gut, get a bunch of different guts together. Get a bunch of different people in the room who can bring everything.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I taught a class at UC Berkeley this semester on leadership, and we talk about that a lot because great leaders develop intuition, right? It's the pattern matching part of experience, right? Where you develop heuristics, which allow you to make good judgments, even if you can't quite explain where that judgment came from, right? That's what the gut is. But it's also, like I said, where bias comes from, where cognitive, like all the cognitive biases, there are a list of 151 of them on Wikipedia. I won't name them, but like all those thorny things that lead us astray, you know, the behavioral economists and social psychologists study, those live in the gut. And so the advice is, you know, when you are looking to check your gut, you have to do that thing.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“Easy to sideline researchers. Like, we don't need them in that situation. We've got our guts. Isn't the gut where a grade PM, a great founder needs to have that gut? And they do, but they need to be open to the fact that your gut is limited and biased and narrow and wrong sometimes.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“And the other thing that is inherent in that quote is I at Airbnb wore mini hats over the years. I was head of research two different times. I was head of design for guest products. And my last job was I was head of the design studio. So UX Research, UX design, writing, localization, they all report it to me. So I've seen this from many disciplinary angles in the UX field. And researchers aren't the only ones who are guilty of this. Like I would say design has a ton of performance. And it comes from the fact that we have like figured out user-centered design, this process or design thinking, which IDEO popularized. Like that's what we're supposed to do, right? Right. Bezos told us that we as PMs had to be customer obsessed, right? So that's what we're supposed to do. It's a really common and damaging thing when we are not genuinely like we don't genuinely have that growth learning mindset. And it's like.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“There is an important place for intuition in product development, of course, like the best designers, researchers, product people develop strong intuition for the product. But like you got to understand, intuition is where all of those biases lie. It's where all your blind spots are. And what great insights people do, what great researchers do when you're next to them all the time, is they'll expose you. I don't have to be the deep throat, right? you have somebody whose professional job is keeping you honest is probably the wrong way to put it but you know somebody who's whose capabilities are about expanding your horizons making it so that your intuition is constantly improving you don't have to rely on it when your intuition and the evidence sort of collide in a way that either affirms or falsifies the product decision you made now something really good is happening so”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“Was we don't validate, we falsify, right? We are looking to be wrong. That is the mindset you should use when you're approaching insights and research. I want to be wrong. I want you to do research that shows we were off base in the following ways. Tell me exactly how and why in a way that allows me to fix it quickly. But many PMs, many designers are not in that place. They do not want to be wrong. They're looking to validate. And that's user-centered performance.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“That PM is not interested in being wrong at all. It's too late in the game for that. We got to ship it, right? What they want is to check the box. So any check the box style research is a wild example of user-centered performance. I would argue every researcher has probably had to do executive listening sessions, you know, because a lot of PMs, founders, product people, but designers too, they want to get close to the customer, right? And so like, can I do some focus groups? I want to be there. I want to ask them questions. This is 97% performance. It's well-intentioned, but it isn't focused on learning. It isn't going to drive better outcomes or more impact. And then there's all these implicit ways that people engage in that kind of user performance too. A lot of it comes down to cognitive biases, confirmation bias, ego. Like one of my big kind of mantras.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“So, user centered performance is a term I made up because it's fun to make up terms. And it refers to customer obsession or user-centered practice that is symbolic rather than focused on learning, right? So it's hugely common, I would argue. It's work we do to signal to each other how customer obsessed we are, not because we want to make a different decision. And like if you're like listeners are like, I don't do that, I'm kind of like think about it for a second, right? Because this is extremely common. It shows up in explicit ways, in implicit ways. So explicitly, I would say every time a PM comes to a researcher at the end of a product process and says, can you just run a quick user study? You know, just to validate our assumptions, that's user-centered performance. It's too late to matter.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“Lenny, the opposite is true too because you got to take the average PM who lives in that land all day every day. And what they do is not in the ven. You know, I think those are people who are also like performing customer centricity and performing user-centeredness a lot when they're really not interested. And so this is like not about researcher. This takes two sides. Fixing this broken system takes everyone, researchers, PMs, designers, everyone at the company, but also the way that organization is structured and integrating itself in a different way. Everybody's got to come to the table.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“If it's a public company, did you listen to the shareholder call? And they're probably like, no, it's full of a bunch of language I didn't quite get. I'm like, yeah. So there you go. That's the language you need to learn. Scour your, you know, Google Drive folder, your internal folder, and look for all of the documents that are about this quarter or this halves or next half strategy. What are the OKRs? Understand the metrics and the funnel, the conversion funnel, like know it back and forward because then what you're doing is you're proposing if you're in the act of conversation, you're saying, cool, I hear you asking that research question. I've identified this is exactly the spot in the funnel where I think we need to do work, right? There's an opportunity here or that competitor is eating our lunch with this group of users. Like I know that because I read the competitive report and I understand it deeply. So those are skills that many some researchers have and a lot are building these days, but historically.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“So much of user experience practice, not just research, but design too. Is focused on empathy, right? And very user centered. This is beautiful. I'm not saying that we should abandon that. I think what I'm saying is there's an overlapping then where you have the user and profit or the business. And what researchers need to do is be way more explicit about finding that overlap. So one thing I often, when researchers ask for advice, they're like, well, what should I do to be more business or private focused? I say something like, Did you read the last quarterly report”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“Meta and an Airbnb, that was my goal individually as researchers build up those tools. And then as a team, build deep expertise that would fill all the gaps”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“I've interviewed hundreds or thousands of researchers. And the way I usually approach that is you want a researcher who's got a Swiss army knife. Because if all you have is a hammer, then everything looks like a nail. And so if you give in the context of an interview, let's say, a researcher, a pretty juicy, open-ended research question, and you want to see how they handle it. And a good answer is usually multi-method, right? We're not going to handle it in any one way. We're going to say, well, here's a couple of ways we could deal with this. Here's that we could do this in a day or a week or a month. I mean, we usually don't have a month, but sometimes big research projects go on for that long. And here are the different sets of methods that we can use. So see where they go. It's actually pretty simple. Most researchers are deeper in one than the other. And sometimes you can make up for those five tools with the team, right? So you have experts who are kind of T-shaped, but maybe deeper in one or several of those ways. But when I built a team at”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“Responses from communities small and large, you can get a lot out of really well crafted surveys. But to do that, you have to have the fourth tool, which is applied statistics. The best research know a little bit of stats. Like you can't interact in a world of A-B testing without knowing basic statistics. And then in the old version of this, the fifth tool was SQL because I think good researchers need to be able to run their own queries. These days, so much of that is dashboarded that the fifth tool may now be prompt engineering, you know, which is a thing we could talk about. But I think it's some maybe that maybe that's the fifth tool is somewhere. Is it technical skills that fall in between querying your own data, understanding it very well in companies that are washed with data and then interacting with generative AI?”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“So, the best researchers, I think, are, first of all, multi method. The first iteration of user research was primarily a qualitative discipline, but a strong opinion that I have is that is largely one of those models that needs to evolve. It's not that qualitative user research is no longer important. It's that the best researchers have five tools. I think they have five tools. And those five tools are number one, what we would call formative or generative user experience research. So looking at innovation focus, really open-ended, maybe more ethnographic. Let's go out into the field and talk to host and guests on Airbnb. Let's see people using our product in the field, stuff like that. So that's formative. The second type is evaluative, right? So more like usability testing. The third tool is a basic rigorous survey design, right? It's the best scaled way to get.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“It doesn't matter, and then it doesn't, you know, it's less impactful research or executives kind of conclude that therefore researchers are not as impactful and then they get sidelined or laid off and the cycle continues. So I think the short circuit is the constant engagement. If you take a great researcher and you insert them consistently in a product process, I feel confident that researcher will drive product improvement, metrics impact, growth, all the things that you want to see as a PM and a product leader. It's just that's the exception of the norm these days.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“There's a vicious cycle that's been happening, right? From where I sit. And this is what I hear from many, many researchers and research leaders, which is a lot of companies hired a lot of researchers with great intentions, didn't quite know how to integrate them. And research is a UX research is a kind of a newer discipline, so maybe that's not surprising. We're still learning how to use it. Cool, let's evolve. But a lot of companies hired these people, but they hired them into kind of like a service discipline. Very reactive, not in the room, not integrated in the way I said. And so they had less input on the questions to ask, or they're included, but only at the end, right? And then they're unable to build those direct relationships, to be there in the room, to actually drive the questions and insert insights because a good researcher is like the repository of insights you need for growth, but they're not there. They don't participate in the decision. So they end up doing research. They have jobs to do. So they do research that is too reactive.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“Reactive in the sense that a researcher in the room listening and participating in the conversation could have a ton of impact on framing exactly the right question that will drive maximum business impact, maximum product improvement at that moment, and then go do it quick and get back and we're on to the next. But they weren't there. The relationship wasn't there. They're not engaged in the project.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“Think the reason why so many PMs ask for those middle range questions is because they haven't really gotten deep with their researcher in a way which can leverage it for maximum impact. So if the question is like, hey, Judd, you just pointed out like a bunch of problems. Like, can you be more solutions oriented? Well, the solution is simple but not easy to me. It's that we need to restructure the way we make products in a way which integrates research much more fully. Like it looks like consistent relationships in which researchers and the work and the insights they provide are a part of the process from beginning to end. That's how you worked. I remember you. Like I know who you worked with. You worked with great researchers. But honestly, most product processes are not that way. And so that's when you like research as a service function. It gets called in right at the end.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“Let's look at a really interesting, reasonable question, right? And we can go out and do research on that, but it's not that specific. It's not really targeted at a business problem yet. It could be, right? Maybe that's a result of the research, but it yields these kind of like middle range insights in which we learn things like, well, hosts want flexibility about their payment options. I'm making this up. And that's a good example where it's like, it's not that that's not an interesting set of questions. It's just not quite pointed enough in order to like, and it's not framed in the language of like the funnel or the business strategy or the okay Rs. It's not quite enough aligned enough to that. It's too globular in that middle level and it ends up not driving impact.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“So, middle range research is usually focused on a more specific set of research questions or a constituency. So if macro is like let's understand the overall competitive landscape, let's do a concept car type project where we really look ahead, let's get involved with strategic planning, which is a wonderful thing for researchers to do, you know, do TAM studies, other things like that. That stuff lives in the macro space. The middle range space is like, what's a good example? We want to know how Airbnb hosts feel about their payment options.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“We spent the last many years hiring, hiring, hiring researchers, but maybe I'm sure most of your listeners are familiar with the idea of a ZERP. Maybe it was a zero interest rate phenomenon where it was okay when the money was easy to hire researchers, even though we were not setting them up properly. We're going to set them up to fail. We set them up as a service function. We didn't know what research was for. We didn't know how to really drive impact with it. And that's where the reckoning comes from. It's like that era is over. Research, I think, is more crucial than ever. Researchers, great researchers are more impactful than ever, but it's in a new space. We're in a new space now.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, it's like the questions turn out to be really interesting, and there are many cases at many companies where it's super impactful. But the problem with those types of questions is they tend to be really like they trigger all the worst stuff that researchers experience, right? So they yield results which are interesting but sometimes hard to operationalize. They trigger the post hoc bias really, really, really well, where like a lot of people can say confidently like, that was kind of obvious. We knew that already. And they fulfill this kind of need for us to feel and be customer obsessed, user-centered without changing anything. And so doing too much of that research to me is a symptom of a broken system, right? And where like companies are really different for me to each other, I heard from so many after this article and they're like, well, at my company and my industry is like this or not like this. But in tech.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“And then you have this middle range, which is this blobular place where the research questions are sort of middle altitude. And a lot of the core kind of, let's say, user understanding questions fall here. A lot of what research is doing is research in that space. It's sort of let's take a group of people and ask some questions about how they think, feel, behave, how they're using a product or not using a product. And it's just this devastating mix of really interesting to many, including researchers and not impactful enough for the business. That's kind of the core thesis. Researchers do it because it's interesting, but honestly, and I think we should talk about Lenny is researchers also do it because it's the kind of work we most often get asked to do.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“Is that we're just doing too much of what I would consider the wrong type of research? And what I mean by the wrong type of research is I have this framework that it's in the article macro middle range and micro research. At least three ways to talk about it. And it's pretty simple, the intuition of what those are. So macro research is like big picture, strategic business focused, forward-looking innovation, you know, look at the market, look at competitors. you know, long-term research to understand where the product should go, stuff like that. And then you have micro research, which a lot of really technical usability falls into this. All the beautiful stuff that researchers do to enable like a really high quality, excellent pixel perfect thing to go out the door. Laser focused research to understand A-B test results, stuff like that.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“So, yeah, I mean, everybody who's paying attention has noticed that there have been a bunch of layoffs. And I think back in the summer, I was thinking, listen, this seems to be hitting UX and UX research particularly hard. Is there something going on? Is there a bigger picture? Is it maybe the reason I use the word reckoning is because to me that's like, hey, a moment to take stock and triggered by the fact that a lot of wonderful humans may have lost their jobs and many more are afraid of losing their jobs. And so if it's a sign, the fact that research has been hit so hard, it's a sign of what. And so the thesis of my article is really, it's a sign that maybe the system is a little more broken than we think. And that research is not driving the value or impact that it should or could. And that's for a bunch of reasons, I think. Some of it is stuff that research can do better and a lot of it is how research is integrated and positioned in companies. And at the root of all that, I think.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“Was the anti capitalist crew? Because one of my points that we'll talk about is that I think researchers need to be more profit focused. And there are a lot of people out there who think that's not, I think they think that's not cool or not research's job. And I'm kind of like, well, what are we doing then if we're not helping businesses succeed? But that was the most surprising critique for sure.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, I mean, I was definitely surprised. I wrote it because I wanted to start a conversation about something I was thinking about. I didn't really know who would read it. And in the end, it turned out a lot of people read it. I learned that using the word reckoning may have been a mistake because it inspires a lot of drama in a conversation that I wanted to be really productive and positive. Overall, I would say, though, that the response was very positive. It seemed to resonate with a lot of people who reached out to me. I spent a lot of time talking to teams, to designers, to researchers. But there were also a ton of critiques. I would say some of it was like people thought I was throwing research or researchers under the bus. Like it's researchers' fault. We're doing it wrong, which I don't believe at all. And that I wasn't taking responsibility as a research leader or a design leader myself and like a real most interesting one, I would say.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“I've never looked at that list, but I'll tell you, I have been so privileged to work with all these amazing humans. I can't take credit for it. They're just outstanding people. And I'm glad the diaspora is out there because these people, rock stars.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source
“User centered performance refers to customer obsession or user centered practice that is symbolic rather than focused on learning. It's hugely common, I would argue. It's work we do to signal to each other how customer obsessed we are, not because we want to make a different decision. If you're like listeners are like, I don't do that, I'm kind of like, think about it for a second. This is extremely common. Every time a PM comes to a researcher at the end of a product process and says, can you just run a quick user study? You know, just to validate our assumptions, that's user-centered performance. It's too late to matter. We got to ship it, right? What they want is to check the box. One of my big kind of mantras was we don't validate, we falsify. We are looking to be wrong. Many PMs, many designers are not in that place. They do not want to be wrong. They're looking to validate. And that's user-centered performance.”
2024-01-04 · Lenny's Podcast · The UX research reckoning is here | Judd Antin (Airbnb, Meta) · IDENTIFIED FROM THE TRANSCRIPT · source