YouSaid · the spoken record

Jackson Shuttleworth

lines on the record: 103
first: 2024-12-15
most recent: 2024-12-15
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“It is actually, I'm going to show you one of the most thoughtful gifts that anybody has ever given to me. This is our Duolingo Serenity or Streak Serenity Prayer. My co-lead Antonia”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“People realize, hey, this is a day, a daily mechanic. And so think about this screen holistically, but every single thing that you're doing on the screen, how can you use it to communicate? What is the point of this feature? How does it work?”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Clarity also doesn't have to come at the expense of delight. And this is something where you hit a milestone and Duo gets, you know, it becomes, we call them Phoenix duo and he becomes awesome and lights on fire. And I think there are things that you can do to still make the experience really exciting and delightful and celebratory. And I would not lose that, but just don't do it at the expense. And I think it's also about figuring out for what you, you know, you can get away with doing more of this for users who are deeper into their streak experience than users who are starting where it's like your goal for the one day street user is just to make sure do they understand how this feature works i mean even something again just like another random experiment at the bottom of the streak screen we have a calendar and over the years it just looks more and more calendar like and that is simply because we find that the more we make it look like a calendar days on top you know little circles the check like the more we make it look like calendar the more the”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Form should follow a function here was a learning for us. And you'll see that now in a lot of places where we're showing streak, where we're really leading with the number, not necessarily the flame.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Encounter insights like this. And so even our screen design, we used to have a flame. It was mostly this flame that would like light up every day. But again, it was like an indication of a metaphor for a mechanic. And when we redesigned it, we did this, Kurt, one of our animators, did this awesome odometer animation where it's like your number would tick every day. It kind of like looked good, but from a product perspective, what was cool is we actually focused the design on the screen to show your number going up. And then it would say, you know, like seven day streak, eight day streak. And I think that as you're thinking about designing around a streak, don't get too bought up or caught up into what is this, you know, like the beautiful story that you're trying to tell at the expense of it being a really comprehensible feature. And so as you're thinking about product design, making that product design a clear distillation of this is what we're actually tracking.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“The other thing that I'll call out with the streak is, you know, it's like we have the image of the streak is this flame. And, you know, we have the streak flame and it's very much corridor iconography. It's important to acknowledge that that's a metaphor for a retention mechanic. Like the idea of keeping a flame lit. And again, if you've, I think we've established the flame as, you know, for a lot of users as sort of their understanding of a streak, which is great. But there's a lot of people in different cultural contexts and different stages of life where the idea of like keeping a flame lit to show your commitment to something makes less sense. We did some UXR in India many years ago, and this was something that just like did not resonate at all there, which was a really interesting learning. And that's something it's like, again, depending upon what your user base is, like the more global UXR you can do to understand how users are actually understanding and experience your feature, the better because you just, again,”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“And to be a little bit of a learning sanctuary was. Now, it's funny nowadays, I think we actually have enough XP-based mechanics and fun things that we can do that like I think actually a lot of our beliefs about the unlesson experience have changed such that something like that could work. But at the time, didn't feel good to keep that around.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“And the focus here is on learning. And now you've added this other thing up there to be distracting for users. And I think the question we talked about roadmaps and strategy here, the question that he had for me, and I didn't have a good answer for at the time, was like, so what else are you going to do with this? Like what's your iteration idea is where's this going to go? Is this going to make the lesson experience more gamified? And what we realized is that honestly, it was kind of just an easy engagement when idea, but we had touched our most sacred space in the app to do that. And so that was a case where it's like, yeah, it was a nice win, but we had added that UI element. And, you know, at least at the time it was less clear what we would do with it. And we realized that like long-term, it was just going to get in the way. And we'd rather, for simplicity's sake, pause that, you know, shut it down and keep the lesson.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Retention doesn't only work on the streak, although you would think most of our work is the streak. We've touched a lot of different surfaces over the years. And there was one experiment that we launched that we talked about XP earlier. In the lesson, the only UI elements are a progress bar at the top. And then how many hearts you have, right? So we keep it really simple. And this very much speaks to the design philosophy of Duolingo, which is simpler UI is better. And we decided, hey, let's add XP in there. And so let's show your XP ticking up as you're going through a lesson. That's going to make the user feel good. It's going to give, you know, show you what you've earned. You're going to be less likely to quit. All of these good reasons to do it. And then you finish the lesson and then we'll show you you've collected all this XP. And at one, I mean, the hypothesis was a good one. But we realized, and I remember having this conversation with Louise, is like, cool, this is our most important screen in the app. It is our lesson is where users learn.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, and we do both. I mean, we'll also look at a lot of times I'll look at for retention day one versus day seven versus day 14. A lot of what I'm looking for is for us to have a better day 14 impact than better day one impact because it means that users are retaining better over time. This is particularly for users that would see a future multiple times. I just like absolute DAU numbers because As long as you're controlling for different biases, like a recency bias or a novelty bias,”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“I would highly recommend that if you are interested in this, give it a read to summarize basically what we found is that If we wanted to drive DAUs and Duolingo cares, our growth North Star is DAUs, the metric that is most effective, where a percentage change in that metric is most effective at driving DAUs is current user retention rate. And this is just users who are not new or resurrected getting them to come back tomorrow. And so most of the work that our teams do, our retention-based teams do, is focused on Kerr. And so the retention team that I lead focused on Kerr, it just so happens the streak is the best feature at driving Kerr. And so this was this experiment was the biggest Kerr wind that we had had. It was like a top three Kerr win anyway for us. Just this little copy, right? And that's what I say, like death copy a thousand different ways. Sometimes it's not the big, beautiful feature, you know, that's going to drive the huge gains. Sometimes it's just something simple as a few words.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, and it's funny. I mean, this was four years ago, but I think it was like in the order of magnitude of over 10,000 DAUs for us. And actually, maybe this small bit of context. So Dulingo really cares about the metric current user retention rate. And actually, our first ever Duolingo post with Lenny was the newsletter that our former head of product wrote, Jorge.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“I think it's still on the app, like start a day to extend your streak, but miss a day and it resets, something like that. That makes sense to me.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Win experiment win when I joined Duolingo was when you start a streak, we have little copy at the bottom of the screen that just, I don't know, it's kind of like flavor copy. We use it to celebrate you or give you context. And I ran some tests that just tried to and eight words explain what a streak was. That was it. And it was a massive win because it really required, like it really dumbed down here is exactly how the streak worked. And it really helped users just understand what they needed to do. And I think this is something that's like, you kind of constantly got to remind yourself, particularly if you work in tech and you're building cool tech features, but you're user base is not a bunch of tech workers to like think about, all right, who is my audience? And for us, it's not just like tech workers. It's not people in America, people all over the world of all ages, of all cultures. And so making sure that you're”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Again, sticking with this one to seven day streak, the idea of a streak, particularly probably to this audience, is like, obviously, like, oh, it's a streak. It, you know, goes, it just counts so many days. We've realized over time that a lot of users do not understand how a streak works. And it can be as big as small as, well, I don't understand how streak freezes work or I don't like my mom the other day was talking to you about it. She's like, oh, I like didn't use Duolingo. And I come back. My, you know, streak's still there. So like there's like certain elements of the feature that, you know, we can do better at explaining. But even like what a streak is, it's tracking how many days that I've used the app, yeah, the more that we can make the feature easily comprehensible to users, the more retentive it is. And we've run a number of experiments to do this. You talked about, you asked about early easy copy changes that we made. Actually, this is my first.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, for us because we might not be able to tell you, hey, you now know 0.01% more Spanish, but we can show you, hey, you've gotten your streak a little bit higher. And so I think this works particularly well when you're an app that is doing something that's going to be sensed or felt over a longer term to help contextualize that progress in a way that makes more sense or at least feels more tangible to a user.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Even if it's like a game, it's like, I don't know, you're like making somebody a moment away from the craziness of their lives. And so I do think, though, that it's contingent on companies who are going to figure out if a streak works for them to figure out how can you frame the streak in such a way that a user does feel good about it. And it's easier for a Duolingo, but I think there's creative ways to phrase this for users. The other thing, maybe just on that that I'll say is The Streak works really well for Duolingo because with language learning, it's really hard to see day-to-day progress in becoming more fluent. Like influency is not even the right word. It's like becoming better at Spanish or whatever. It is a years-long process for someone to get better at a language. I mean, Duolingo makes it easier, but you still got to put in thousands of hours if you're going to reach C1 or C2 fluency. And that is really hard to track on a day-to-day. And so the streak works really.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, that is definitely a benefit if I had an app that is actually why I think a lot of mobile games do streaks differently because, you know, to say that you played a mobile game, and as somebody who plays a lot of mobile games, that you've done it for 3,000 days in a row, like, I don't know, maybe that hits a little bit different than you've learned Spanish for 3,000 days in a row. I think the comparison set is much larger, though, than a lot of companies give them credit for themselves credit for. And I think that there's, I mean, there's a lot of ways that companies think about there are very few companies I imagine out there in the world saying, oh, we don't do some degree of good for our users.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Term goal in mind. I do think, though, and we do this every now and then with the streak, eventually you just hit a local maxima and you say, like you talked about lunch and neutral experiments. This is a great example where it's like, all right, cool. Now we need to throw a bunch of this stuff out based on all the learnings. Can we reset this real estate? Can we reset this UI, reset this feature in such a way that is just as good as what we have now, but is way more plain or simple that we can again start to layer on. Those are really hard experiments to”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, our product review structure where we've got our head of product design, one of the lead, you know, the product management leaders, and then Luis NPR. Because they see everything, they have a really high product barm. And so that helps, I think over time, we just as PMs have to look at, okay, where is our feature headed? And so, you know, we do this with the streak, at least on a quarterly basis, to look at, okay, well, what have we learned? you know, how is our streak developed and how do we imagine this going in the future? I don't think, you know, it's easy to do end in some sort of awful local maxima if you're not constantly looking at your roadmap and think you don't have a clear strategy. For me, it's like if you have a clear strategy for where your features going, hopefully all of those AB tests are not just done to get some cheap gain. They're done with a long.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“There's nothing better than being told by Luis. I don't think this is going to win, but sure, if you want to. And a lot of times, you know, to his credit, he's right. And a lot of times, you know, our intuition was right. But it's just so cheap to do it. I think when the lift is smaller, it's great to have hypothesis, but you don't need to beat it up too much.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Going back to that goal screen, we used to say continue our like standard CTAs continue. And we changed that to commit to my goal. And it was like a massive win. Even just, you know, and again, it was like, okay, users tapping on that. What are we asking them to do? Commit to the goal. What is that going to lead them to do commit, not, you know, churn? Just that little copy change, that one time right there led to huge wins. And copy changes are so cheap. Like it's just you translate us, you know, for us, you know, we have a lot of users all over the world, a lot of UI languages. But like just come up with a bunch of ideas, translate some strings. This is one where the feedback that you'll typically get from Louisa, so all of our changes at Duolingo go through product review that are reviewed by Luis. So Luis reviews every single change that we propose or every experiment that we run. To be with copy, he's just like, I don't know, test it.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Just try everything. And I think that how strongly you feel about the hypothesis directly ties to how hard that experiment is. Like with copy, for instance, we've actually set up really good infrastructure for copy testing. I'm of the opinion that companies should run as long as you have the user base to do it, like copy test constantly. Like the amount of copy test that we've had that is just like that have won and you, I don't know, you just try things and you figure out what wins is definitely legion. And if ship like massive wins from little copy changes.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Do look, and we're like, you know, a lot of the apps that we look at actually are games themselves. So it's like, all right, you're playing royal match or there's the new Pokemon trading card game that I'm spending way too much time on. You look at these games and see what they're doing. And it's really good fodder for what we can do. But a lot of times you're at least going with a strong hypothesis based on what you've seen work elsewhere.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, and I think, you know, I shared that how this idea came partially from a monetization win that they had. I think there's a lot of because Duolingo runs, I don't know what percentile we're in, but it's got to be a very high percentile of per capita experiments run for a company based on company size. We're just constantly learning so many things. And there's a really great cross sharing of monetization say, hey, this thing worked for us. Is that something you can use? So I'd say it's rare that we go into something where it's just like, let's just try something. Typically, we have some insight because of all these experiments that we've run that, hey, if we do this, I don't know, it's one here or it's worked here or it's driven this user engagement. If we sort of massage that and make it try in this, you know, scenario or like a different screen, you know, we come in at least with a strong hypothesis this will work. A lot of times we”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“I feel very much like an amateur armchair psychologist with everything that at least as far as people who want to learn languages on their phone go. I really understand those folks.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“That optionality was a huge insight. And so because of that, we built a goal setting feature where you could choose between different goals, giving users that optionality was likewise a huge win. I'll say one final learning on this. Again, you talk about like friction and good and bad friction. Once we built a goal picker screen where you could pick between different streak lengths, we were like, oh, well, let's recommend that users do a hard, you know, a harder goal. Thinking that, okay, well, a harder goal is going to be better retention. And we'll pre-select the harder goal for them. And this, based on all the learnings that I just shared with you, you probably imagine, lost pretty significantly. We realized that like, yes, we could speed users through the screen more by virtue of picking a goal for them. But like that act of selecting, I think it's 30 days, I think it's 14 days, was where we were getting so much of the engagement from this feature.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“But we captured just what happens if we add an opt out button. And adding an opt-out button, and you would think, like as a PM, oh, I'm like, you know, now users might not engage with my feature. That's a bad thing. But it was a huge win to let them do that. And the learning here was that this intentionality of saying, no, I want it. Previously, it was just a continue button. But now it's like, no, I want to hit 30 days. And having that be an intentional decision for them, yes or no. Even though, again, this had no impact. No impact past this screen. Everything that I'm talking about now was, you know, just that screen that day and then was all thrown out.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“We're all good, but they appealed to different users. And so we started to realize, all right, well, we probably need to be more thoughtful about who we give these different options to. And so then we followed that experiment up with, all right, let's start with 30 days, and then we're going to let you opt out. You know, we'll say, no, I don't think I can hit it. And our thinking was, and then if you say no, we'll hit you with an easier goal. Because we just wanted to get you to commit to a goal. And this was a fascinating one because it was a good win to give users that easier goal to try to catch them before they said no. But it was almost just as big a win to add that opt-out button. So we tested separately, and I'm a huge fan of.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, we found a different thing. And now let's just beat the heck out of it. So we followed that up with another experiment where we tested different lengths. So we'll test 14 days and 50 days. And we found that”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“In this case, trying to finish the course was a huge win. So, this is where we started, and we're like, ah, goal setting. All right, we should go much harder on this.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“From our modernization team. So one of the strings that they had, the pieces of copy that they had worked really, really well was, I think it was your like 5.6 times more likely to finish the course if you subscribe to plus now it's super our subscription. And it was a really good hook that if you really cared, you'd sign up. And so we ran a similar, we had a similar thought where it's like, oh, let's just tell you how much more likely you are to finish the course if you get a 30 day streak. And so we started with that and I think it was like you're seven times more likely to finish the course if you have a 30 day streak. And just that message when you started your streak, I was telling you that was awesome. Huge win. Indicating an outcome. And like Duolingo doesn't have, we don't, but we have a gem economy, but we don't actually have, you know, it's just, it's all you learning. But being able to actually talk about it in terms of the outcomes that a user would think.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Get users to that point where loss aversion kicks in, and then they don't want to lead the app. One of the fun ones that we did, and it's honestly as much about process as it is about the feature itself, was we have a feature called Streak Goal. And it is, again, so much of the stuff seems so obvious in retrospect, but it was really novel at the time. We had this idea of like, hey, maybe we'll just goal users to hitting a certain street length, as you can imagine, this is pretty powerful, you know, user psychology. And we started with the simplest version of this. And this is how Duolingo does a lot of our testing. Rather than design the big complex feature for V1, just do like the simplest encapsulation of what that feature can be. See if it has legs, and then just add to it iteratively over time. This is partially how we get to 600 plus experiments on streaks. They're not all big ones. But we started, and it was funny, we actually took a learning.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, maybe I can talk about a few different ways that we structure, you know, a few different themes that I think we lean on. So the first is focusing on the zero to seven day user experience. And I would say this is, if you look at the kind of whatever breakdown of experiments are, we run definitely more than average number of experiments on getting users to go from a zero to seven day streak. And a lot of this is because we've looked at the data for our retention curves. And what we found is that once you get to seven days, Lasha version kicks in and you retain. So going from a one to a two-day streak, huge jump in retention, two to three day streak, slightly less but still huge. And it's up until day seven. Once you hit day seven, it flattens out. It's not to say that like, you know, if you're a 30 day streak, you're way more attentive than day seven, but non-order of magnitude that it is from day one through seven. So we do a ton of work to”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“It really depends if we're adding something and it's neutral. We tend to shut it down because it's just more cognitive load. It's something that we're going to have to start building around, a new UI element that we have to figure out how it fits into a system. I'd say when we do ship a neutral experiment is something that we have real conviction around, okay, yeah, maybe this was neutral, but it's going to give us a new platform to then build on top of. That experience might be neutral, but now we can build these DAU positive experiments. My general take there is, though, in that case, build that in as part of your V1 so that you make sure that at least your hypothesis around this roadmap has play should probably be the case. But in general, shutting down a neutral experiment so you don't introduce more complexity to the app is the way we tend to go.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Exactly. No, we, I mean, and we test every, so we, I was looking at the numbers as well. We've run in the last four years over 600 experiments on the street. So every other day, effective 100. We're running an experiment. And they range from big like this, like changing how the mechanic works to like, let's swap a string with another string and see if that copy is better, you know, for users. So we're constantly testing on everything. I do think that I'd be more careful running that experiment now. Like at some point, your street gets big enough that, again, I got 9 million users on the street. I got to be really careful. Those are our best retaining users. You know, you got to be careful. But in the early days of the streak, I'd say like test everything. I mean, see what, don't get don't get super caught up and it has to be like this. Just test a bunch of stuff and see what speaks most to users. Because I think, you know, again, you will constantly be surprised by the insights that you get from whether you, we shut down about half of our experiments. So half of our experiments lose. We still learn a ton by virtue of running them. So super, super valuable.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“We had dumbed down the unit of, and nobody thinks about, I just want to come you do one question on Duolingo. Nobody thinks about that. So we had more clear unit of measure that we were basing our streak around. And the users that we were capturing with our streak, you come, you do one or two questions on Duolingo, then you leave, where like the least engaged user is imaginable. And so I think that's something also to think about as you're building your streak is like, what is the user that you're solving for? So not only what is the kind of habit that you're building for, but like what is the level of commitment? And that was an example where we overendexed on a type of user who we honestly just weren't going to keep. That was a very easy shutdown decision.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Duelingo has very much has a strong tested philosophy. We're willing to test a lot of different, honestly, we'd much rather test it than debate it for days and days. So we actually followed up this experiment with, and this is a little bit later with, hey, what if we make it even easier to extend your streak? And so we actually tested, hey, if you do one exercise, just one exercise in a lesson, we'll extend your streak. And a lot of like the insight was good. Like you look at the funnel. Hey, there's a lot of users who are starting but not finishing lessons or not extending your streak. The loss aversion doesn't kick in. They don't, you know, come back. So this kind of followed that train of thought. What we realized when we ran this experiment is DAU's moved not one bit. And what we were doing by, and this, if we go back to like unit of measure, you know.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Mean, we track a lot of this with the work that we do, we make sure that as we're making changes to the street, we're not hurting the learning experience and we don't have a ton of interaction with it. So we're constantly thinking about this. But thank you, Luis, for saying that.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“As an engagement PM, that was like the coolest thing he could have ever tried. It's well, again, very much as subjectively, I guess, as an engagement PM. I mean, that's how I've, you know, I'm sure my learning, the learning folks at Dueling Gold cringe when they hear it, like I see myself as a learning PM as much as an engagement PM because the easiest way not to learn on Duolingo is not to come back the next day. And so if we don't make the app retentive, you will have no opportunity to engage with our learning features. I do think that there is a long tail of learning where if you start to dumb down, and honestly, this is something we wrestle with the streak as well, you start to dumb down the experience and your users aren't actually learning, they're not going to care. I mean, streaks work best when they're sitting on top of an app, you know, that users care about. But yeah, I mean, if you can't come, you know, if you don't come learn Unduolingo, you know, if you don't come back to Duolingo, then you're not learning. So, no, we.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, you do stuff on Duolingo, and then based on what you did and how well you did it, we give you experience points. This actually drives a number of our features in our app. So leaderboards is the big one. We have a leaderboard system where you try over the course of the week, you battle with 29 other people, and you want to win. That's all driven by XP. So we do have other features in the app that really benefit from this XP system. The streak is just no longer one of them.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Have a million goals or objectives for what you want your feature to be and potentially build a more complex feature. One lesson the streak, it's just easy for more users to understand.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Is just making it easier to extend your streak. But I think really importantly is still meaningful, right? The unit of use and as you're thinking about building a streak, I think it's really important to think about what the unit of use of your app is. The unit of use for DuLingo is like doing a lesson. And so if what we care about is users coming back every day and doing a lesson because it shows that they're actually engaging with the app, then it doesn't hurt us to make our streak focus on just do one unit versus multiple. And so that was like probably the big sea change experiment that we ran at the time was moving from an XP-based streak to a one lesson streak. It's also simple. And I think that's like one of the things to think about with streaks. It's always easy as a PM to like.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Forward user base initially. And so this whole idea of an experience point space streak system made sense. But what that also meant is that you could have a user come and use the app, do multiple lessons a day. Maybe they just set too hard of a goal for themselves and then lose their streak, which you don't need to be an expert in streaks to understand. That's probably not good. The nice thing with how we initially set it up, though, is it really did connect with what your goal was. So if you were serious, let's track how good you are at being a serious language learner. But I'd say one of the most impactful experiments we ran was about, this is actually as I was joining, or just after I joined, we had run this experiment, was to move it from a XP-based streak to just do one lesson a day and you'd extend your streak. And, you know, the risk that you can sort of imagine is, well, then users kind of care less about it because it's not connected with their goal. And we saw none of that. I mean, this was huge driver of DAU.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah, so the oldest streaks are as old as Duolingo itself. When we launched Duolingo, we launched with the street. And I say we, I was like just graduating undergrad when Duolingo launched. So this is well before my time. But we launched with a streak feature. You know, initially how the feature worked was you'd come to Duolingo and you'd set a goal for yourself. And it was an XP-based goal. So Duolingo, a little bit of like, you know, nomenclature, we have an experience point space system that drives a lot of our features in the app. And the way you'd set it is you'd say, you know, based on what your language learning goal was, maybe you have a 10 experience point goal versus a 50 experience point goal. And so extending your streak would be, hey, can you hit 50 experience points? If that's what you set your goal to be. And this worked well. I think this also speaks to like how Duolingo initially grew. Luis launched it with a TED talk. We probably had a more tech.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“I mean, if you look at the numbers, I think pretty objectively has been our biggest growth lever for driving DAUs. And also say a lot of it's just related to how we think about growth at Duolingo. And a lot of what we try to do is organic growth. you know just as much as bringing new users onto the platform is not losing them if you're just bringing people onto the platform then they churn that's not going to be sustainable and so as much as we can do to keep our users coming back and actually retaining as users it's going to make it it's going to give us a much easier base to continue growing dAUs off of”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“I was looking up some stats before I came on, and it's pretty crazy. So we have right now over 9 million users with a year plus streak. So 9 million of our users have had used Duolingo every year almost every day for well over a year, which is pretty, I don't know, I like to imagine things in terms of like, you know, okay, well, like if you put all these people in a city or, you know, in a place where it would be, and like, it's like a very large city, nine million people.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“This is, and this is not just the subjective retention PM talking. I think this is our biggest feature with the exception of the lessons. And I think it's actually worth start that. Streaks are a great engagement hack. I'm kind of of the opinion that any team, you know, any app out there can introduce a streak. And if you figure it out, it probably works to retain users. But at the core, you have to have an app that people want to use. And people really like using Duolingo. It's fun. It's delightful. You learn something. And so it allows us to layer an engagement mechanic on top of that. The streak is really powerful. So it ships a disgusting amount of DAUs. Again, it is just, it is one of our golden geese. And again, what's cool is that you look at notifications. Notifications for Duolingo is massive.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, presumably, people know Dueling Go is a language learning app. The Duolingo Streak tracks currently anyway. How many days in a row you've done a lesson? So you come to Duolingo, you do a lesson. Your first lesson, you know, you'll start a streak, and then every consecutive day that you come use the app, you'll extend your streak. And I should actually put an asterisk around how many days in a row because we also have built flexibility into the feature. So we have these things called streak freezes. It's like insurance for your streak. So it's a pretty simple feature in theory. But over time, we've layered on challenges and goal setting and rewards and, you know, social features. A lot of our notifications are tied to the streak. So pretty simple feature to understand, but it's been a really rich feature for us to build on top of.”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, internally. So when we onboard new folks to the team, I actually just did this with somebody that joined the team recently where we spent, you know, I was like, hey, let's spend an hour just talking about the streak. We got through an hour and I got through about 30% of what I wanted to share. There's just so much. We'll talk about this, but there's so much that we've learned over the years. But never anything externally like this. I think we've shared bits and pieces of learnings, but this will be like the motherload of learnings, hopefully. Dueling goes”
2024-12-15 · Lenny's Podcast · Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) · IDENTIFIED FROM THE TRANSCRIPT · source