YouSaid · the spoken record

Sander Schulhoff

lines on the record: 105
first: 2025-06-19
most recent: 2025-06-19
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“Lack of details, lack of any prompting techniques that is the reality of a large part, the vast majority of the conversational prompt engineering that I do. There are cases that I will bring in those other techniques, but the most important places to use those techniques is the product focused prompt engineering. That is the biggest performance boost. And I guess the reason it is so important is like. Have to have trust in things you're not going to be seeing. With conversational product engineering, you see the output. It comes right back to you. With product focused. millions of users are interacting with that prompt. You can't watch every output. You want to have a lot of certainty that it's working well.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“I guess just quickly, maybe. Maybe a reality check is like the way that I do kind of regular conversational prompt engineering is I'll just be like, you know, if I need to write an email, I'll just be like Ret email, like not even spelled properly about, you know. About whatever, I usually won't go to all the effort of showing it my previous emails. And there's a lot of situations where I'll paste in some writing and just be like, make better, improve. So that like super, super short.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“I don't know. It's just one of those kind of random LLM things. But I had to add in that thought-inducing phrase like, you know, make sure to write out all your reasoning in order to make sure that happens. Because I wanted to make sure to maximize my performance over my whole test set. So what we see is that new model comes out. People are like, ah, you know, it's so good. You don't even need to prompt engineer it. You don't need to do this. But if you look at scale, if you're running thousands, millions of inputs through your prompt, oftentimes in order to make your prompt more robust, you'll still need to use those classical prompting techniques.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“All of the major labs are still publishing publishing, still productizing, producing non-reasoning models. And it was said as GPT-4, GPT-40 were coming out, hey, these models are so good that you don't need to do chain of thought prompting on them. They just kind of do it by default, even though they're not actually reasoning models. I guess a weird distinction. And so I was like, okay, great. you know, fantastic. I don't have to add these extra tokens anymore. And I was running, I guess, like GPT-4 on a battery of thousands of inputs. And I was finding like... 99 out of 100 times it would write out its reasoning, great, and then give a final answer. But one in a hundred times, it would just give a final answer, no reason”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Yeah. So this is classified under thought generation to a general set of techniques that get the LLM to write out its reasoning. Generally not so useful anymore because as you just said, there's these reasoning models that have come out and they by default do that reasoning. That being said,”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“And so you think kind of all right, like the soccer historian guy and the internet search one say they give back 13 and the English professor is like four. So you take 13 as your final response and one of the neat things about roles as we discussed before which may or may not work is that they can kind of activate different regions of the model's neural brain and make it perform differently and better or worse on some tasks. So if you have a bunch of different models you're asking and then you take the final result or the most common result as your final result you can often get better performance overall.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“You might say to one of them, okay, you need to act as an English professor and answer this question. And then another one like you need to act as a soccer historian and answer this question. And then you might give a third one no role, but just like access to the internet or something like that.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“But if you know random forests, these are kind of a more classical form of ensembling techniques. So anyways, a specific example of one of these techniques is called mixture of reasoning experts, which was developed by a colleague of mine who's currently at Stanford. And the idea here is you have some question, it could be a math question, it could really be any question. And you get yourself together a set of experts. And these are basically different LLMs or LMs prompted in different ways where some of them might even have access to the internet or other databases. And so you might ask them like, I don't know, how many trophies does Real Madrid have?”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Give the LM a math problem. I'll give it this prompt technique with the math problem, send it off. And then a new prompt, new prompting technique, send it off. And I could do this with a couple different techniques or more. And I'll get back multiple different answers. And then I'll take the answer that comes back most commonly. So it's kind of like if I went to you and Fetti and Gerson to a bunch of different people, and I asked them all the same question, and they gave me back slightly different responses, but I kind of take the most common answer as my final answer. And these are kind of historically a historically known set of techniques in the AI ML space. There's lots and lots and lots of ensembling techniques. It's funny, the more I get into prompting techniques, the less I remember about classical”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“There's certain ensembling techniques that are getting a bit more complicated. And the idea with ensembling is that you have one problem you want to solve. And so it could be a math question. I'll come back at again and again to things like math questions because a lot of these techniques are judged based off of data sets of like math or reasoning questions simply because you're going to evaluate the accuracy programmatically as opposed to something like generating interview questions, which is no less valuable, but just very difficult to evaluate success for in an automated way. So ensembling techniques will take a problem and then you'll have like multiple different prompts that go and solve the exact same problem. So I will take maybe like a chain of thought prompt, like let's think step by step.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“There absolutely is. And you know what? I actually consider myself something of a Prompting or Gen AI historian. I won't even say consider myself I am very, very straightforwardly. And there's these slides I presented yesterday that go through the history of like prompt, prompt engineering. Have you ever wondered where those terms came from? They came from, well, a lot of different people, research papers. Sometimes it's hard to tell, but that's another thing that the prompt report covers is that history of terminology, which is very much of interest to me.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Special ways of architecting your prompt or like special phrases that kind of induce better performance. And so there are parts of a prompt which like the role, that's a part of a prompt. The examples are a part of a prompt additional information is part of a prompt. The directive is a part of a prompt. And that's like your core intent. So for you, it might be like give me interview questions. That's the core intent. And then there's stuff like output formatting and you might be like, I want a table or a bullet list of those questions. You're telling it how to structure its output. That's another component of a prompt, but not necessarily prompting technique in and of itself. Because again, the prompting techniques are like special things meant to kind of induce better performance.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, the way we formatted things in this paper, the prompt report, is that we went and kind of broke down all the common elements of prompts. And then there's a bit of crossover where like examples, giving examples, examples are a common element in prompts. But giving examples is also a prompting technique. But then there's things like giving context, which we don't consider to be a prompting technique in and of itself. The way we kind of define prompting techniques is like”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Well, I guess I will mention that we actually have gone through some more advanced techniques, depending on your perspective. Yeah, what would you call it?”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Depends, and this also can kind of get into are you going to fusehop prompt with different pieces of additional information? I usually don't. There's no need to use the XML brackets. You feel more comfortable with that if that's the way you're structuring your prompt, anyways, do it. Why not? But I almost never include any kind of structured formatting with the additional information. I kind of just toss it in.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“And so that's one really big reason to do it at the beginning. And then the second is that sometimes if you put all your additional information at the end of the prompt and it's like super, super long. The model can forget what its original task was and might pick up some question in the additional information to use instead.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“I would say so. Yeah, that is pretty much my advice, especially in the conversational setting when, I mean, frankly, when you're not paying per token. And Naval latency is not quite as important, but in that product focus setting, when you're giving additional information, it is a lot more important to figure out exactly what information you need. Otherwise, things can get expensive pretty quickly with all those API calls and also slow. So latency and cost become big factors insiding how much additional information is too much additional information. And so usually I will put my additional information at the beginning of the prompt. And that is helpful for two reasons. One, it can get cached. So subsequent calls to the LM with that same context at the top of the prompt are cheaper because the model provider stores that initial context for you as well as kind of like the embeddings for it. So it saves a ton of computation from being”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“I actually took the original email the professor had sent me describing the problem and pasted that into the prompt. And it performed pretty well. And then sometime down the line, the professor is like, hey, probably shouldn't publish our personal information in the eventual research paper here. And I was like, ah, you know, that makes sense. So I took the email out and the performance dropped off a cliff without that context, without that additional information. And then I was like, all right, well, I'll keep the email and just anonymize the names in it. The performance also dropped off a cliff with that. That is just like one of the wacky oddities of prompting and prompt engineering. There's just small things you change that have massive unpredictable effects. But the lesson there is that including context or additional information about the situation was super, super important to get a performance prompt.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“People say, you know, I'm going to kill myself, stuff like that are not actually indicative of suicidal intent. However, saying things like, I feel trapped, I can't get out of my situation are. And there's a term that describes this sentiment. And the term is entrapment. It's that, you know, feeling trapped in where you are in life. And so we're trying to get GPT-4 at the time to classify a bunch of different posts as to whether they had the entrapment in them or not. And in order to do that, I kind of talk to the model, like, do you even know what entrapment is? And it didn't know. And so I had to go get a bunch of research and kind of paste that into my prompt to explain to it what entrapment was so I could properly label that. And there's actually a bit of a funny story around that where”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“So back in college, I was working under Professor Philip Bresnick, who's a national language processing professor and also does a lot of work in the mental health space. And we were looking at a particular task where we were essentially trying to predict whether people on the internet were suicidal based on a Reddit post, actually. And it turns out that comments like”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Trying to get the model to do some task. You want to give it as much information about that task as possible. And so if I'm getting emails written, I might want to give it a list of all my kind of like work history, my personal biography, anything that might be relevant to it writing an email. And so similarly with different sorts of data analysis, you know, if you're looking to do data analysis on some company data, maybe the company you work at, it can often be helpful to include a profile of the company itself in your prompt because it just gives the model better perspective about what sorts of data analysis it should run, what's helpful, what's relevant, so including a lot of information just in general about your task is often very helpful.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“I guess we could get into parts of a prompt. So including really good Some people call it context. So giving the model context on what you're talking about, I try to call this additional information since context is a really overloaded term and you have things like the context window.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Another one is a set of techniques that we call self criticism. So the idea here is you ask the LM to solve some problem. It does it, great. And then you're like, hey, can you go and check your response? Confirm that's correct or offer yourself some criticism. And it goes and does that, and then it gives you this list of criticism. And then you can say to it, hey, great criticism. Why don't you go ahead and implement that? And then it rewrites its solution. So it outputs something. You get it to criticize itself and then to improve itself. And so these are a pretty notable set of techniques because it's like kind of free performance boost that works in some situations. So that's another kind of favorite set of techniques of mine.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“And so go run a database check on that and then confirm what kind of car they have, confirm what date they checked it out on, whether they have some kind of insurance on it. So those are all the subproblems that need to be figured out first. And then with that list of subproblems, you can distribute that to all different types of tool calling agents if you want to get more complex. And so after you solve all that, you bring all the information together. And then the main chatbot can make a final decision about whether they can return it, if there's any charges and that sort of thing.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“So I do distinguish them. And I think with this example, you'll see kind of why. So, a great example of this is like Like a car dealership chap hot. And somebody comes to this chatbot and they're like, hey, I checked out this car on this date, or actually it might have been this other date. And it was this type of car. Or actually, it might have been this other type of car. And anyways, it has the small ding and I want to return it. What's your return policy on that? And so in order to figure that out, you have to like look at the return policy, look at what type of car they had, when they got it, whether it's still valid to return, what the rules are. And so if you just ask them all to do all that at once, it might kind of struggle. But if you tell it, hey, what are all the things that need to be done first? Just like kind of what a human would do. And so it's like, all right, I need to figure out, first of all, is this even a customer?”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Before answering it, tell me what are some sub problems that would need to be solved first. And then it gives you a list of sub problems. And honestly, this can help you think through the thing as well, which is half the power a lot of the time. And then you can ask it to solve each of those sub-problems one by one. And then use that information to solve the main overall problem. And so again, you can implement this just in a conversational setting, or a lot of folks look to implement this as part of their kind of product architecture. And it'll often boost performance on kind of whatever their downstream task is.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Helpful. So decomposition is another really, really effective technique. And for most of the techniques that I will discuss, you can use them in either the conversational or the product focused setting. And so for decomposition, the core idea is that there's some tasks, some task in your prompt that you want the model to do. If you just ask it that task straight up, It might kind of struggle with it. So instead you give it this task and you say, hey, don't answer this.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Giving more context. Exactly. And so that's why that one might work, might have worked. And for the kind of threats and promises. Seen explanations of like, oh, the AI was trained with reinforcement learning. So it knows to learn from rewards and punishments, which Is true in a rather pure mathematical sense, but I don't feel like it works quite like that with the prompting. Like that's not how the training is done. During training, it's not told, hey, do a good job on this and you'll get paid and then that's just not how training is done. And so that's why I don't think that's a great explanation.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“The math professor one would actually get easier to explain. Telling it it's a math professor could activate a certain region of its brain that is about math. And so it's thinking more about math. It's like context.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Order to get true statistical significance, you need to run some pretty robust studies. And so I think that this is really the same as role prompting on those older models. Maybe it worked on the more modern ones. I don't think it does. Although the more modern ones are using more reinforcement learning, I guess so maybe it'll become more impactful, but I don't believe in those things.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“That's a great one to discuss. So there's that. There's like the one, oh, I'll tip you five dollars if you do this. Anything where you give some kind of promise of a reward or threat. Some punishment in your prompt. And this was something that went quite viral, and there's a little bit of research on this. My general perspective is that these things don't work. There have been no large scale studies that I've seen that really went deep on this. I've seen some people on Twitter ran some small studies, but”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“I ended up being right. And a couple months later, one of the researchers who was involved with that thread who had written one of these original analytical papers sent me a new paper they had written. I was like, hey, we reran the analyses on some new data sets. And you're right. There's no effect, no predictable effect of these roles. And so my thinking on this is that At some point With the GP3 early chat GPT models, it might have been true that giving these roles provides a performance boost on accuracy-based tasks, but right now it doesn't help at all. But giving a role really helps for expressive tasks, writing tasks, summarizing tasks. And so with those things where it's more about style. That's a great, great place to use rules. But my perspective is that roles do not help with any accuracy-based tasks whatsoever.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“I do remember at some point we put out a tweet and it was just like row prompting does not work and it went super viral. We got a ton of hate. Yeah, I guess it was probably this way around. But anyways.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Right, right. Yeah, exactly. And so at some point, people were like arguing on Twitter about whether this works or not. And I got tagged in it. And I came back. It's like, hey, you know. Probably doesn't work. And I actually now realize I'm going to told that story wrong. And it might have been me who started this big debate. Anyways, I.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Your general researcher, and what they seem to find was that roles with more interpersonal ability, like teachers, performed better on different benchmarks. It's like, wow, that is fascinating. But if you look at the actual results data itself, The accuracies were like 0.01 apart, so there's no statistical significance. And it's also really difficult to say which roles have better interpersonal ab”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Looking in the GPT3 early chat GPT era, it was a popular conception that you could tell the AI that it's a math professor. And then if you give it a big data set of math problems to solve, it would actually do better. It would perform better than the same instance of that LM that is not told that it's a math professor. So just by telling it it's a math professor, you can improve its performance. And I found this really interesting. And so did a lot of other people. I also found this a little bit difficult to believe because that's not really how AI is supposed to work. But I don't know. We see all sorts of weird things from it. So I was reading a number of studies that came out and they tested out all sorts of different roles. I think they ran like a thousand different roles across different jobs and industries. Like you're a chemist, you're a biologist.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Role prompting is really just when you give the AI you're using some kind of role. So you might tell it, oh, like you are a math professor, and then you give it a math problem, you're like, hey, like, help me solve my homework or this problem or whatnot. And so”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“This is perhaps the question that I am most prepared for out of any you will ask because I've spoken to this over and over and over again and gotten into some internet debates around. Do you know what role prompting is”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“That's another thing if you don't even necessarily have the inputs and the outputs. In your case, you just have, I guess, outputs that you're showing it from the staff.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Still use QA even when there is no question or answer involved just because the LMs are so familiar with this formatting due to I guess all of the historical NLP kind of using this and so the LMs are trained on that formatting as well and you can combine that with XML there's yeah there's a lot of things you can do there”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Four minutes. The usual way I format things is I'll have I'll start with some data set of inputs and outputs. And it might be like ratings for a pizza shop and some binary classification of like is this a positive sentiment? Is this a negative sentiment? And so this is going back more to classical NLP, but I'll structure my prompt as like Q colon and then I'll paste the review in and then A colon and I'll put the label and I'll put a couple lines of those and then on the final line I'll say Q colon and I'll input the one that I want to like the LM to actually label the one that it's never seen before and Q&A stand for question and answer and of course in this case it's there there are no questions that I'm asking it explicitly I guess implicitly it's like is this a positive or negative review but people”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Researchy approach, but just take some common format out there that the LLM is comfortable with. And I say that kind of with air quotes because it's a bit of a strange thing to say, like the LM is comfortable with something, but it actually comes empirically from studies that have shown that formats of questions that show up most commonly in the training data are the best formats of questions to actually use when you're prompting it.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“My main advice here, although actually before I send my main advice, I should preface it by saying we have an entire research paper out called the prompt report that goes through all of the pieces of advice on how to structure a few shot prompt, but my main advice there is choose a common format. So XML, great if it's like Know like question colon, and then you kind of input the question, then answer colon, and you input the output. That's great too. It's a more research.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Across the industry and across different industries, there's like different meanings of these, but ZeroShot is no examples. One shot is one example and few shots multiple.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“So, my best advice on how to improve your prompting skills is actually just trial and error. You will learn the most from just trying and interacting with chatbots and talking to them than anything else, including reading resources, taking courses, all of that. But if there were one technique that I could recommend people, it is few shot prompting, which is just giving the AI examples of what you want it to do. So maybe you wanted to write an email in your style, but it's probably a bit difficult to describe your writing style to an AI. So instead, you can just take a couple of your previous emails, paste them into the model, and then say, hey, you know, write me another email, say I'm coming in sick to work today and style it like my previous email. So just by giving examples of what you want, you can really, really boost its performance.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Absolutely down here. And most of the research is on those, I guess. Now you've coined it as product focus prompt engineering.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Each day, I need this one prompt to be perfect. And so a good example of that, I guess going back to the medical coding, is I was iterating on this one single prompt. It wasn't over the course of any conversation. I just take this one prompt and improve it. And there's a lot of automated techniques out there to improve prompts and keep improving it over and over again until something I've satisfied with and then kind of never change it. And I guess only change it if there's really a need for it. But those are the two modes. One is the conversational. Most people are doing this every day. It's just kind of normal chatbot interactions. And then there is the normal mode. I don't really have a good term for it.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“So, this was actually a bit of a recent development for me in terms of thinking through this and explaining it to folks. But the two modes are First of all, there's the conversational mode in which most people do prompt engineering. And that is just you're using Claude, you're using ChatGBT, you say, hey, can you write me this email? It does kind of a poor job. And you're like, oh no, make it more formal or add a joke in there and it adapts its output accordingly. And so I refer to that as conversational prompt engineering because you're getting it to improve its output over the course of a conversation. Notably, that is not where the classical concept of prompt engineering came from. It actually came a bit earlier AI engineer perspective where you're like, I have this product I'm building. I have this one prompt or a couple different prompts that are super critical to this product. I'm running like thousands, millions of inputs through this prompt.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“So recently I was working on a project for a medical coding startup where we were trying to get the Gen AI's GPT-4 in this case to perform medical coding on And so I tried out all these different prompts and ways of kind of showing the AI what it should be doing. But at the beginning of my process, I was getting little to know accuracy. It wasn't outputting the codes in a properly formatted way. It wasn't really thinking through well how to code the document. And so what I ended up doing was taking kind of a long list of documents that I went and coded myself or I guess got coded and I took those and I attached kind of reasonings as to why each one was coded in the way it was. And then I took all of that data and dropped it into my prompt and then went ahead and gave the model like a new transcript I had never seen before and that boosted the accuracy on that task up by I think like 70% so massive”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source
“Intelligence. I imagine you're familiar with the term social intelligence, which kind of describes how people communicate, interpersonal communication skills, all that. We have recognized the need for a similar thing, but with communicating with AIs and understanding the best way to talk to them, understanding what their responses mean, and then how to adapt, I guess your kind of next prompts to that response. Over and over again, we have seen prompt engineering continue to be very important.”
2025-06-19 · Lenny's Podcast · AI prompt engineering in 2025: What works and what doesn’t | Sander Schulhoff (Learn Prompting, HackAPrompt) · IDENTIFIED FROM THE TRANSCRIPT · source