YouSaid · the spoken record

Suhail Doshi

lines on the record: 28
first: 2024-04-18
most recent: 2024-04-18
sittings or episodes: 1
sources: podcast

Every line below is reproduced as it was said and linked to the record it came from. Nothing here is summarised or generated. Directory · Search · Corrections

“Flow is just like the swing of lyrics to a beat, or like a, you know, you hear like actually really good lyrics that feel like very emotional, have the right breathiness, doesn't sound like it's all made on like auto-tune, I guess. And so I have this like little flow where I like make a song in Suno and then I use a different AI tool to AI tools all the way down, I guess, to like split the stems and just grab the lyrics, but then throw away the instrumental. And then I get to like make a song with the instrumental and the vocal anyway. So I put some songs on my Twitter where like I basically tried to do this and it sounds, you know, so I can get to like a higher quality song, I guess, because I make the instrumental. There are still some like weird errors in the songs. But that's been like a really cool way to use AI, in my opinion.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Beats are those instrumentals are fairly easy to make and easy. What's hard is to get lyrics in vocals. And that's always been like a difficulty of mine. Like, how do I find a singer? And then how do I get them to like write lyrics? That's a much more scarce resource in the music world. And so for the first time with something like Suno AI, it was really cool because it's the first time that I heard them be able to make like a rap song where the rapper has like good flow.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Yeah, I love audio. That would be the other kind of thing that I would go work on if it weren't for Playground. Partly I didn't work on music because the music industry is like only, the whole industry is $26 billion. So it was a little hard for me to figure out how big a music thing could be. But I definitely think audio is going to be enormous. Things like 11 labs are very interesting. But anyway, yeah, I mean, the way that I've been, I've been trying to find ways to figure out how to use it as a user because that gives me like a really stronger sense of maybe where things are going. And so one thing that I've been waiting for for many years is that instrumentals in music are actually very easy to get or to make. You know, there's a wide variety of quality, of course. But generally, instrumentals in a song, like if you hear a song from Taylor Swift or whoever rap song, those”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Yeah, I mean, there's an assumption, maybe one assumption I tend to question is whether the inner data is sufficient. Like the internet is very big, but maybe there's some kind of boat collapse, even with internet data. Whereas with vision, at least you can make a robot that just travels down the street and just keeps taking pictures of everything. You can get infinite training data with vision, but it might be trickier to filter and clean internet data, especially as more synthetic data ends up on the internet.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Has like a lower ceiling than, say, vision because it's very easy to get lots of pixel data. And that pixel data is like very, very high density.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Between me and you. It's like a compressed way that you and I can converse with each other at a higher, somewhat of a higher bandwidth, right? Like we have an abstract view of what those words mean. So I think that the models, there has to be something like language is really great because it's compressed information and then like vision is really great because it's so information rich, but it's been hard to annotate until recently. It's only because vision language models exist that it's now suddenly a lot easier to like sort of label or annotate or understand what's going on in an image. So I think that these two things are very are going to be very likely married. The only question is does language, to me it's kind of a question of does language has this wonderful trait where it's like you can use language to control things, which is pretty cool because of its low dimensionality. But my question would be like, I wonder if language will hit a ceiling.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Definitely think the models are going to be multimodal. And in fact, that's kind of what I mean about some of these models that are just strictly trained through division transformer. Like a division transformer that's only taking caption image inputs is just like completely lacks sort of some knowledge. And then conversely, if you look at just like the language models, we know that language is at a much lower dimensionality than say like an image which has like all these pixels that like sort of tell us about lighting or physics or spatial relationships or size and shapes. So for example, if you were to like take a glass and like shatter it on the floor and then I asked you to describe it and I asked and then I described it, we would both come up with like completely different descriptions if Elad had to go and like draw it right so we know that like pixels have an enormous amount of high information density compared to language and language is just really”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Great, amazing knowledge of a language model, and then just using something like DIT, which is completely trained from some kind of video caption or image caption to an image. Because there's not enough interpretable knowledge, I suppose. You're not able to interpret anything about the input, which a language model is really great at. But then there's these models that are just trained on these captions that emit images. And it's kind of unclear how we might marry these two things. And so it sure would be nice if somehow we could combine these two things. So I think the architecture is mostly going to change. I don't think that DIT is like the right architecture, but Transformers certainly.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Kind of like controversial take perhaps is that there's this thing called DIT, which people allegedly believe, like Sora is based on. And then there are variants of DIT. There's this thing called like, I think MM DIT, which I think Stable Division 3 is supposed to be based on by that research team at Stability AI. And my overall feeling is that transformers are definitely, I think transformers are definitely like the right direction, but I don't think that we're going to get a lot of enough utility if we're not somewhat trying to figure out a way to combine”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Maybe you have like a video camera or surveillance system or something, and it's like able to understand what's going on in that. But I think right now we're really focused on graphics.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“And B, it's like actually somewhat efficient computationally to do. So long term, I think that we're trying to make a large vision model. There's not really like a word, I guess. Like we have LLMs, but I'm not really sure what the word is for vision or pixels if you're trying to make a multitask vision model. And so the goal would be to try to do three areas of a large vision model would be to be able to create things, edit things, and then understand things. And so understanding would be like GPT-4V or if you're using something open source like COGVLM or there's all these amazing vision language models that are happening and then editing and creating are things that we've kind of talked about. But it would be really amazing at some point that if you made this like really amazing large vision model that it could do things like not just like create things like art but like maybe like help some kind of robot like traverse like some sort of path or like maze and then there's like things in the middle that are sort of like”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“If people are out there kind of like working on scaling text, we're basically trying to focus on scaling pixels in the first area that we're basically started on is just images. And the reason why we're working on images instead of, say, something like video or 3D or something like that is one part, one issue with 3D is that it tends to be better to work on 3D if you're like making the content, like you're making Pixar movies. The tools in 3D tend to not make as much money. And then the other thing with video is videos is just extraordinarily computationally expensive to do inference or even training on. And a lot of the video models first train like pre-train with like a billion images first anyway to like have a rich semantic understanding of like pixels. We just think that video that images is like maybe the”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“It feels like it's too much effort, I guess, to get something that you really, really want. So I think more where we're navigating is like, how can we help you take an image that you love? Maybe it's your logo, maybe, or incorporate something like your logo or put it in some sort of situation that you would prefer. Text synthesis is like something that we want to do, for example. Those are some areas that we want to head towards where there's like higher utility and less you make an image and you just post it to Instagram or something like that.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Maybe like probably like number two, I suspect, I guess, at text to art at the moment, just because we're training these models from scratch and we're closing the gap really rapidly as rapidly as we can around all the various kind of use cases. But I think that we'll probably diverge from some of the other companies in part because we care, I think we start, we're going to care a lot more about editing. People just have like a lot of images on their phone or they want to take some image that they love, whether that's made as art or something that they found and they want to like tweak it a little bit. It's a little annoying that you make this image and then you can't really change too much about it. You can't change the likeness of it. Maybe there's like a dog or like your face or like something character consistency issues. It feels a little like a loot box right now. And I think that because it's so much of a loot box, it feels...”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Generally, we try to keep something very simple because we know that users are there to make images. They're not there to necessarily help us label images. And so or annotate things or tell us everything about their preferences. And so we kind of have like a very sophisticated process of how we sort of curate images and how we're collecting data from these users to help us kind of rank and sort of like make sure we're choosing the right sort of things that we want to curate. And so I think these things are, they might seem like very simple when you encounter it, but like beneath that is like something very, very complex. But yeah, it's a little tough to go into it too deeply because, yeah, it does feel like a little bit of a secret sauce, I suppose.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“That could be like homework, like solving an LSAT or a bio test or a math test. And so some of these evals just don't have like the necessary coverage. So I think with things like judgment and taste, my feeling is that overall the evals need to get like way stronger. And so one thing that we tend to do is we just tend to like really look at a lot of images across a lot of grids. And we're really being exacting about what thing could be off, but you have to look at like thousands of images across different grids across different checkpoints to basically find and pick like sort of release candidates. But I still think that our own emails are not sufficiently strong enough. And they could be better at like world knowledge, whether that's like its ability to reproduce a celebrity if that's what you want or paintings. Sometimes paintings are difficult or like 3D or like illustrations or logos. Those kinds of things are all.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Thing that I've noticed is that every time we do an eval, we try to make our evals better and we try to make them better than the predecessor eval. And so one thing I always notice, though, with each successive run is that I find out much later after eval that the model has like all these gaps. So an example of a gap that we recently had was like we did well in our eval, but one area that I thought we did poorly that I wish we had done better on was photorealism. Sometimes it would make faces look like they hadn't gone to sleep for three days or something. I think that most evals in the industry are relatively flawed. Like a lot of them are doing like benchmarks on things that maybe are valuable from the purposes of marketing, but are not necessarily well correlated with what maybe users care about. And so a simple example would be like with large language models, like there's probably a good reason, there's a reason why they're probably good at homework. It's because a lot of the evals are related to things that are related to home.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“But I think the number one trick is like really just like that last phase of a supervised fine-tune where you're finding like really great curated data. And it's hard to say how much of that is a trick because it's actually just a lot of meticulous work. So I think there's a kind of a combination of some of these things being tricks and techniques. And then there's just like this other thing that's just really hard, meticulous. Like there has to be deep care. And with images, maybe more so than language, there has to be like taste and judgment.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Yeah, I mean, there's just so many different dimensions of these models. I mean, one is just like it's understanding of knowledge, but then for like aesthetics, it's really tricky. Honestly, I think the field itself is just so nascent that every month there's like a new trick. There's like a new thing that we all sort of develop or find out. I think there's an element of some of that being like a lot of different tricks. Like there's like this new trick that hasn't been well employed or well exploited yet by this guy named Teo Karras. And he basically does this weird thing called power EMA. Anyway, like basically helps converge training really fast. And so that's like one trick. And then there's this EDM trick and there's this thing called offset noise. And so there is a lot of tricks for things like color and contrast. There's even a trick called DPO for like that I think works in the language model world and also the image world, right? So I think there are all these, there are like lots of tricks that sometimes get you like 10, 20, sometimes two X improvements.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“The new model. And so we employed this thing called this EDM formulation, which samples the noise slightly differently. And it's a really clever kind of math trick. And there's a paper that you could probably read on it. But it's surprising how this one little very clever trick can produce images that have incredibly great color and contrast where like the blacks are really vibrant with like a bunch of different colors and this average brightness kind of goes away So that's like one thing.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Which is like a UNET, right? And Clip and the same VAE that Robin Rombach trained all this stuff. And then we sort of said, okay, what if we try to get something that's just at least better than SDXL, better than the open source model? And we weren't really sure by how much. And so our only goal was to just be better and try to deliver on the number one state-of-the-art open source model that we could release. And so we kind of learned two things. One is that when we looked at some of the images from something like SDXL, we noticed that there was sort of this average brightness. It was really confusing. It didn't quite have the right kind of color and contrast. And in fact, I became so used to this. I became so surprised about the average brightness when comparing it to the images of our model that I thought it was a bug during eval. I literally was like looking at the images and I was like, these cannot be the right images. And my team was sort of like, hey, I think you're actually just getting used to the images.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“It turns out that I think when a set of strong engineers, like their first thought is that you just take a model architecture, you find a lot of data, you fund yourself with enough compute, and you just sort of throw these things into a mixture of sorts and out comes out something like Dolly 2 or Dolly 3. It turns out that it's just way more complex than that and way more complex than I even imagined. I had a sense that it was more complicated than that, but then it's still further, it's more complicated than even that. You know, so I think there are a couple things that we did. One of the things that we were really focused on with that model was that we wanted to see how far we could push the architecture of something that already existed. This was mostly like a test. It was a test to see whether how far we could get as a research team before the next model change. And so we wanted to take something that we knew as a recipe that worked already, which was stable diffusion, Excel's architecture.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“The difference is that these models haven't quite reached the potential of what maybe its utility could be. Right now, for the most part, we formulate a prompt, which is really just a caption of what the image is, and then it diffuses into an image, a set of pixels, but a lot of those pixels are primarily used for art. But what we haven't done is we haven't done anything beyond that. We haven't really done something like editing, for example. Like, why can't we take an image that you already have? And why can't we sort of insert something into that with the correct lighting and stuff? Why can't we stylize an existing thing? Why is there not a blend of real and synthetic imagery into a single image that could then be used for a lot more things than just pure art? And so right now it's a lot of just people making art, but not a lot of people. Sometimes that reduces its practicality or its utility.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“I think one thing that has sort of been surprising, and it hasn't changed too much, actually, from maybe around June or July of August 2022, was that like a lot of people think about text to image as just, it's right now it's kind of not, it's not even text to image, it's like text to art.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“I just thought, like, hmm, there's just not enough people that want to do this one thing and do it really, really great. So I think for me, it was just about, were there enough capable people that wanted to do this?”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Think with language, there's like, I don't know, I don't know how many language companies there are. You guys would probably know better than me, but it seems like there's like over 20. And then maybe like five or like eight of them have a billion dollars worth of funding. I also didn't want to work on something if there were already extremely passionate people really working hard at that thing, people that I really respected that were working on that thing. And so at the time with images, there was just sort of, I think there was mid-journey, there was OpenAI doing some Dali stuff. And then you saw sort of stable diffusion. But for some of these companies, it didn't seem like there was going to be a longstanding concerted effort to keep making them better. It was sort of unclear who was doing this as a fun demo versus who was doing this as something they would like spend and invest tons of their time in. And so once I had kind of figured out, you know, to what extent OpenAI was going to invest in it or to what extent it seemed like the folks at Stability AI were like sort of focused on like seven different kinds of things.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“Yeah, I was, I mean, there have been a lot of people that thought I should do something in music, but I just, because music has been like a huge hobby of mine for like six years or so, I like produce music. But I just couldn't wrap my brain around useful thing I would end up making for people, although now there's like a lot of very interesting, cool, useful things for music. And then it seemed like a lot of people were very focused on language. And I had really enjoyed, I already work with lots of creative tools like when I was in high school, I used to make logos or I would make music or whatever. So I was excited that finally I could find something where it was a combination of creativity, tooling. Images have really amazing built-in distribution people want to share those kinds of things. So it ended up just like being this perfect thing that I was excited to work on.”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT
“I think back in April of 2022, I think that was just to place that time. It was like GPT 5 kind of came out and then Dolly 2 came out. And I was actually working on the second company Mighty. And at that time, I was trying to figure out how to do something with AI inside of a browser address bar. But when I saw Dolly 2 came out, it was just this very big, strange eye-opening moment where I think a lot of people didn't think that we'd be able to do like weird, interesting art things so soon. And so, and I think then soon after that, I think stable diffusion came out around June or July of that same year. And I got early access, maybe a couple weeks access to early to SD14. And I just kind of blew my mind what people could do with that. And I just thought that it seemed odd that all of this was being done in a Google Colab notebook. Shouldn't there be like...”
2024-04-18 · No Priors · The Future of AI Artistry with Suhail Doshi from Playground AI · IDENTIFIED FROM THE TRANSCRIPT