Skip to content

Podcast

Building Snipd- The AI Podcast App for Learning

Latent Space: The AI Engineer Podcast

Source ↗ ← All highlights
  • Snipd on Apple Watch
    • Snipd users often listen while multitasking, making note-taking difficult.
    • The Apple Watch app lets users snip insights without interrupting their activity. Transcript: Kevin Smith But yeah, it’s mostly… swyx Yeah, the watch thing, it was very funny because in the Lanespace Discord, most of us have been slowly adopting Snips. You came to me a year ago and you introduced Snip to me. I was like, I don’t know. I’m very sticky to overcast. Kevin Smith I’m doing it slowly with Switch. Why watch? So it goes back to a lot of our users, they do something else while listening to a podcast. And one of us giving them the ability to then capture this knowledge, even though they’re doing something else at the same time, is one of the killer features. Maybe I can actually, maybe at some point I should maybe give a bit more of an overview of all of the features that we have. Sure. So this is one of the killer features. And for one big use case that people use this for is for running. Yeah. So if you’re a big runner, a big jogger or cycling, like really, really cycling competitively. And a lot of the people, they don’t want to take their phone with them when they go running. So you load everything onto the watch. So you can download episodes. I mean, if you have an Apple Watch that has internet access, like with a SIM card, you can also directly stream. That’s also possible. (Time 0:18:25)
  • Substack and Apple Watch
    • Substack-hosted podcasts can’t be played on Apple Watch, regardless of the app.
    • This limitation is due to Substack’s policies, not Snipd’s. Transcript: swyx Error playing episode Substack, the host of this podcast, does not allow this podcast to be played on Apple Watch. Kevin Smith Yeah, that’s a very beautiful thing. So we found out that all of the podcasts hosted on Substack, you cannot play them on an Apple Watch. swyx Why is this restriction? Kevin Smith Don’t ask me. We try to reach out to Substack. We try to reach out to some of the bigger podcasters who are hosting the podcast on Substack to also let them know. Substack doesn’t seem to care. This is not specific to our app. You can also check out the Apple podcast app. It’s the same problem. It’s just that we actually have identified it and we tell the user what’s going on. swyx I would say we host our podcast on Substack, but they’re not very serious about their podcasting tools. I’ve told them before, I’ve been very upfront with them. So I don’t feel like I’m, you know, shitting on them in any way. And it’s kind of sad because otherwise it’s a perfect creator platform. But the way that they treat podcasting as an afterthought, I think it’s really disappointing. (Time 0:19:47)
  • Snipd Tech Stack
    • Snipd uses Python for the backend, Google Cloud, and Flutter/Dart for the frontend.
    • Flutter enables cross-platform development with good performance. Transcript: swyx Very interested in the tech stack. There’s a big data pipeline. Could you share what is the tech stack? What are the most interesting or challenging pieces of it? Kevin Smith So the general tech stack is our entire backend is, or 90% of our backend is written in Python. Okay. Hosting everything on Google Cloud platform. And our front end is written with, well, we’re using the Flutter framework. So it’s written in Dart and then, but compiled natively. So we have one code base for, that handles both Android and iOS. You think that was a good decision? It’s something that a lot of people are exploring. So up until now, yes. Okay. Look, it has its pros and cons. Some of the, you know, for example, earlier I mentioned we have an Apple Watch app. Yeah. I mean, there’s no flutter for that, right? So you build native. And then, of course, you have to sort of like sync these things together. I mean, I’m not the front-end engineer, so I’m not just relaying this information, but our front-end engineers are very happy with it. It’s enabled us to be quite fast and be on both platforms from the very beginning. (Time 0:28:30)
  • Early Snipd Transcription
    • Snipd initially used open-source models like Wave2vec for transcription before Whisper.
    • Wave2vec’s application of transformers to continuous audio data inspired Snipd’s creation. Transcript: Kevin Smith Know that was just to give you a bit of an overview. I think the more interesting things are, of course, on the AI side. So we, like, as I mentioned earlier, when we started out, it was before ChatGPT, before the ChatGPT moment, before there was the GPT 3.5 Turbo API. So in the beginning, we actually were running everything ourselves. Open source models, try to fine tune them. They worked, the results, but let’s be honest, they weren’t. What was the Soda before Whisper? The transcription? Yeah. We were using Wave to work. There was a Google one, right? No, it was a Facebook one. That was actually one of the papers, like when that came out, for me that was one of the reasons why i said we we should try something to start a startup in the audio space for me it was a bit Like before that i had been following the nlp space uh quite closely and as i mentioned earlier we we did some stuff at the startup as well that i was working at before and wave to work was The first paper that I had at least seen where the whole transformer architecture moved over to audio. And a bit more general way of saying it is like it was the first time that I saw the transformer architecture being applied to continuous data instead of discrete tokens. Okay. And it worked amazingly. And like the transformer architecture plus self-supervised learning. Like these two things moved over. And then for me, it was like, hey, this is now going to take off similarly as the text space has taken off. And with these two things in place, even if some features that we want to build are not possible yet, they will be possible in the near term with this trajectory. So that’s a little side note. (Time 0:30:47)
  • Diarization Techniques
    • Snipd uses open-source diarization tools with custom tweaks and heuristics.
    • Leveraging podcast structure improves diarization accuracy. Transcript: swyx Wanted to double click on diarization. It’s something that I don’t think people do very well. So, you know, I’m a B user. I don’t have it right now. And they were supposed to speak, but they dropped out last minute. But we’ve had them on the podcast before, and it’s not great yet. Do you use just PyAnote, the default stuff, or do you find any tricks for diarization? Kevin Smith So we do use the open source packages, but we have tweaked it a bit here and there. For example, if you mentioned the BAI guys, I actually listened to the podcast episode, which was super nice. And when you started talking about speaker diarization, and I just had to think about their use case, like with all of the different environments, it can basically be anything it’s Completely out of domain like there’s no there’s no data for this yeah i mean i was feeling with them because like our advantage is that we’re working with very high quality audio yeah It’s very controlled usually recorded in a studio this is quite an exception i guess it is kind of a studio it’s like pretty quiet there’s consistent background noise which you can edit Out uh this new york yeah it’s nice it’s a character um no so that that of course uh helps us uh another thing that helps us is that we know certain structural aspects of the podcast for example How often does someone speak? Like if someone, like, let’s say there’s a one hour episode and someone speaks for 30 seconds, that person is most probably not the guest and not the host. It’s probably some ad, like some speaker from an ad. So we have like certain of these heuristics, yeah, exactly, that we can use and we leverage to like improve things. And in the past, we’ve also changed the clustering algorithm. So basically how a lot of this, the speaker diarization works is you basically create an embedding for the speech that’s happening. And then you try to somehow cluster these embeddings and then find out this is all one speaker, this is all another speaker. And there we’ve also tweaked a couple of things where we, again, used heuristics that we could apply from knowing how podcasts function. And that’s also actually why I was feeling so much with the BAI guys, because like all of these heuristics, like they, like for them, it’s probably almost impossible to use any heuristics Because it can just be any any situation any uh anything um so that’s that’s uh one thing that we do yeah another thing is that we actually combine it with llms so the transcript llms and And the speaker diarization like bringing all of these together to recalibrate some of the switching points like does the speaker stop when does the (Time 0:36:03)
  • User-Centric AI
    • Focus on user needs, not complex AI, for consumer apps.
    • Simple features can delight users more than cutting-edge technology. Transcript: swyx Just say Spotify is not very good at podcasting. I have a documented dislike for their podcast features. Just overall, really, really well integrated. Any other LLM focused engineering challenges or problems that you want to highlight? Kevin Smith In the direction of handling the uncertainty of LLMs. So for example, with last year, at the end of the year, we did sort of a snipped wrapped. And one of the things we thought it would be fun to just to do something with an LLM and something with the snips that a user has. And three, let’s say, unique LLM features were that we assigned a personality to you based on the snips that you have. (Time 0:48:04)
  • LLM as a Judge
    • Use a smarter LLM as a judge to select the best output from a cheaper model.
    • This improves quality while managing costs. Transcript: swyx Yeah. Interesting. I think this year I’m very interested in LM as a judge being more developed as a concept. I think for things like Snips Wraps, it’s it’s fine like you know it’s it’s it’s entertaining there’s no right answer i mean we also have it um we also use the same concept for our books Kevin Smith Feature where we identify the the mentioned books yeah because there it’s the same thing like 90 of the time it works perfectly out of the box one shot and every now and then it just uh starts Identifying books that were not really mentioned or that are not books or yeah, starting to make up books. And there basically we have the same thing of like another LLM challenging it. Yeah. And actually with the speakers, we do the same now that I think about it. Yeah. (Time 0:50:15)
  • The AWS of AI
    • Kevin sees OpenAI and similar providers as the AWS of AI.
    • They offer serverless intelligence on demand, simplifying scaling and infrastructure. Transcript: Kevin Smith So it’s a bit similar how back before AWS, you would have to have your servers and buy new servers or get rid of servers. And then with AWS, it just became so much easier to just ram stuff up and down yeah and this is like the taking it even even uh to the next level for AI yeah I am a big believer in this basically swyx It’s you know intelligence on demand yeah we’re probably not using it enough in our daily lives to do things I should we should able to spin up 100 things at once and go through things and Then stop. I feel like we’re still trying to figure out how to use LMs in our lives effectively. (Time 0:54:57)
  • Voice Integration and Habit
    • Focus on existing user habits as triggers for new features.
    • Voice integration allows Snipd to seamlessly incorporate learning into the listening flow. Transcript: swyx Yeah. I think your framing of this is very powerful because i think this is where you are a product person more than an engineer because an engineer would just be like oh it’s just chat with your Podcast it’s like chat with pdf chat with podcast okay cool but you’re framing it in a different light that actually makes sense to me now as opposed to previously i don’t chat with my Podcast like why i just listen to the podcast right but for you it’s more about retention and learning and all that um and because you’re very serious about it that’s why you started a Company um so you’re focused on that whereas yeah i’m still me like i will admit i’m still stuck in that consume, consume, consume mentality. And I know it’s not good, but this is, you know, my default. Which is why I was a little bit lost when you were saying all the things about Duolingo and you’re saying the things about the trigger. This is my trigger for listening to the podcast is, you know, I’m by myself. That’s my trigger. But you’re saying the trigger is not about listening to the podcast. The trigger is remembering and retaining and processing the podcast I just listened to. Kevin Smith So, no, so, so what I meant, like you already have this trigger that gets you to start listening to a podcast. Yes. Like this you already have. Yes. And so do, I don’t know millions of people yeah so there are more than half a billion monthly active podcast listeners okay um so you already have this trigger that gets you to start listening But you do not have this trigger as you just said yourself basically you do not have this trigger that gets you to regularly um process process this information right and um voice basically For me is is uh the ability to hook into your existing trigger with the trigger that i was talking about is basically your podcast ends and you’re just still listening so we just continue And we can now spend you know this can be two minutes like i’m not saying now this is like a 60 minute process. I think like two minutes, three minutes that can just come on completely naturally. And if we manage to do that and you start noticing as a user, like freaking hell, like I’m just now spending three minutes with this AI companion, but like your retention is more taking This much away. And it’s not… And like retention is one thing, but you like… You start to take what you’ve learned and apply it to what’s important to you, like your thinking. Yeah. If we get you to notice that feeling, then… (Time 1:04:26)
  • YouTube for Podcasting
    • YouTube is the best podcasting platform due to its social layer and recommendation engine.
    • Snipd aims to incorporate backgroundable video for discovery. Transcript: swyx We are focusing a lot more on YouTube this year. YouTube is the best podcasting platform. It is not MP3s. It is not Apple Podcasts. It is not Spotify. It’s YouTube. And it’s just the social layer of recommendations and the existing habit that people have of logging onto YouTube and getting that. That’s my observation. You can riff on that. The only thing I would just say is like, when you were listing your list of priorities, you said audiobooks first over YouTube. And I would switch that if I were you. Yeah, like as in YouTube, video, video podcasts. Kevin Smith I mean, it’s obvious, yeah, video podcasts are here to stay. Not just here to stay, bigger. Yeah. What I want to do with Snipped is obviously also add video to the platform. Oh, yeah? The way I see video is I do believe it’s, I like this concept of backgroundable video. I didn’t come up with this concept. It was actually Gustav Söderström. The CPO of Spotify. Exactly, exactly. When I speak with people, it remains true that they listen to podcasts when they do something else at the same time. This is like 90% of their consumption. Also, if they listen to on YouTube. But every now and then, it’s nice to have the video. It’s nice if you’re, for example, just watching a clip. It’s nice if they sometimes mention something, like they show some slides or they show something where you need to have the visual with it. It helps you connect much more with the host as a listener. But the biggest benefit I see with video is discovery. I think that is also why YouTube has become the biggest podcast player out there because they have the discovery and discovery in video is just so much easier and so much better and so Much more engaging. So this is the area where I’m most interested about when it comes to video and Snips, that we can provide a much better, much more engaging and much more fun discovery experience. (Time 1:10:02)