Skip to content

Podcast

DeepMind’s RAG System With Animesh Chatterji and Ivan Solovyev

Software Engineering Daily

Source ↗ ← All highlights
  • File Search Removes RAG Infrastructure Burden
    • File Search is an integrated, opinionated RAG solution that removes infrastructure overhead for developers.
    • Ivan Solovyev emphasized upload-and-query simplicity plus simplified pricing that charges only for indexing and model tokens. Transcript: Sean Falconer Yeah, absolutely. Ivan Solovyev So file search tool, first of all, is an integrated RAG solution that makes it super easy for you to take lots and lots of data, text, PDFs, codes, whatever you have, upload it into Gemini And start asking questions about your data. There are plenty of Rack pipelines available on the market, like we have Vertex, Rack Engine, and there are other providers who do support this feature. So it’s nothing new. And what we focused on in file search in particular is accessibility and simplicity of use. We made some opinionated decisions. We removed a lot of complexity in terms of configuration, setup. You don’t need to set up your database. You don’t need to set up your infrastructure. The tool is just there. You just upload your data and you’re going to use it right away. (Time 0:02:35)
  • Simplify RAG Pricing To Lower Friction
    • Simplify billing to reduce developer friction by charging for indexing and tokens only.
    • Ivan explained they removed storage and other line items so costs are easier to predict and typically cheaper. Transcript: Ivan Solovyev We removed most of the things that you are paying for, and we focused on two simple aspects. So first of all, you are paying for indexing. So whenever you upload the file, we do need to do a lot of complex processing, we need to do embeddings. So you’ll pay for that. And after that, whenever you do a query to Gemini, you’re just paying for tokens. Obviously, there’s going to be some addition from file search, adding data into the context, but that’s it. You’re not paying for storage, you’re not paying for anything else. Sean Falconer Why make that change around pricing? Is it primarily to just really try to simplify things for the users of this? Why was it needed to kind of, I don’t know, buck the trend of how perhaps people have used to been paying for RAG in the past? Ivan Solovyev Yeah, I think in Gemini API and AI Studio in general, we are aiming a lot for simplicity. And we do hear a lot of feedback from developers that it’s hard to deal with lots and lots of products. It’s hard to deal with different billing models and billing cycles and how the whole cost is being calculated. So we do see it as a decent improvement over other products. And the price is actually much cheaper. (Time 0:03:56)
  • RAG Still Crucial For Large Data Sets
    • RAG remains essential for large corpora and enterprise use cases despite longer model contexts.
    • Ivan argued long contexts help small datasets, but RAG lowers cost and enables working with huge codebases and legal docs. Transcript: Ivan Solovyev In terms of where we are, I think RAC is a fundamental capability. RAC been there from the very beginning. Whenever those models get really popular in use, it was always a staple whenever you wanted to process the data and hype cycles were going up and down. We always see this with different features related to LLMs. But I feel that the track was always there and it was always useful to some extent. In the latest years, we saw improvements in the context size, obviously available to LLMs. And this does help a lot with use cases with limited data sets. And we do see much better quality whenever you try to do simple retrieval tasks on a small data set that fits into the complex. And we usually do recommend to use that approach. However, whenever you start doing any enterprise use cases, whenever you have a huge code base, whenever you have a large file sets, like any legal documentation, anything like that, Having RAC becomes very, very beneficial. First of all, you can work with the whole database without building the complicated pipeline or infrastructure to actually juggle the data in and out of the context. You can work with the whole data set. The costs become much better with Rack. If you put everything into the context, it becomes expensive very, very fast. And especially if you’re using the higher tier of models, like pro models, it’s getting expensive. And with file search or other RAC solutions, you are able to actually reduce this cost. (Time 0:05:44)
  • Embedding Improvements And Smarter Chunking Evolved RAG
    • RAG techniques evolved: better embedding models, smarter chunk selection, and refrag-style selective expansion.
    • Animesh Chatterji noted embeddings improved multilingual retrieval and new methods pick which chunks to expand to the model. Transcript: Animesh Chatterji Well? Yes, I think the use case has evolved. So I think to your question, whether the long context models have straightened the proposition of RAG, I would say that in fact, they have encouraged RAG to cover even more use cases. Like there are a lot of EDO use cases where people want to upload documents worth their entire semester and there is now even more focus on making RAC efficient because even though long Context like retrievals are working fine we still see sometimes this context rot or lost in the middle syndrome where models are not great at retrieving data which is kind of in the middle Of the context so now there are new techniques on making sure that how can we improve even the chunks that we are feeding into the model from rag so there was a recent paper last year which Is called a refrag where what they are trying to do is instead of passing the chunks as it is to the model they are trying to embed the chunks and give all these embeddings to the model and Let the model decide which of these embeddings may seem more interesting and only expand those cases. So yeah, from the kind of initial vanilla rag where we kind of find everything, give it to the model, we are trying to make that more smart by figuring out which chunks to give. Also the embedding models themselves have improved, the way we are able to represent data that has significantly improved. So now we are better at understanding the context. We are doing better in languages other than English. So internationalization has also picked up. So there are these different parts using which we can say that RAG as a product is improving. (Time 0:07:30)
  • Embed Corpus And Query With The Same Model
    • Use consistent embedding models for corpus and queries and tune retrieval size via sensible defaults.
    • Animesh described chunking, embedding with Gemini embeddings, and default knobs they evaluated for latency-quality tradeoffs. Transcript: Animesh Chatterji Think what we do is when you provide the data, we chunk it and we embed it using the latest Gemini embedding model. And then we basically index it internally. And then at the time of the query, when the user provides the query, we again embed it using the same embedding model, and then try to figure out the relevant chunks from that corpus that The user has uploaded. And then we have some knobs on which embedding model to use or how many chunks do we want to retrieve and pass it to the model. And we have run a bunch of evals to kind of find the sweet spot in terms of latency of how many chunks we want to retrieve versus the quality we see. And there are some knobs that the users can provide in terms of how do they want to chunk the data, but mostly it’s the default settings that we have iterated upon. We have tweaked the SI to make sure that the model actually triggers this tool when it actually feels that it is necessary to get more context and it’s not unnecessarily triggering. (Time 0:10:19)
  • Prefer Strong Defaults Over Many Config Options
    • Favor opinionated defaults over exposing many knobs to help most users get good results quickly.
    • Ivan said their one-size-fits-most approach returned ~five chunks and fit many legal and code search scenarios. Transcript: Animesh Chatterji Kind of kept it open. We don’t document how many chunks we want. But yeah, it’s not too many at this point of time. Is there some use case dependency on that? Or can you actually have sort of more of a universal approach to this? Yeah. So right now we are going with the one solution fits all approach because we want to keep it simple. And when we hear use cases of customers who feel the need that they want more of these chunks retrieved, it’s easy to expose that as an option in the api we don’t want to do that right now But if needed we could do that if the threshold at which you want to retrieve the chunks or the number of chunks you want to feed to the model those are all things we could tweak around ivan Ivan Solovyev You’re adding something yeah so so far what we saw from the partners integration is that the default configuration actually fits most of the use cases. (Time 0:11:37)
  • Embeddings Drive Retrieval Quality
    • Embedding quality drives most retrieval outcomes; configuration matters far less.
    • Ivan estimated roughly 80% of quality comes from embeddings and 20% from retrieval configuration, justifying simplified UX. Transcript: Ivan Solovyev Think most of the quality actually comes from the embedding model. So it’s like you should think about this as like 80% of quality is embeddings, 20% is your configuration. So as long as we have the best embedding models, which we believe we do, the rest is less relevant to the quality of the outcome. So we do believe that for most people, playing with those configurations will not yield significant improvement and the time is better spent in building their own pipelines. So that’s where we focus on. At the same time, we never say don’t use any other RAC pipelines. We actually say file search is the simplest tool. You should try the first thing. You should work for the majority of people. But if you really need the configurability, let’s say your use case is very, very complex. You’re processing very well-structured data, specific tables, specific graphs that our system does not yet recognize well. (Time 0:13:17)
  • Start With Text And OCR Then Add Multimodal
    • Start with text-first ingestion and OCR for images, then add multimodal support later.
    • Ivan explained File Search currently indexes text, runs OCR on images, and plans native image, video, and audio retrieval support. Transcript: Ivan Solovyev I would love to say all of them, but we are indexing text files mostly. So PDFs, docs, code files, anything with text. We are currently doing OCR on images. So we’re not fully ignoring images within PDFs and other files, but we are going through the OSR system. We’re extracting text out of them and putting that into the context as well. And we are actually working on getting the multimodal support in as well. So we want to support native image processing, video processing, and at some point native audio as well. Gemini models are pretty good at reasoning on top of image and video data. (Time 0:14:25)
  • Handle Document Updates By Inserting New Versions
    • Provide document management APIs to handle updates and let developers delete or insert new versions.
    • Animesh explained they treat updates as new inserts and recommend clients remove old versions if needed. Transcript: Animesh Chatterji There are two parts to this update, right? One is basically you kind of calling our API to ingest those documents. So we try to make sure that we are highly parallelized in terms of our ingestion latency. So we pretty much can parallelize at a chunk level and ensure that all of those are ingested into the database. And then Google has the Spanner, which is also exposed externally as the Cloud Spanner, which provides very strong consistency guarantees. So once you pretty much write the data, it’s almost instantaneously available to be indexed. And we leverage that capability of Spanner to make sure that we can pretty much read our writes as soon as they are available. That significantly reduces the delay in reading the indexes and reading the embeddings. Sean Falconer Do you have to, like if I’ve already indexed a particular page though, or a document, and then I’m re-indexing it, do I have to blow away the initial indexing indexes in order to re-index It? Or is there essentially the equivalent of like an upsert in the vector world? Animesh Chatterji So essentially you have the corpus, you can add your new document to that corpus, which would just mean that the new chunks are indexed, the rest of the index remains as it is. (Time 0:19:46)
  • File Search Uses Semantic Search Now But Hybrid Is Coming
    • File Search currently uses purely semantic (vector) search but may add keyword/hybrid search later.
    • Animesh noted user requests for keyword search and indicated Spanner-backed indexing makes hybrid search a natural extension. Transcript: Animesh Chatterji Right now, it’s it’s purely semantic search, which is vector-based. We have had some requests of users wanting a keyword-based search, and that is something we’re considering adding to the roadmap. Given the indexing capabilities that Spanner offers, we think it’s like a natural extension to the offering, and yet something which should not add too much complexity to our system. Ivan Solovyev We also looked at the GraphRag systems in the past, but for now, it’s to me at least feels a little bit more complex for the product that we’re trying to build. (Time 0:21:46)
  • Map Model Citations To Chunk IDs For Grounding
    • Use model-generated citations plus stored chunk IDs to map answers back to original documents.
    • Animesh explained the model cites unique indexes for each returned chunk so post-processing can map citations to source metadata. Transcript: Animesh Chatterji Yeah. So right now it’s the models are trained to cite the responses. So when they generate the responses, they actually cite every sentence that they have used the original corpus to generate from. And then it’s just a matter of post-processing that response, removing those citations and adding that separately as grounding data. So essentially, it’s models generating citations to the data that they refer to. Sean Falconer Yeah, so the model is trained to essentially figure out or to provide a reference back to where that text or what the source text was. And then you have to map that source text, I guess, back to the database chunk in the original source in order to inject the link or something like that that refers to the citation. Animesh Chatterji Is that right? Yeah, so basically when the flow is something like this, when the model realizes that it needs to use file search, it will emit a query that I want this query to be answered by the file search Tool. You run the query and you give the responses. Each response in some sense is indexed uniquely. So if the model is receiving five chunks of data, it knows that each of them is a different index. And this index can vary per turn as well. So now when the model responds, it kind of cites the exact unique index using which we can kind of figure out which chunk it was referring to and then figure out which document it was part Of and add more metadata about that. (Time 0:23:48)
  • Beam Uses File Search To Teach New Game Developers
    • Beam used File Search to onboard game developers by indexing engine code and documentation.
    • Ivan described Beam pulling docs into agent context to guide novices on modules, animations, and scripts during development. Transcript: Ivan Solovyev I think their use case is pretty neat and simple at the same time. So they have lots and lots of new developers coming to the platform who want to build games with AI, and they don’t necessarily experience developers. They are mostly learning. And AI helps Beam to educate and help those developers to create their first game. So the way they are using file search is they have a huge code base that is their engine plus the documentation on top of that that is talking about how each component is used, how animations Are happening, how scripts are implemented, et cetera, et cetera. It’s a rather big data set. So what they do is they put it all into the file search, they index it, and whenever the user starts experimenting with the agent and the agents that supports it, they will naturally ask Questions how to do specific things. And through file search, they can very quickly pull all the relevant documentation into the context and actually present to the developer that, hey, you probably want to use this module. Here is how it works. Here is the old documentation. And they’ve been able to close this education loop for their customers, receive great feedback. (Time 0:25:30)
  • RAG Latency Is Small But Quality Varies By Domain
    • Retrieval latency is a few seconds and retrieval accuracy varies by domain, sometimes reaching ~85% correct hits.
    • Ivan said retrieval latency aligns with model latency and quality depends on the dataset and use case. Transcript: Ivan Solovyev Is somewhat in line with the model latency, a couple seconds for the retrieval. In terms of quality, it will depend on the use case. If I recall correctly, we saw up to like 85%, depending on the use case of the retrieval, like correct hits in terms of documents redrift. (Time 0:26:49)
  • Three Practical Levers To Improve RAG Accuracy
    • Improve accuracy by using better embeddings, tuning retrieval vs latency, and training models to trigger tools only when needed.
    • Animesh listed embedding quality, retrieval strategy, and model tool-triggering as the three main levers. Transcript: Animesh Chatterji Think there would be like a few things right one would be naturally be the embedding model that yvan talked about and kind of called out the importance the second is your retrieval strategy Like sometimes you would want to optimize quality for latency whether you want to kind of go through your entire database find all relevant chunks or kind of figure out the first few Relevant chunks and give it to the model so that would kind of be the other aspect on like trading of latency versus quality and third i think is just the model training on triggering the Search only when relevant and also not hallucinating the answers those are kind of orthogonal to file search those apply to any tool that we have trained Gemini with. But I would say those are kind of the three aspects, the emailing quality or retrieval quality, and just the model quality, which probably is of utmost importance. Ivan Solovyev Yeah, that’s mostly on our end, and that’s what we are working on in terms of improving the quality. For developers, what I saw is some developers actually implement the post-processing. So they would implement file search calls in a separate, in a sub-agent or a separate flow. And they will do filtering on top of the returned results. So they will call Gemini one more time. They will have a prompt that is doing the verification of the results in comparison to the context that the model already has. (Time 0:27:25)
  • Rerankers Usually Don’t Improve Results
    • Re-ranking returned chunks rarely helped their experiments; models can choose among returned chunks without extra rerankers.
    • Animesh noted they apply a quality cutoff but haven’t seen benefit from reordering results for the model. Transcript: Animesh Chatterji Not so much. In fact, I mean, that just adds more complexity. And if you kind of extrapolate the question that we were talking about a bit earlier about whether even do we need drag, I think if you kind of take that logic and apply it here, that once You give the relevant chunks and as long as your context is not blowing up too much, I think letting the model figure out what is relevant probably is better. Yeah, in terms of like, if you are retrieving too many chunks then we have some threshold on what is the quality score of this chunk right and then we have some threshold below which we Don’t return those to the model but that’s not like re-ranking between chunks it’s just like a vanilla cutoff beyond which we don’t get any more chunks. (Time 0:29:03)
  • Fine-Tuning Is Often Not Worth It
    • DeepMind advises against fine-tuning for most use cases because base models improve too quickly to justify it.
    • Ivan warned fine-tuning benefits are short-lived as new embedding/model versions often outperform custom fine-tuned models within months. Transcript: Ivan Solovyev Have a general recommendation in GDM. And I think we made this a year or so back that people shouldn’t do fine-tuning in most of the cases. The speed of progress of the models is so much faster than what individual smaller labs can do in terms of fine-tuning that it’s almost irrelevant. And by the time you actually have a fine-tuned model, and it will probably perform better for a use case for like a month or two, we’re going to have the next 003 embedding model. It’s going to be better across the board, like 15% on all the benchmarks. And fine tuning won’t be that relevant anymore. With that said, we do see people using fine-tuning in specific use cases. I think it does make sense if your use case is very, very niche and you own a very particular data set, which you don’t expect Google or anyone else to pay attention to anytime in the future. That may yield good results. (Time 0:30:05)
  • Use Multimodal Embeddings And Truncation Tricks
    • Use multimodal embeddings and consider truncated Matryoshka embeddings to trade quality for storage.
    • Animesh described multimodal support and embeddings that can be truncated to reduce storage while retaining reasonable accuracy. Transcript: Animesh Chatterji I mean, we have added the multimodal embedding support now, which would really improve the quality of understanding of things beyond text. So that is one thing. And I think it’s in public preview right now. So we are hoping to do a GA launch for that. The other thing which kind of we launched, I think in the last version of our embedding model or the one before that, I don’t remember, is we started representing embeddings as this Matryoshka Representation, which basically means the embedding vector that is generated. It very easy for the end user to just truncate the embedding so in case like the embedding is it’s a 3k dimension embedding vector and you don’t want to store as much size you could just Truncate it at any point and it would still give you an accurate enough representation of the entity so some of those things have been really helpful and some of those we can actually Explore like we were talking about the knobs that we could give to the user. That could be another knob in future we could give to the users if they want to reduce the size of their storage by using a truncated embedding instead of the full embedding at the cost Of some quality. So based on their use case, users can choose to pick one of the two knobs. (Time 0:32:48)
  • Remaining Challenges Include Multimodal And Structured Data
    • Hard problems remaining: multimodal retrieval, chunking structured data, and multilingual quality.
    • Both guests highlighted tables/graphs parsing, image/video retrieval, and non-English performance as active work areas. Transcript: Animesh Chatterji Think, Ivan, please add on, but like multimodal, I’m sure we’ll kind of add on more capabilities on the multimodal side. That is one thing. We talked about chunking. That’s still an area I feel we can get more benefit out of by kind of capturing the structure better in certain kind of use cases. The multilingual or like the internationalization, that is another aspect. I feel we are getting great at solving English related queries, but there are certain languages where certainly we could do more. And as the user base expands to countries across the world, we have more of these internationalization use cases. Ivan Solovyev So that is another aspect where i feel we can certainly improve upon yeah just in general i think getting the quality higher hit rates better retrieval is something that we always pursue Multi-modal is very interesting aspect text is working great for a lot of use cases but there is a lot more multi-modal use cases that people are like thinking right now. Looking for images, looking through video opens up a lot more consumer products that can be built on top of that. (Time 0:34:07)
  • Migrate By Evaluating Embeddings First
    • To migrate, first test Gemini embeddings in your existing pipeline, then load a small corpus into File Search for side-by-side evaluation.
    • Ivan recommended starting with the embedding model standalone and running evals before full migration. Transcript: Ivan Solovyev Well, migration itself, I hope, is fairly easy and straightforward. You upload your data. But the reality is, I think, what I would recommend them to start with is probably use the embedding model that we provide. It is available as a standalone service. So if they have their own pipeline and they want to run the evals and experiment with the Gemini infrastructure, I would recommend using the embeddings model first with part of their Data and just comparing the results. (Time 0:35:50)
  • Prototype Fast Using AI Studio And Code Samples
    • Try File Search via AI Studio applets and code samples to quickly prototype uploading data and querying with Gemini.
    • Ivan noted File Search is GA for Gemini 2.5 and 3 families and AI Studio has runnable examples that generate code using File Search. Transcript: Ivan Solovyev Absolutely. File search is actually generally available for our 2.5 model family and recently our 3 model family. So 2.5 models are in GA. So it’s both models and the two combination are generally available. Flash 3 and Pro 3 models are still in preview, but file searches as an API is generally available there as well. Sean Falconer Okay. And then what is next besides, you know, moving all the stuff to getting it in the hands of more users? Like what can you share about some of the things that you’re thinking in terms of additional problems to attack or things that you want to continue to make investments around making The product really, really easy to use? Yeah. Ivan Solovyev So as we mentioned, multimodal support is the big push we’re doing. We want to invest into a better understanding of structured data, and we keep collecting these examples from our developers in terms of tables and graphs and whatnot that they’re trying To process. So I think that’s going to improve the quality and the applicability of this a lot. And then latency and being able to work with a much bigger data sets. We do limits at one terabyte right now for the highest tier, but the latency go down quite a bit if you start consuming all of the water. So we want to invest in that as well and improve the retrieval latency. Is that one terabyte of total storage? Sean Falconer Is that right? Ivan Solovyev Yes, that’s one terabyte of total storage, of course, all of your file stores. Sean Falconer Is there a limitation around the size of a single file that can be processed? Animesh Chatterji Other than, I guess it needs to be smaller than a terabyte. It’s 100 MB right now. So that is what we have kept on the file. And for making sure that the retrieval latencies are kind of acceptable and in the good range, we recommend users to keep their individual corpus to about 20 GB. So their total data across corpuses can be one GB, but each corpus or each file search store as we call it should be relevant to 20 GB. And then at the query time, you can provide multiple of these file search store IDs. So we can find out those queries in parallel, but the size of an individual file search store performs good till the limit of more 20G. Sean Falconer Okay. And if I want to play with this, how do I get started? Ivan Solovyev The simplest way, I think, would be to go to AI Studio and play with the file search applets. I think the link is available in the blog post that we published. And the other way is to start hitting the Gemini API. Sean Falconer Okay, great. Animesh Chatterji We have code samples that would make it really easy for somebody to just start playing around, just upload their data using the upload API and hit the Gemini API. Ivan Solovyev Oh, and our wipe coding environments in AI Studio also fully supports the file search. (Time 0:36:38)