I know it feels like we've been on this slide forever, but it's such an important slide.
Ten Juicy techniques.
Number six.
Query expansion.
This is this is one that a lot of people use.
This is just saying.
Oh, okay.
Look, um, um, we could we could, uh, add to step five.
Don't just ask the model to write one query.
Ask it to write a bunch of them, a bunch of things that we could call our database lookup for.
Let's have three different Rag queries and collect similar documents for each.
That's called query expansion.
Super common.
And then Reranking Reranking is saying we might have ended up with lots of different chunks from our
data store, particularly if we've done number six query expansion, we might have a lot.
We will have people like to say we have k times n, you have k the, the, the, the amount that you
retrieve every time times n the number of queries you've got a potentially a bunch of chunks.
Well, we could then take all of these chunks and try and reorder them in terms of the ones that's most
relevant for the question at the top and all the way down to the least relevant for the question.
And how are we going to do that?
Well, we could have an LLM do that.
We could call an LLM and say, you're not responsible for answering the question.
Someone else is going to do that.
You're responsible simply for ordering these chunks in order of most important to least important.
Do that step.
And then potentially we might chop off some of the unimportant ones when we finally send it to an LLM
to actually craft the answer.
So that is reranking.
Again, super popular.
And you can imagine, particularly if you've done some of the previous steps, Reranking can be really
important to avoid polluting the context with tons and tons of chunks.
Okay.
Number eight, we're on the home stretch hierarchical rag.
This is trying to fix a particular Achilles heel with rag that I mentioned before, which is that rag
is really bad at answering questions which span lots of documents.
It has a really hard time with that.
And so there are some, pretty simple, uh, hacks that try and get around this.
And one of them is to summarize your knowledge base at various levels of roll up, like a hierarchy,
have a kind of a summary of all products, a summary of all employees.
And then when you do your Rag lookup, do a Rag lookup across the summaries first to get the kind of
coarse grained information and then drill down to to chunks that come from the fine grained information.
And that means that if, for example, you could have an employee summary document that lists all employees
and their salaries in one condensed document.
And that way, if someone asks a question like how many employees have a salary of under X, then hopefully
that chunk with this compressed data, this summarized data about salaries would be surfaced.
And so the the the Rag, the LLM would have a hope of answering it otherwise.
Without that, it's impossible to answer.
Uh, how many of kind of question because it would require chunks from so much of the knowledge base.
So that's a pro move hierarchical rag.
And you can probably tell like it is very hand-wavy, very hacky because you could roll up in one way,
but that might not be aligned with the question that someone asked.
They might ask a question that's like how many documents starting with a letter A do you have?
And obviously you haven't you haven't figured that one out.
You haven't put that in in some hierarchy, uh, by by alphabet.
So, so, you know, it's it's always a whack a mole.
It's always someone asks a question that spans your documents.
You don't have a good answer to it.
So you think about a new hierarchical summary document that you could slip in.
And that's that's the point of hierarchical rag okay.
And we end with two fancy ones that people love to talk about.
One of them is Graph Rag.
Graph rag.
It's actually it's it sounds really, really fancy, but it's a fairly simple idea, which is that often
the documents in your knowledge base aren't isolated.
They have some relationship with each other.
So like an employee might might have a boss, a manager, which is another employee.
So you could really think that that your documents frequently or even your, your chunks are related
to other chunks.
And when you're doing your your rag query, if, for example, you remember there's a metadata associated
with each chunk, if we'd been careful in that metadata to specify other chunks that are related could
be as simple as other chunks in the same document, or it could be something like the chunk associated
with a boss or something like that.
Then.
Then when you look up that chunk, you could also then shove in the context the related chunks that
are kind of one hop or two hops away in the in terms of these relationships that you've recorded in
the metadata.
And of course, taking it a step further, as you probably guessed, I was going to say there are special
databases called graph databases like Neo4j's and many of them that are specially designed to store
Information as as nodes with edges.
A way to think about things in terms of relationships between entities.
And if you store it in a graph database like that, then it's even easier for you to find a chunk that's
close to a vector, and then include that and include all the chunks that are the sort of 1 or 2 neighbors
away in the graph database.
So it's a trendy thing to do.
It tends to work only in reasonably.
It tends to work well only in quite specific situations where your data does have a lot of relationships
between it.
And I would argue that most of the time you can handle this just by using the metadata.
The need for a graph database, trendy though they are, are not as huge unless you very specifically
got a problem that lends itself well to that kind of relationship based database.
So definitely one to watch and potentially experiment with.
Uh, but but uh, yeah, it's it's quite a fancy one.
And then last but not least is a gentic rag.
You may have heard of it.
What is this?
Well, it's it's a different way of thinking about it.
It's much the same stuff.
But what it's saying is, look, we've designed this, this very linear idea that the user asks a question.
Then we go and do a vector based lookup.
And we always shove that in the context.
And then we call the LLM.
Well that's a bit you know 2024 whatever.
It's you know, things in ancient history is 2024 these days.
Uh, it's a bit 2024, uh, that nowadays the way that we like to, to think about things is, is let
the LLM make the decisions.
So instead of just doing a vector lookup and putting all that in the context, call an LLM and give
it some tools, give it a tool that could do a vector lookup based on a query, maybe give it some more
tools, like a SQL tool that can run SQL on a database, or if it's the files that we've got in our
knowledge base, just do a string lookup in the files if it wants to give it access to a bunch of tools
and then call this like your retrieval agent or something, and then say, here's a user question.
Go to town, go and figure this out.
You can come up with your queries, you can make your vector lookups.
And you can also do a SQL query.
You can also look for ticket prices by by calling this API.
You can do all these different things and use this as your way of retrieving context.
Now in large part it is it is equivalent to some of the steps before, uh, if you just give it a vector
database then it's going to call back.
It's very similar you can imagine to doing query expansion, query rewriting and then query expansion,
and then it's running the queries and then it's basically reranking because it's going to figure out
which ones matter.
And that's what will eventually be used to answer the question.
So it's doing the ones before number ten.
Number ten is about doing the ones before it, but letting the LM decide which to do and in which order.
And potentially it can be in a loop so that if it does a few of them and it's not getting results,
it can keep going.
So a genetic rack has all of the great things and some of the caveats about working with this kind of
more, more flexible approach.
It can be very powerful and get answers where you've not been able to get them before, because it can
keep working at it.
Keep experimenting until the model decides to stop making tool calls and then it satisfies.
But also it's of course, less predictable.
It's harder.
You can put metrics around it and measure it and get good results, but then you could run the same
thing tomorrow and get bad results and that would be horrible.
So, you know, it has the downsides in terms of repeatability, robustness and so on that you often
find with these more what we call autonomous kind of kind of approaches.
And if you're interested in more about this then of course my agenda course covers some of this.
In the last week we have a graph based uh, memory, along with uh, also a vector based memory as well.
We cover that a few times that that is the sort of that the new agentic approach to doing the same thing.
But but very much keep in mind that it is still in large part carrying out many of the previous steps
on this list, just doing it through tool calling rather than through R code orchestrating step by step.
Finally, I will end this slide.
I know I've been on this slide for 20 minutes.
I'm going to end this by saying a lot of people are saying that rag is dead.
You hear that a lot.
You see lots of memes on this.
And there's typically two different reasons for this.
People come at it from two directions.
One of them is that rag is dead because context windows are so big.
Now that you can put all your knowledge base in a context window and let the transformer use attention
to figure out what's what's relevant, that's surely the best.
That's one reason people say that rag is dead.
And I think that's that's, uh, that's nonsense.
Because, uh, sure, you could say that, but but clearly there are going to be cases when your knowledge
base is much, much, much bigger, like in week eight of this course, when we're going to have a monstrously
large knowledge base.
And okay, maybe you could put all that into an LLM call, or maybe you could carve it up and make many
LLM calls, one after another and then another one to judge it.
But wow, that's not going to be an efficient use of time.
At the very least you could get down to you could throw out 90% very easily with some kind of a vector
based search or any other, uh, on this list.
So it just seems to me that however big the context window gets, this, this kind of approach to try
and remove irrelevant context is always going to be important.
But the other reason people say that rag is dead is because of number ten, because of a genetic rag.
Now that we have agents, this kind of of pipeline where you do a vector based lookup and then you surface
relevant content, that approach is kind of old school.
The new approach is to equip an agent with the ability to, uh, to itself decide how to dig into the
data to do the research itself using any of these techniques and figure out how best to retrieve relevant
context and then answer the question.
That's a super interesting point.
But my answer to that would be, uh, predictably enough, you knew I was going to say long live, right?
It's such a familiar trope.
Uh, but the reason I say long live is that come on, that's just rag by a different name.
Sure, you could call it a rag and sound like you're so modern, so fashionable.
But but okay, it's still vector based or encoder based retrieval of relevant context.
Or maybe you're using a graph database.
Whatever you're doing, you're using techniques to retrieval techniques to augment your generation from
an LLM.
And so it's still retrieval augmented generation.
It's still rag even if it's rag, whatever it is, by whatever name you choose to call it, as far as
I'm concerned, that's still rag.
Long live rag.