Okay, let's run an experiment. 好吧，让我们来做一个实验。
So we come in now to the implementation folder. 现在我们进入实现文件夹。
This is where we're going to live now because all we need to change is either the ingest approach or 这就是我们现在要生活的地方，因为我们需要改变的只是摄取方法或
the answer approach. 答案的方法。
We don't need to change anything else and then rerun evaluation. 我们不需要更改任何其他内容，然后重新运行评估。
So let's go into ingest. 那么让我们进入摄取。
So first thing you often go to is chunking. 所以你经常要做的第一件事就是分块。
Good old chunking that that big old hack. 很好的老分块，那个大老黑客。
So we're currently chunking into chunk size of 1000 characters give or take. 因此，我们目前正在将块大小划分为 1000 个字符的块大小。
Let's let's just try changing that to 500. 我们尝试将其更改为 500。
Let's have smaller chunks. 让我们有更小的块。
Uh, now that means we'll have like twice as many chunks. 呃，现在这意味着我们将拥有两倍的块。
Now, when we do this, uh, it's worth remembering that we're surfacing five chunks each time. 现在，当我们这样做时，呃，值得记住的是，我们每次都会显示五个块。
And it seems to me that if we're going to have smaller chunks, it's only fair if we surface more chunks. 在我看来，如果我们要拥有更小的块，那么只有我们表面上有更多的块才公平。
Uh, otherwise there's less context overall. 呃，否则整体上下文就更少了。
So. 所以。
So if we go to answer pie at the top here, retrieval k is five. 因此，如果我们去回答顶部的饼图，检索 k 就是 5。
That's saying that we always try and and collect five relevant documents you see right here. 这就是说，我们总是尝试收集您在此处看到的五份相关文件。
K is five when we call retriever dot invoke. 当我们调用检索器点调用时，K 为 5。
So let's make that ten. 所以让我们把这个数字定为十。
So instead of five documents of 1000 characters we're now going to surface ten documents of 500 characters. 因此，我们现在将显示 10 个 500 个字符的文档，而不是 5 个 1000 个字符的文档。
That seems fair. 这看起来很公平。
Uh, so now same amount of context will use up in both cases. 呃，现在这两种情况都会用掉相同数量的上下文。
All right. 好的。
That's good. 那挺好的。
Let's let's go and test this out. 让我们来测试一下。
Okay. 好的。
So I'm going to bring up a new terminal. 所以我要打开一个新的终端。
I'm going to go into week five. 我将进入第五周。
I'm going to go into implementation. 我要进入实施阶段。
And then I'm going to do. 然后我就去做。
You've run ingest dot pi. 您已经运行了摄取 dot pi。
Remember this rebuilds the the vectors. 请记住，这会重建向量。
Uh re-ingest. 呃，重新摄取。
So before we had 400 and something. 所以之前我们有 400 之类的。
So now we're expecting 8 or 900 or so vectors to be created and put in our data store. 所以现在我们期望创建 8 或 900 个左右的向量并将其放入我们的数据存储中。
And there we go. 我们就这样吧。
970 vectors. 970 个向量。
Because there's overlap that you would expect there to be more than double. 因为存在重叠，您预计会有超过一倍的重叠。
Uh, so so there we have it. 呃，所以我们就有了。
Um, lots of vectors, 970 of them. 嗯，很多矢量，970 个。
Now let's go and rerun the evaluation now. 现在让我们重新运行评估。
Actually, if you've got your screen still running from before, you have to to quit that and create 实际上，如果你的屏幕仍然在运行之前，你必须退出并创建
a new one because we've regenerated the database. 一个新的，因为我们已经重新生成了数据库。
So you will need to to launch a new Gradio app. 因此，您需要启动一个新的 Gradio 应用程序。
So let's go and do that right now. 那么我们现在就开始吧。
And then let's, let's run an eval okay. 然后让我们运行一次评估吧。
So I do CD up up I go up to week five and now I do you've run evaluator.py. 所以我做 CD 向上，直到第五周，现在我做你已经运行了 evaluator.py。
It's going to bring up our evaluator. 它将调出我们的评估器。
The MRI that we're comparing with is 0.7298. 我们所比较的 MRI 为 0.7298。
That's what we have to beat. 这就是我们必须击败的。
Let's press run evaluation. 让我们按运行评估。
Let's see what we get. 让我们看看我们得到了什么。
Drumroll please. 请击鼓。
And we do do better. 我们确实做得更好。
Smaller chunks mean that we do better. 更小的块意味着我们做得更好。
Uh interesting. 呃有趣。
Okay. 好的。
Let's let's take a note of that. 让我们记下这一点。
It's all all gone, gone yellow. 一切都消失了，变黄了。
Uh, I'm just just for to keep us focused on one thing. 呃，我只是为了让我们专注于一件事。
Both. 两个都。
Both MRI and all three metrics have improved. MRI 和所有三个指标均有所改善。
We've gone right into the green with keyword coverage with more smaller chunks. 我们已经通过更小块的关键字覆盖率进入了绿色阶段。
Let's we're going to try bigger chunks. 让我们尝试更大的块。
But let me take a note of this. 但让我记下这一点。
So this is with 500 I'm keeping a little note tab. 所以对于 500，我保留了一个小注释标签。
And so should you try and try and write this down so we don't lose track. 所以你应该尝试把它写下来，这样我们就不会迷失方向。
Because if we're going to be scientific we might as well be scientific. 因为如果我们要科学化，我们不妨也科学化。
Okay. 好的。
Now let's go and make bigger chunks and try again okay. 现在让我们去制作更大的块，然后再试一次好吗。
So I'm going to I'm going to close that. 所以我要结束它。
I'm now going to go back to ingest I am going to uh change the chunk size. 我现在要返回摄取，我要呃改变块大小。
We're going to go up and I'm not going to double its 2000, I'm going to go to like one, six, six, 我们要上升，我不会将 2000 翻倍，我会上升到一、六、六，
seven. 七。
And you're like, why? 你会想，为什么？
Why are you doing that? 你为什么要这么做？
Uh, why such a weird number? 呃，为什么这么奇怪的数字？
Because I want to stay true to filling up the same amount of context space I'm going to go to to answer 因为我想忠实地填充我要回答的相同数量的上下文空间
dot pi and say, let's go down to three, K is three, because three of one, six, six, seven will 点 pi 并说，让我们减到三，K 是三，因为一、六、六、七中的三将
come to the same number to, to uh, to 10,000, which means sorry to 5000. 达到相同的数字到，呃，到 10,000，这意味着对不起 5000。
Uh, is that right? 呃，是吗？
Yes. 是的。
Three times one six, six, six seven 6667 would be 5000. 三乘一六、六、六七 6667 就是 5000。
So we are now filling up the same amount of context. 所以我们现在正在填充相同数量的上下文。
So basically it's three bigger chunks versus the five chunks that we had to start with versus the ten 所以基本上它是三个更大的块与我们必须开始的五个块与十个块
chunks of much smaller. 块小得多。
So we've got consistency there. 所以我们在那里保持了一致性。
Okay. 好的。
So we're comparing apples and apples. 所以我们正在比较苹果和苹果。
All right. 好的。
We need to rerun ingest CBD implementation run ingest dot pi. 我们需要重新运行 ingest CBD 实现 run ingest dot pi。
Off it goes. 关了就走了。
Let it do its thing. 让它做它该做的事。
We're expecting fewer much fewer now uh 235 vectors okay. 我们预计现在呃 235 个向量会更少。
And now we're going to go to uh upper directory to the parent directory. 现在我们要转到上层目录到父目录。
And we're going to do you've run evaluator dot pi. 我们将让您运行评估器 dot pi。
And it will bring up a fresh interface for us to to run with. 它将为我们带来一个新的运行界面。
And we'll see what you have to make a bet. 我们会看看你必须下注什么。
What do you think. 你怎么认为。
Better or worse, better or worse. 好还是坏，好还是坏。
We'll soon find out. 我们很快就会知道。
Running evaluation. 运行评价。
And we find it is interesting. 我们发现这很有趣。
How about that? 怎么样？
Were you expecting that? 你期待吗？
Because I wasn't. 因为我不是。
It's. 它是。
I was thinking it was going to be worse, but it's not. 我以为情况会更糟，但事实并非如此。
It's slightly better. 稍微好一点了。
So let's review. 那么我们来回顾一下。
So having a chunk size of, um, uh, of one six, six seven, um, is, is uh, .7475. 因此，块大小为，嗯，呃，一六，六七，嗯，是，是呃，0.7475。
That is the MRI, which is better. 那是MRI，效果更好。
So having a chunk size of 1667, a bigger chunk size is better than a chunk size of 1000, and having 因此，块大小为 1667 时，较大的块大小优于块大小 1000，并且具有
a chunk size of 500 is better than either of them. 500 的块大小比它们中的任何一个都要好。
That's best of all. 这是最好的。
So there's clearly more experiments to be had here. 所以这里显然还有更多的实验要做。
You can get better numbers by continuing to experiment with it, and it's important to try and keep 通过继续试验，您可以获得更好的数字，并且尝试并保持这一点很重要
keep all things being equal. 保持一切平等。
Assume that you want to try and keep roughly the same amount of information in the context window, 假设您想尝试在上下文窗口中保留大致相同数量的信息，
so that you're only varying one parameter. 这样你就只改变一个参数。
If you're willing to have more in the context window, that that would change the picture a bit. 如果您愿意在上下文窗口中显示更多内容，那么情况会有所改变。
All right. 好的。
Well, that was interesting. 嗯，这很有趣。
You should do many more experiments. 你应该做更多的实验。
This stuff is fascinating. 这东西很有趣。
But now we're going to try different kinds of of chunking strategy. 但现在我们将尝试不同类型的分块策略。
Now one of the reasons people love Lang chain, and I totally see it is that it comes with so much stuff. 现在人们喜欢郎链的原因之一，我完全明白它是因为它配备了这么多的东西。
It's a real, what they call a batteries included kind of framework that has lots of things that come 这是一个真正的，他们所谓的包含电池的框架，有很多东西
along with the mix, and one of them is a ton of different types of splitter things that can divide 随着混合，其中之一是大量不同类型的分离器，可以将
up your chunks in different ways. 以不同的方式增加你的块。
And in our case, our knowledge base is actually made up of markdown documents. 在我们的例子中，我们的知识库实际上是由 Markdown 文档组成的。
And it turns out, would you believe it, that there is a markdown text splitter that is designed for 事实证明，你相信吗，有一个 Markdown 文本分割器是为
splitting markdown documents? 拆分 Markdown 文档？
There's actually two of them. 实际上有两个。
There's a markdown text splitter that just figures it out, and there's one called markdown header text 有一个 Markdown 文本分割器可以解决这个问题，还有一个叫做 Markdown 标题文本
splitter when you specify how you want it to do it. splitter 当您指定您希望它如何执行操作时。
But we're going to go with a simple one. 但我们将选择一个简单的。
So basically we find our recursive character text splitter. 所以基本上我们找到了递归字符文本分割器。
And we will comment this line out. 我们将注释掉这一行。
And instead of that we will say text splitter is look at that. 相反，我们会说文本分割器是看那个。
It's filled it in for us. 它已经帮我们填好了。
But I don't think you even need to specify this. 但我认为你甚至不需要指定这一点。
I think we can just do it like that and it will just figure it out itself. 我认为我们可以这样做，它会自己解决的。
That's that's how good it is. 就是这样就好了。
So let's see if I'm right. 那么让我们看看我是否正确。
Let's do UV run ingest.py, uh, and we'll see if that just runs and just does some, some documents 让我们运行 UV run ingest.py，呃，我们会看看它是否只运行并且只执行一些、一些文档
splitting. 分裂。
It's done it 109 vectors. 它已经完成了 109 个向量。
Let's go test them. 我们去测试一下他们吧。
And I should say that I'm going to leave this uh answered py with k is three because these are clearly 我应该说我要离开这个呃回答 py k 是 3 因为这些显然是
bigger chunks with fewer vectors. 用更少的向量得到更大的块。
So just to roughly try and have apples and apples will keep retrieval k at three. 因此，只要粗略地尝试吃苹果，就会将检索 k 保持为 3。
And we will now go into the parent directory and do uv run evaluator.py evaluator.py. 现在我们将进入父目录并执行 uv run evaluator.py evaluator.py。
Up comes our evaluation framework. 我们的评估框架就出来了。
And we'll see whether this markdown splitter does a better job or not. 我们将看看这个降价分割器是否做得更好。
Here it is. 这里是。
Run a valuation. 进行评估。
Off it goes. 关了就走了。
And we get 0.738 is aha. 我们得到 0.738 啊哈。
So it does better than our original MRR when we had simple character splitting at the 1000 chunks. 因此，当我们在 1000 个块上进行简单的字符分割时，它比我们原来的 MRR 效果更好。
But it's worse than either of our of our subsequent ones. 但这比我们后续的任何一个都更糟糕。
So for whatever reason going with the the markdown splitter has not worked out well. 因此，无论出于何种原因，使用 markdown splitter 效果并不好。
And it's not for whatever reason we can tell exactly what's going on. 无论出于何种原因，我们都无法确切地知道发生了什么。
What's happening is that when we just focus on the headings now, our chunks are way too big. 发生的情况是，当我们现在只关注标题时，我们的块太大了。
Uh, and so that's, that's, uh, that's why we're seeing, uh, performance suffering there. 呃，所以，这就是，呃，这就是为什么我们看到，呃，性能受到影响。
But it's nice to know that we've got this fancy markdown splitter, should we wish it. 但很高兴知道我们已经有了这个奇特的降价分割器，如果我们愿意的话。
Okay. 好的。
Are you getting bored of experiments now? 你现在对实验感到厌倦了吗？
Have you had enough? 你玩够了吗？
Well, I haven't. 嗯，我没有。
I'm only just getting started. 我才刚刚开始。
I. 我。
One more, one more set of experiments to do. 还有一组实验要做。
This is fun. 这很有趣。
This is what. 这是什么。
This is what engineering is all about. 这就是工程学的全部内容。
It's about the experiments. 这是关于实验的。
So, uh, I have reset everything back to the chunk size of 500. 所以，呃，我已将所有内容重置回块大小 500。
The recursive character text splitter. 递归字符文本分割器。
That's our best result so far. 这是我们迄今为止最好的结果。
We're now going to change the embedding model, the encoder. 我们现在要更改嵌入模型，即编码器。
So we've been using the all mini Ml6 v2. 所以我们一直在使用all mini Ml6 v2。
So instead of that let's import OpenAI embeddings. 因此，让我们导入 OpenAI 嵌入。
And let's do embeddings. 让我们进行嵌入。
Is OpenAI embeddings. 是 OpenAI 嵌入。
We'll use as the model the small the text embedding three small. 我们将使用嵌入三个小文本的小文本作为模型。
And I will comment that out. 我会对此发表评论。
And notice one little tiny thing here. 请注意这里的一件小事。
If you do this, be sure that you do this after you've loaded your dot EMV so that your OpenAI key is 如果您这样做，请确保在加载点 EMV 后执行此操作，以便您的 OpenAI 密钥
set up. 设置。
Okay. 好的。
Copy that. 复制那个。
Let's go over to answer dot pi. 让我们过去回答点pi。
We also have to change the embeddings here as well. 我们还必须更改此处的嵌入。
There we go. 我们开始吧。
I've done that. 我已经这么做了。
We might as well comment out the line above it and make sure that we are importing. 我们不妨注释掉上面的行并确保我们正在导入。
Yes we are importing it right up here. 是的，我们正在这里导入它。
Excellent. 出色的。
We're in good shape. 我们的状态很好。
That's all it should take. 这就是所需要的一切。
We should now be able to try using OpenAI's embeddings model. 我们现在应该能够尝试使用 OpenAI 的嵌入模型。