Okay, so it's time to evaluate who did best and what better way to evaluate who did what best than 好的，现在是时候评估谁做得最好了，还有什么更好的方法来评估谁做得最好？
to use an LM for this task. 使用 LM 来完成此任务。
It would be tedious to have to read through all of these and decide. 必须阅读所有这些内容并做出决定，这将是乏味的。
First, let's just understand where we are. 首先，让我们了解一下我们现在所处的位置。
So we had two lists. 所以我们有两个清单。
One was called competitors and one was called answers. 一种称为竞争对手，一种称为答案。
I love the way cursor does that. 我喜欢光标的方式。
And so we can just print them out and we'll see that we've got these two lists. 所以我们可以将它们打印出来，我们会看到我们已经得到了这两个列表。
There they are. 他们就在那里。
These are our competitors. 这些是我们的竞争对手。
I hope you recognize them. 我希望你能认出他们。
And these are their answers. 这些就是他们的答案。
So one one thing that's nice to do right now, it would be sort of nicer to be able to pair these up 所以现在要做的一件事很好，如果能够将它们配对起来会更好
and say which one is which. 并说出哪个是哪个。
There's a really useful Python function called called zip that is worth knowing about. 有一个非常有用的 Python 函数，名为 zip，值得了解。
It's like a pro thing to use, and you can iterate through these two collections together like this 它就像一个专业的东西使用，你可以像这样一起迭代这两个集合
I can say for look haha, what am I doing here? 我可以说看看哈哈，我在这里做什么？
I just press tab. 我只是按 Tab 键。
You can say for competitor and answer. 您可以为竞争对手说出并回答。
Those are the two we want to be iterating through competitors and answers in. 这是我们希望通过竞争对手和答案进行迭代的两个问题。
And if you say zip competitors answers, then it will just iterate through through these two lists together. 如果您说 zip 竞争对手的答案，那么它只会一起迭代这两个列表。
And competitor and answer will have each one. 竞争对手和答案将各有其一。
And it's written all the code. 并且它已经写好了所有的代码。
For me, that's exactly what I wanted to do. 对我来说，这正是我想做的。
So if I do this now, you'll see that it's printing each competitor and its answer one after another. 因此，如果我现在这样做，您会看到它正在依次打印每个竞争对手及其答案。
Very nice. 很不错。
Okay, but let's bring it together into one string called together. 好的，但是让我们将它们组合成一个字符串，称为 Together。
And I also wanted to show another construct here called enumerate. 我还想在这里展示另一个称为枚举的结构。
If you have a list like answers, a list of things and you want to iterate through them, but you also 如果您有一个类似答案的列表，一个事物列表，并且您想要迭代它们，但您也
want to track the index number as you do, you probably like people that don't know about enumerate 想要像你一样跟踪索引号，你可能喜欢不了解枚举的人
often have a kind of count equal zero. 通常有一种计数等于零。
Count plus equals one in your loop. 在循环中计数加等于一。
Not needed. 不需要。
Enumerate is a nice little trick you can say for index, comma, answer in enumerate answers, and then 枚举是一个很好的小技巧，你可以用索引、逗号、枚举答案中的答案来表示，然后
you will just step through. 您只需迈出一步即可。
And so we will say response from competitor. 所以我们会说竞争对手的回应。
And we're going to add one to the index so that the first one is is competitor number one rather than 我们将在索引中添加一个，以便第一个是第一名竞争对手，而不是
competitor number zero, which just doesn't sound as nice. 零号竞争对手，这听起来不太好。
So we run that and then uh, why don't I just just show you what that looks like? 所以我们运行它，然后呃，我为什么不直接向你展示它是什么样子呢？
But I'm sure you believe me. 但我确信你相信我。
It looks like this. 看起来像这样。
And if I print it, we'll see it looking a bit more pleasant than that. 如果我打印它，我们会发现它看起来比那更令人愉快。
Here you go. 干得好。
So you can see response. 这样你就可以看到响应。
Oh, look at that. 哦，看看那个。
There's a mistake. 有一个错误。
Just as well. 也罢。
I did print it, isn't it? 我确实打印了，不是吗？
That's the benefit of printing these things. 这就是打印这些东西的好处。
You do that, then you get to see, uh, see what's going on. 你这样做，然后你就会看到，呃，看看发生了什么。
Wow. 哇。
Okay. 好的。
I hope you you caught that. 我希望你能抓住这一点。
That's embarrassing. 那就尴尬了。
But I think I'll keep it. 但我想我会保留它。
So you get to see that, uh, that not only can one make mistakes, but this gives you a great way to 所以你会发现，呃，人们不仅会犯错误，而且这为你提供了一种很好的方法
inspect what's going on and see what's happening. 检查正在发生的事情并看看发生了什么。
So when I print that now, we see response from competitor one. 因此，当我现在打印该内容时，我们会看到竞争对手一的回应。
And presumably down below it will have response from competitor two. 大概在下面它会得到第二个竞争对手的回应。
All right. 好的。
So now we're going to have a new bit of text. 现在我们将有一段新的文本。
You are judging a competition between that many competitors. 你正在评判这么多竞争对手之间的竞争。
Each model has been given this question. 每个模型都被提出了这个问题。
Your job is to evaluate each response. 您的工作是评估每个回复。
And there is the. 还有就是。
We say we want you to respond in JSON and only in JSON and follow this format a JSON with results and 我们说我们希望您以 JSON 格式且仅以 JSON 格式进行响应，并遵循此格式的 JSON 以及结果和
then a competitor number, a second best competitor number, third best competitor number, and so on. 然后是竞争对手编号、第二最佳竞争对手编号、第三最佳竞争对手编号，依此类推。
And here are results from the competitors. 这是竞争对手的结果。
And now respond with the JSON with the ranked order of the competitors. 现在用 JSON 响应竞争对手的排名顺序。
Nothing else. 没有别的了。
Do not include number formatting or code blocks. 请勿包含数字格式或代码块。
So just a few things to point out about this. 关于这一点，有几点需要指出。
First of all, you may not be familiar with the triple quotes. 首先，你可能对三重引号不熟悉。
If you used triple quotes in a Python string, you get to have an entire block of text without needing 如果在 Python 字符串中使用三引号，则无需使用三引号即可获得整个文本块
lots of quote marks and pluses and things. 很多引号和优点之类的东西。
So this is a nice trick for a block of text. 所以对于文本块来说这是一个很好的技巧。
It's also, of course used for for for docstrings, so I'm sure you have come across it before. 当然，它也用于 for for 文档字符串，所以我相信您以前已经遇到过它。
Another thing to know is that I've sort of assumed you're familiar with f strings before, but one nice 另一件需要知道的事情是，我假设您以前熟悉 f 字符串，但有一个很好的
trick to know is that if you actually want to have a curly brace within your string, then you can do 要知道的技巧是，如果您确实想在字符串中包含大括号，那么您可以这样做
that by having two curly braces in an F string. 通过在 F 字符串中包含两个花括号来实现。
So that's why there are two curly braces here, because we actually want one curly brace to appear in 这就是为什么这里有两个大括号，因为我们实际上希望一个大括号出现在
the string itself. 字符串本身。
We're not. 我们不是。
We don't want we don't want this to be interpreted as code. 我们不希望我们不希望它被解释为代码。
All right. 好的。
So with that, if I print this to to show you what on earth I'm prattling away about, what you should 因此，如果我打印这个来告诉你我到底在喋喋不休，你应该做什么
see is you're judging a competition between six competitors. 看看您正在评判六名参赛者之间的比赛。
Each model has been given this question. 每个模型都被提出了这个问题。
Your job is to evaluate. 你的工作是评估。
Here is the how to give us the results. 以下是如何向我们提供结果。
And here are the responses from each. 以下是每个人的回应。
And there you see it in there I'm also going to mention you see at the end here I say do not include 你在那里看到了，我还要提到你在最后看到的，我说不包括
markdown formatting or code blocks. Markdown 格式或代码块。
It's always worth doing that. 这样做总是值得的。
Otherwise these models love to add in a little extra JSON tag around things. 否则，这些模型喜欢在事物周围添加一些额外的 JSON 标签。
So if you use this text then you make sure you get pure JSON back. 因此，如果您使用此文本，那么您将确保获得纯 JSON。
Okay, so now that we've got this all set up, we're going to put this into messages. 好的，现在我们已经完成了所有设置，我们将把它放入消息中。
And then we're going to call an LM and get our results. 然后我们将调用 LM 并获取结果。
Let's do that. 让我们这样做吧。
Okay. 好的。
It's judgment time. 审判时间到了。
So we're going to put this judge text into a messages list as usual with the normal structure. 因此，我们将像往常一样将这个判断文本放入具有正常结构的消息列表中。
And now let's choose a judge. 现在让我们选择一位法官。
So I'm picking O3 mini a little bit pricier I think than some of the others. 因此，我认为 O3 mini 比其他一些产品贵一点。
So they should check the vellum leaderboard. 所以他们应该检查牛皮纸排行榜。
And you may want to just go with a with a cheap one. 您可能只想选择便宜的。
Like like like using GPT four or mini. 就像使用 GPT 4 或 mini 一样。
I'm going to go with O3 mini because we're going to try and have something that really pays attention 我会选择 O3 mini，因为我们会尝试做一些真正引人注目的东西
to this. 对此。
Now, I realized the first competitor in the list is GPT four mini, and so it's a bit -- that 现在，我意识到列表中的第一个竞争对手是 GPT 4 mini，所以它有点 - 那
some of the same OpenAI models are judging themselves. 一些相同的 OpenAI 模型正在自我评价。
And it'll be interesting to see whether it thinks that the model from its own family is the winner, 看看它是否认为来自自己家族的模型是赢家将会很有趣，
and we're not telling it the model names, we're only giving it a rank number, so it's not going to 我们不会告诉它型号名称，我们只会给它一个排名号码，所以它不会
know. 知道。
So we'll see how number one fares. 那么我们来看看第一名的票价如何。
Actually, the number one's answer did look really good. 事实上，一号的答案看起来确实很好。
So it might do very well. 所以它可能会做得很好。
We will see. 我们会看到。
But we're letting O3 mini make the call. 但我们让 O3 mini 来决定。
Let's see how it does. 让我们看看它是如何实现的。
Off it goes. 关了就走了。
It's thinking about it. 它正在考虑这个问题。
It's taking its time. 这需要时间。
It's doing some reasoning. 它正在做一些推理。
And here's the answer. 这就是答案。
So it doesn't put the uh GPT four mini at the top. 所以它并没有把呃 GPT 4 mini 放在首位。
It puts model number three. 它把型号放在第三位。
And you probably remember AI model number two was uh, was anthropic model number three, I forget. 你可能还记得第二号人工智能模型是呃，第三号人类模型，我忘了。
Was it deep league? 这是深度联盟吗？
We'll soon find out. 我们很快就会知道。
So we are now going to, uh, load this, which has come back in perfect JSON just as we wanted. 所以我们现在要，呃，加载这个，它已经按照我们想要的那样以完美的 JSON 形式返回。
We're going to load it into a dictionary. 我们将把它加载到字典中。
We're going to pluck out results. 我们将提取结果。
So now we're going to have this. 所以现在我们要得到这个。
We're going to iterate through it using again this enumerate approach. 我们将再次使用这种枚举方法来迭代它。
For each one we're going to look at this string And subtract one and look it up in the competitors list 对于每一个，我们都会查看这个字符串，然后减去一，然后在竞争对手列表中查找
and print the results. 并打印结果。
So if you're following me, this is just going to print out from best to worst the names of the models 所以如果你关注我的话，这只会从最好到最差打印出模型的名称
in our results. 在我们的结果中。
Drum roll please. 请打鼓。
Here we go. 开始了。
So it was Gemini two zero flash. 所以这是双子座二零闪。
That is our winner. 那是我们的赢家。
And GPT four mini came second. GPT 4 mini 排名第二。
Llama 3.3 came third and then Deep Seek and then Claude 3.7. Llama 3.3 位居第三，然后是 Deep Seek，然后是 Claude 3.7。
But bottom of the list was the flailing llama 3.2. 但排名垫底的是“flailing llama 3.2”。
That kind of gave up halfway. 就这样半途而废了。
So there are the unscientific results of our judgment. 所以出现了我们判断的不科学的结果。
Of course, it will be great fun for you to try this in a more scientific way. 当然，如果你能够以更科学的方式尝试这一点，将会非常有趣。
For example, you could have each of the different competitors come up with the rankings and then use 例如，您可以让每个不同的竞争对手给出排名，然后使用
that to take averages. 取平均值。
That kind of thing would be rather more interesting way of assessing it than just simply going with 这种评估方式比简单地使用更有趣
whatever oh three mini tells us. 无论《哦三迷你》告诉我们什么。
So anyway, this was a really interesting experiment designed to show you how you can collaborate between 所以无论如何，这是一个非常有趣的实验，旨在向您展示如何在
LMS. 学习管理系统。
So hopefully you were paying attention during this and the last lecture, and you've been able to identify 所以希望你在这堂课和上一堂课中都集中注意力，并且你已经能够识别出
which agentic workflow patterns were used in this example. 本示例中使用了哪些代理工作流程模式。
Pattern or patterns? 图案还是图案？
There might be a couple. 可能有一对。
If you're not sure, then go back through it and have a think and look back at the diagrams. 如果您不确定，请回顾一下并思考并回顾图表。
And it might be some sort of hybrid or mish mash between a couple of them and the exercise for you. 它可能是其中几个与适合您的练习之间的某种混合体或大杂烩。
First of all, do that. 首先，这样做。
And then secondly, please pick one of the patterns that interests you and add that to this mix. 其次，请选择您感兴趣的模式之一并将其添加到此组合中。
The goal of this lab was was twofold. 该实验室的目标是双重的。
First of all, to experiment with different APIs and see for yourself how the OpenAI API is used so 首先，尝试不同的 API，亲眼看看 OpenAI API 是如何使用的
frequently across many models, except anthropic, and also to experiment with with the sort of the 经常跨越许多模型，除了人择模型，并且还尝试使用
basic API structure and the prompts. 基本 API 结构和提示。
And the second goal was to experiment with this orchestration between models, asking a question, having 第二个目标是试验模型之间的这种编排，提出问题，
multiple models answer, getting another model to assess the output all this stuff is about interactions 多个模型回答，让另一个模型来评估输出所有这些都是关于交互的
between models, dividing up a bigger problem into smaller problems, and a little bit of autonomy in 在模型之间，将一个更大的问题划分为更小的问题，并且在模型中具有一点自主权
terms of coming up with whatever question the first model wished to. 提出第一个模型想要提出的任何问题。
So the, uh, yeah, add another agentic design pattern. 所以，呃，是的，添加另一种代理设计模式。
I'd love to see it. 我很想看看。
And then once you've done that, consider doing a PR and putting it in community contributions. 一旦你做到了这一点，就可以考虑做一个 PR 并将其放入社区贡献中。
There'll be instructions in the resources on how to do that. 资源中会有关于如何执行此操作的说明。
It's a good tip is to note to delete outputs of your notebook before you do it. 一个很好的提示是，请注意在执行此操作之前删除笔记本的输出。
So there's not lots of junk in there. 所以里面没有太多垃圾。
And be sure that you're only pushing things in that community contributions folder. 并确保您只将内容推送到社区贡献文件夹中。
There'll be instructions. 会有指示的。
It'll be fabulous to see what you're doing, and to see some interesting design patterns added in to 看到你正在做的事情，并看到一些有趣的设计模式添加到其中，真是太棒了
this kind of exercise. 这种练习。
And then final thought for this lab is about commercial implications. 该实验室的最终想法是商业影响。
I do want to always bring it back to how you can, how you can think about this in a commercial setting. 我确实想始终把它带回到你如何能够，如何在商业环境中思考这个问题。
And to be honest, I it's hard to be specific here because this is so universally applicable. 老实说，我很难在这里具体说明，因为这是普遍适用的。
Really, any time that you have something that you want to to generate something, you want an LM to 事实上，任何时候你想要生成一些东西，你都希望 LM 能够
take care of these kinds of patterns. 照顾这些类型的模式。
Being able to send the same request to multiple LMS, being able to then evaluate responses and either 能够向多个 LMS 发送相同的请求，然后能够评估响应，或者
select the best or perhaps select a couple of best ones, or use that to give feedback. 选择最好的或者可能选择几个最好的，或者用它来提供反馈。
All these kinds of patterns are used to increase the robustness and the accuracy of models, and to 所有这些类型的模式都用于提高模型的鲁棒性和准确性，并
be able to solve harder and harder problems. 能够解决越来越难的问题。
So really this is, as I say, universally applicable. 正如我所说，这确实是普遍适用的。
So you should be able to pick whatever commercial problem you can think of that you're applying AI to, 所以你应该能够选择你能想到的应用人工智能的任何商业问题，
whether it's a summarization kind of problem or a generative problem, writing an email, building a 无论是总结型问题还是生成型问题，写一封电子邮件，构建一个
document, writing a business requirements document. 文档，编写业务需求文档。
And you can think of how you could apply these kinds of techniques. 您可以考虑如何应用这些技术。
Sending the request to multiple models. 将请求发送到多个模型。
Voting on the best outcome. 投票选出最佳结果。
This is something that you should be able to apply to your projects right away. 您应该能够立即将其应用到您的项目中。