Okay, it's week four and I have some good news. 好的，第四周了，我有一些好消息。
And I have some good news. 我有一些好消息。
The good news is that we've only got one more week to go until rag week, which is where everyone wants 好消息是，距离大家都想要的服装周只剩下一周的时间了
to be. 成为。
And the other good news is that this week is really quite short and quick and sharp and, uh, yeah, 另一个好消息是，本周真的很短、很快、很尖锐，呃，是的，
it's definitely, uh, going to be easier than the last two weeks. 呃，这肯定会比过去两周容易一些。
So prepare for interesting quick fun week and then rag week. 因此，请为有趣的快速欢乐周和破烂周做好准备。
And don't skip this week because it's quick and fun. 不要跳过这周，因为它又快又有趣。
And we'll get to ragweed really fast. 我们很快就会找到豚草。
Okay. 好的。
Let's get started okay. 让我们开始吧。
So today we are going to be talking about the most important question that we will cover on this course, 所以今天我们将讨论本课程中最重要的问题，
which is which is the best LM that's the one that you'll be able to answer at the end of today. 这是最好的 LM，您将在今天结束时回答这个问题。
And what you can already do is confidently code with front end models, with the chat Completions API, 您已经可以做的就是使用前端模型和聊天完成 API 自信地进行编码，
build a multimodal AI assistant, and build solutions with the hugging face Transformers library, both 构建多模式 AI 助手，并使用拥抱脸 Transformers 库构建解决方案，两者
at the of the pipelines level and at the tokenizers and models level. 在管道级别以及分词器和模型级别。
Okay. 好的。
Let's talk quickly about how we got to where we are. 让我们快速讨论一下我们是如何走到今天这一步的。
You started out. 你开始了。
Oh, just a few weeks ago, we covered the foundations of Check Completions API. 哦，就在几周前，我们介绍了 Check Completions API 的基础知识。
We covered front end models, API's tool calling Multi-modality. 我们介绍了前端模型、API 的调用多模态的工具。
And then last week we really dug into hugging face. 然后上周我们真的开始拥抱脸了。
This week selecting models and also generating code. 本周选择模型并生成代码。
And then next week is rag week. 下周就是服装周。
Did I mention that next week is rag week? 我有没有提到下周是服装周？
I might have mentioned the next week is rag week. 我可能已经提到下周是服装周。
We will do that next week. 我们将在下周这样做。
It's gonna be so great. 这将会非常棒。
And then the week after that we go. 然后下周我们就出发了。
The last three weeks are really intense fine tuning, a frontier model, traditional data science and 过去三周确实是激烈的微调、前沿模型、传统数据科学和
modeling, and then fine tuning an open source model. 建模，然后微调开源模型。
A really big week, and then the finale. 这是非常重要的一周，然后是结局。
A gigantic AI, a beautiful product. 一个巨大的人工智能，一个美丽的产品。
So much to do at the end of which you will be able to plant the flag of success and say that you have 要做的事情有很多，最后你将能够插上成功的旗帜，并说你已经完成了
mastered LM engineering. 掌握LM工程。
And this brings us to the question du jour, which is which is the best model? 这给我们带来了当前的问题，哪个是最好的模型？
What's the best LM to pick? LM最好选什么？
And I know you've been wondering this, and of course, the answer is that that is not an acceptable 我知道你一直想知道这个问题，当然，答案是这是不可接受的
question. 问题。
It's an ill posed question. 这是一个不太恰当的问题。
There is no best LM. 没有最好的LM。
The right question is what's the right model for the task at hand, given the problem you're trying 正确的问题是，考虑到您正在尝试的问题，当前任务的正确模型是什么
to solve, which is the best model for that? 要解决这个问题，哪个模型是最好的？
And in answering that question, there are there's a whole strategy that we're going to cover for, 在回答这个问题时，我们将讨论一个完整的策略，
for how you go about making sure you pick the right one. 了解如何确保选择正确的产品。
And of course, it starts with understanding your requirements. 当然，首先要了解您的需求。
And once you've understood the requirements that we'll we will cover later, you then look at a model 一旦您了解了我们稍后将介绍的要求，您就可以查看模型
in in two ways. 有两种方式。
Look at two aspects. 看两个方面。
One is just the basic information about a model. 其中之一只是有关模型的基本信息。
And the second is digging into the benchmarks. 第二是深入研究基准。
And when I say looking at the basics, I'm saying things like how many parameters does it have? 当我说看基础知识时，我指的是它有多少个参数？
What's the context window, how much does it cost, what's the license, that kind of stuff. 上下文窗口是什么，它要多少钱，许可证是什么，诸如此类的东西。
You first use that to develop your candidate set of models that you want to look in more detail at, 您首先使用它来开发您想要更详细地查看的候选模型集，
the ones that you can afford, the ones that you can build against quick enough, all that kind of stuff. 那些你能负担得起的，那些你可以足够快地构建的，所有这些东西。
And once you've done that, the next thing you do is you look at the benchmarks, and the benchmarks 完成此操作后，接下来要做的就是查看基准测试和基准测试
are the published metrics in which the models are assessed against different things, like how good 是已发布的指标，其中根据不同的事物评估模型，例如有多好
are they at coding, how good are they at understanding language, at reasoning those benchmarks? 他们擅长编码吗？他们理解语言、推理这些基准的能力如何？
And you can look at them on leaderboards, which is which are the sort of tables which rank the different 你可以在排行榜上查看它们，这是一种对不同的排名进行排名的表格
models against different criteria. 针对不同标准的模型。
And there are also arenas where models are put head to head, which are some fun places to look. 还有一些模特们面对面的竞技场，这是一些有趣的地方。
And we will look at all of this over the next couple of days. 我们将在接下来的几天内研究所有这些。
But first, let's drill into the basics. 但首先，让我们深入了解基础知识。
So what do I mean by looking at the basics? 那么我说的“看基础”是什么意思呢？
Well, first of all, you want to look at the models that you're thinking about and say, are they open 嗯，首先，你想看看你正在考虑的模型，然后说它们是否开放
source or closed source paid models? 源代码还是闭源付费模型？
Are they chat or are they reasoning models or are they hybrid? 它们是聊天模型还是推理模型还是混合模型？
And you've now got a pretty good sense of what what that means. 现在您已经很清楚这意味着什么了。
And the pros and cons. 以及优点和缺点。
But but certainly it's not always the case that that a reasoning model is better. 但可以肯定的是，推理模型并不总是更好。
Reasoning models often perform better on benchmarks. 推理模型通常在基准测试中表现更好。
They often perform better in terms of their their intelligence because they can think things through. 他们的智力通常表现得更好，因为他们可以把事情想得很清楚。
And that's one of the two ways you can scale. 这是您可以扩展的两种方式之一。
But they tend to be slower. 但它们往往速度较慢。
They tend to not be so good at generating creative content. 他们往往不太擅长生成创意内容。
So there are definitely limitations with reasoning models as well. 因此推理模型也肯定存在局限性。
And then there are hybrid models which which are able to choose whether they want to chat or do reasoning 还有混合模型，可以选择是聊天还是推理
depending on the situation. 视情况而定。
And and often that's a great choice. 而且通常这是一个不错的选择。
But sometimes you don't want a hybrid model because you only want a chat model, for example. 但有时您不需要混合模型，因为您只需要聊天模型。
So that's one decision. 这就是一个决定。
There's also the date when the model was released, of course. 当然，还有模型的发布日期。
How recent is it? 最近多久了？
When is its knowledge cut off? 其知识何时断绝？
When is the time? 时间是什么时候？
Up until it was trained? 直到训练完成为止？
That will tell you what kind of information is part of its training set that will already know, without 这将告诉您哪些信息是其训练集中已经知道的信息，而无需
needing any inference time techniques. 需要任何推理时间技术。
In other words, if it was trained up until July of this year, then built into its training, knowledge 换句话说，如果它的培训一直持续到今年 7 月，那么将其纳入其培训、知识中
already will be facts. 已经将是事实。
Up until that point, it will know about all that information just right away. 在那之前，它将立即了解所有这些信息。
Out of the box you can just ask that question, but if you wanted to know about more recent information, 你可以直接问这个问题，但如果你想了解更多最新信息，
you would need to add that to the prompts. 您需要将其添加到提示中。
You would need to arm it with that information when you ask it the question, so that it's equipped 当你问它问题时，你需要用这些信息武装它，以便它配备
to answer it either either through prompts or through tools. 通过提示或工具来回答它。
And when I say inference time techniques, that's what I mean. 当我说推理时间技术时，这就是我的意思。
You would need to to come up with ways to, to inject that expertise into the model. 您需要想出一些方法，将专业知识注入到模型中。
Okay. 好的。
Then of course, the number of parameters that gives you this indication of, of, of its, of its capability 当然，还有参数的数量，这些参数可以让您了解其功能
of, of how, uh, how much information it's been able to absorb during training time. 关于，关于如何，呃，在训练期间它能够吸收多少信息。
And we, we went through that timeline of how many parameters there are in larger models. 我们，我们经历了较大模型中有多少参数的时间线。
And of course, I did point out that sometimes it can be a mistake to overfocus on the number of parameters, 当然，我确实指出，有时过度关注参数的数量可能是错误的，
because in recent days people have done well at getting more into slightly smaller models. 因为最近几天，人们在将更多的东西融入到更小的模型方面做得很好。
But as a rule of thumb, generally bigger is better. 但根据经验，通常越大越好。
More parameters means smarter. 更多参数意味着更智能。
And then the training tokens is just how much data was used to train this model. 然后训练令牌就是使用了多少数据来训练这个模型。
More data, of course, means it's got more knowledge. 当然，更多的数据意味着它拥有更多的知识。
Uh, so that's that's something to look for. 呃，这就是要寻找的东西。
And the context window, you know this one. 还有上下文窗口，你知道这个。
Well, this of course, is the the total capacity the model has to look back on information from the 嗯，这当然是模型必须回顾来自
conversation. 对话。
It's not just the most recent prompt. 这不仅仅是最近的提示。
It's not just the current prompt it's responding to. 它不仅仅响应当前的提示。
It's got to fit the first message you sent it, the reply, the next message, the reply to that, everything 它必须适合您发送的第一条消息、回复、下一条消息、对该消息的回复，所有内容
that's going in this memory of prior conversation, of prior messages, as part of this conversation, 这是在之前的对话、之前的消息的记忆中，作为这次对话的一部分，
up to and including the current response that the model is generating, It all has to fit into the context 直到并包括模型生成的当前响应，一切都必须适合上下文
window. 窗户。
Yes, I know you understand this. 是的，我知道你明白这一点。
So that's another thing to look for. 所以这是另一件事要寻找。
And typically you can get all this information by looking at something called the model card, which 通常，您可以通过查看型号卡来获取所有这些信息，该卡
is something that every provider like OpenAI publishes with its model. 像 OpenAI 这样的每个提供商都会发布其模型。
So you can just Google just to look up the model card. 所以你可以通过谷歌来查找型号卡。
But there are also some leaderboards which lay out this information so that you can compare it. 但也有一些排行榜列出了这些信息，以便您可以进行比较。
And then there are some other basic facts about models that you'd want to know to be able to compare 然后还有一些关于模型的其他基本事实，您想知道这些事实以便能够进行比较
them. 他们。
And some of these aren't available on the model card. 其中一些在型号卡上不可用。
You'd have to look them up. 你必须查一下它们。
One of them, of course, is the cost. 当然，其中之一就是成本。
How much is it going to cost to run this model? 运行这个模型需要多少钱？
And there's two possible costs there. 这里有两种可能的成本。
It could be the API cost. 这可能是 API 成本。
If you're using like a cloud, a frontier model on the cloud, or if you're running it locally, there's 如果您像云一样使用云上的前沿模型，或者如果您在本地运行它，那么
there's this sort of sense that if I'm running a local model, then it's free. 有一种感觉，如果我运行本地模型，那么它是免费的。
And in a way it is. 从某种程度上来说确实如此。
But remember, you're still paying for the compute one way or another. 但请记住，您仍然需要以某种方式为计算付费。
If it's running on your machine, it's going to be hammering your computer. 如果它在您的计算机上运行，​​它将会严重影响您的计算机。
If you're running it on your on your company's servers, it's still got a cost. 如果您在公司的服务器上运行它，它仍然需要付费。
So you have to factor in the runtime it computes. 所以你必须考虑它计算的运行时间。
There's the cost to do any extra training that you need to do. 进行任何您需要进行的额外培训都是需要付费的。
So if you're assuming you're going to take this model and you're going to need to train it more, as 因此，如果您假设要采用这个模型，并且需要对其进行更多训练，如下所示
we will do in week seven, and it's going to be great, then you need to factor in that cost. 我们将在第七周完成，这会很棒，然后你需要考虑到该成本。
If you're going to be building something like Rag that we're doing next week or building a product around 如果您要构建类似于我们下周要做的 Rag 的东西，或者围绕它构建产品
it, then you need to understand the build cost. 那么您需要了解构建成本。
And that's something that you'll need to to work out. 这就是你需要解决的问题。
And then very much related to build cost is the time to market, which is of course, how long is it 然后与构建成本密切相关的是上市时间，当然是多长时间
going to take you to build the product that you need to build? 会带您构建您需要构建的产品吗？
And you might say, what's that got to do with the model? 你可能会说，这和模型有什么关系？
Well, these things are all related, because if you're using something simple, like if you're using 嗯，这些事情都是相关的，因为如果你使用一些简单的东西，比如你使用
GPT five or Claude for five on it, then then you know, you're getting such a head start that the amount GPT 5 或 Claude 5 就可以了，然后你就知道了，你已经领先了，金额
of build you have to do is relatively light. 你要做的构建相对较轻。
You can very quickly, uh, roll something out that's using a Top frontier model like that. 您可以非常快速地推出使用此类顶级前沿模型的产品。
But if you're trying to use your own open source model, or if you're wanting to use a smaller model, 但是，如果您尝试使用自己的开源模型，或者想要使用较小的模型，
a cheaper model, your runtime costs will be lower. 更便宜的型号，您的运行时间成本会更低。
But you'll need to spend more time doing the build and experiments and figuring out how you get the 但是您需要花费更多时间进行构建和实验并弄清楚如何获得
business performance you need from a smaller, cheaper model, and so that that would affect the time 您需要更小、更便宜的型号来获得业务性能，这会影响时间
to market. 推向市场。
So that is a that is absolutely a factor here. 所以这绝对是一个因素。
The larger, more expensive models you should be able to get out much faster. 越大、越贵的型号，你应该能够更快地卖出去。
Rate limits is a factor. 速率限制是一个因素。
So some open some some frontier models. 所以有的就开放一些一些前沿的模式。
Some closed source models have limitations on how often you can call them. 某些闭源模型对调用它们的频率有限制。
And so you need to be make sure you're cognizant of them. 因此，您需要确保您了解它们。
Uh, typically if you're willing to pay more then then you can have higher rate limits. 呃，通常如果您愿意支付更多费用，那么您可以有更高的费率限制。
So there's there's usually an opportunity to spend more money there. 所以通常有机会在那里花更多的钱。
But but not always even even open AI has limitations even if you spend to the top tier, there are some 但即使是开放式人工智能也不总是有局限性，即使你花到了顶级，也有一些
limits. 限制。
Although I imagine there's a call me at some point that you could probably have an enterprise deal if 尽管我想在某个时候有人打电话给我，如果您可以达成一项企业协议，
you wished. 你希望的。
Still, you need to know about what the constraints are and then speed. 尽管如此，您仍然需要了解限制是什么，然后是速度。
Speed how fast the model will respond. 加快模型响应的速度。
Different models have very, very different speed characteristics, as we'll see later. 不同的模型具有非常非常不同的速度特性，我们稍后会看到。
But but things like for Gemini, flash light is blazingly fast. 但对于双子座来说，闪光灯的速度非常快。
GPT five nano is quite fast. GPT 5 nano 相当快。
GPT four one nano is really fast, and it's great to have that perspective on these things. GPT 四一纳米确实很快，能够对这些事情有这样的看法真是太好了。
And of course, yes, there are leaderboards that compare these so you can quickly pick the model that's 当然，是的，有一些排行榜可以比较这些模型，这样您就可以快速选择适合的模型
going to have the performance characteristics that you need. 将具有您需要的性能特征。
And then closely related to speed is latency. 与速度密切相关的是延迟。
Uh, or people sometimes talk about the mean time to the first token. 呃，或者人们有时会谈论第一个令牌的平均时间。
And it's particularly, uh, prominent with, with the reasoning models that tend to think before they 尤其是，呃，突出的是，推理模型往往会先思考
start responding. 开始回应。
So the time until it starts bringing back responses that you might be streaming onto your UI. 因此，直到它开始带回您可能会流式传输到 UI 上的响应为止的时间。
That has become increasingly important. 这已经变得越来越重要。
And so people need to look at both speed and latency, or the time to the first token, side by side 因此人们需要同时考虑速度和延迟，或者同时考虑第一个令牌的时间
when considering whether a model is suitable for their business purpose and then license often forgotten 在考虑模型是否适合其业务目的时，经常会忘记许可证
about, but particularly when you're working with open source models, different models do have different 关于，但特别是当您使用开源模型时，不同的模型确实有不同的
licenses. 许可证。
You need to understand some of them are very permissive and you can use them for any purpose. 您需要了解其中一些是非常宽松的，您可以将它们用于任何目的。
Some of them have limitations in terms of your ability to use it for to for commercial gains. 其中一些在您使用它来获取商业收益的能力方面受到限制。
So there's there's some revenue caps, uh, and then some of them, like metas need you to, to sign 所以有一些收入上限，呃，然后其中一些，比如元需要你签署
various agreements, as you've discovered, uh, which limits your ability to use them for evil and 各种协议，正如你所发现的，呃，这限制了你利用它们进行邪恶的能力
things like that. 诸如此类的事情。
And also, I think perhaps to compete with meta. 而且，我认为也许是为了与元竞争。
Uh, but, of course, that's where you get the lawyers involved, and you make sure that whatever license 呃，但是，当然，这就是你让律师参与进来的地方，并且你确保无论有什么许可证
is associated with the model suits your business goals. 与适合您的业务目标的模型相关联。