Welcome to day three.
I have bad news for you.
Today is one of the very rare days when we are not actually going to do a lab.
There's no lab today, but I have something better.
As you will see.
But first, let's quickly do what we always do.
Recap what we've done so far.
You know the progression to become proficient.
You've got your first impression of frontier models OpenAI, Gemini, llama, various models through
llama like Deep Distilled and Llama 3.2.
And you've built something that can summarize web pages using OpenAI API and hopefully your llama as
well.
What you're going to be able to do by the end of today is compare a bunch of frontier models.
You'll know the difference between chat models and reasoning models.
You'll appreciate what they do well and where they struggle.
Let's get to it.
So there are three different breeds of llms that that reflect what they've been trained to do, the
tasks that they've set out to accomplish.
And the starting point is known as a base model, is an LM that's just there to take a sequence of information
as the input and to predict what would come next.
That's all it does.
And these base models, you don't come across them very often.
We will do later in the course when we get to work with our own.
Um, but they are, they are before it's been taught how to do things like chat with someone.
It's just about completing the sequence.
But you have a base model yourself and you use it probably quite often and it's in your pocket.
If you if you bring up your phone and you go to to write a text message and you use predictive text,
if you say sort of hello there, and then you see what it prompts, you would come next and you repeatedly
press that button that is like you're using a base model.
It's just completing the sequence, giving you the most likely output that could come next after a particular
input.
Repeatedly.
And every time you select a word that goes into the sequence, and then it's predicting the next thing
to come after this sequence, that is a base model.
And before ChatGPT came out in 2022.
Before that was was the earlier versions of the models, like GPT three.
That was just a base model and people that were using it like myself at the time, we were very familiar
with, like, you put in some text and then it would start predicting what would come next, and there
were various ways that you could force it to try and answer questions.
You'd say, like question in the prompt.
You'd give it like Q and A question and then a colon and the answer Q colon, another question, a colon,
an answer.
And then you'd put q colon and the actual question you wanted to ask.
And then you'd put the letter A, and that would prompt it to be thinking more like it's in question
answer mode and to give you an answer.
And and that was the way a lot of people used these base models.
And OpenAI had a brainwave and thought, hang on, we could train our models a bit more with data structured
in this way with kind of one message response, one message response, and and doing that became known
as making a chat variant or also known as an Instruct variant, a chat model, or an Instruct model.
It's a model that's been trained to work in this kind of prompt style.
And they came up with this idea that there should be one overall piece of information that describes
the whole chat, and then there should be like a message from, from, from one user to the AI and its
reply and then another message and that, that first that overall construct became known as the system
prompt.
Uh, you knew I was going to say that.
And the the next message was called the user prompt.
And then the assistant reply and user prompt assistant reply.
And that's training.
A model in this way became known as making a chat variant.
There was a particular approach they used called reinforcement learning from human feedback RL, and
it was that that got from normal GPT to ChatGPT.
It was the chat variant.
And so then chat models were all the rage.
And we will be chatting with with ChatGPT.
And quite quickly people noticed that there were some tricks, some prompt engineering tricks to get
more out of the model.
And one of them, which became known as chain of thought.
Prompting was just as simple as if you ask the model to do something, you add as the last sentence
uh, please think step by step.
And just by by virtue of saying that you'd get something that would apparently do better, it would
go through things methodically, and the sequence that it would predict would end up being more likely
to solve the problem just because you told it to think step by step.
Which seems super hokey, but it kind of works.
And so again, that gave people a brainwave.
Maybe we could train the model with lots of examples that show it thinking step by step, and then show
it getting to a conclusion.
And so doing that training.
Training a model so that it would a chat model would then would then think through what it was doing
before it did.
It became known as making a reasoning model or making a thinking model.
And that led to these reasoning models.
They are models that have been trained to first output their thinking steps and then give the answer.
And we saw that I think in the very beginning, in the first day, the instant gratification part,
when when we saw OpenAI's OSS model thinking things through before it gave its reply, and that that
is what a reasoning model is.
It does the thought process first and now the the some of the most modern models are what's known as
hybrid models.
But that's not really.
It's just really a variation on the reasoning thinking model that's that's able to decide how much thinking
it does.
And in some modes it's much more similar to just being a chat model.
It hardly does any reasoning at all.
And in others it will go ahead and reason quite a lot, and it decides how much to reason based on the
question you ask how how much of a puzzle it is, or if it's just in a chat mode, kind of just just
a simple hi there, then it won't bother reasoning the answer to that.
And so this kind of model that's able to decide how much thinking to do is what's known as a hybrid
model.
And Gemini Pro 25 is a hybrid model.
So it's got four and so is GPT five.
These latest models are all examples of hybrid models.
Um, and and then the latest version of the open source model, when they have both a hybrid model and
they have model which, which is just chat and reasoning in case you want to select that, you just
want one in a chat mode.
And the amount of reasoning that a reasoning model does is sometimes called its reasoning budget or
its reasoning effort.
And there's a technique called budget forcing.
When you make a model a reasoning model, think longer, you make it do more reasoning.
And there are various tricks to achieving this.
And they are quite remarkably, many of them are remarkably hacky and hokey.
And it's I think when I, when I explain this to you, you'll be like, really?
Uh, but it turns out that there's a great paper called S1 that you can Google.
I'll put, put put a link in the resources.
Um, S1 explains that it was a discovery from January of 2025.
Uh, that what you could do, uh, when a model comes up with its thinking.
Trace, if you wanted to do some more thinking, not not just to get to its conclusion, but to to to
go back and do more thinking, there was this very complicated mathematical trick that involved some
really complex calculus.
No it didn't.
It involved just add the word weight in to the thinking trace.
So it's come up with something it said like like I should I should consider blah, blah, blah, blah
blah.
And then you just insert in this sequence the word weight.
And by virtue of doing that, it kind of continues that sequence as the most likely words to come after
weight.
And it says like, wait, I should rethink this, am I sure?
Let me reconsider.
And So they discovered these scientists that just simply by adding the word weight periodically into
the sequence as you're making it predict the next tokens, causes it to reflect on its reasoning and
reason a bit deeper, and challenge itself and weigh up the arguments that it's made and consider whether
they're still accurate.
And so just just adding the word weight got better outcomes.
And that technique is known as budget forcing.
And there are sort of various other ways of doing it.
But that's that's the best known one.
And the first time you hear that, you're like, surely it's got to be something more sophisticated
than that.
Well it's not read the paper and you'll see.
And so, generally speaking, reasoning models with a high reasoning budget perform better in all of
the different benchmarks really almost across the board.
And we'll be looking at benchmarks and leaderboards and everything else in week four.
So there'll be plenty of time to do that.
But generally they do very well these these reasoning models.
And so you might be saying to yourself, why don't we always use a reasoning model?
Why even have the hybrids?
Why have chat at all?
And there are some some reasons.
It's not always better reasoning models are better for problem solving.
Uh, particularly if you don't mind wasting a bit.
They will always perform better at puzzles, for sure, and they're just just generally more intelligent,
like they score higher on any of the kind of intelligence related tests, chat models.
They are, of course faster because they don't need to produce all of this reasoning a trace.
So they're better for interactive use cases if you want to have chat, and the chat models are better,
shockingly.
Uh, so they are they are better in that use case.
They are obviously faster and cheaper because they don't need to produce all of these, these, these,
these, uh, these bits in the middle, the thinking tokens.
We'll talk more about the costs and so on later.
Um, but so they are they are better at that.
And, uh, this is I put a question mark here because people aren't sure about this, but it does seem
that chat models are often better at more creative kind of just just content generation.
If you want something to write an email for you, uh, sometimes you find that the reasoning models
kind of overthink.
Perhaps what they come up with is quite cold and and analytical, whereas chat models tend to produce
more, um, fluid content.
But this sounds hand-wavy.
There's not any there's not really good metrics on this.
It's more of an anecdotal way that that people think about it.
You should try it yourself and see if you agree with that.
And a base model is better in a very specific case of when you are trying to train a model to do something
different, to give it a new skill, which is something we'll be doing later in the course.
And when you're doing that, it's better to start with a base model.
You don't necessarily want to be in a chat construct or in a reasoning construct.
You want the opportunity to train it to have some different construct.
And in that case, better to start with a base.
So that's when you use each of those three flavors of model.