Now there's a constraint.
When working with LMS, you need to be aware of, which is known as the context window.
And the context window is the maximum number of tokens that any particular model can fit that it can
it can look back on.
It can consider for when it's generating the next token.
It's the maximum length of the input that it can that it can handle.
And if you try and pass in more input than that, then it will fail.
It will say bigger than my context window.
Can't do it.
Now here's something to keep in mind that that thing that gets that needs to fit within the context
window isn't just your message you've just given to the LM.
But as you now understand, it's what the actual input sequence isn't just the most recent message,
it's the whole conversation.
So far.
It's the first thing you said.
Hi, my name is Ed and then nice to meet you, Ed.
And then what's my name?
All of that has to fit in the context window.
And actually it's a little bit more than that too, because you have to pass in all of that.
And then GPT generates the most likely next token, which is perhaps the word your is going to say.
Your name is ed your, it produces your.
And then the whole of that input sequence is fed back in again to GPT with the word your at the end
of it, and it generates the next token of name.
And then in goes all the input with your name and it generates is and then your name is Ed.
It generates tokens one at a time and the whole input is passed back in.
And that means that what you need to be able to fit in the context window is the original input, the
replies, the next input, the final message you're giving it, and then all of the generated tokens
that it comes up with up until and not including the very last token that it generates.
All of that has got to fit in the context window.
And so obviously the context window governs how much the background the model can remember about references
and content and context.
And as I gave the example before.
If you're shoving all of the ticket prices to different cities in the prompt, then it allows it to
be able to remember a certain number of those references.
And it's particularly important for a technique called multi-shot prompting, which is a technique where
in the input you give a series of examples of example questions and answers for it to kind of draw from,
as it's trying to figure out what sequence to generate.
And it's also important for techniques like Rag that of course we will get to.
But many of these inference time techniques we talked about about training time and inference time scaling,
many of these inference time techniques make hefty use of the context window, and that's why it's important
to keep it in mind.
And people that have used Claude code say, are very aware of what happens as you start to fill up that
context window.
And if you happen to have a question on the complete works of Shakespeare, well, you would need a
context window of like a million tokens to be able to handle that.
And as we'll see at the moment, it's only models like Gemini that are able to handle that much of a
context window.
And that brings me to everybody's favorite topic API costs.
So look, the chat products like ChatGPT typically have a free tier and a paid tier or many paid tiers.
They range from $20 a month to $200 a month.
And that gives you a monthly subscription.
It means that you can use it with no charge per use, but some kind of rate limiting that is completely
unrelated to using the API.
If you use the API, it doesn't matter whether or not you have a subscription, you pay per use of that
API, and the idea is that you could use that to to build your own product, your own ChatGPT to rival
ChatGPT, and you'd be monetizing it, but you'd also be paying per API call because you have to pay
the compute bills.
These these API costs go into paying for the inference compute.
There's trillions of calculations happening.
And presumably a little bit goes towards paying OpenAI back for the $100 million plus that they spent
on training this model.
So that's why there are costs.
Typically, the cost depends on how many input tokens you passed in and the output tokens that you generate.
And there's two little catches in there.
One of them, of course, is that the input tokens needs to include the full sequence so far, including
all of the fake memory that you've inserted in there and anything else you put in there like rag.
And some people think that that sounds unfair because costs will start to accumulate.
But again, remember you need it to do this compute.
You need the transformer to look back on all of this in order to predict the most likely tokens.
So you should be happy to pay the bill because you want the result.
If you wish, you could just pass in the most recent message.
Forget the history.
Don't pay for that compute.
But then your your results are not going to be good.
So that's why.
And it does make sense.
The other catch is that when you pay for these output tokens that Includes any reasoning that the model
is doing for these reasoning models that generate output that describes their thought process.
Those are output tokens that you need to pay for, and in the case of gpts models, you actually don't
even get to see these outputs.
It's all it's all happening behind the scenes, but you still pay for them.
And again, people sometimes think that that sounds a bit mean.
But at the bottom line is that compute needs to happen, that processing needs to happen.
Those those calculations need to happen.
And so it's only fair that we do need to pay the compute cost for calculating that reasoning trace.
But it can lead to a little bit of unpredictability associated with costs when you don't always get
to see the reasoning that's happening.
So those are the two catches to watch out for as well with API costs.
One of the things that I'm going to cover a lot are things called leaderboards, which is where you
get to see ranked comparisons of different llms, particularly in week four.
We do that, but one that I'll show you right away is a leaderboard called vellum which is at vellum
leaderboard.
You should go check it out.
And this is a really convenient it's got lots of leaderboards there.
Lots of great things to look at.
We'll look at many of them later, but in particular there's a very handy one right in the middle which
shows you the context window and the API costs of many major models.
And you'll see there that I'm showing that GPT five, it has a context window of 400,000 tokens.
That's big.
That's a lot that you can cram in there.
The input cost is one point is $1.25, and the output cost is $10.
And you might think, okay, that's quite a lot $10.
But do keep in mind that that is $10 per million output tokens.
So there's a there's a lot there, uh, that you can fit in a million output tokens basically give or
take.
You generate the complete works of Shakespeare and you've spent $10.
Uh, and that would be a lot of content to generate.
And so, you know, it's definitely worth understanding that these things are they come at a cost.
And if you're looking to operate this at scale, where you have lots of concurrent conversations happening,
then you need to take a lot of care to understand what are going to be your unit costs for each of your
users, and how do you factor that in.
But when it comes to individuals using if you're just experimenting, prompting yourself some of these
models with just hi, my name is Ed, then divide those numbers by like like almost a million by like
100,000 for ten tokens.
You can see that the costs involved in individual small API calls are really very small indeed.
And so whilst people I understand it's frustrating to have to put in the $5 up front to OpenAI, that's
annoying.
But recognize that the actual costs that you make for everyday API calls if you if you're not building
a bigger, scalable system or an agent loop which can eat through the tokens, then the costs involved
are relatively small, so it's important to get a handle on that and feel comfortable with what are
these API costs.
And I also want to mention that when you see GPT five there, that is the big version.
If you scale all the way down to the tiny version, GPT five nano, if you work with that, then the
input cost per million tokens is $0.05 $0.05 per million tokens.
So for each token is a millionth of that.
It's a number too small for me to say, and the output cost is 0.4 of a dollar per million.
It's $0.40 per million output tokens.
So if you got GPT five nano to generate the complete works of Shakespeare, you'll be spending less
than a dollar.
A couple of other things to mention.
There's also this this idea called caching, which is that if you send in the same input twice within
a few minutes, then you pay less.
It's cheaper because some of that information is cached with GPT.
That's automatic.
With Claude, it depends.
It can be.
It's more of a story.
But basically in many cases, there's a way if you're sending the same inputs frequently to models,
there's some ways to get those costs even smaller for the input costs.
But there are some some tricks to be aware of.
And we'll talk about that another day.
Another thing to point out is just take a look at some of those context windows.
You'll see GPT five again 400,000.
The Claude range has got a 200,000 context window.
You'll see that GPT OS, the open source model, is around the 130,000 and famously Gemini.
Gemini 2.5 flash has a 1 million token context window, which means, again, that you could almost
fit the complete works of Shakespeare in in one prompt to Gemini and say, tell me, give me a give
me a quote about from this play and it will be able to look back and do that, which is just insane.
So it's good to get that perspective and have that kind of sense of what is the range of context windows
offered by these different LMS?
Goodness gracious, somehow you're already 10% of the way through.
How about that?
Wow.
So what you could already do, you can already write code to call OpenAI and a llama.
Make a summary.
You can contrast the leading frontier models.
And now at this point, you've had this introduction to transformers, to tokens to context, windows
to API costs, and you've just got this good grounding, this good foundations to everything that will
be working on, including the illusion of memory by the end of the next time.
At the end of this week, you'll be confident with the OpenAI API, the chat completions, API, chat
completions.
You'll you'll know about this thing called one shot prompting.
You'll know about streaming markdown JSON results.
You'll have implemented a business solution in a matter of a few minutes.
It's going to be all practical.
It's going to be sleeves rolled up.
We're going to be building.
I will see you then.