Okay.
We are now going to talk about abstraction layers also just called frameworks.
LM frameworks, they go by lots of names.
And by far the most famous of them is one that we will come back to for sure in week five.
It goes by the name Lang Chain, and I imagine many people take this course just for long.
Chain.
Now here's the thing.
I myself am a bit of a I'm a bit of a lang chain naysayer.
I'm not a huge fan of it.
I'm gonna go along with it.
I will do work with it.
We will.
I will absolutely teach it and I will be.
There's some parts about it I really like, and I'll be clear about them.
And there's some things that I, that I find a little bit more of a headache and I'll tell you them
too.
But I know there are big die hard fans out there.
So.
So don't get angry with me if you're a fan, I get it.
I get that it has positives too.
It's powerful.
It's mighty.
It's quite heavyweight.
Um, so this is what it looks like.
We're going to just just just ask GPT five nano to tell us a joke.
Oh, let's go with mini.
Let's, let's spend a little bit more and get a better joke.
Lang chain OpenAI Import chat OpenAI.
We create a chat OpenAI object, and then we say responses LLM invoke and we pass in the tell a joke,
uh, collection and then we'll print the results.
This is the the list of dicks.
Um, and you might say this doesn't look too heavy.
Wait.
And sure.
Not yet.
Right.
Wait till week five.
Uh, um, but yeah, you know, there is quite a lot to learn.
It's a it's a it's a it's a big abstraction layer.
There are a lot of important abstractions which allow you to work in the Lang chain environment with
lots of different models, but there is a fair bit to learn.
Okay.
Uh, how many LLM engineering students does it take to change a light bulb?
One to ask the model for 10,000 possible ways, one to filter the outputs and the rest to argue over
the exact prompt wording?
That's not bad at all.
I had that that had an unexpected twist at the end that I was all getting ready to be like, ah, but
it was quite funny at the end of it.
Uh.
Very nice.
Good.
Good for, uh, GPT five mini.
Uh, of course.
It's got nothing to do with Lang Chain.
Just that Lang chain helped us make the call.
But quite respectable performance from GPT five mini.
And this is your first look at using Lang Chain to call an LLM.
And of course, the thing for Lang Chain is that you could then import other providers with Lang chain,
underscore and another provider, and then create different objects here that you would connect to.
But by contrast with lang chain, almost the opposite extreme is an abstraction layer called light LM.
And a light LM does what it says on the tin.
It is very light.
It is a lightweight abstraction layer that just gives you a simple interface to any model.
And personally, I really love light lm I use it a lot myself.
It gives you such an easy way to switch between different models.
I mean, you can also just use the OpenAI client library, but light LM is just particularly simple.
So this is how it looks.
You just import this one thing.
Completion.
And then you just say response is completion.
It's a bit like OpenAI, but just the word completion.
You pass in the model and the messages and then you get back the same as OpenAI response choices, zero
message content.
And with what you pass in here, you pass in a particular format, which is a provider slash, a model
name and light.
LM has all of the lists of all the different providers.
And the great thing about this is that if you if you're interested in using bedrock, if you're using
AWS and you want to call a model on bedrock, you can just use a different format here with bedrock
Slash and called bedrock models.
If you're using Azure, it's the same you can and vertex for Google.
So for any of the managed services you can call all of them through light LM as well.
It lets you connect to anything.
So that makes it so convenient.
Uh, so anyways, I should stop talking.
And uh, this is, uh, going with the big GPT 4.1.
So, uh, the prior model.
Why did the LM engineering student break up with their language model?
Because it kept predicting their next move.
Yeah.
That's right.
The next token.
Next move, I see.
Yeah, that's fair enough.
That's I, I would say that it's not done as well as five minutes, but that's respectable.
Uh.
Not bad.
Okay.
But the reason, the reason that I want to show you later is because in addition to doing this, it
can also have this super useful utility which will tell you how many input tokens and output tokens
and the cost of what you just did.
So the cost of that little run that we just did right then was 0.023 $0.04.
And just in case you're getting confused and you think I'm saying $0.0234 and that this, this API call
cost, you think it cost $0.02.
It didn't.
It costed it costed uh uh two To, uh, hundredths of a cent it cost 2/10,000 of a dollar.
Absolutely tiny.
So people do tend to get quite, quite, uh, focused on API costs.
And it's important to remember, as I said before, when you're thinking about the unit economics of
a big set of APIs, then it matters to understand how much each one costs.
But when you're dealing yourself individually with individual calls, even with a big model like GPT
4.1, it's absolutely tiny.
The numbers involved, you should always check them.
Light LM gives you a way to check every single call, but you should always also monitor the websites
the platforms themselves.
But but do keep in mind that the typical costs are like per million tokens.
They add up to be very small for your everyday API call.
Okay, now I want to use light LM to to show off a professional feature a pro feature.
So the pros you can set up, you can you can turn me down from two x to 1.5 x.
I've got something here for you.
Look at this.
So let's suppose that I that I check out a file that I've got locally that happens to be called hamlet
dot txt.
I wonder what it might contain.
It contains, of course, the complete, uh, play Hamlet by William Shakespeare.
So I'm just going to read that into a variable called hamlet.
And I just want to print out a little bit of it, which is when, uh, lettuce says, uh, speak, man.
Uh, sorry, sorry.
Uh, King says speak, man.
And later says, where is my father?
And the king says, dead.
Uh, so that's just a little just an extract from it.
So let's ask a question to a model.
We'll say in hamlet, when Laertes asks, where is my father?
What is the reply?
So we're going to start by we're going to use, uh, the light lm completion.
We're going to ask Gemini 2.5 flash light, exactly that question.
And let's see what it says.
Here comes the answer.
He says A father blessed that hath a knowing son.
Well, that's not true.
Ha ha ha!
It is dead.
Uh.
And, uh, the, uh.
So so back comes a, uh, super confusing but confident answer.
And of course, this is a warning sign for you.
Recognize that when llms get it wrong, when they're backed into a corner, they have to answer something
and they don't know.
They will typically tend to confidently give the wrong answer, because that seems like the most likely
next token.
And so this is an example of a confident hallucination.
Uh, while we're here, we can just quickly look at how many tokens are being used, and you'll see
that a total number of tokens was 122.
It was 19 input and 103 output tokens.
And that total cost was 0.004 $0.03.
Very cheap indeed.
Uh, it was uh, yeah.
Uh, this is, uh, not an expensive question to have asked, but that is about to change.
Okay.
What we're now going to do is I'm going to add to the question for context, here is the entire text
of hamlet.
And I'm going to shove in all of hamlet.
So it's now a rather bigger question, and we're going to call Gemini Flashlight again.
It's going to take a little bit longer and it gets the right answer.
Dead.
And this is an example of what happens when you put in the right context.
If I print out what happened here, we had that many input tokens 23 output tokens and it costs half
a cent.
It's still not going to break the bank, but it did cost half a cent.
Now I'm just going to run that same question a second time again and print the tokens again.
And this time, when I print it again, you'll see that there's something different.
It costs about five times less.
It costs about 0.1% instead of 0.5 of a cent.
And when I printed out the number of tokens, you'll see there's a difference there.
Uh, the the same number of input tokens, 53,000.
But look, Thousand 200 of them are called cached tokens.
That means that it detected that we made a second call to the model within a few minutes, and it had
already cached inputting those into the model, and as a result, you pay a lot less.
In fact, about five times less on the input tokens.
And so this is a pro feature called prompt caching.
And it's something which can automatically allow you to pay less if you prompt with the same input context
or similar input context.
Um, several times in a row within a certain period of time.
So there is a pro feature for you, and I've given a little bit more explanation about the different
rules.
There's some fine print for OpenAI and Anthropic and Gemini.
The the important thing to know is that typically and certainly for OpenAI, the beginning of the prompt
must match identically all the way up until the point where you no longer need to cache.
So if you have something like today's date that you insert in your Certain your prompt.
If you insert that the date and time at the very beginning, you will never get prompt caching because
it will be different from the very start of your prompt.
So you want to put that kind of information that might change at the end of your prompt, not at the
beginning, and put your static content at the start.
So if you have a number of different questions on hamlet, you shouldn't put the question and then all
of Hamlet's text, you should always do it with Hamlet's text.
And then the question, because that way this will all be cached every single time.
So that's a trick to be aware of, to know about, or a trap, really.
Um, with anthropic it doesn't work the same way.
It won't automatically prompt cache.
You have to tell it that you want to be doing caching.
You want to prime the cache and you pay more when you prime the cache 25% more.
But then when you reuse from the cache, you pay ten times less.
It's a much bigger saving, so there's a little upfront cost and a much bigger saving.
And Gemini actually supports both.
There's like an implicit mode and an explicit mode, and I'm not going to go into it now.
You can read all about it through these links.
So if you're very cost conscious and you're sending big prompts with lots of input, context, and you're
thinking carefully about the unit economics of the calls that you're making, then you should look into
prompt caching.
It's a great way to reduce the costs of your input context.
All right.
That's the end of the pro piece.
And it's particularly useful because light LM makes it so easy to see what's going on, to see how many
tokens you're using from the cache and to track the costs of each call.
And I've built systems, production systems where I use light LM and I keep track because you can switch
between different models.
It keeps track of the tokens and spends for every user request.
So we can have a sort of admin function where we can watch and see how much we're spending on each of
our clients.
And we can compare that to, to the, the, the revenues, and use that to be able to make sure that,
that our unit economics are favorable.