So this is a moment of truth.
There's going to be a lot of moment of truth moments this week, but I do hope that you still have to
hand somewhere.
The prop from last week, with all of your notes written down of where we got to in our price, we were
able to get a three handle with 39.85, was our best price from our specialist model, and that that
is what we are up against here.
How does the strongest model on the planet equipped with a Rag knowledge base lookup, compare with
a model that is more than a thousand times smaller and quantized down to four bit, but fine tuned.
So this is definitely the battle of the Titans.
This is the function.
It's as simple as this.
Let's first do a quick experiment.
You remember that the price of that distortion pedal is $219.
Let's do Ragnar.
So when I run this one thing, it's going to it's going to first look up in chroma, find those similar
items, and then it's going to make that call to GPT five one in the cloud to come up with an estimate,
and it comes up with $229.
So that one is $10 off.
That would that's certainly very close.
That is that is surprising.
But what we don't know is, is whether this is going to whether this is going to be true across the
board, whether they're going to be outliers.
Let's find out.
Let's run this off it goes.
It's it's going off.
There's already there's some greens and there's some reds.
You can see it's moving nice and quickly.
And I'm too this is too exciting that I'm not going to do the thing of putting you on pause and coming
back because because we need to to live this moment together and be part of of all of this as it ticks
up.
Um, and, uh, yeah, I should say that even though we've set the seed here, I noticed that there
is some a bit of randomness in here, so I don't know exactly what answer we're going to get.
It's a bit different every time I run it.
We are two thirds the way through.
We're in the home stretch coming up 30 left to go, 20 left to do ten.
The answer is about to appear, which is going to be better training versus inference.
Here's the result visually.
And there's the number 30.19 rag for the win.
Wow.
So, uh, shockingly and amazingly, uh, the frontier model plus rag has crushed even our fine tuned
model.
And this is a different result to this time last year, where last year the specialist fine tuned model
was still stronger even than Rag.
But not anymore.
Now, the combination of having the frontier model with GPT five, one's power, with all of its worldly
knowledge and all of its abilities, combined with the with the expert knowledge of being able to look
up relevant content in a chroma database has allowed it to be really, really sharp.
Be within $30.
Uh, not within within $31 $30.19 is the error.
This is really tremendous.
The visual says it all.
We have just built we have just surpassed all of our previous numbers.
This is now the strongest model on the planet.
Yes, I hear you talking me down the strongest model on the planet for this very specific task.
And it's not really that.
It's the model that's the strongest.
It's that this workflow, the model plus the rag lookup together, has allowed us to be the strongest
on the planet.
So maybe I'm getting ahead of myself, but still, it's really awesome.
It's a terrific result, but it turns out that we can do slightly better.
You're like, what?
Ha!
How could we possibly do better than this?
Only slightly better.
But we can do slightly better.
And it's because of this mysterious thing called an ensemble.
If you have different models that come up with different answers, then you can often build another
model, which is a combination of those models and have it beat all of their performance.
Now, something about that might sound counterintuitive to you.
You might say, hang on, how could that possibly work if all of these models are out by more than than,
I don't know, let's say 100, then how could some combination of them get an error that's less than
100?
How could that possibly work?
And that that is the magic of something called an ensemble.
And and I mean, the data scientists amongst you for this is all old hat.
But famously, ensembles really, really came into being with something called the Netflix prize, which
is when Netflix famously won or offered $1 million to anyone that could could be the best, uh, beating
the existing model they had to recommend what movie you might want to watch next.
And the winners of the of the Netflix prize did it by building a big ensemble model.
Um, but ensembles are things that data scientists love and build all the time.
And in fact, the random forest model that we that we saw before is in itself an ensemble model.
So how is it that bringing together multiple models is able to do better than any of the constituents?
Well, one easy way to explain it is this supposing that we do have a model that is estimating prices,
and suppose that actually the price of all of our goods are exactly $100.
All of them $100 for every single product.
And this model is always off by exactly $10.
It's either $10 too much or $10 too little.
So it always either guesses 110 or 90.
Every single time, it's always off by ten.
Then when we run this model, the total average error is ten because it's off by ten every single time.
So the average is ten.
And let's say we have another model and it has much the same properties.
It's also always off by ten.
And it's also always either gets it wrong by too much or too little.
So both of these two models, model A and model B, both have an error of ten because they're both always
guessing off by ten.
Now let's suppose we combine these two models simply by taking the average.
We just take this model plus this model divided by two, the average of those two models.
What's going to be the error of the combined model?
Well, if it's if we're most unlucky and every time model A guesses too much by $10, model B also guesses
too much by $10.
And every time A is down, B is down.
Worst case.
In that eventuality, we would always again be off by ten in the average, and so the combined error
would be ten.
But if there's ever a case that that they go in opposite directions, then immediately the average error
would be less than ten.
And that that hopefully I give you this example to give you some intuition.
It sounds it sounds a bit counterintuitive, but but actually it makes sense.
It's quite possible there'll be ways to combine different models that would iron out the noise between
them, particularly if they use different methodologies, and it would allow the hole to be better than
the sum of the parts in some way.
And that's the sort of theory behind ensembles.
And that that is what we're going to do right now.
So let's start by going back to our our modal pricer our specialist.
I got a little function here specialist that is going to to to call modal in the cloud and return a
summary from modal.
I've also got this little thing here.
Get price that strips out a number when some of our models return text.
Okay.
Now there's another model that that that we're going to use as well, which is the neural network that
I unveiled at the end of week six that you may have also trained yourself if you wish to.
I've uploaded the weights to a file, deep neural networks, PyTorch, and it's up on the Google Drive
right here that you can download those weights.
And you should put it right here in this directory deep neural network.
It's just over a gigabyte.
So it might take a little bit of time to download.
And I'm not going to put it in git because it's too big for GitHub, so you can download it and put
it there.
And then I've made a class deep neural network inference, which you can just load in.
We don't actually need I don't think we need that.
Uh, we can just load this in and then we can load in our weights.
And that will give us another way to, uh, to to run this.
This will give us a way to, to to run deep neural network.
So now I've got a, a specialist model and a deep neural network model.
And we can now have an ensemble model.
And it's going to take the frontier rag as price one, the specialist model as price two and the deep
neural network as price three.
Now, what you're supposed to do in these things is now do a linear regression.
Now run this over a ton of data points and run linear regression to say, okay, what weighted combination
of these different models would give you the best outcome on some validation data.
And in fact, that is what I did in this course last year.
But it's a bit of a of a distraction.
It takes some time to do.
If you're interested in doing it, then I would I would very much encourage you to give that a try.
It doesn't take long, but it's about half an hour of faffing around with a linear regression model.
And I don't think it's important to illustrate the point because I could just quickly rough, rough
strokes say, look, let's pick 80% of the first price, 10% of the second price, 10% of the third
price.
Combining these three prices, the 80% goes to our frontier model with Rag, 10% goes to our specialist
model, 10% goes to the deep neural network.
Let's just try combining them and see what happens.
So what we can then do is simply run evaluate our usual test with the ensemble model with this test
test data set and see how it performs.