And we will continue our exploration of Tokenizers by looking at a few others we're going to look at
five for from Microsoft Deep SEQ, 3.1 from Chinese upstart deep Sky and encoder 2.5 from Alibaba Cloud.
So I will load in those variables.
As always, it's the hugging face username Microsoft followed by the name of their model five for mini
instruct.
So it's an Instruct variant and then we will take some text.
I am curiously excited to show hugging Face Tokenizers in action to my LLM engineers.
Let's turn that into tokens or really token IDs and then decode them.
And we'll do it for Lama and then for five four.
So for Lama you can see here there's like a beginner text token.
And then I am curiously excited to show her hugging face.
Tokenizers.
And if you look down here, what you'll notice we're looking at the five four version.
So the numbers are completely different.
That's the first thing to notice, just different IDs.
It's been trained with different numbers and that doesn't matter at all.
So whatever numbers it's been trained with have to be consistent.
And it doesn't start with a special token.
So they've got a different approach, a different structure.
They've experimented, they've decided this suits their model better.
And you'll also notice that mostly the words matter the same way.
Except look at that hugging face when it came to llamas.
Tokenizer was hugging face, but when it came to Phi four it was hugging face.
So different.
Different tokenizers slightly different behavior.
And it's just worth appreciating that they're different but not worrying about it.
Uh, and uh, this is showing the apply chat template.
Uh llama we get these beginner text start header.
You'll notice llama has something that they also insert in there.
Uh but then and then there's these these headers.
The five for one is quite a lot simpler.
It has system end user and assistant.
And then the response.
So that's the difference between them.
The apply chat templates look completely different.
And all that matters is that this was used consistently during training okay.
And now we look at deep six tokenizer.
Um and so what I've got here is just showing side by side llamas tokens.
And this is five four tokens.
And this is deep six tokens for the same sentence.
Just to illustrate again the numbers are completely different.
Different vocab different IDs.
All that matters is the consistency with the training data.
Again if you look at deep six chat template it looks totally different.
It has a special token for beginning of sentence and then user and then assistant.
And the assistant prompt is just comes at the front of the whole thing.
That's how it's organized.
For deep seek you can see these subtle differences between them.
And then the final one I'm going to show you is a tokenizer for code.
So this here is the tokenizer for for um Quinn coder.
And so we bring in the tokenizer I've got some code as def hello world person print.
Hello, person.
Uh, a very complicated Python program there.
And we are going to then tokenize it and then just print out separately the, uh, the results.
Here you go.
You see, what I've done here is I've put the name of a token and then what words it represents.
And you can see that, uh, for example, underscore world is something that maps to some token.
And yeah, you can see that there are some constructs like, uh, close brackets.
Colon gets one token.
So and that seems to be quite, quite an early up token.
And I haven't done a deep analysis on this, but I rather imagine that there are much more common tokens
in this tokenizer that are related to common coding constructs than there might be in a tokenizer that's
not designed for a coding model.
So they might have optimized their tokens to make sure that they can get the most meaning associated
with the token in the way that they tokenize.
So just worth understanding that the tokenizer is associated with the model.
Different models might have a different strategy for how you convert natural language into these token
IDs, because its token IDs that models understand the numbers.
It's not like language or even lists of dicts get passed in to these models.
So today was a bit of a gentle day.
We just got through that pretty quickly, but I will make up for it tomorrow.
Tomorrow will not be gentle, but what you can do now, you can code with frontier models.
You can build a multi-modal AI assistant.
But importantly, not only can you use hugging face pipelines, but you now know about Tokenizers and
how they turn language into numbers, words into numbers, and fragments of words into numbers.
Next time we then dig into hugging face.
We use the models.
We use hugging face models to generate some text, and it's basically doing the same thing as we did
with the pipelines.
But now we are actually driving the model inference ourselves, which is really fun.
And there's other stuff for me to show you tomorrow.
So, so prepare for a big day and I will see you on the other side.