Okay, a fine.
Welcome to week eight.
Day three.
Today is about leveling up yet again with a new skill.
Structured outputs.
Such a such an important skill in the way that that LMS work and the way that particularly with with
Agentic workflows already, of course you can use frontier and open source models, you can use tools
and rag, you can fine tune both frontier models and open source models.
Importantly, but also you can now deploy a fine tuned model with modal.
You can use Rag with frontier models with a big setup with 800,000, uh, similar items and and add
together an ensemble model that combines the best of multiple models.
Okay.
Today, today it's going to be about structured outputs.
And and importantly, one of the applications of this, there are many, many, many applications of
structured outputs.
One of them that we will do today is using it as a way to parse unstructured data and turn unstructured
data into a structured form, which is a business task, a commercial task that you come up against
all the time.
And llms are just great at it.
Let's get into it.
So structured outputs then.
So there's there's a way that that it feels like structured outputs work.
And then there's what they actually are.
It's a bit like tool calling.
When tool calling feels like the LLM is able to suddenly make calls to tools, but in practice it's
something a little bit more pedestrian.
So the way it feels with structured outputs is that you can define a Python class, and you have to
make it like a subclass of this, this thing called base model, which is a pedantic object which,
which we've touched on a couple of times.
So.
So you make a Python class, it's a subclass of something called base model.
And when you're doing that, it means that you define each of the attributes of this class fairly carefully.
So so you define a class that way.
And you can tell OpenAI OpenAI invented this and now most of the other providers support it as well,
but not all.
But you can tell a provider, hey, when you when you do, do what I'm about to ask you to do in my
prompting, when you respond, don't respond with natural language.
Respond with an object, a populated object, a Python object with a fields filled in.
That's weird right?
It's like it's a natural language model.
How is it suddenly able to not predict tokens, but instead craft an object, a Python object, and
that's what you get back?
Well, that's the magic of structured outputs.
And that's how it feels.
And that's that's really cool.
And it allows you to do so much.
It allows you to put structure around these, these somewhat very flexible calls to llms.
It allows you to constrain what you get back.
It allows you to then take actions based on it.
So it allows you to to put so much, uh, organization and structure around how you orchestrate between
multiple llms.
Uh, but, um, yeah, this this is how it feels.
But the reality, as is often the case, is a little bit more pedestrian.
So here's the reality.
The reason why you make a subclass of a pedantic base model is that what pedantic is all about is defining
a JSON schema.
If you if you have something that's a that's a subclass of base model, you can convert it to JSON and
you can convert the JSON back to the object again.
So the pedantic class that you set up is used to generate a JSON schema, something which says that
the JSON must conform to the following spec and it lays it out.
It needs to have these fields.
And you put like a natural language description by your fields, and that gets included in the JSON
schema.
And OpenAI just shoves something extra in the system prompt that says, hey, when you're generating
your response, you have to respond in JSON and it needs to conform to this schema, please.
And so the model just generates tokens.
All that generates tokens.
And the tokens it generates are JSON.
And models love generating JSON because they're trained on tons of it.
And it conforms to the schema because it's in the system prompt.
And then the OpenAI Python client library then goes and takes that JSON and converts it to an instance
of the class that you want populated with those JSON fields.
And lo and behold, you get back, apparently a Python object.
That's that's how it actually works.
It's not particularly magical, but it gives you this feeling that you can talk Python objects with
models instead of just talking natural language.
Except there is one thing about it that actually is quite magical.
There's one thing about it that is like an invention that is like, wow, that's clever.
One thing about structured outputs that you, even if you knew about everything so far, you may not
have known this, this little, little feature.
At least this is how it's implemented with OpenAI and with some other providers.
And it's really, really clever.
And to tell you about it, I need to remind you of the way that inference works.
Remember, when a model generates the next token, it doesn't literally generate a token.
That's not what comes out of the neural network.
What comes out after you apply the softmax function thing is a set of probabilities, all the probabilities
of all the possible next tokens.
And then there is some code which says okay, I'll pick the most likely or okay, I will sample from
these according to their probability distributions.
Well there is a trick.
And there's a trick that's called inference time constrained decoding.
That's the technical name for it.
And even from those words you probably already know what I'm going to say.
Basically when when the model generates this probability distribution, OpenAI have written some Python
code that says, hang on a second.
This model needs to conform.
This needs to generate something which conforms to a particular JSON spec.
Let me look through all the possible next tokens and see if there are any tokens here that would break
the spec a token.
That would mean that what comes out wouldn't conform to the spec, and I would simply zero out the probability
so that the model, whatever happens, it's not going to generate a token that will break the JSON spec.
It's only going to be able to generate.
I'm only going to keep probabilities for things which would stay in in line with the spec.
And then and then we'll generate the.
Then we'll select our next token and that trick.
And isn't that clever.
It's so simple.
Once you've heard it you're like, I would have thought of that, but you didn't think of it.
But once you heard it, it's like, of course, of course.
And that means that actually the model is constrained, that it has to generate text that conforms to
the spec.
The worst case is that it doesn't generate it never generates a conclusion.
It sort of keeps going or something.
So there's still some failure scenarios.
But but aside from that, when you're using OpenAI, it will always generate an output that is consistent
with the Pydantic JSON schema.
And so it is a way to guarantee that the response from the model will populate your Python object.
And that's really clever.
Okay.
And with that we're going to go straight to the lab.
We're going to get back to to coding.
And we're going to focus on the scanner agent I will see you in cursor.