-: As I mentioned at the beginning of this section,
whether you know how it works or not,
you can still get quality prompts
and quality outputs from ChatGPT,
but understanding these nuances and these subtleties
is just gonna give you a more well-rounded education
on how to effectively use it for your own personal life.
So let's now talk about
how does Chad GBT actually answer your questions.
So GPT stands for Generative Pre-Trained Transformer,
and it doesn't read words in the way that humans do,
but it processes them
through a series of computational steps.
And this is gonna be an overview of how it computes
and handles these words from your prompts.
First, we have tokenization.
Initially, the text that is inputted from a user
is broken down into smaller pieces,
much like splitting a sentence into individual words
or even smaller parts.
This process is called tokenization.
For example, the sentence, Chatgpt is great,
might be split down into three pieces,
ChatGPT, is, and great.
Each of these pieces is referred to as a token,
and this token breakdown makes it easier for the model
to analyze the text and understand its structure.
In short, it's taking a large amount of information
and breaking it down into smaller chunks
that it can dissect and further understand
by looking at things individually
and then looking at how these different parts and pieces
relate to each other.
This leads to the embedding phase.
After the text is broken down into smaller pieces,
each piece is then converted into a numerical form,
like a unique code.
So this way, the computer can understand
and work with it more efficiently.
This process is known as embedding.
And it's like translating the words
that you inputted into a language
that the model can understand,
where each word or individual piece
gets its own unique numerical identifier.
This way, GPT can process the text
in a format that's suitable for mathematical operations,
which are used in the later steps
to analyze the text and generate a response.
This is necessary because these models
do not understand words in the same way a human mind does.
So with these unique codes,
the numerical values are pertaining to semantic
and syntactic information.
Semantic information relates to the meaning of words
and their relationship with other words.
For instance, consider the words doctor,
nurse, and hospital.
Semantically, these words are related
because they all pertain to healthcare.
In the embedding space,
they might be represented by different vectors
that are close to each other,
indicating their related meanings to each other.
On the other hand, a word like pineapple,
which is unrelated to healthcare,
might have a vector far away from the words doctor, nurse,
and hospital.
So it's a Way of grouping information and relationships
that words have with each other.
For syntactic information,
this is about the arrangements of the words and phrases
to create formed sentences in a language.
It's like the framework or rules that dictate
how sentences are constructed.
In English, the typical word ordering for a sentence
is Subject-Verb-Object, or SVO.
So in the sentence, John eats apples,
John is the subject, eats is the verb,
and apples is the object.
Syntactic information helps identify
the correct order of words that make meaningful sentences.
In combination, these two types of information
help GPT to understand the context and nuances
of a user's input,
which leads to step three, which is the encoding process.
This is where tokens are fed into the GPT model
and are essentially moving through a multi-layered network
that refines the understanding of each token
based on the tokens around it.
So let's break down what this actually means.
So first, we have our layers,
and GPT has a series of these layers
that are similar to the floors of a building,
and each layer performs a special operation
on the incoming tokens.
So imagine each layer as a kind of workshop
where tokens get refined, reshaped,
based on the surrounding context of information.
GPT-3, for example, has 175 billion parameters
distributed across 93 transformer layers.
Now, in each of these layers, tokens are transformed
through very complex mathematical operations,
and you can think of this as kind of a translation
from one language to another,
helping to build a richer understanding of each token.
We then have the attention mechanism.
So within each layer, there's a mechanism called attention,
which allows each token
to kind of look at other tokens in the input
and adjust its own representation
based on what it, quote, unquote, "sees,"
from looking at other tokens.
For instance, the word bank might adjust its representation
based on whether the surrounding words
relate to a financial institution bank
or the side of a river bank.
This type of attention mechanism
is incredibly important for GPT
to understand how all of the inputted words
relate to each other.
This is all leading to what's called contextual adjustment.
Now, the attention mechanism
is helping in adjusting the representation
of each token based on its context.
So this is to make sure that the model understands each word
in a way that makes sense,
given all of the surrounding words around it.
So now these more refined tokens
move up to the next layer,
and they go through a similar process
again and again and again,
with each of these layers that it passes through,
there's a deeper understanding of the context
and the information that the model gains
through this process.
So as these tokens are moving through these layers,
they aggregate and gather more and more information
surrounding the original tokens,
which helps to continuously build a rich
and contextual understanding
of the entire output as a whole.
So by the time the tokens reach the final layer,
they have been significantly refined
and carry a detailed understanding of the input text
provided by the surrounding words.
Each layer has contributed
to building this deep understanding of the input,
making the tokens ready for the next step,
which is the output generation.
After this multi-layered process
of processing the user's input,
GPT begins the task of creating a response.
It starts with the tokens provided in the input
and then works to predict what should come next
one token at a time.
So let's dive into this process.
So we start with our prediction and selection.
For each new token,
GPT looks at all the tokens that have come before it,
which includes the user's input
on top of any other tokens
that were previously generated inside that specific chat,
and this is gonna help predict the most likely next token.
It makes this prediction based on patterns
it has learned from the vast amount of data
it was trained on,
and then GPT selects the one that it calculates
to be the most likely next token
based on its understanding from the training data.
So when processing the input phrase,
Albert Einstein was the world's most...,
GPT would go through the following steps
to predict and select the next word.
First, it would go through the tokenization process
that we mentioned.
The input phrase is tokenized into individual tokens,
aka Albert is separate, Einstein is separate,
was, the, world's, and most are all individually looked at.
Then we have embedding,
where each token is converted into a numerical value.
Those values are then processed through the multiple layers
of the GPT model.
Through the self-attention mechanism,
the model then identifies the relationship
between those tokens,
understanding, for instance, that Albert Einstein
is referring to a notable individual in history.
We then have the probability distribution.
Based on the context,
the model computes a probability distribution
over the vocabulary for the next token
after the word most.
Some common continuations for this probability
would likely be most famous, most influential,
most brilliant.
These would all receive high probabilities.
Now, what exactly is a probability distribution?
It's simply put is like a rule
that tells you how likely different outcomes are
in a situation involving some sort of chance or randomness.
So imagine you have a jar of colored candies,
red, blue, and green.
If the candies are mixed together all completely equally,
the chance of you pulling out one color
is gonna be the same, which is one out of three,
or 33.33333333%.
Now, if you write down these chances for each color,
we get a simple probability distribution.
The chance of red is 33.3, the chance of blue is 33.3,
and the chance of green is 33.3.
In the case of GPT, generating a word,
think of it like the model has a huge jar of words,
and some words are more likely to be picked
based on the previous words.
The probability distribution is the rule that tells GPT
how likely each word is to be picked next,
Simply put, the probability distribution
is the rule that tells GPT
how likely each word is to be picked next.
So GPT goes through this process, it chooses the next word,
and then we have continuation,
where the model continues this process
for each subsequent token using the growing context
to inform the prediction and selection process
of the next token.
For instance, after choosing the word influential,
it might predict next that the proper word
would be physicist, with a high probability,
given that the context of the prompt
is about Albert Einstein.
With this, the model continues generating tokens
until a stopping criteria is met,
such as the end of a sentence, punctuation,
or the maximum token limit has been reached.
This process continues
until the model would likely create the phrase,
Albert Einstein was the world's most influential physicist
of the 20th century,
whose work revolutionize our understanding
of the fundamental laws of the universe.
So it's really incredible to know
that when you put in a prompt and instantly get your output,
this is the process that's going on underneath the hood
that involves billions of parameters,
training data and information,
that is synthesizing all this complex information
and giving you your desired output.