What is going on, guys?
My name is Hossein, and this video I want to discuss, I caution that some of you asked me when do
I, Shahd, my database and short answer for those who are impatient.
Last thing you want to do is the Shard, your database, there are so many other things that you can
do before complicating your life with Shaadi.
So with that said, let's just jump into it, guys, if you're new here.
Welcome.
My name is Ozanian.
This channel we discuss all sort of back in engineering.
So if you're interested to be on better software engineer, consider subscribing to that Billikens.
So you get notified every time I upload a new video.
With that said, let's just jump into Sharding.
When do you share your database, guys?
Sharding the idea of starting a database is I know it sounds that the that term sounds very sexy.
Arrives as I.
Oh, I'm sure my database makes you sound intelligent, but if you think about what you're doing with
Sharding, it is a very complex architecture and complex operation to implement.
So.
In order to talk about, as you have so many other things you can optimises, like why are you doing
it to begin with?
Right.
So.
The start over, you have a beautiful database.
And I'm now talking about mostly relational databases, because that's my specialty.
I very rarely work with no more SQL databases, but they work similarly.
But here's the thing.
And in relational databases, you have a single instance database, right, and you're probably building
a Web application, so you need a Web server or an API framework such as Narges, Django, all that
jazz go application, your custom application that essentially speaks SDB protocol that talks to the
browsers or some rest clients or.
Any job, easy job, easy, but what is the thank you.
Thank you.
Yeah, so anything that's basically HDB and you make request and that request goes to the Observer and
the Web server makes queries to the database and response gets her back responses and kind of changed
the representation to something that the client understand could be JSON, XML, anything, and then
ship it back to the client.
That's their best representation, right, in the representation.
All right.
So that looks good.
What's the problem with this, you get a lot of requests, a lot of people are reading, I say I was
like, oh, I'm getting give me all these employees table or give me I'm reading this profile.
I'm reading this for I'm reading this tweet.
I'm reading this.
So you're hitting that database a lot and you're reading a lot.
Is it OK that you're reading from a table right at that table is large.
It's getting large because you're inserting a lot of stuff in it has one million, two million, three
million, four million twenty fourteen hundred million rose in it.
And you might say Hosain doesn't matter because I have an index on it and that index is going to be
smaller.
Right.
And I'm going to head the index because I'm smart.
I am going to index the fields and I'm only quoting based on those fields that are indexed.
So my queries are fast, but then they're saying to you.
Only so far, because the larger the table gets, the larger the index gets and almost you're doing
and full index can, which can also be slow.
So the solution is not to just split everything into multiple servers.
You're doing it.
You're overengineering too early.
And that's what the problem with most people are just overengineering too early.
You can still do so much stuff.
I hope you have this.
This is their problem now, right?
You have to identify with the problem first in order to solve it.
You can't just solve something that you don't know.
The problem is right.
Identify the problem.
So let's say this is my problem.
My reads are slow.
I didn't say anything about this law just reads, because writing is always fast in writing to the end
of the table.
And even if you're using a Vetri data structure index.
Yeah, if you're using and it's this idea that could be.
Harmful for this is because you're rebalancing that tree sometimes if you're riding a lot, and that
could cause some shifts in the desk.
I just know that as bad as Reed, Ellis is optimized for rights.
But that's another story rights.
So I'm reading a lot, Miles, getting Lawrence Lawrence, it used to take one milliseconds now is like
20 and now it's a hundred milliseconds.
What do you do?
Tables large, what do you do with it?
There's something called most databased supports called partitioning.
And I'm going to talk about just the horizontal partition.
So imagine this.
This is your table and I talk about horizontal and vertical partition.
I'm going to reference the video here.
Go check it out.
But in a nutshell, you have a huge table, hundreds of millions of rows.
You just slice it in half.
Horizontally, that's that's the horizontal slice it in the middle.
And you say this piece from this.
I.D. to this ID go to this stable, right, and we're going to call this a partition and this idea to
this idea goes to this table I'm going to go partition to and this this idea, this idea go to partition.
So now you instead of having one table, you have three different tables that represent the whole logical
table.
But when you query, we can find out based on something called a partition key, which usually is that
is the primary key of the table.
Doesn't have to be there.
But based on that, you can know which table to hit.
Now you have smaller tables, you have 10 smaller tables.
And guess what?
With a little better knowledge from the client, we can know exactly which small table or partition
to hit.
And now I'm working with a smaller index and my queries will be fast instead of having this huge index.
Now I have a smaller index and I know exactly what quite so.
My queries are fast, just like that.
By just enabling partitioning and most databases.
Postgres, for example, does that blindly for you if you enable it.
And the client doesn't have the even need the knowledge to know that, oh, I need to have this table
table partition.
Underscore what?
No, it's just happened for you previously and all the releases you had to have this knowledge.
Oh, this is this table, this table, this.
Remember, guys, I'm still in the same server as one instance, one database instance, but with little
bit of a tweak of a feature of my database, I, I fix the problem.
Right, you don't need charting.
We're going to talk about Sharding in a minute.
What does it really mean?
But.
You come back, you say, Hussein, no, that's not good, because, yes, I am, my singular reads
are fast.
I'm selecting staff from the stable, and yet it's very fast now because I partition all that stuff.
But my dad, a base server, cannot handle because all these TCP connection to a single server just
read stuff.
It's overwhelming my server and it's the file descriptors are blowing out of proportions and I cannot
handle that anymore.
Right.
So what do you do?
Well, I'm not even introducing the idea of caching.
You can obviously introduce a layer of REGISTE cache and all that stuff, but.
It comes with its own problems because you'll end up with inconsistency and you have to invalidate the
cash and all that stuff, but you can't play with that.
And without introducing that extra layer, I have some tricks up my sleeve to fix that problem.
All right.
And most of the stuff you guys know it probably, but.
Replication, why is nobody talking about this?
You have a master backup, so.
And you some some databases have this by default, this this features where you have one master and
create another server and make this server a backup of that master and what it will do as periodically
the master will push the edits.
Right.
The rights to that.
To the replica, to the backup, so you can get as many backup as you want.
And multiple servers with partitions.
And.
Do all that stuff.
Now, you segregated multiple servers, but you have one right to do a ride, you go to that dang server,
you don't want to write in multiple servers, that will confuse a lot of things.
And we're going to talk about it in a minute because reading is easier than writing.
Reading is doesn't really cause conflicts or writing.
If I'm writing to the same rule you have in the seven different servers.
Bad things happen.
How do I reconcile these changes?
Right.
It's just you can't solve it, but it becomes a very challenging problem to solve.
So you want to avoid complex problems and why would you do it?
Well, I'm gonna write to myself because my rights are not as bad.
I don't I often barely write anything, but I read a lot.
So you can just multiple servers, you have one master server and just push all that stuff to that replicas
or the backups.
And now your clients or the Web servers, but a reverse proxy at proxy in Ingenix Haertsch.
And if you want and then start funding or you want to read, go to this database or you know, read
you what, as the backup, go to the backup and now you're balancing the rights, but the rights still
goes to this database that is the master and some people who will come to or know who say, and you
don't know me because I write a lot and I need to write a lot.
And my single server that does the right no longer is sufficient is going to die because I write so
much.
Do it.
Really doubt it first, because if you're riding a lot, you will be riding a lot to saturate a single
server.
I'm assuming you have a big server that does that stuff right.
But let's say you do.
You're writing a lot.
You still have some tracks before the final thing, which is sharding.
You can do multiple replication, a two way replication, but have two masters on different regions
in in in the US.
This is very common.
This is a U.S. E and this is a U.S. W.
People from Florida writes to this server anything that has to do, customers with the with the E just
deal with this, anything it has to do with customer.
The W deal with this and most of the time they don't overlap.
Right.
Really.
So if you have this aggregation, use it to your advantage and then synchronization of this can happen
very easily because you're barely going to have any conflict.
And if you do, you're going to resolve it.
And if this is so complex, you finally go and do sharding, which is the following sharding is the
idea of having a single key.
The after all of these are exhausted.
Right, come down and Lusardi, right, if and if all of this stuff is also done, Goodhew Sharding,
if none of them, then probably you don't have to do Shabaan because training is the most complicated
thing to accomplish.
And let's talk about what is become easier these days.
But let's talk about Sharding.
So what is shocking?
Sharding the idea of splitting like partition, like horizontal partitioning, but put the tables that
are partitioned, not in a single in a different table.
In the same instance, same server.
No, both those partitions or shards in a different server altogether.
Wuk.
Yeah.
We're partitioning, you can still do transaction, by the way, guys, the database supports transactions.
Well, thing for for the Sharding was sharding.
My, my, my tongue has been swallowed will sharding, you can all do transactions.
Forget about transactions.
Bye bye.
Roll back all this beautiful comet staff Essid at Tomasetti, bye bye, no sharding for you.
You cannot do all this because what do you do if you want to try to multiple this is different servers
write.
And you cannot maintain the asset transactions.
I know some people don't like as a transaction and the favor the others that with eventual consistency,
but the old school kind of guy.
All right.
So so what do you do is I am about to update customer number hundred and does the number 100, you say,
OK, the client, you are the client, which is the world's sole observer in this case is OK.
Customer number one is between hundred, between one and ten thousand.
So yeah, as server number one, let me establish the ECB connection with our number one and execute
that right or the right.
Right.
And and you received another request, but this request has to do with customer number ten thousand
and seventeen.
All it does is 17 is between 10000 and 20000.
And based on my Sharding Grant arrange, this is server to subject his connection to the number two
and executed and come back and result.
And yep, you might say, yeah, I'm going to use puling because I don't want to establish D.B. connection
every time.
Anderson and I, I absolutely agree with you have to use pooling.
You don't have to do this cold start of TKIs.
Hopefully one database platform will get to use quick and all this problems will go away.
They call stop and all that stuff, but.
Yeah.
Sharding is very complex, that logic that we just talked about is in your client application.
That's hard, right?
It's very hard.
By the way, I made a video about Sharding guys and actually show you how it's done.
Check it out here.
But what do you do, guys?
Right.
This is do you do you really want to do this with your life and sacrifice rollbacks and comments and
transactions and all this stuff just to do a stinking sharding?
Just to do what?
Scaling.
You want to scale the rights?
I agree it's beneficial, but sometimes it's an overkill.
You get a really neat thing.
Do you really need to, Shahd?
Right.
All right, that being said, you two run into this problem as you might, as you might think, because
YouTube is a.
Millions, millions of videos which with billions of watch hours and hours uploaded, right, so thousands
and thousands of hours are uploaded to this platform and comments and and I you to listen to a podcast
with the lead of YouTube talking about how they solved it.
They have a MySQL database on the back.
And by the way, they manage all that.
So and the videos are I believe in I forgot where it is.
Maybe it's three.
I might be wrong there.
Before that was before Google acquired it and all.
That's not right.
So they moved everything to YouTube, to Google infrastucture and all that stuff.
But they for a long time they had a single MySQL server and they managed to do fine until the rights
became absolutely disastrous.
So they started doing what we talked about and until they couldn't handle.
So they had to do sharding, which is called.
What we talk about is application level.
Schoening, the application is aware of the Shahd, which is this piece of the partition.
And when, you know, when you are aware that's a problem because you create a coupling and coupling
is the worst thing you want in software because you want software to be as isolated as possible.
You want software to have social anxiety, social distancing in these days through social distancing
for software like Gabrielsson.
For a part, they shouldn't know stuff about each other.
That's just a general idea about separate.
Coupling is bad about software.
Come back now.
What they did, it's like they had this application level for a long time and says, look, I know this
is ridiculous, we cannot maintain this because if they want to change some logic and everything breaks,
right, they want to forget about regarding the idea of rationing is like, oh, okay, I want to change
the range from 10000 because 10000 is really low.
Let's make it 100 thousand.
Right.
Each card has 100000 rolls.
Oh, forget about that.
Nobody can do that because I will break every single client.
You've got to go to that client.
And yeah, so it's complicated.
So they created this technology called Vitez and I'm going to reference it here.
And the test is essentially a middleware that you send your query on top of my and it's I believe it's
open source and it's Cuban and the Israelis and all that jazz.
And you put it on your MySQL.
Architecture and is going to do the magic sharding for you, and that is awesome, because you send
a beautiful school query and they pass that query and they understand what charge to hit.
And you don't need anything better than that if you absolutely need, because nothing's for free.
If you want if you have the maintenance cost of implementing the task, go ahead and do it.
Any technology you implement there is a cost for it.
If you're ready, go ahead.
But ask yourself, do you really need it?
Sometimes you do not need it was an awful British accent.
Impression.
All right, guys, that's it for me today, that was an answer, 20 minutes video answering went to
Shahd.
What do you guys think?
Am I feel B.S. or do you agree?
I am welcoming all the comments, guys, I really learn a lot from.
From you guys, I learned a lot from you because.
We'll each develop and and.
Advance, when we listen to each other, we learn you.
I might have said something that is absolutely wrong here and someone and it happened countless of times
like Hosain.
That's B.S. the moment you said administration.
That's incorrect.
And here's why they tell me.
Here's why.
And here's that proof for it.
And I absolutely cutted them.
I heard that comment.
And I sometimes spin it because I admit not perfect.
We're all learning.
We're going to learn a lot and keep learning, guys.
All right, guys, stay safe out there and see in the next one, dibby.