1 00:00:01,047 --> 00:00:02,157 hello and welcome back. 2 00:00:02,247 --> 00:00:05,877 In this video, we are going to see properties of pub sub and 3 00:00:05,877 --> 00:00:09,057 how it can effect your choices while designing a system. 4 00:00:09,567 --> 00:00:10,587 So let's get started. 5 00:00:10,857 --> 00:00:13,287 So the first point here is about topics. 6 00:00:13,557 --> 00:00:17,967 So we have seen the topics act as intermediaries between the publishers 7 00:00:18,087 --> 00:00:22,257 and subscriber, but there is one more important point that the topics 8 00:00:22,317 --> 00:00:24,747 also act like a database solution. 9 00:00:24,897 --> 00:00:27,117 So how they act like a database solution. 10 00:00:27,297 --> 00:00:31,587 So whenever you're going to publish messages to your topics, then the messages 11 00:00:31,617 --> 00:00:36,597 are going to be stored in a persistent storage, so you can use them later. 12 00:00:36,837 --> 00:00:41,067 So with that, there's one more property attached, that is, you 13 00:00:41,067 --> 00:00:45,537 are guaranteed at least once delivery of messages, focus on that. 14 00:00:45,687 --> 00:00:47,967 At least once delivery of message. 15 00:00:48,177 --> 00:00:51,807 It means that if a subscriber subscribes through a topic, it is 16 00:00:51,807 --> 00:00:56,847 guaranteed that subscriber is going to receive your message at least once. 17 00:00:57,192 --> 00:01:01,152 Now, if I talk about subscribers, there is one property attached to them. 18 00:01:01,362 --> 00:01:05,022 Whenever they are going to receive a message, which would be pushed by a topic. 19 00:01:05,402 --> 00:01:09,182 Then those subscriber are likely going to send an acknowledgement. 20 00:01:09,302 --> 00:01:12,332 Or you can say an ACK to the topic why they do it? 21 00:01:12,412 --> 00:01:16,842 because as we have just discussed that a message is guaranteed 22 00:01:16,912 --> 00:01:19,072 to be deliver at least once. 23 00:01:19,312 --> 00:01:22,392 So we need to tell the topic, okay, we have got this 24 00:01:22,392 --> 00:01:25,032 message once, so you can stop. 25 00:01:25,362 --> 00:01:27,012 So that's how it works. 26 00:01:27,312 --> 00:01:29,832 Now with that, there is one more problem. 27 00:01:30,042 --> 00:01:30,912 What is the problem? 28 00:01:31,043 --> 00:01:35,483 So when I say at least once delivery being guaranteed, but sometime your 29 00:01:35,483 --> 00:01:40,163 message might be actually be sent more than once, because notice I 30 00:01:40,163 --> 00:01:42,533 didn't said Exactly once delivery. 31 00:01:42,683 --> 00:01:47,093 But what I said, at least once delivery, both are two different terms. 32 00:01:47,223 --> 00:01:50,553 So if I would have said exactly once delivery, it means that your 33 00:01:50,553 --> 00:01:54,293 messages are going to be delivered exactly one time, no matter what. 34 00:01:54,938 --> 00:01:57,248 But pub sub system doesn't work like that. 35 00:01:57,278 --> 00:02:01,088 It works on at least once delivery, where it is guaranteed. 36 00:02:01,148 --> 00:02:04,448 You're going to receive your message at least once, but your 37 00:02:04,448 --> 00:02:08,378 message can be pushed out to your subscriber more than once also. 38 00:02:08,738 --> 00:02:11,753 So now let's see a situation where it can happen. 39 00:02:12,263 --> 00:02:17,303 So let's say you have a message M1 which gets pushed out to all of the subscribers. 40 00:02:17,393 --> 00:02:21,863 And let's say there is a subscriber S1, which subscribes to that message. 41 00:02:22,433 --> 00:02:27,043 And then what happens that subscriber S1 loses a connection, maybe due 42 00:02:27,043 --> 00:02:30,643 to a network called or some kind of issue going on in your network. 43 00:02:31,033 --> 00:02:32,863 So with that, what happens? 44 00:02:32,953 --> 00:02:38,087 That subscriber S1, get the message, but that subscriber is not able to send 45 00:02:38,087 --> 00:02:40,307 an acknowledgement to the topic back. 46 00:02:40,577 --> 00:02:42,077 With that what can happen? 47 00:02:42,077 --> 00:02:47,027 Your topic doesn't know that your S1 subscriber has received a message or not, 48 00:02:47,107 --> 00:02:52,357 because your S1 subscriber has not sent any acknowledgement back to your topic. 49 00:02:52,627 --> 00:02:53,527 So what can happen? 50 00:02:53,737 --> 00:02:58,947 So when your S1 eventually reconnects, the topic may realize, okay, the m1 message 51 00:02:59,037 --> 00:03:04,317 hasn't been sent to S1, or maybe that subscriber S1 hasn't received it yet. 52 00:03:04,737 --> 00:03:08,367 So what your topic is going to do, the topic might send 53 00:03:08,367 --> 00:03:09,987 that message a second time. 54 00:03:10,107 --> 00:03:14,517 So in this case, you can see clearly how that subscriber S1 received the 55 00:03:14,517 --> 00:03:20,337 message multiple times just because the subscriber S1 was not able to send an ACK 56 00:03:20,547 --> 00:03:22,857 or acknowledgement back to your topic. 57 00:03:23,127 --> 00:03:27,267 So that's why it is a very important concept to know about, because here 58 00:03:27,267 --> 00:03:31,587 we are talking about at least once delivery where we are sure your 59 00:03:31,587 --> 00:03:35,997 subscribers are going to get messages at least once, but we cannot guarantee. 60 00:03:36,517 --> 00:03:40,927 After one time, how many times they can get the message more. 61 00:03:41,137 --> 00:03:43,387 So that is something that we have to keep in mind. 62 00:03:43,747 --> 00:03:45,607 And it's a very important concept. 63 00:03:45,667 --> 00:03:49,307 And we have to be aware of while dealing with the pub-up system. 64 00:03:49,487 --> 00:03:54,557 Now with that, we also get to a topic which is known as item potent operation. 65 00:03:54,737 --> 00:03:57,587 So what is an item potent operation, maybe, you know, 66 00:03:57,587 --> 00:03:59,357 this already, or maybe not. 67 00:03:59,507 --> 00:04:03,647 So basically the item potent operation is a kind of operation that has 68 00:04:03,647 --> 00:04:08,537 same ultimate outcome, regardless of how many times it is performed. 69 00:04:08,697 --> 00:04:12,627 It means that if we perform an operation and the result is always 70 00:04:12,627 --> 00:04:16,887 same, then that's an item, potent operation you can think of an scenario. 71 00:04:17,187 --> 00:04:21,207 So let's say we have message m1 and that contains some form of 72 00:04:21,207 --> 00:04:22,947 operation and what we are going to do? 73 00:04:22,947 --> 00:04:25,407 We are going to perform that operation multiple times. 74 00:04:25,797 --> 00:04:30,267 So after performing that operation multiple times, if it has the 75 00:04:30,267 --> 00:04:34,797 same outcome, no matter what, how many times you perform it, then 76 00:04:34,797 --> 00:04:36,537 that's an item potent operation. 77 00:04:36,740 --> 00:04:38,390 so let's take an example of that. 78 00:04:38,900 --> 00:04:42,320 Think about, you're trying to set the status of something to complete. 79 00:04:42,980 --> 00:04:45,050 Now that's an item, potent operation. 80 00:04:45,230 --> 00:04:45,740 Why? 81 00:04:45,830 --> 00:04:49,850 Because if you set someone's status to complete, even if you perform 82 00:04:49,850 --> 00:04:53,690 that operation five times, your status is still going to be complete. 83 00:04:53,810 --> 00:04:54,980 It's not going to change. 84 00:04:55,220 --> 00:04:57,140 So nothing else is going to happen. 85 00:04:57,170 --> 00:04:59,090 So that's an item potent operation, 86 00:04:59,376 --> 00:05:02,506 but now let's discuss about non item potent operation. 87 00:05:03,056 --> 00:05:06,421 So if you have an operation that increases some kind of counter. 88 00:05:06,571 --> 00:05:08,881 For an example, a like on YouTube video. 89 00:05:09,031 --> 00:05:13,501 So that's not an item potent operation because here what's happening because of 90 00:05:13,501 --> 00:05:17,371 that operation that you are performing, your value is getting changed. 91 00:05:17,611 --> 00:05:21,081 So for an example, if you perform liking of video operation five times, 92 00:05:21,111 --> 00:05:25,281 instead of once, then the counter will always going to gets incremented 93 00:05:25,381 --> 00:05:27,271 by five times instead of one. 94 00:05:27,511 --> 00:05:30,691 So that is a non item potent operation. 95 00:05:31,051 --> 00:05:34,021 So again, we have to keep in mind this thing when we are dealing 96 00:05:34,051 --> 00:05:37,451 with pub sub system, that we are guaranteed at least once. 97 00:05:37,451 --> 00:05:38,621 Delivery of message. 98 00:05:39,101 --> 00:05:44,411 So if we deliver a message multiple times, Or it contains some kind 99 00:05:44,411 --> 00:05:46,421 of non item potent operation. 100 00:05:46,631 --> 00:05:50,141 That could be bad for our system depending on the operation perform. 101 00:05:50,321 --> 00:05:54,371 So for an example, if you are doing some kind of operation and you just want to do 102 00:05:54,371 --> 00:05:59,171 that operation only once, then the pub sub system might not going to work for you. 103 00:05:59,921 --> 00:06:03,941 And in those cases, the up system will have drawbacks and 104 00:06:03,941 --> 00:06:05,351 your system can fall apart. 105 00:06:05,351 --> 00:06:07,931 So you have to keep in mind when using pub sub system. 106 00:06:08,021 --> 00:06:12,341 If you want your operation to be performed only once, then pub sub 107 00:06:12,361 --> 00:06:14,401 system might not be a great choice. 108 00:06:15,091 --> 00:06:17,831 Now let's get back to the properties of pub sub system. 109 00:06:18,091 --> 00:06:19,561 We will talk about topics. 110 00:06:19,651 --> 00:06:22,561 So topics also give us ordering of messages. 111 00:06:22,591 --> 00:06:24,331 And that is very cool thing about it. 112 00:06:24,601 --> 00:06:28,021 So if you're going to send or publish messages to the topics, 113 00:06:28,286 --> 00:06:31,196 they're going to be pushed to the subscriber in the same order. 114 00:06:31,346 --> 00:06:32,336 So you can think of that. 115 00:06:32,366 --> 00:06:36,386 If you're publishing five messages to a topic, then subscriber will 116 00:06:36,386 --> 00:06:40,616 receive those five messages in exact same order as they were published. 117 00:06:40,886 --> 00:06:43,346 So that's a very cool thing about a pub subsystem. 118 00:06:43,586 --> 00:06:47,366 So you can think a pub subsystem like a queue where the first piece of 119 00:06:47,366 --> 00:06:51,536 data that goes into it , is the first one that comes out of it as well. 120 00:06:52,166 --> 00:06:53,876 So let's take an example for that. 121 00:06:54,236 --> 00:06:58,466 So suppose you are dealing with a stock execution order system 122 00:06:58,466 --> 00:07:01,736 and you have a pub sub system for that to execute those orders. 123 00:07:02,116 --> 00:07:05,696 So if you perform those operations sequentially, you can use a pub sub 124 00:07:05,716 --> 00:07:10,716 system because a pub sub system is going to give you that guarantee of 125 00:07:10,716 --> 00:07:13,386 operation being executed in order. 126 00:07:14,279 --> 00:07:17,759 Now the next example, maybe you are sending some stock prices to the 127 00:07:17,759 --> 00:07:21,329 client and maybe you want to send those stock prices to appear in 128 00:07:21,329 --> 00:07:23,909 the sequential way in your order, 129 00:07:24,039 --> 00:07:28,239 So that's where it can be useful and another really good use case of pub 130 00:07:28,299 --> 00:07:32,319 subsystem is, if you want to replay your messages in a pub sub topic, 131 00:07:32,739 --> 00:07:34,149 if you want to, you can do that. 132 00:07:34,399 --> 00:07:38,149 So for an example, let's say a subscriber request for a particular 133 00:07:38,149 --> 00:07:40,459 message again to do it's processing. 134 00:07:40,729 --> 00:07:43,549 So you can do it with a pub sub system very easily. 135 00:07:43,989 --> 00:07:46,899 Now let's discuss about another property of a pub sub system. 136 00:07:47,209 --> 00:07:51,239 So your pub sub system can help you to achieve clear separation of 137 00:07:51,239 --> 00:07:55,529 concerns and separation of duties by publishing to a different topics. 138 00:07:55,589 --> 00:07:56,399 What does that mean? 139 00:07:56,759 --> 00:08:00,579 So if you have a big system, you will surely have a lot of work to do 140 00:08:00,579 --> 00:08:05,139 in it, and if you want to separate the task between multiple servers, 141 00:08:05,259 --> 00:08:07,109 you can easily do it with pub-up. 142 00:08:07,329 --> 00:08:11,169 So what you can do, you can have a topic for each kind of task. 143 00:08:12,009 --> 00:08:13,569 And what can happen. 144 00:08:13,599 --> 00:08:16,719 You will have a subscriber which effectively is going to be a server, 145 00:08:16,749 --> 00:08:19,909 which is going to consume that topic and can perform the work. 146 00:08:20,119 --> 00:08:22,839 For an example here, if you're dealing with a stock based system. 147 00:08:22,839 --> 00:08:26,509 So what you can do, you can publish the stock prices, maybe to the topic T1 148 00:08:26,689 --> 00:08:28,969 and news alert, maybe to the topic T2. 149 00:08:29,059 --> 00:08:32,149 So you can see that clearly we are defining the boundaries for 150 00:08:32,149 --> 00:08:34,459 our news alerts and stock prices. 151 00:08:34,644 --> 00:08:36,474 They're not going to be mixed together. 152 00:08:36,684 --> 00:08:41,354 And our subscribers will always know where to look for the specific kind of data. 153 00:08:41,564 --> 00:08:46,754 So with that pub sub system can help you to achieve clear separation of concerns 154 00:08:46,814 --> 00:08:51,234 and separation of duties in your system and it is really helpful, especially if 155 00:08:51,234 --> 00:08:55,939 you're dealing with a distributed system where you want your task to be assigned to 156 00:08:55,939 --> 00:08:58,849 a specific servers so they can work on it. 157 00:08:59,359 --> 00:09:04,249 Now, if you're thinking about what are the popular pub sub systems out there. 158 00:09:04,399 --> 00:09:07,259 So one of the very popular one is Apache Kafka, 159 00:09:08,089 --> 00:09:12,499 and another one may be Google cloud pub-up and AWS as well. 160 00:09:12,769 --> 00:09:17,029 So these solutions that are listed out here give you a lot of things out of box. 161 00:09:17,129 --> 00:09:20,549 So when I say out of box, I will take an example of one thing. 162 00:09:20,819 --> 00:09:24,899 So maybe they give you solutions, like end to end encryption. 163 00:09:25,079 --> 00:09:26,069 What does that mean? 164 00:09:26,129 --> 00:09:29,489 It means that whenever you're going to send messages over to the network, 165 00:09:29,549 --> 00:09:31,109 they are going to be encrypted. 166 00:09:31,319 --> 00:09:35,399 So it means that your subscriber will only know how to read them. 167 00:09:35,429 --> 00:09:37,529 No one else can read it. 168 00:09:37,709 --> 00:09:42,439 So this has been given by the Google cloud platform or AWS, and there are 169 00:09:42,439 --> 00:09:46,489 a lot of other functionality as well that comes with your pub sub system 170 00:09:46,579 --> 00:09:49,039 out of the box and free as well. 171 00:09:49,309 --> 00:09:52,279 So one more example, like Google cloud gives you auto 172 00:09:52,279 --> 00:09:54,079 scaling options for your topic. 173 00:09:54,229 --> 00:09:59,269 It means that Google cloud system says to you, don't worry about your topics, how 174 00:09:59,269 --> 00:10:01,669 to scale them, just publish the messages 175 00:10:02,034 --> 00:10:06,924 and take care of what kind of data goes to in which topic, and they say that we 176 00:10:06,924 --> 00:10:08,694 are going to take care of how to scale it. 177 00:10:08,754 --> 00:10:10,074 You don't have to worry about that. 178 00:10:10,154 --> 00:10:14,324 So a lot of things can be given to you by your cloud provider out of the box. 179 00:10:14,504 --> 00:10:17,654 You just have to look out to the different options that are available. 180 00:10:18,014 --> 00:10:21,824 So with that, I will say that publish and subscribe pattern is a very 181 00:10:21,824 --> 00:10:25,694 powerful pattern that you should use in your system design interview. 182 00:10:25,954 --> 00:10:30,424 So if you're excited to know about how to use this pub sub system in a system design 183 00:10:30,424 --> 00:10:35,174 interview, then we are just very close to our system design interview questions. 184 00:10:35,689 --> 00:10:40,009 Be sure you also check them out so you can know how all the things are tied 185 00:10:40,009 --> 00:10:42,259 together, that we are learning until now. 186 00:10:42,379 --> 00:10:43,999 So thanks for watching this video. 187 00:10:44,089 --> 00:10:45,439 I hope that you enjoyed it. 188 00:10:45,739 --> 00:10:46,249 Goodbye.