gRPC Community, 1 Aug 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: gRPC July Meetup/ Demo: System Subjugation by Preeti Sharma

Description

🐶 System Subjugation hands-on session 👀

A

So thanks a lot for introducing me uh hello, everyone how how you all are doing. I hope everyone is good. Everyone is okay, so my topic for the presentation is system subjugation. um I know it's a very complex topic like reading it from here, but I hope so within the end of slide, you will get the clarity what I am going to tell you so uh the system subjugation I am dealing with. uh I am talking about the part of slave notes and master slave architecture.

A

Moving on with a slide.

A

So my uh ppt, my presentation will consist of introduction to relational and non-relational database a network traffic, single point of failure, remote procedure, connection system, design, scaling, load, balancing database, sharding cost or reliability issues, caching, masternodes slave node master slave architecture, master, safe communication, master, save architecture with caching conclusion and comparison moving ahead.

A

So um what uh you guys understand about uh by system subjugation so uh reading it as whole system subjugation? uh We guys confused a lot like what actually it is in whole, so try to divide it into parts system and subsystem so system you can see it's a system is like a whole component and so, for example, our body system, our computer system. So it basically consists of components with a mechanism, a logic, and there is a access through a particular interface designing.

A

This all consists of system. But uh what is a sub sub subsystem? Subsystem is a system in itself. Okay, so what does it actually mean? It actually means when a particular system inherits some properties or a particular parent system, using some predefined access capability rules, or you can see protocols with it within some abstractions within some necessary details only so it could contains in the subsystem moving ahead. I hope you get the clarity. What actually system subjugation is moving ahead so uh and then what about the connection?

A

Okay, we heard about system and we have a subjugation, but how will how they are all are connected to each other? So if we get this question a lot, so uh they are connected to a basic constraint. For example, data security data is a very uh integral part of life, but we don't. uh We don't uh know what its actual significance is. So in a system subjugation, it is also very important part because it is how it communicate uh it.

A

It becomes a scalable, portable, compa, compatible to the user and developers, and the architecture also depend upon the individual categories. So now, let's we deep dive it up into the individual categories and uh let's go through some basic uh system design concepts uh before deep, diving into the actual uh motive of the presentation. So uh uh there is a user uh he or she wants the traffic to be clear, isn't it so?

A

Who wanna wait in the long queue and suffer like, for example, when we are, you know, in a hurry to rush to our destination, we stuck to traffic and we wanted to be clear and reach your destination as soon as possible, and that's that's also same thing happens in system design also, so I hope so uh some of you have unknown about ttl error, control, flow control and some of the up and a little bit idea of data link layer or network layer.

A

So uh if you, if you don't know about it, no issue, because it's too much computer network- and it's actually not required- you heard about bandwidth right, uh for example, you go somewhere, you have something, so uh you are talking about a person um when we are in the work. uh We uh we see that uh this person have that much bandwidth to complete this work. This much person have that much money. So what actually bandwidth is so talking about in the system design concepts?

A

It is basically a payload okay, you can see the data, the traffic, so how much payload or traffic is moving from one sender to the receiver per unit. Prime through any channel uh is the traffic for a network. Okay, and it is also known as the bandwidth. So, okay cool it is the data fine. So uh how will you deal with the data why it's slow? Why? uh How it's fast? What are the network providers? Is there any reason behind it? So we will talk about uh in much detail in the further slides moving right.

A

Okay. So since we talk about data, how can we forget about database? So, let's travel back to time, machine sql versus mongodb, where we have the structure schema, we have 10 tables and inserted records in it and and updated as per time, obviously, and also selected and returned the data requested by some requested by a user.

A

Looking cool, okay, fine structure data is readable uh from a developer point of view, uh but what what happen is the scheme of the structure uh if the schema change or the original structure can our attribute be very, um for example, uh there are certain parameters when a person person uh receives something uh from uh got to go, uh wanted to get a scholarship, so it uh depend upon certain parameters.

A

uh It's not only about his uh scores, it's not only about his name or uh uh the parameters of students like the ob that there are many uh parameters within the student that could be acknowledged when uh getting to a scholarship okay. But why is this comparison yeah? So uh you accept attributes can vary right because uh depend upon a different use case. So that's how schema schema is the design of the attributes like the table design. So it's also can vary, obviously so moving ahead with it discussing it further.

A

So do we need to create a separate tables and used to have a many joint statements like that? Is it really feasible like it? It can consume so much of the time. So here comes a picture of no sequel.

A

If a person says I don't need a table, I can rather handle unstructure or semi-structured data. So for that we have no secret. So developer is happy uh because it's what they need uh less time and more out more output. So uh comparing between the two sequel is far reactable and comparative uh compatible and faster, but no sequel, it depends so they are. They both have two properties. uh No sequel for uh base uh is a property for nosql and asset.

A

It's a property for a sequel, so uh this is not always uh satisfiable when we um talk about different use cases, uh but acid uh atomicity consistency, isolation, durability. These are these are very a basic, uh uh very uh satisfiable. When you uh talk about various use cases, so uh partial consistency can never lead to all what happens is there is sometimes the data? Is there sometimes not there? So this is a partial consistency that happens. That is always not best, but you have to choose between sequel or no sequel.

A

It is subjective a matter of time. It depends upon what use case we are dealing with, but guys did the comparison really matter to the end user we are discussing here. He needs this data to be served and guided in the most efficient way to him. My sequel is: if not working, you should use other sequel right. So talk up talking about that. uh So since we are talking about data, how can we not talk about traffic?

A

So uh when we are talking about in system design, traffic is a very data, is a very important concept here, because what happened is if a data get lost uh in at a certain point of time. All of our, for example, if a person is posting a pic with a with with his friends in a cafe- uh and uh there is a point of failure- occurs, all the data has been gone. So what's the actual use of it? Why are we discussing all about it?

A

So since we, since uh it's very centralized, there is no different device, we are using for storing our data. Our system is not robust; our machine is crashing, there could be faulty, so um actually, what's the use of it. Please tell us about okay, so we are discussing here is remote procedure, connection, okay, something that does not exist in our address space and it is for using a distributed, client server technique.

A

One of the best way to communicate between the server and system is through a network only right, so we will discuss about uh this further uh fabricate this further uh title in a uh different forms in a future slides moving ahead. So, but before that, tell us why are we actually reading about it by actually discussing about system design? So when we are on a website, we have a particular uh model. We view we, we view our website and there is a controller behind it, how it's navigating how it's rooting?

A

Okay, so for that the model is handled by the developer. Obviously the user. The view is very close to the user point, but the controller is a bridge to connect model in view. Okay, so it's a very simple term. It's a very efficient way for the scalability compatibility security, delay uh security, reducing everything here so basically scalability impor compatibility security is that we all need when we are discussing about system design- and we are disc uh dealing with a use case.

A

So that's how uh a a developer um will think about whenever creating a website application or anything that's of use to the user and provide a user uh experience to him or her. So moving ahead with this. So okay, um we heard about this.

A

We we listened to this concept like scalability com, but what they actually are tell us that okay, so uh first of all, scaling uh is basically um when we used to have a measure to the particular request that we are getting so it can be of two types it has to be. It could be horizontal scaling. It could be vertical scaling, for example there uh when we add um more than one number of physical machines, so that our request could be met, request meets the, for example.

A

The user wants something from the website, there's a particular request, just presenting right so for that, if we have more than one physical machines uh on our system, so it is horizontal scaling. But when it comes to vertical scaling, we are adding the load on a single machine. Okay uh and the cap, we are increasing its cap capacity according to the number of requests. So moving ahead with this, but uh okay, we heard about that.

A

But what is the load that you are talking about right, so load uh load and we are increasing the size as well. So what are? What are we doing is we are getting the request and we are balancing it out. Okay, on the sink through a single uh single machine right, we discussed about vertical scaling earlier. So what is that?

A

So? What we are doing is we are balancing the load through uh the distribution of request we are getting and that is all served on the machine based upon a mechanism, a hashing mechanism, as you can see, is there uh three to four clients uh that join through internet, the all the data is going through the load balancer, and then only it will be distributing through to the servers, because there are three clients that there are three to four servers. So this is how the load balancing works.

A

It will distribute the request and to be served upon various machines based upon a hashing mechanism, because the distribution can't be random right, so moving ahead.

A

Okay, we heard about that, and uh you also told about database um database sharding. Okay, when it comes to uh system design. We heard about it a lot okay. So what does data be starting? So we we talk about traffic. We we talk about failure of a particular server, but uh we don't talk about. How can we deal with it? Okay, uh we apply a load balancer, but still there is no traffic uh to each of the each of the server right.

A

So for that we use database sharding database sharding was used where instead of core in texting or hashing methods, the database record got distributed across system. Okay and we didn't understand okay. So what happens? Is we try to reduce our data through this mechanism? Sharding means we are cutting. We are uh reduces shrinking so, uh for example, a chart.

A

You have a particular index and key based upon the sharding techniques that look up scaling and key based, injecting these are some techniques that use in sharding, basically in a very lame way. You can say that it is used for reducing a database and then send it to a system through some. There are some indexing or hashing methods that have been involved in it and there are some techniques for it. As you can see here, uh as you can see here, there are certain servers.

A

There was abc and various servers right and uh uh going to the uh the data is going to the uh their data is sharding and then it will. It is going to the server. So so each of the data is uh presenting in the servers, but it is reduced so moving ahead.

A

Okay, so we feel about data we heard about scalability, uh but uh if a developer has to work upon it, he has to keep in mind upon the cost or how much reliable our data is right. So, yes, cost is very uh important, but reliability is not always so if the end user accessing any database server in a chart can be overwrite it um overriding take too much.

A

uh Obviously we think about it because uh overriding with too much cost- or uh it may not reflect in our records uh that these issues will come like the cost or reliability, but uh there the replication of data is, could be uh shown in the in our data a database. So these are certain parameters on which we uh we think that what database we should use uh how feasible it is for our use case so moving ahead.

A

Okay, uh you also told about caching caching right since we are talking about replication. um I hope you guys have heard about caching in the browser. We have a cache history, but actually it is, uh if you guys have ever noted or wondered uh the there are some copies of our uh copies of uh what are searched, or maybe our history that came under it.

A

So caching is like that only initially uh what it stored is a temporary replicate version of database in a cache, so that that's all that's only I'm talking about is uh the database that the data we have or the temporary version of it uh that got stored is known as caching by temporary knows it is the copy it could be deleted right so, but how many caches? It is really costly. No for each chunk of records, our cache will increase.

A

So costs become a very major issue because for each of the uh data it got replicated. So how are we dealing with that map so moving ahead, we have a master, node, okay. This is the main highlight that uh so please listen carefully.

A

So uh master is something um who says like I'll kept each and everything into myself, and I can allow only read and write data both on me, but what, if uh it crash? Who will take care of that? The master will never share it, lock fire with the slaves right master know everything, okay master can, uh but he can only uh he can. He will not share his data or log files to the slave right. So how can? How can sleep not come into picture?

A

So it said that okay, I'm here, but you can do one thing. If you can't share the whole log file, you can replicate the data among us. Okay replicate its copyright among us. We promise we won't allow anyone to modify the data and write on us. We will just allow them to read you're getting my point in the masternode. The date and write permission is there, but when it comes to, when we are storing the replication of data, they have only the power to read. uh The user can only read the data from them.

A

Okay, that means the slave data will store the replicated data from the master database storage, but only will have the data. This is not the right access and then master became happy. Okay, I'm more powerful. Obviously I'm the master. I have the reading right, whereas the slave is only the right access. So I'm moving ahead. This is the basically a diagram you can show. There's a client there's a master and there are two slaves master c. The master is only one, but there are two slaves involved. Why? Because data is big tata.

A

Okay, data is big. Data needs to be lead, but whereas in the master master can master have both taxes of reading, as well as the writing, so called the data can be stored in the master itself.

A

Okay, so slave has the uh the client can read from steve's, whereas the client will write or read in the master or vice versa, because it can read as well as right. You're getting my point. I guess so. The client will have the read and write access for master, but in the slave it can only read okay, but the data is replicated in both the slave nodes, which is a very good point: okay, so, but but how the replication work in my sequel.

A

So we have a particular database, okay, so in database we have a ddl dml commands. uh I don't I don't uh go in much detail in them, but it there's a modification or definition language that we're talking about, so they has a particular dumps the dumps of data okay.

A

So uh if we have a terms of data, we have a log as well. Okay, so when the log is good, uh when the log is got situated to a database, so we take the data from that log. So that's how uh sql is work. So what happens? Is the slave connects to master and there's an input output thread for the data? And this is my my binary log generation log is basically, for example, you will say uh I'm very speaking in a very general term when there is a word log. So there is there.

A

Are there are many pieces? That's how there are many data like you can see metadata. Okay, so log is like that, and there is sql thread applies to your data. Okay, so we have. We already told about the master slave communication is, for example, a client can read and write and master master stored, a copy to the slave okay, you can, you can have my copy, but you can't have the read and write permission. You can only have to read permission: that's how master slave works.

A

We can create uh okay, so there are heterogeneous database replication. um We can create a replication. So that's uh these are some use cases. We can create the copy from the sql based master to no sql based slave okay. So this is just to look into it uh and I don't need to go in much detail input, okay, moving ahead, master, save architecture with caching. uh Now um here we are, we will also talk about cost right. Cost is not a huge headache. Okay, this model can be really useful.

A

They further store the temporary data for all the server and it can increase the capacity for the faster access. Okay, because you can have many sleeves and if a particular person need to access the data you can go to in either any of the slaves or it what if, whatever it will do, it will increase the access of the user from the particular slave node and the data can be easily look into and cache coherence and that's how the reliability of the structure can also be increased.

A

A person can reliable on this particular architecture right moving ahead now uh the slides are already been completed. I have already told about how the communication is happening. What are the significance? What are the parameters that we are going to uh be dealing with when we are looking into system design and the communication of master and slave nodes? So um when we deal about uh now, you go into uh your use cases your projects and you talk about database, sharding, centralized storage or master safe. What will you prefer right?

A

You will think about it, see I will going to centralize that we're going to distribute it or I am going to test database sharding. So it depends. Frankly, it depends upon the requirements that you were building. Okay, now we talk since we're talking mostly about cost reliability. So how please tell me what what will be?

A

What will be cost effective, so cost effective is a centralized thing, but when we talk about network traffic, fault, tolerance and scalability database sharding is very important because it will reduce the data optimize or basically optimize our uh uh data and help to uh send the traffic over to the system.

A

Okay, now, when we talk about this reliability, maintaining the security and fall tolerance, okay, so we are, we are going from end user access. You can say so that's how we uh come into picture is master sleep. It will be helping uh all of our uh communication, uh reducing the for reducing the error. We can it's very reliable as well, and the security also issues can also be reduced. So to conclude, all data equally important, but it depends upon the use cases and requirements as well.

A

Okay, that's all uh about the pptp! um Now uh I'll go into hands-on session. Before that any question, any queries you feel free to ask.

A

I think we can start with the hand sensation uh pretty. Thank you sure sure, okay! So before going into it, I was hoping that you guys have my sequel since we are talking about uh database and everything, so my sequel is prime importance. So uh what I'm doing here is, I am just forget all about it, and this I will just tell tell what I'm actually doing. Please don't look into the code, firstly, okay.

A

So what I am doing here is, I am setting up a small master slave with two slave and one master on your local host and then I'm making a connection in my sequel. I am naming naming, it is a master server and I am assigning a localhost port number and I have a particular credential for it, because each database, when we are connecting, we have a root uh username as well as a password, and then I created a database. Then I create table and everything moving on.

A

If you don't have anything right, you can just go to this uh home page. There will be um okay, I will just show it from here only. But if I go to the okay, uh you can go to connect to database right. If you have nothing right, connect to database and there it will come, the standard connection is the localhost connection.

A

You can say uh see the localhost connection. There is a port assigned to it. There is a username as well as a password, it will store in the vault, but initially you have to give a particular uh password to your username okay. So what we are doing here is in a master. We have created a database, we have created database customer details and we have created some tables in it. For example, here I have created a customer details database in that I have table customer right. You can see here see here.

A

I have provided an id title address and a character. Okay, so.

A

See you can see here uh call uh it's the id title address, okay, so after that uh there are certain values also uh assigned to it. Let me just a second um okay. I will just tell about what actually it's happening. Okay, then we have to create another two connection. These are the slave nodes for creating um the slave nodes. These are some basic insert queries when we have to add values to our particular database table. We use this insert queries, okay, so it has certain values to it.

A

uh Then we have created a view for it as well as so, but we are here. We are talking about the slave nodes. Moving on to the slave nodes, we have created a particular user. This is the username as well as with a particular password. It got assigned to okay, so uh two slave nodes we have created, and we have a assign a particular ports for them to connect and the host is not see.

A

The host is the same as localhost, but the ports are different. You can see here right. So, okay, we got this, but we have a different username as well as password for them, so that you can come back to a master server. But uh since we are also uh creating the slave nodes, we have to grant. We have to give the privilege to user for your database in master server.

A

Firstly, for example, uh uh for each of the uh master can only write and slave can only read, so uh we have to create user as slaves. For your sleep, for example, uh you have to give access master, you have the read and write access right, but you have to give access to the slave notes so that they have the slave are only dependent upon the master node right for uh access for getting the access to the data.

A

So that's how we are giving the privilege to the uh to the slave by using grant as a command and it uh it will go from table to this particular uh particular username. This this particular slave node, these two slave nodes, we are giving uh the command and it can be created through create a grant it giving the privilege to that user for reading only and whereas create user.

A

Like you can say here is it for the access of the user grant is for giving privilege and create is for creating up giving the access to the user? Okay. Now we have done everything we are checking that in the slave oops, again, um okay, it's already uh been developed. That's why giving the output like this?

A

Okay, I will just tell it from here because uh yeah- um I will just tell it from here, because it's already being created to give the issue so uh in the uh in after this, you can check whether the slave you can connect and only uh read and not light, and the master that you have created is a read or light permission right. So.

A

Wait a second. Let me.

A

Actually, okay.

A

I've already created this, that's why it's not! um Basically, that's that I think so. You guys can um easily try this hands on and try to connect your master and slave note to the particular uh to the particular uh tables and the queries that was that will be created. If there is any issue, let me know: that's all from my side.