Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: What's Next for etcd Cluster Management? - Gyuho Lee, Amazon Web Services & Sam Batschelet, Red Hat

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

What's Next for etcd Cluster Management? - Gyuho Lee, Amazon Web Services & Sam Batschelet, Red Hat

etcd allows for distributed systems to be built using its provided primitives, but etcd itself is a distributed system. Let’s dig in and see what it takes to make that happen: whether it’s bootstrapping a three-node cluster or expanding from a single node to five. Finally, we’ll look at what improvements could be made to make cluster management easier in future versions.

To Learn More: https://sched.co/GrYD

A

Okay, thank you all for. Thank you all for coming. My name is Kehoe I work at AWS and then also working on a project called STD.

B

A

Today we will talk about the project s CD, which is now CN, say, project CN CF project, and then we will present some of the initial ideas that we are working on to improve our cluster operation of SC d.

A

So typical acidic acid is distributed so typical city cluster is distributed over five to I, mean three to five, not for high availability and then our city. So a city, server implementer after consensus algorithm so raft is like a leader based so like so in terms of a cap theorem a CDCP system, so a city for high availability as it is distributed while week, I practice the consistency and then for tolerance.

A

So what that means is cities to provide one logical cluster view of many physical servers. So as long as you maintain the connectivity with the quorum, the cluster should continue to work even with the faulty network. So raft is little based, so data is replicated from leader to follower and then follower forward proposals to leader and then what it means.

A

In the raft course or implementer after consensus, algorithm.

A

So when you operate city, a lot of things can get thinking wrong.

A

So let's say they can go this one. So let's say we knew arrow a new member to next. We know the cluster or criticize becomes so when you add a new member to another cluster cluster size becomes from 3 to 4.

A

So in this case, so when you so let's say you had a new member to 3 node cluster and they went leaders Network like so when you add a new member, the new member joins the cluster with empty data, which means they require more update from the leader note, and then then leaders network becomes easily overloaded and then, when the leaders Network pen, this is all you stop. The the leader heartbeat may block will drop in the case the follower may election timeout and then a trigger or new leadership. Election.

A

So in a city is leader based, so when you do not have a leader due to the leadership election se, the cluster cannot process any clamp requests so body. So when you operate a city cluster as it is distributed, so you must be prepared for network partition.

A

So when the network partition happens so in solving when you have our like leadership, now a network partition so in this, the exterior of the cluster.

A

So you like, as long as you maintain the true active, maintain the active quorum in the cluster. It's a city should continue to work so in this 3 node cluster is still the leaders partition still maintained so active quorum. So as this cluster should continue to work, but what if the leaders node becomes isolated? So in this case leader does not maintain the reactive quorum. So, leader like has to revert back to follower after election timeout, and then they can effect a cluster availability.

A

But what, if the leader, what if we have both what if we have a total partition and the membership reconfiguration, so they would depends on the network topology also, it depends on the cluster size and they also depends on the size of quorum. So when you add a new member to three node cluster cluster size becomes 4 and then size of the column becomes from 2 to 3, and then in this case we still have our. So this is like 3 & 1 partition so photo 3, not partition.

A

We still maintains two active quorum, so leader does not lose the quorum, so that means like we still like can process the likely request.

A

But this is, this is not okay, so this is true and to partition. Neither of the partition maintains two active quorum. So this the leader, will soon election timeout and they revert back to follow work, so they can affect our cluster availability and then the client would need to wait until the new leader gets elected.

A

So what if network of partition happens first and then new member gets added.

A

So in this case, so membership reconfiguration is a two-step process. So you issue a member, add comment. First and then you start a new sed process. So before we run the member add comment, we had two active members out of three node cluster, but after we run the member add comment, the cluster size becomes four and then in this case we only have our two active members out of four. So little lose the active quorum like from its partition and then revert back to a follower again affect my cluster availability.

A

So these the same logic applies to the one node cluster. So let's say you error a new member to the one node cluster in the cluster size comes to, and then you only have so before you run the member at comment. You only have a one active node, one node cluster, which is okay, but after you run the member add comment. The cluster size becomes two and then you have only one node out of two node cluster, which means that you just lost a quorum after member add comment.

A

So this getting worse, if you paste or if you run the member, add comment with the wrong your air. So let's say you run the member add comment with invalid, your air, so the member add comment will still be applied for it. A raft well, the since the new member was misconfigured in the first place.

A

Don't you remember would never be able to join the cluster, which means that the the cluster will never be able to elect a new leader and then next, since you lost lost a leader, you cannot even like revert membership reconfiguration either.

A

So same applies to the multi node cluster, so we have here neck three node cluster, and then you error, you run the member add comment with invalid URL. So as soon as you applied on member edde comment, the cluster size becomes 4 and then corn size becomes 3. So here like in door number two, we still have our 3 active node out of 4 node cluster. Well, like this cluster can now only tolerate one flaky member.

A

So, even though one node going down can make the whole cluster down, because in that case we would only have our two active members, our four node cluster, so a city has become the critical component in communities, so we want to do better. So we want to protect a city against miss configuration as I just explained, and they we also want to maintain the high availability during membership reconfiguration.

A

So a city learner defined as a new node state in raft and the learner know, joins the cluster as a non-voting member. So what that means it? The loner node neither vote or does not count towards the quorum, and then what that means there isn't adding a loner node with none effect. The cluster availability as much.

A

So to simplify the workflow, you just run the member ed and yvette and then like once, the loner node is ready to be promoted. You just want the member promote comment to promote the existing neck loner node to the voting member.

A

So once promoted the lunar node now a voting member would count would be counted towards the quorum and then the home site would change so for safety checks, so sed server will validate the promoter request for some safety checks. So in the case so let's say the lunar node has not caught up to the leaders. Log yet like still behind in the case like SAT server, will reject a promote request.

A

And to simplify, simplify the initial implementation, the Lana would have only a limited set of features, so you cannot transfer leadership to the lunar node and then you cannot request the read or write from the lunar node. So this is only meant to serve as a naked standby note.

B

Thank you very much, okay, so just a quick raise of hands how many of you out there have used @cd operator, okay and how many are currently still using that see the operator. So a few last is anyone using it. In production, I saw someone. Okay, all right, so I'm gonna be talking about persistent and durable at CB clusters using at CD operator, so just kind of an outline of what I'm gonna talk about.

B

We're gonna go over the definition of an operator and then we're going to go through the critical components of the operator and we're also going to then look at what ezio operator does great and then what really we can improve with SCD operator and then finally, just a roadmap of where the projects could be going in the next few months, as some people have probably noticed, Etsy operator was a little bit in an unmaintained State for a period of time and I've been working with some of our team members to resolve that so, hopefully, with those seeing some improvements soon.

B

So thank you for your patience. Okay, so an operator is, represents human operational knowledge in software to reliably manage an application. So what does that really mean? Okay? So in the case of the diagram here we have a desired state. We have current state and then we have our controller. So the controller is basically going to attempt to reconcile the difference between current state and desired state and to do that. There's two main components.

B

So the first component is the custom resource definition, so the CR D gives us access to gives us the capability to extend the kubernetes api, and this gives us access to custom resource objects. Much like the built-in objects resources such as a pod.

B

This would be an example of a building resource and the the second component is the custom controller. So the custom controller is the logic that, combined with the custom resource definition, gives us access to a declaration.

B

Here is very powerful.

B

Okay, so to install it see, operators very straightforward. This is literally just a clone from the the repository I'm using the example this deployment. If we were to apply and then do a get against custom resources, we would see that the package programmatically defined three custom resources for us. So at this point we have access to SVD clusters. We also have access to my CD restore and that's it e backup and for the remainder of the talk I'm going to focus mostly on the that city cluster resource.

B

Alright. So let's go back to the diagram and now that we have our resources available to us we're going to define a desired state. So in this example, we're going to do three node cluster and we're going to specifically use at CD version 3.3 times.

B

As far as the controller is concerned, it's it's tasks are to populate the configuration to facilitate a cluster and then also to to deploy the pods to kubernetes as well. The yellow representation of that definition is this is this is what this would look like. So you can see that we're making a call against the NC. The cluster resource in the spec we're defining the size of three and a specific version of that CD as well.

B

So we were to apply this. We could see if we did and then did it get against pods. We can see that there's three at CD instances running so this is great. This is a very quick process and it worked as expected, but we still want to do a little bit of a sanity check to make sure this is a valid cluster. So we just did a quick member list against one of the pods and we can see that there are three members in the cluster and that one of those members is leaders.

B

So this looks pretty sane, so at this point, at CD operator has done done its job. It has taken the desired state of a three node cluster, reconciled that against kubernetes to give the current state of a three node cluster. So now that we're at this point, let's kind of take a little bit of a look into how sed operator did this right. So what were the those steps?

B

So if we were to describe the first pod that was initialized we'd see it's actually a single node cluster, which you know you may or may not have assumed. So the reason for this is because when we declare the resource we can declare a size and to scale up and scale down the cluster we're going to use the cluster API, so that would involve using a, for example. Member add right. So this is how at C the operator scales cluster.

B

So if you look at the second node you'll see that that increments and then, of course the third would would do the same. So this is it's a smart right, so in the case of a failure they can recover.

B

So that's that's work. Alright, so we've went through kind of where at CD operator does great. As far as you know, kind of easy task, starting the three node cluster, we define a version, that's pretty straightforward, but now let's look at some failure cases. So, in the case of a single failure, the sed ed city still maintains quorum, so we can still use the cluster API.

B

So, in this case, the controller task would include first removing the failed member, removing the pod and then doing a member, add and and launching a new pod, bringing the total number of active members back to three.

B

And then, on the code end of things we can, if we take a look, the codes very easy to follow. This is against. This is a snippet from cluster reconcile, and if we take a look here, we can see that we're actually doing calls to Etsy D using client v3. So if you look at the top there you'll see, this is actually a member, add command and then below that.

B

If you look at you'll, see a create pod and if we were to kind of follow that back far enough, we would see eventually that this was a call to 2v1 pod. So we're talking about using the kubernetes client go.

B

So that's the operator has succeeded. Right has taken a failure state of a single node that would have required manual interaction with the API, and it has solved for that right. So this is. This is a great example of what it does well, so.

B

So now, let's kind of go through where this can go a little sideways. So in the case of a multi node failure, then things get a little more complicated because we've lost quorum now we can no longer interact with the API and get get out of this situation with a cluster API so that to resolve this situation will require a snapshot restore. So, although there is a restore operator with that's it e operator, if there was to be say, all three nodes were to fail, the the data and the positive cells are ephemeral.

B

So if we were to lose all of that, we would only the last known good state of the data would be the last snapshot, so there's a possibility of some data loss there we're to use that wrong, and the inevitable thing here is that this will fail and, as the operator cannot solve this problem by itself, so how can we make this better per system volume support?

B

So this is this was a big addition, so the initial support was added via PR 1861 and basically, by having persistent volume support, we can have the data directory persist outside of the lifecycle of the pod right, and that's really important in this case, because the data underlying is what at CD needs to operate.

B

So in this case, so now, that's any operator has another tool in its tool chain. Instead of just doing a restore or do using the cluster API, we can now look at recycling the pods right, so so, how would that work? So, let's go back to the multi node failure.

B

So in the case of this multi node failure at CD is lost quorum and- and this would, as far as the operator goes required, some type of restore from snapshot, but if we were to actually just unmount the PVC from the pod restart all three pods with the same configuration and name that CD would come back up and that's what we would operate as if there was no problem, because there isn't so that problem would be solved. So the result is a healthy cluster. This is, this is pretty big.

B

It's a lot more complicated than it looks. There's failure cases that we need to work through, but you know it's it's pretty exciting, so just a few of the future goals for the project, so we are maintaining it more now we're hoping to get some of the features there's a lot of outside APRs we're trying to work through. Of course, we need to solve for corner cases in PV and PVC failures, and we definitely want to add learner, support for durability and efforts.

A

So upgrading a city is another challenge when you work on you strip your systems, so right now.

A

So SCD, so a city has a concept called a cluster version. It is our stream value agreed by the home, like majority of the cluster, so this is how a CD of weight works so leader first now ghetto ghetto versions over budget value from each peer and then little picks the lowest sovereign value out of all those virgins and then little pig store Louis over virgin value as a cross, diversion and the broadcast cluster version value to the peers and then each peer.

A

Now, the follower we try to applied it cluster version value to the server, and then this will fail if the cluster version is a downgrade to the server. So in order to prevent or downgrade as of like STD 3.3, so the new sed member would fail to join the cluster if the cluster version, if the like new members of our version is lured in the cluster version.

A

So in order to verify all these steps, upgrade must happen incrementally. So when you upgrade its sed cluster, you just upgrade one by one.

A

So this is how a city, so this is how, like litter, picks the cluster version. So first litter fetches the server versions from each peer and then in this case leaders gonna pick the lowest like bottom value. So when you have a three point, three in the three point, four the cluster version will be selected and so a three point three.

A

So here we have our new member with, like software version 3.2. So if this member tried to join 3.2 3.3 cluster, it would fail in order to prevent or downgrade.

A

So when you, when you upgrade City, there are still a lot of things that can go wrong, so we want to do better. So.

A

We are introducing introducing the downgrade feature, so you, when you run the sed Koro downgrade command like we can temporarily. While is this like lower version value like a target version and then we're gonna provide API that you can check the progress of downgrade and then we were. We were also provide the comment that you can cancel the neck down great feature. So when gia from Google is working on this in there we gonna add this feature to the STD 3.4.

B

All right so extensible discovery, so let CD offers various bootstrap options. Probably the most interesting I think is SRV discovery, but what we want to be able to do is support multiple discovery mechanisms, unfortunately doing that within the server really isn't maintainable, so we're looking at so basically, what I'm talking about is there's some ideas for for basically moving discovery mechanism out of the server and into the client.

B

Back to the SRV discovery, SRV discovery is really interesting because you can define the a lot of your information in the SRV details of the DNS entries. So at that point you can minimize your configuration because that data will be consumed during the SRV query. The problem is: we've gone, probably 1/2 or 3/4 of the way. If you look at the the highlighted areas, there's still quite a bit of dynamic information that would still need to be populated in this config right. So, for example, we're listening to ABBA.

B

We may need to listen to a specific IP, so all this information needs to be populated in the config. So this is where we're looking that we can, we can do better. So this is an initial proposal, so it's still in its infancy but I hope to get something together in the next few weeks on paper and basically the idea is name cluster in it and basically how this works is first of all. What if this was the configuration required for SRV discovery? If you take a look at this configuration, it's completely static, there's! No!

B

There is nothing identifying features of the node itself, no IP domain names, etc. So this greatly simplifies deployment. When you are deploying this config, you don't have to worry about the values and the reason. Why is because the discovery has already occurred before @cd even starts, so this is done on a client level and then, by having the code on the client side, it's easier to extend right. So if you want to write your own discovery, it's not that big of a deal because the server doesn't have to support it.

B

So, let's kind of go through how that works.

B

So again, this is generally at high level. The client would just have a single flag, and this is basically those of you familiar with SRV discovery. The client already has this mechanism, so we're just going to consume the results, and from that list we can we. We are certainly smart enough to know which IP is ours right or we can do a DNS query to see which which of this list it represents this particular node. So we can extract that information and then persist it to the member bucket of the data store.

B

So once that's persisted on the server side, we can or something different, but basically just letting that CD know that those values could be at the store and then, if they aren't then fall back to the existing functionality so kind of a visual of what I just kind of went through there, we'll kind of show the different layers. So at the top layer we have our DNS as our V records. Okay and then, as we move down into the client, the client is going to do a SRV query against the DMS and the results.

B

Then it's going to use to populate the member struct, so this is very similar to what happens in the server, but we're going to also persist the client URL as well.

B

So at this point, we're gonna, save that into the member bucket and and that's the basic process.

B

So this is all great and dandy, but let's talk about a basic use case. So if we were to go back to the operator example where we were deploying an initial cluster and we had to wait for each node to increment well, if we wanted to provision if we wanted to provision new new data layer as well, you know this can all take time right.

B

So, in this case, with with the with what I talked about, we could do it in a NIC container and basically, when the initialization took place, we could do our discovery and populate the data, basically seed, the data for the members during the startup process, and so at that point we could start all of the nodes at the same time right. So if this was five nodes, if it was three, we wouldn't have to wait. So this is just an example of a.

B

A

Yeah we have five minute for cleaning.

B

Well, so the server of the client still needs to have that capability to basically start a raft, node and and persist to the dowser, so you're doing the same operations of the server, but it's much more condensed right. So it's it's not a full server implementation, but as far as writing to the member bucket doing a put, you know that that would have to be part of the client.

B

Well, because the underlying data is still intact right. So as far as the data directories are concerned, they are on the persistent volume themselves right. So if the pod, if the pods were to fail, you know we can basically the lifecycle of that pod isn't connected directly to the to the persistent volume. So if we were to restart those pods with the exact same names and the persistent volume per PBC, when that starts, then the at CD as far as its concerned would still be would run.

B

Does that make sense I mean it doesn't know that that happened.

A

Anyway, for day use case, we have something some feature for the Chimaera maker, so he has like all like low latency. So you just like consistently like watch from the existing cluster in there, you just a minute our data from the other cluster, and then you can use there like a mirror, like mirrored node and so forth. Back but like in terms of your like deco replicated like the data center like deployment, you can still tune, though, like a heartbeat prymaat and the election time out. So there are like many other ways to do.

A

It well photo like. If you have the lunar node, you can also use like Lohner to have our like four pack, like a node in different eras in.

B

So I think they're good. So do you mind repeating that.

C

So what we are trying to do is that there already we are trying to upgrade the HDD host vm. So in order to do- and we have a persistent volumes now to do that- we unbounded persistent volume. We upgrade our host vm and then remount at at CD rebounded volume to the HDD and then try to do add. Member so like we are trying to add it to at CD cluster, which already has the data. It's not like a it's. A new member with all the data exists, king.

B

Right so so, to answer yes, so so, basically we're not using this to extend the cluster, we're just we're. Basically, in the case of a failed pod, we can recycle the underlying dinosaur, but we're not extending the size of the cluster. So we can't just replicate it n times and then start at CD that wouldn't work, because we still have to populate the.

B

A

B

I'm really just getting into this and there's definitely a ton of corner cases, even just in the last few weeks of work, so I mean.

C

There are concerns.

B

So it's something that you know that we'll have to solve around but yeah. There is concern.

A