Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Zero Database Downtime with etcd-operator - Tyler Lisowski & Kodie Glosser, IBM

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Zero Database Downtime with etcd-operator - Tyler Lisowski & Kodie Glosser, IBM

Zero downtime upgrades, multi availability zone redundancy, automatic instance recovery, periodic backups, and restores in the case of disaster scenarios are all possible when you automate etcd operations with the Etcd-operator. This talk will walk through a live example of a multi availability zone etcd cluster managed by Etcd-operator and how Etcd-operator automates database administration tasks. We will visually step through how these operations are executed by Etcd-operator and how attendees can integrate it into their existing architectures to eliminate downtime and drastically reduce the time they spend on database administration tasks.

https://sched.co/Zeqj

A

Hi, my name is tara lisowski and I am presenting with cody glosser.

A

Today we will illustrate how entity operator, automates, ncd database management and eliminates database downtime etcd is the primary data store of kubernetes and holds all the kubernetes cluster state internally at ibm. Our first entity deployment model for coop clusters consisted of deploying and managing etcd instances on vms, using ansible.

A

As we scaled the number of kubernetes clusters, we managed this deployment model became a bottleneck to the scale we could operate at due to the amount of engineering hours it took to maintain this automation suite at scale have any of you all had the experience of going away for vacation for a week and coming back to an inbox of a thousand plus emails that you have to process and respond to that's the same feeling our engineers were experiencing as the various database creation and administration tasks queued up on a weekly basis.

A

In addition, deployment and upgrade times of ncd clusters took, on average one to two hours to complete, which increased the total time it took to get a kubernetes environment functional we needed to re-architect the way we cluster deployments in order to scale the amount of kubernetes clusters we could manage.

A

Moving to an entity deployment model centered around a cd operator allowed us to eliminate these bottlenecks.

A

To put it in perspective, we manage a thousand times more etcd clusters with our entity operator based solution than our ansible base solution. In addition, over the past year, we have spent significantly less engineering hours, maintaining all lcd clusters managed by a city operator compared to those managed by our ansible base solution.

A

Now, let's illustrate the power of entity operator by walking through a live demo of the entity. Cluster deployment process.

A

A quick note for those that have downloaded this slide deck the number in the top left hand, corner of my screen corresponds to the slide number that has the content we are currently covering and will be updated throughout the demo.

A

We are going to create a three node, highly available multi-az ltd, cluster managed by etcd operator. We first need to define the database that entity operator is going to create for us in a language. It understands that language is the etsy cluster custom resource definition.

A

The crd provides entity operator with structured data that outlines all the requirements for the ncd cluster, which allows ncd operator to take action toward creating a cluster that meets all those requirements.

A

The process is conceptually similar to that of an engineer, working a github issue that outlines the requirements for creating a three-node ncd cluster. I have an example.

B

A

Once the engineer has read the requirements he or she will begin executing actions toward achieving the desired result like provisioning, three vms and then installing and configuring entity on each individual vm to create a functioning cluster.

A

The engineer will provide status updates in the issue that will track their progress on the task and ultimately close the issue when the task is completed.

A

The github issue is the same idea as the entity. Cluster crd and the engineer is the lcd operator for all the non-engineers in the room. Think of a person gathering three items in a grocery list at the grocery store, the person is going to read the piece of paper outlining the items and take three discrete actions that consist of finding each item and placing it in the cart. The grocery list is the structured data in this situation and the person gathering the items is the operator.

A

The key point in these analogies is enough. Information needs to be communicated to the operator in order for he or she to properly deliver the result. A lot of the same data you saw in the github issue that the engineer worked is going to appear in the.

B

Edcd cluster crd.

A

Resource there's, a few major sections of the ncd crd to highlight. First, is the size and version fields which are going to control the size and version that the lcd cluster is deployed with.

A

The next section are the scheduling rules which are going to build upon kube's scheduling logic to ensure the pods that run individual entity instances end up on three different nodes and three different availability zones.

A

Lastly, there are customizations to the configuration of individual ncd instances, including the supported cipher suites.

A

We are going to apply this entity cluster custom resource definition into the cluster.

A

Which entity operator will then read to to execute and create the database entity operator will execute the following series of logic loops after it detects the creation of the.

A

Crd in the first logic, loop entity operator has been told to create a cluster of three fcd instances at version 342 and currently has no existing instances. Etsy operator will create one.

B

Lcd instance.

A

To initialize the cluster and give it time to get healthy, sd operator will also update the crd to reflect what it has accomplished so far.

A

Now the next logic loop starts, etsy operator has been told to create a cluster of three fcd instances at version 342 and has one healthy.

B

A

Entity operator will call into the fcd cluster and configure it to add a new member. Then it will spin up an ncd instance that will function as that member and give the cluster time to stabilize.

A

Now the final logic loop starts that will execute an action entity operator has been told to create a ncd cluster of three entity, instances at version 342 and has two healthy instances.

A

Etsy operator will call into the ncd cluster and configure it to add a new member. Then it will spin up an ncd instance that will function as that member and give the cluster time to stabilize.

A

Finally, xd operator has been told to create a cluster of three entity, instances at version 342 and has three etsy three healthy instances.

A

Its work is finished, and now it takes some much-needed time off, puts on a vr, headset and watches the tulips in amsterdam safely from its home.

A

At this point, we have accomplished our goal of creating a functioning multi-z ltd cluster that is ready to serve requests.

A

In this theoretical situation, the cluster is put into production, use and results in downtime. If it goes down, the cluster is also under strict compliance guidelines that the organization managing it has to me. Security vulnerabilities can be disclosed at any time that impact the entity.

B

A

We are currently running. This is just one of the many reasons why we need to have a system that automates the entire ncd cluster upgrade process without any downtime etsy operator makes this process as simple as one command an update to the version field in the ncd cluster crd resource.

A

After we apply this change into our environment,.

A

Entity operator will detect that on the next reconciliation loop to begin the upgrade process, the upgrade process will consist of the following logic: loops.

A

In the first logic, loop etcd operator has been told to create a cluster of three fcd instances at version 343 and has three.

C

Healthy instances.

A

Running three four two etsy operator will patch the image of one etcd instance to be 343 and then give the cluster time to stabilize the execution of this logic. Loop can be identified when a pod in the entity, cluster registers, a restart.

A

The execution of the next logic loop can be identified by the restarts of two ncd pods in this loop entity operator has been told to create a cluster of three std instances at version 343 and has three healthy instances.

A

Two running three, four two and one running three four three etsy operator will patch the image of one lcd instance to be 343 and then give the cluster time to stabilize.

A

The execution of the final logic loop, that executes an action can be identified by the restarts of three entity.

A

Pods entity operator has been told to create a cluster of three entity instances at version 343. It has three healthy instances. One running three, four two and two running three four three etsy operator will patch the image of one of the xd instances to be 343 and then give the cluster time to get fully healthy.

A

At this point, the cluster is fully healthy, since all the pods are in two of two ready state and we have completed the upgrade process.

A

You'll note: throughout the entire upgrade process, we had two pods ready to serve traffic at any given time, which ensures, writes and reads, can be processed throughout the entire upgrade.

A

To recap, we were able to set up a ha multi-azee cluster in under four minutes and execute an entire database upgrade with zero downtime in under four minutes as well. Now I'll turn it over to cody to take us through the power etcd operator can provide in disaster scenarios.

C

Thank you tyler. My name is cody glosser. The next thing I'm going to talk about is what makes sed highly available and go through a few different disaster recovery scenarios, as tyler pointed out before the fcd cluster custom resource definition has pod affinity, rules that prevent paws from being scheduled into the same zone and host based off the node labels.

C

As you can see in the image here, the pod affinity rules look at a topology key that specifies the zone. Let's go through a few different scenarios to help visualize high availability.

C

In this scenario, our lcd cluster pods are scheduled across three zones. Del 10 del 12 and dial 13. each zone has four worker nodes. Now, let's say we have a power outage and we lose nodes in dial 10, including the node that the sd cluster pod is running on.

C

The scd operator will detect that the pod is dead and the actual peers does not equal the desired peers. So we'll attempt to schedule a pod we'll look for a running and ready node in dial 10 to be able to schedule to.

C

As you can see, in this example, the new pod came up on worker 1 in dell 10.. Now, let's do the same scenario interacting directly with the kubernetes cluster.

C

As you can see here, I have the same setup with four nodes in dial, 12, 4 and dial 13 and 4 and dial 10.. Additionally, you can see the lcd pods are scheduled to a node in each zone. Now we're going to simulate a zone failure in dell 10 on these two nodes.

C

With the watch on the lcd pods, you will see that the lcd pod starts to go into a terminating state and then eventually the fcd operator spin back up a new healthy pod in dell 10.

C

For scenario, 2: let's see what happens if all nodes go down in dell 10., we either have the option to open up a new zone or wait until we get a healthy node in dial 10 up. While we are at two or three pods, the fcd cluster is still healthy and achieves quorum. So there's no outage to the instance.

C

If we were to lose another zone or lcd pod, then quorum would be broken back to our kubernetes cluster. We will demonstrate this scenario. As you can see, we have all the fcd pods back up and running. Next we will simulate killing all nodes in the zone.

C

With this watch up, you will see that the scd pod in dial 10 goes into a terminating state. As you can see, all the nodes below in dalton have entered a scheduling, disabled state and will eventually hit a not ready state.

C

A new lcd pod could spun up, but will be an impending state until a new node becomes healthy in dell.

C

C

Describing the pending pod you'll, see zero of twelve nodes are available with four nodes. Unscheduleable and eight nodes do not meet. Are pod affinity, slash anti-evidently rules for our next scenario. What happens if we break quorum quorum? Is the minimum number of members of an etcd cluster necessary to have it in a functioning state? Quorum is achieved from having majority of pods in a running and healthy state in our case is two of three running pods.

C

As you can see. In this scenario, we lose all of our nodes in dell 10 and a node in dell 12 that has our other lcd pod running on it with only one etsy pod running the data is still being persisted in the pod and is in a read-only state to the lcd operator instances.

C

If we were to lose the final pod in dell 12, we will not be able to recover the data and manual recovery is needed to recover from the most recent backup. Now, let's go back to our kubernetes cluster and demonstrate both of these cases when losing quorum.

C

As you can see, the fcd cluster has three or three running pods. Now let me write some data to the ncd database, let's put in kubecon 2020 as the value in demo as the key. Now let me simulate failure in dell 10 by killing all the nodes in that zone and one node in dial 12.

C

As you can see, two of the three pods have gone into a terminating state. The third running pod will stay in this running state to allow us to take a backup of the most recent ncd data. Let's try to read and write to the scd database.

C

Again, let me write the value lost data to the key demo.

C

And, as you can see, I'm getting a context deadline exceeded error. Let me try to just read the key demo.

C

And, as you can see, I'm getting the same context deadline exceeded error with this one at cd pod up. Let's attempt a backup and restore. So now we need to construct an cd backup, custom resource definition. You can see an example of that here. The sections highlighted in blue is information on where this backup will be stored. It contains the backup location, the path and the credentials to get to that location.

C

The section highlighted in red is information on how the etc backup operator can contact the fcd instance. This end point is etcd headless kubernetes service endpoint for this fcd cluster back to the kubernetes cluster. Let's apply the fcd backup resource and wait for it to succeed.

C

You can see that it succeeded by looking at the status field of the fcd backup resource. Now that the lcd data is backed up, we can restore the sd cluster without losing any of our data before we do a restore. Let's recover our nodes.

C

Now we need to construct an sd restore custom resource definition. An example of that can be found here. Section highlighted in blue is information on where the backup is stored, notice how this is the same path we used for the fcd backup resource.

C

The section highlighted in red is the name of the fcd backup resource that the data will be copied to back to the kubernetes cluster. Now, let's apply the fcd restore resource and wait for all the lcd pods to come back.

C

C

Now that the pods are back up and running, let's try and read that key from sce to confirm. We did not lose any data.

C

So again, I'm going to read from the key demo and we can see that kubecon 2020 is still written in that field titan's still intact, and everything is back to functioning normally for our last.

B

C

Scenario 4: we lose all our etcd pods and break quorum. As you can see here we lost all our nodes in dell 10, one node in dial, 12 and another node in dell 13. back to the kubernetes cluster. As you can see, the fcd cluster is back to three to three pods running now. Let's write data to the scdub database. Again, let's put in lost data and write it to the key demo.

C

Let me simulate a complete failure in dell 10 by killing all the nodes, one node in dial 12 and one node in dial, 13.

C

Now that you can see all the pods have died, let's attempt to do a backup with the same ammo we used before.

C

As you can see in the status of the backup we were unable to complete it because the scd backup operator is not able to contact the lcd cluster.

C

The next step to do is to restore from the most recent backup the one we saved in the previous scenario, but first, let's recover our nodes again now that the nodes are recovered, let's apply the same lcd restore resource again and wait for all the lcd pods to come back up.

C

Now that all the pods are up and running, let's try and read from our ncd database. Let's read the key demo again and, as you can see, the most recent data we wrote lost data is no longer there and it was reverted back to kubecon 2020..

C

That completes my demo on ncd high availability and disaster recovery.