Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Stayin' Alive: PodDisruptionBudgets for Maintenance and Upgrades - Matthew Robson, Red Hat

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Stayin' Alive: PodDisruptionBudgets for Maintenance and Upgrades - Matthew Robson, Red Hat

As we work harder to automate our clusters, it becomes more and more difficult to guarantee the availability requirements of our applications. In large clusters, operations teams may not have the insights to ensure an applications minimum capacity requirements are maintained. Without that understanding, you may inadvertently bring down or inhibit applications through routine maintenance activities. Enter the Pod Disruption Budget (PDB). Simply put, PDBs allows application owners to define the minimum requirement for a service to operate in a stable manner. In this Lightning Talk, let me walk you through the benefits, usage and implementation of PDB’s. As an attendee, you will walk away with the necessary knowledge on how to use PDBs to define enforceable operating requirements of your applications.

https://sched.co/ZemK

A

Hi, everyone welcome to my lightning talk: staying alive, pod disruption, budgets for maintenance and upgrades. My name is matthew robson and I'm a principal technical account manager at red hat, working on openshift and kubernetes. If you have any questions following this lightning talk, feel free to reach out to me on twitter at matt j robson.

A

So what exactly is a pod disruption? Budget, a pdb for short, is an application owner created object that defines the minimum number of replicas that must be available for an application to operate in a stable manner during voluntary disruptions.

A

With that said, why would I want to use pod disruption budgets first off they're owned by the application team application owners best understand the requirements and performance characteristics of their services. This is also supportive of the operations team, because it bridges the gap of application. Knowledge to operational execution pdbs also define your availability requirements.

A

This could refer to maintaining things like quorum requirements for an sla or the minimum number of pods to support a specific workload.

A

Pdbs are also respected by the eviction api. This means anything like drain, auto scaling or the descheduler can leverage disruption budgets. Finally, all your favorite controllers, like deployments replication controllers, stateful sets and replica sets, are easily integrated with anything. There are some caveats: involuntary disruptions, like node crashes, hardware, failures and network outages cannot be prevented with disruption budgets.

A

Equally, if you explicitly delete your pods or you scale, your deployment to zero pdbs won't help. When dealing with single replicas, I would recommend not using disruption budgets, because they're burdensome on the operations team for any drain to occur. The pdb would need to be deleted. This can lead to things like indefinite hanging with drains.

A

Finally, do not overlap your selectors if you're creating multiple disruption, budgets. This can confuse things and lead to drains hanging as well.

A

The requirements are quite simple for disruption budgets, first of all, a meaningful name so that it's attributable to the pods that it oversees. Secondly, a match label corresponding to your particular controller selector. The third piece, either a min available or a max unavailable. Setting min available refers to the minimum amount of pods that must be available where max unavailable refers to the maximum number of pods that can be deleted at any point in time.

A

The definition looks like this as follows: first, we have our spec selector match label which is app django ws. You can see how this corresponds to our deployment selector of app django ws.

A

Then we have our min available or, conversely, our max unavailable setting. This can be an integer or a percentage when dealing with percentages. If the number of pods is not an even number, it automatically gets rounded up to the nearest whole integer in practice. Pod distribution budgets look as follows. You can see. Currently we have three pods our desired or min available is two meaning. We have an allowed disruption of one.

A

If we go ahead and drain our first worker, you can see, the node is cordoned, our pod django ws is evicted and then the drain is completed. Looking back at our disruption, budget object, you can see. Our current is now two our desired is two meaning. Our allowed disruptions is now zero. We can no longer tolerate any more pods being deleted if we try to drain a second worker where another one of those django pods is running.

A

You can see that it's going to fail because it would violate the pods disruption budget using a timeout of 10 seconds. We can see that the drain did not complete and we can reevaluate and try again to recap, encourage your application owners to define their operating requirements with pod disruption. Budgets, leverage your voluntary eviction tools, such as food control, drain for all of your maintenance and upgrade requirements, and remember the caveats and watch out for the bad practices that we discussed.

A

Thanks for listening to my lightning talk and enjoy the rest of kubecon cloud native con virtual europe 2020. thanks.