Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 20,000 Upgrades Later: Lessons From a Year of Managed Kubernetes Upgrades - Adam Wolfe Gordon

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

20,000 Upgrades Later: Lessons From a Year of Managed Kubernetes Upgrades - Adam Wolfe Gordon, DigitalOcean

Upgrading to a new release is one of the most disruptive operations we regularly inflict on our Kubernetes clusters. There are multiple strategies for doing an upgrade, but they all require rescheduling workloads and restarting cluster components. We started offering upgrades on our managed Kubernetes platform, DigitalOcean Kubernetes Service (DOKS), in May 2019. Since then, our customers have kicked off about 20,000 automated patch and minor release upgrades on their clusters. Most of those upgrades went well, but some didn't and we've learned a few things from the ones that went wrong. In this talk we will share lessons from a year of automated Kubernetes upgrades: what we got right, what we got wrong, workloads that caused us trouble, and changes we've made to make the process smoother. We hope these lessons will help others avoid pain in their Kubernetes upgrades.

https://sched.co/ZepW

A

All right, hi everyone, I'm adam wolf, gordon, I'm an engineer at digitalocean. I currently work as the tech lead for our kubernetes and container registry products, and I'm going to talk today about kubernetes upgrades and our experience with them.

A

I'm going to talk about how we do upgrades at digitalocean things that we got right and things that we got wrong in that process.

A

But, more importantly, what I want to talk about today is what we've learned about doing upgrades and some lessons that you can apply as either a cluster operator, someone who could be upgrading clusters or as a developer, who's deploying your applications into a kubernetes cluster. That's going to be upgraded. These are things that hopefully will help your upgrades be easier and keep your workloads running as expected. As you do upgrades on your clusters, I want to start with a little bit of a story about how this talk came to be so.

A

This talk really starts about. A year ago, in barcelona, at kubecon, eu 2019 and in barcelona we announced the general availability of our managed kubernetes product, and, if you stopped by our lovely booth in barcelona, one of the you would have one of the things we would have told you is that our product is now ga and you might have asked us what that meant.

A

We would have told you a bunch of things that we had in our ga blog post. Here. um We would have told you that it's production ready ready for your workloads to be trusted.

A

We would have told you that it has an integrated monitoring service and right at the sort of end of the second paragraph. We would have told you that it has automated patch version upgrades and what we didn't tell you was that you couldn't actually upgrade your cluster. Yet we hadn't enabled any upgrade paths for our customers.

A

um We had tested our upgrade process quite a bit. We run lots of upgrades on test clusters, but we hadn't exposed our upgrade process to the full richness of possible configurations and workloads that our customers might have, and we didn't want to do that. While we were all in barcelona, we wanted to wait until we were back at normal work. So uh we, if you went to your control panel on digilotion at that time, looked at your cluster.

A

It would have told you your cluster was up to date, regardless of what version it was actually running.

A

So, as you can probably guess when we did enable upgrades the next week or so, we started learning things and that's when I had the idea that maybe I can give a talk about this one day uh and that's why this talk is lessons from a year managed kubernetes upgrades and when I wrote the the cfp for this talk the end of 2019.

A

I ran some numbers and I figured that by kubecon amsterdam, uh where we're supposed to give this talk, um we would have done about 20, 000 upgrades and that's a nice big round number. So I put in the title of the talk and in preparation for today I ran the numbers again and um we've now done about 35 000 upgrades, so we've accelerated a little bit over what I thought we would have done and uh that's we've had upgrades enabled for about a year.

A

So that's about 100 upgrades a day across thousands of clusters, and I'm not saying this just to sort of brag about how many upgrades you've done. Although I think it's an interesting number, I'm saying this mostly just to let you know that we have a pretty good set of data to draw some lessons from, and I feel like we've done enough upgrades that we can really talk about what we've learned at this point.

A

So that uh leads me to my favorite slide, which is disclaimers, and I have two disclaimers for this talk. One is that uh these are lessons from our upgrade process.

A

There are lots of different ways to upgrade kubernetes and I'm going to talk about how we do it, but, depending on how you choose to do it, some of these lessons will be applicable to you and some of them won't, and if what you take away from this talk, is that you don't want to do it the way we do it, because you don't want to hit those problems, then that's a completely fair takeaway.

A

The other disclaimer is that this is things we've learned from upgrading our customers clusters and their workloads are probably different than your workloads they're, definitely different than our workloads, depending on your environment. How you run things you might also hit different things.

A

So let's talk about how you upgrade kubernetes and what you actually do when you upgrade it. There are basically two parts of the kubernetes cluster: there's the control plane and there are the worker nodes. So when you upgrade kubernetes first you're going to upgrade the control plane, then you're going to upgrade the work nodes and that's it you're done you've upgraded group headings.

A

Of course it's not actually quite that simple.

A

So here's an expanded set of steps of things that you have to do to upgrade when you're upgrading the control, plane, you're, really upgrading a lot of things. So the first thing you're going to do is update any resources that aren't supported in your target version. Then you're going to upgrade mcd. If you need to then the actual control plane components. First, the api server, then the controller manager, then the scheduler.

A

Then you can update your cni plug-in if you're using cni for networking. Finally, you can upgrade any provider specific components of the cloud controller manager, csi driver and then the cubelet and cubectl on the control plane. Assuming that you're running things using those once you've done that, then you can upgrade the worker nodes into upgrade a worker node you're going to cordon and drain it so that there aren't any workloads running on it. Then you're going to update the cubic configuration.

A

uh If you need to upgrade the cubelet uncord in the node, let worklets come back onto it and you're going to repeat for each node in your cluster.

A

If you're running kubernetes on virtual machines and not bare metal, there's a bit of a shortcut you can take and that's rather than upgrading each individual software component in the cluster you're, going to just replace all of the nodes with new ones that have the new software installed on them, so that first step is still applicable.

A

You're still going to read the release, notes find out that everything is supported in your new version, update it if you need to, but then to replace your control, plane, you're, going to destroy the old control, plane, virtual machine and provision a brand new one with all the new software and new configuration.

A

This does assume that your xcd data is persisted somewhere outside of that control, plane, node or nodes, so either you're replicating it across multiple nodes or you've, got it stored on a persistent volume or you've got it stored somewhere entirely outside your cluster. But assuming that's the case, you can do it in this one step. Just provisioning a new node same with the worker nodes, you're still going to coordinate and drain each one, but then you can destroy a node provision.

A

A brand new one in its place if you've worked with kubernetes clusters quite a bit, you can probably see some potential issues with this scheme of doing upgrades and there definitely are some issues, but there are also some advantages, and this is uh the upgrade process that we use at digitalocean for our managed product.

A

We replace all of the nodes in a cluster when we want to do an upgrade.

A

There's a few advantages to this, and this is why we chose to do it this way. First off. It means that every node in the upgraded cluster is a clean slate. There's no chance that there's been customized configuration on that node. There's some customizations made to the node, that's going to persist across an upgrade and cause problems in the new version.

A

It's also a nice process to automate, there's not that many steps and, if you've built any automation for managing your cluster, you've, probably already automated the steps that you need draining a node, destroying a node, creating a new one waiting for a node to join a cluster. So automating. This upgrade process is just a matter of combining those things that you've already automated into the right order, adding some sanity checks and things, and um that was the case for us. We already had the the primitives that we needed to make this happen and automating.

A

It was pretty easy. uh The other nice thing is that this works the same across all kinds of upgrades, regardless of whether you're doing a patch version upgrade a minor version upgrade which components need to be upgraded, it should all work.

A

I say it mostly works for all release, types and all different kinds of upgrades, because there are definitely some cases we've hit where we have to do really specific things, but for the most part, this process has worked for us, regardless of what upgrade we're doing, which version- and I say all of these things that we decided to do- are specific a little bit to our environment, of running many many clusters and doing completely automated upgrades.

A

It's your trade-offs are going to be a little bit different depending on how many clusters in your operating environment,.

A

So I said we were going to talk about things. We got right and things we got wrong in building our upgrades, and this is the first thing I think we really got right is we did our upgrades via node replacement, um so it if you're, going to upgrade if you're going to manage a lot of clusters and build automation, I'd recommend considering this as a way to do your upgrades.

A

The downside of doing a node by node replacement upgrade is that once you've upgraded a node you're left with a brand new, completely different node, so any custom configuration on the node is going to be lost. It's going to have a new name in kubernetes, it's going to have a new ip address and if you had set any labels or any taints on it through kubernetes api, those are going to be lost as well.

A

So this bit some of our customers, um people who are scheduling their workloads to a specific node based on its name or who were accessing nodes directly by their ip because they were routing traffic to their cluster. That way, they they had some issues when they first did upgrades with this process.

A

So a lesson for for operators here. Other people who are upgrading kubernetes clusters is, if you can it's great to reuse, the node names and the ips of your nodes, I even if you're replacing the nodes to upgrade them. um This isn't really something that people should expect from their kubernetes cluster, but it is a tool that's available in kubernetes, so someone's going to use it and you'll make their life easier.

A

If you keep your note names and your ips stable, regardless of whether you're keeping those node identities stable, you definitely want to provide some way to set persistent labels and taints on nodes.

A

uh Those are a great tool to use for scheduling in kubernetes, and people expect them to work so providing some kind of abstraction to let people set labels and taints on their nodes is important. That's something that we've done in the time since we initially introduced upgrades.

A

The other piece is provide some simple way to get traffic into a cluster.

A

Getting traffic into a kubernetes cluster is worth its own talk, and I'm not going to talk about all the details here, but the easier you make it, the less people less likely people are to build their own solution on top of their cluster, and uh it's really people building their own solution that are going to run into trouble if their node ips change. For example, during an number.

A

There's also some lessons for developers here in terms of what you can, but your workload can tolerate for change. So the first lesson is: if you do need to customize your nodes use kubernetes tools to do it. There's two great ways to do this: one is to deploy a privileged daemon set to your cluster and that's going to run on all the nodes. When a node comes up, it's going to make the necessary customization and then your node is in the state.

A

You expect the other way is within a net container on one of your workloads. If you have a workload that requires specific node configuration, you can do a configuration from an init container and when it's done, your workloads are ready to run.

A

Secondly, please don't use node names for scheduling. It is a tool you can use in kubernetes, but at some point a node is going to go away. That's kind of how kubernetes clusters are um you don't want your workload to be left unschedulable, because a node with a particular name is going over there wherever possible, use your provider supported or operator supported label and taint settings. So on some providers, that's going to be just setting labels and taints through the kubernetes api on other providers.

A

It might mean creating a node pool or some other abstraction to set labels and contains for your nodes and same with load balancing. If you use your provider or operator supported, load balancing, then you're much less likely to run into problems during an upgrade, because those are well integrated with your cluster.

A

There are definitely some things we got wrong in our upgrade process, which I'll talk about next, the first big one is doing. We implemented our node replacement in a break before make fashion, so uh we destroy a worker node, and then we provision a replacement for it. um We did this for some reasons that are specific to how our product works, but this has definitely caused a lot of trouble for users, and this is something that we're working on changing right now, where we will create new nodes before we destroy old ones.

A

The basic problem that this causes for our customers is that I, when they, when we go to drain a node, there might not be anywhere to drain that node too either, because there's not enough capacity in the cluster or because they're running a single node cluster, for whatever reason um and and so draining the the node can get stuck, and that causes a problem for the upgrade.

A

um This also just causes extra churn for workloads. When you drain the first node you're guaranteed that those workloads are going to end up on another node, that's running that old version and those nodes are going to need to be drained right away as well. So this your workloads are all going to end up being evicted from a node more than once, instead of just once during the upgrade process, and that's unfortunate for quotes.

A

So some lessons for operators here, um if it's all possible, figure out a way to create new nodes before you delete old ones. um This might be a little bit more complicated for you to automate.

A

It is in our case because we want to keep the number of nodes sort of stable in the cluster, but it is a better experience for users, you're less likely to have workloads evicted more than once during the upgrade, and if you really can't do that, because you're on hardware or upgrading in place or whatever consider reserving some capacity in the cluster to make sure that you're always able to drain a node.

A

Likewise, for developers, um if you're deploying things to kubernetes cluster, try not to run it right to the limit, leave some room so that you always can drain a whole node worth of workloads, because at some point a node is going to go away. Even if it doesn't happen during upgrade. It's going to happen some other time because of hardware failure, maintenance or whatever, and you just want to make sure that that's not going to cause a problem for your application.

A

Another thing we got wrong is that we replace nodes exactly one by one, so we destroy one node and then create one node then destroy one node and then create one node, and this is where it works. Just fine in a three node cluster. It's a lot less good in a 300 node cluster and it's really bad. If we run into timeouts draining nodes, we initially had set our drain time up to an hour and we dialed it back to 15 minutes, but even at 15 minutes.

A

If you have a 20 node cluster and each node takes 15 minutes to drain, that's a five hour upgrade at least right, because that's just the time to drain the nodes, um and that's not a good experience for for a user.

A

So to restate that problem, a little more succinctly, replacing nodes one by one is slow and it can be made even slower if you get stuck evicting a workload upgrades are going to take some time, naturally, because we do have to shuffle workloads around and wait for things to come up, but you want them to be as fast as you can.

A

Users are going to want to watch their upgrades happen and uh they're, or they only want to watch for so long. You don't want to have someone watching their upgrade happen for five hours.

A

So a lesson for cluster operators here um try and replace multiple nodes at once. If you can this, this will really help make an upgrade on a big cluster faster. This kind of requires that you do make before break like I was talking about a minute ago.

A

Your users might have set aside some capacity for a note to be drained, but they're, probably not expecting like 10 nodes at a time to be drained, so you kind of have to add capacity to the cluster before you reduce.

A

It um also make sure that you set reasonable drain timeouts, uh where it does take some time to drain a node, but it shouldn't take an hour and uh that you kind of have to figure out the right balances for your workloads in your environment, um but setting those timeouts to a reasonable value is going to help. Those upgrades be a little bit faster for developers. There's not much! You can do about how your provider or your operator replaces your the nodes in your cluster.

A

But there are a couple things you can do to make sure that your workloads can be evicted. You want to make sure of two things, one that your workloads can be evicted safely, so use pod disruption. Budgets make sure you understand how your workloads can be affected during an eviction and make sure that your workload can be evicted quickly.

A

So have your application respond to the signals that kubernetes sends when it depicts an app when it depicts evicts a pod so that you shut down as quickly as you can and let the with the draining process move on, and my big tip is just test this at some point, your workload is going to be evicted if you're running under kubernetes, it's a great thing to test and just make sure that it works, as you expect make sure that when you delete a node in a cluster, your application keeps running as you expect, and you don't have any unexpected problems back to the positive side of things.

A

This is another thing that we got wrong, but we were actually pretty happy. We got wrong. I mentioned earlier that when we did rga, we offered automated patch version upgrades, so that means like 114 1 to 114 2, not 114, 2, to 15 0..

A

um We started out that way because patch version upgrades are a little bit simpler. Resources aren't supposed to change between patch versions, so everything that you have running in your cluster should continue working after the upgrade, um and it was a good idea for us to start out that way. But when we did get to doing minor version upgrades, we discovered that everything pretty much worked as expected, it was much less uh fraught than we expected. So um that's that's. The lesson here really is just don't worry too much about minor version upgrades.

A

They aren't as scary as you might think. They are at least they weren't for us and assuming that you've designed a good process for doing upgrades, I won't be that different from a patch version upgrade.

A

One thing we did do right that helped these minor version upgrades go well, is that we leave alpha features disabled as much as we can. Alpha features are disabled by default in kubernetes, and we try not to enable them unless we have a really good reason too. Alpha features are the most likely things to change between releases and leaving them. Disabled just eliminates a possible source of problems, as you do upgrades, especially lighter version upgrades so for operators.

A

The lesson here is really simple: just wait until it features beta before you enable it, and if you do really want to enable an alpha feature, because you have a good business use case for it. uh Consider that the upgrade trade-off you are probably gonna have to put some work into upgrading clusters that have that alpha feature. um That's the the trade-off that you're gonna make by enabling it for developers the blessing is similar just be aware of using alpha features if you are using alpha features, uh really upgrade the release.

A

Notes really carefully before you upgrade make sure you understand, what's happening to the feature as it becomes beta and any changes that you might need to make any changes that your operator might need to make. um As long as you you understand what's happening, you should be okay,.

A

So, let's spend the rest of our time here on two common classes of problems that we've seen with upgrades, and the first is problems around container storage, interface or csi.

A

So for those of you who aren't familiar, csi is a plugable way to allow workloads in in kubernetes or another container orchestrator to consume storage from an arbitrary storage system.

A

Kubernetes clusters that are hosted in digitalocean, whether they're part of our managed product or managed outside of it by a user, can use our open source csi plugin to attach our block storage to their workloads, and this is uh how we recommend our users get persistence if they need persistence for an application.

A

Csi has been problem for us in a few different ways, as it relates to upgrades the the biggest one is just the general immaturity of csi on the versions that we started with in our product. So the first version of kubernetes that we supported at digital ocean was kubernetes 1.10, and that was the same release in which csi became a beta feature. I moved out of alpha so in that 1.10 time frame, the kubernetes components for csi were relatively new.

A

The csi drivers, including ours, were relatively new and there were some bugs there, and basically, the problem that we ran into in upgrades is that we hit bugs there's a lot of csi action going on during an upgrade because you are draining nodes and that means uh detaching volumes from workloads that are being evicted and reattaching them elsewhere.

A

As we schedule the workloads on other nodes, um we would end up hitting bugs where the state of kubernetes and the state of the real world were out of sync and we couldn't drain a workload off of a node or we couldn't start a workflow on another node, because the volumes aren't throwing place.

A

That's that's the basic problem that we hid. Luckily, everything has matured a lot since those days I would say in kubernetes, 114 plus, we really see very few csi problems, even during upgrades. So if you're on a newer version, you should be okay. This is a good reason to upgrade in itself, even though you might experience some pain on older releases, um but yeah, it really has sort of taken care of itself in more recent releases.

A

One other problem we hit- and this is one place where we had to introduce version specific code for some of our minor version upgrades- is the csi driver name change in earlier versions of the csi spec drivers were identified by reverse fqdn, notation, so com.example.csi in later versions of the spec. They changed that to be forward fqdn, so csi.example.com, and this driver name gets used in various places in kubernetes, including being attached to each volume that the driver creates. That's how kubernetes maps between uh volumes and the drivers are supposed to manage them.

A

The this name for good reason is immutable in kubernetes, so once you've installed a csi driver into your cluster, you can't change its name. um The problem that we ran into is we changed the name in our driver to match the new spec and uh we then weren't able to manage the old volumes that had been created by the previous driver.

A

um Our solution was to just detect whether our an existing kubernetes cluster was using the old name or the new name and make our software configurable uh for the name. So we can keep using the old name in a newer version of the driver when we need to um that's, unfortunately, a part of our upgrade process that will probably be with us forever in this little version specific thing, so that's uh one place where the process didn't sort of just work.

A

The lesson here for both operators and developers is just be a little bit careful around csi. Your persistent data is probably important to you, so you don't want to have csi problems and persistence problems during an upgrade.

A

um If you are using csi, be careful with your upgrades, make sure you test them watch for workloads getting stuck and use a newer version of kubernetes if at all possible, uh like I said, 114 plus, we've really had basically no csi problems, so I'd highly recommend if you have a choice using one of those newer versions.

A

The last problem I want to talk about is probably the most common problem, we've seen with upgrades on our platform and that's problems with admission control web hooks. These problems are possible in any environment and with any upgrade process, they're really not specific to us. So I'm going to spend a little bit of time going through what the problem is.

A

Some ways to avoid it for anyone who hasn't seen your mission controlled by books, I've got a really quick overview, uh they're, basically a configuration you can make to have the kubernetes api server call out to a service and decide whether a particular resource should be created or not so the sequence diagram here shows what happens. I you go to create a resource in kubernetes. It's going to ask the web hook service whether or not it should be created. If it is, then everything goes ahead as planned.

A

If not, it gets rejected and it's very common to host these services inside your cluster. Although you can't host them externally,.

A

The problems happen if the web hook service isn't running and it can't respond to the api server. So what happens in this case? Where the service is not running and it's not able to respond?

A

What happens here depends and it depends on the failure policy configured for the web hook. If the failure policy is fail, then, when the web hook service isn't available, the api server will just disallow the creation.

A

If it's ignore, then the api server is going to act as if the web hook wasn't there at all and allow the operation to go forward.

A

I'm going to come back to this in just a sec, but I want to talk about how it affects upgrades.

A

The problem with web hook string upgrades is that during an upgrade, we update various system components that run as workloads, so things like coordinates or q proxy or our cni driver, which is psyllium.

A

These are usually things that run in the cube system namespace, but they could run on other namespaces too, and the problem is that hooks can prevent updates to these services from happening, and that causes a problem for the upgrade, because now we're not able to update some of the managed components in the cluster.

A

So, just to illustrate what can happen if we go back to this configuration, we have this web hook. That applies to pod creations in any namespace, and let's say we have a cluster here. That's running the webhook service on a node and also on that node are the normal cube system, things, q, proxy and psyllium.

A

In this case, when we go to do an upgrade, we're going to drain that node, and so it's going to evict the web hook service and the kubernetes scheduler is not going to be able to create a new pod for that service, particularly specifically, the the deployment controller isn't going to be able to create a new pod for that service, because the web hook's not running so it tries to create a new pod with the api server.

A

The api server fails to reach the service because it just depicted the service and now we're not able to create a new pod for it.

A

Now we create a new node and we go to schedule the system components on this node, so things like cube proxy and psyllium, and we also can't schedule those the daemon set controller. Can't create pods because the web hook services are writing. So now we have a node, that's essentially useless. It's not running the stuff.

A

It needs to run to be a full part of the cluster, um and if this happens, your clusters can be in pretty bad shape, because now you have no nodes that are usable, um or at least your new nodes aren't usable your upgrades either going to be stuck or your clusters can be broken.

A

So the simple solution here is to change the failure policy to ignore um that actually can cause another problem. That is not very obvious. um That problem is the timeout. It turns out that pretty much all the default timeouts in kubernetes are 30 seconds. That includes the tempo for webhooks. It also includes the api server timeout. So if you have a webhook with the failure policy of ignore, but its timeout is 30 seconds because you haven't specified one um you that timeout is not going to end up battering it's.

A

The api server is going to wait 30 seconds for your webhook service to return an answer, but by the time it hits that timeout. The request itself is timed out and it can't go ahead, so the the recommendation here is to keep your timeouts much lower than 30 seconds, regardless of what your failure policy is, and this is actually the recommendation- that's in the official kubernetes docs too. uh So it's not just me saying this.

A

This is really the guidance from kubernetes upstream community as well, and the configuration I'm showing on the slide now with the smaller timeout and failure policy of ignore that's never going to cause any problems during upgrade.

A

If you actually do want your failure policy to be fail, you can configure your web hook to ignore the cube system, namespace and any other critical namespaces.

A

You want to also make sure that you're ignoring the namespace that your webhook server itself runs in as long as you're doing that the the webhook won't apply to those things and you'll be able to schedule your important services on the cluster during an upgrade.

A

So the lesson for operators here webhooks can can be trouble. You want to just make sure you check the configurations before you start an upgrade. We use a tool we built called cluster link for this. It's an open source tool.

A

It will throw a warning if we are likely to encounter problems with web folks during an upgrade.

A

The other solution to consider is having a mutating web hook that will mutate the web hook, configurations as they come in and basically make sure that they ignore keep system make sure they ignore anything else, have an appropriate timeout all of that stuff. So that's a good way to avoid the problem in the first place.

A

Lesson for developers is uh basically the manifest that was on the slide. Just be really careful with web hooks check. Your configuration um exclude cube system and the name space that the web hook service itself runs in, exclude any other critical name namespaces and you you might want to again consider the mutating web book solution just to make sure that you don't configure a web hook. That's going to cause problems later, and you might want to also consider running your web hooks outside of your cluster. If you can, because that's less likely to cause problems.

A

So, just to wrap up here's all the kind of advice that I gave today on one slide. The first is to consider upgrading via node replacement and the kind of tips that I made gave to make that go smoothly.

A

Secondly, as a developer make sure that your workloads can be evicted when you're doing an upgrade, try and upgrade more than one node at a time, if you can minor versions are probably upgrade, are probably easier than you think they are especially if you've avoided alpha features, so don't be so scared of minor version. Upgrades um csi is, I would say, just now becoming mature.

A

It's really mature in the last few versions of kubernetes, but you'll want to take extra special care when you're upgrading, a cluster that uses csi and finally, admission control books are lots of trouble. uh You want to run some checks on those and make sure that you don't have any configurations that are going to cause that are going to prevent you from deploying your system components into the cluster. As you do your upgrade. um It's a big problem that we've seen with our customers clusters.

A

And that's all I have for today.

A

All right, I think it's time for live questions, so there's a bunch of questions that came in during the talk and I'll deal with those those first.

A

um So the first one is: how often should we upgrade, and what do I recommend in terms of which version to stay on that's going to depend a lot on your environment and your comfort level, with taking up new versions and how disruptive your upgrades are for your your workloads um as a service provider and someone who's running a managed product for kubernetes.

A

We always try and provide the latest minor version. So currently that's 118 soon that'll be 119., and then we support the two trailing versions, uh just like the upstream kubernetes community does for bug, fixes and security fixes.

A

So I would say you definitely want to stay within that supported range if you can um so like the current release or the the two previous ones in terms of like how quickly you want to upgrade when a new release comes out, like I said, that's going to depend a lot on how confident you are in your in your workloads tolerating the upgrade and uh whether you can test it um in your environment. So I think that's that's!

A

A it's gonna depend a lot on on your own environment and how your team feels about your kubernetes clusters.

A

uh Another question is: how did we solve the taint problem so that I talked about persisting, node pool labels and taints or node labels and taints?

A

We've done this in our managed product by allowing users to set uh persistent labels on a pool of nodes, and so when we create a new node in the pool, it'll get those labels.

A

uh We haven't actually rolled out the taint part of that yet, but we are working on that right now, so the idea is uh rather than setting them directly to kubernetes api you'll, set them to a managed product and then any new nodes that we create for an upgrade or when you scale off a pool or any other reason, we'll get the same labels and teams that you want.

A

uh What did we do to for a drain timeouts if you have a pod disruption budget set to one, for example, um what we the way that we drain a related question was uh around how we drain nodes. um We, when we drain a node, we have two modes, so initially we're in our eviction mode. So we will just try we'll just mark each pawn for eviction and that's where we set a timeout. So we we said we mark each pod for eviction.

A

We wait uh 30 minutes or 15 minutes, whatever our timeout is and when we get to the end of that, we're just going to delete the plot off the notes with a hard delete and uh then we're just going to delete the node. So the font's going to go away. um So that's that's what we do if you have a pod that just will not be evicted for whatever reason um it's going to go away, eventually we're not going to keep the note around forever.

A

uh Another good question is uh whether our upgrade tool lives inside the cluster or outside the cluster. In our case, it lives outside of the cluster. We, uh since it is a managed product, we don't run.

A

We do have a component that runs in the cluster to make uh kubernetes changes to the api, but um the the upgrade automation itself is outside of the cluster. So it's it's operating on the cluster from the outside um there's another one around uh balancing the availability of applications as we upgrade nodes and replace nodes, that's something that we as a managed provider.

A

We kind of leave that to the customer like we expect that the customer will have pod disruption budgets if they need them that they will have multiple replicas of uh their applications if they, if they need availability, and we respect all of that like it it when, when we evict a pod, um it is going to uh respect any affinity or anti-affinity. It's going to respect the pod disruption budget. So as long as it can be evicted and not like I said, we do now um enable the the surge upgrade option.

A

So you can have new nodes created before we delete the old nodes. All your pods are going to be drained to those hopefully to those new nodes um and that that will keep your app available um if you're doing your own upgrades. That's just something to be mindful of that. You want to make sure that you are leaving some time for nodes to be drained and applications to restart on new nodes. Before you can proceed with your upgrade.

A

Basically, um there are a couple around checks or tests that we might do to make sure that uh an upgrade is going well or that it's working, um we kind of have a couple of phases in our upgrade, so we we upgrade that control, plane, node and we won't start upgrading the worker nodes until we know that that's up and healthy and then likewise with worker nodes, as we do, each worker node or each set of worker nodes, if we're doing a search upgrade, we won't continue with doing more nodes until we until our new nodes are up and healthy.

A

So if something did go wrong with the control plane upgrade, your workloads would still be running on the worker notes and you wouldn't have uh application unavailability.

A

It would basically pause, and we would have to you know- do something to resolve the issue.

A

We do take a snapshot of the etcd data before we start the upgrade so that if something really went wrong and the lcd volume was destroyed by a bad, a bad upgrade in some way, I've never had this actually happen, but it's possible.

A

um We do take that snapshot, so we could roll back to it if we needed to- and uh we've only got a couple of minutes left so I'll, maybe answer one more question, but also feel free to hop on slack in the two kubecon operations channel and I'll hang out there for a while to answer some more questions.

A

um But uh let's see we have one one question around vm images: um we build new vm images for each uh of our each version that we release and um that image is essentially like a golden image like we. We don't make any any changes to it once it's deployed, it's it's deployed as a worker node or has a control, plane node, and we do the setup once and then it lives for as long as the cluster lives. So that's um yeah, that's the that's!

A

How we deal with those images and we're always working on those images and updating them. uh We try and release updates at least sort of once a month uh with any new kubernetes patch versions and also any general improvements we've made.

A

So I think we'll we'll wrap it up. There, like, I said, feel free to come and ask some questions on slack and uh also feel free to to find me on twitter or email or elsewhere, and I'm happy to chat more about upgrades thanks. Everyone.