Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 24 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How Does Google Release Kubernetes in GKE - Kobi Magnezi & Josh Hoak, Google

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

How Does Google Release Kubernetes in GKE - Kobi Magnezi & Josh Hoak, Google

Working on the GKE, we have been releasing new versions of Kubernetes since before Kubernetes was 1.0. Although closely related to the Kubernetes release, the release of Kubernetes in GKE involves ensuring fleet-wide consistency through upgrades, qualification and customer feedback. We will talk about the history of the GKE Release process and the lessons we have learned working with Kubernetes. We will close with open source improvements we're making to systems and tooling so that others can benefit from what we have learned.

https://sched.co/MPZ7

A

All right something to do with.

A

Daily meetings to place very personal and must both it was a cartoon, often don't go being so tired.

A

Alright selecting an expectorant decision today, one we're going to speak about introduction to interpretation and how, in between community successors in the cloud and how we integrate to rise into GCD, we will speak about.

B

A

Gpb sports, the reasons.

A

A

This is justice.

B

And so like, let's look at the version number itself, so the version number has I mean you've seen this before it has two parts: it has what we call the vanilla, kubernetes version, that's the one 13 5, the major/minor patch, and then we have our gke specific patch number on top of that, the gke 10, and so all of our version numbers in gke look like this and that's because, at the end of the day that the gke 10 is specifying the patches that we've provided on top of it.

B

And it's because this version number is our version number for all of our software. It's the version number for our the OS image, the cube, API server, cubelet logging and so on. It's our kind of our bundle ID and what makes this challenging is well.

B

Of course, kubernetes is releasing very very fast I mean if you compare it to like, go which releases every say six months: kubernetes is releasing every quarter and that's that's so that's quite challenging, but you know when I think about the release of kubernetes I'm, not even just thinking about the release of say, minor versions. I'm also thinking about like the milestones that we've seen and we've seen a ton of huge milestones.

B

So just give you a little story when I came to gke I came in 20 sixteen, and it was like two months later and I was like sitting on my desk and like someone was playing on their phone. I was like they're playing this game.

B

It's like oh, that's kind of funny, and then someone down the hall was playing this game too, and then I walk outside, and some people are playing this game and then later that evening, like everyone is playing this game and like what am I talking about like I'm talking about Pokemon go was launched on gke and that was like well. What is this thing and I was like for the me. This was incredibly impactful. Cuz I was like oh gke is not just a toy thing, it's it's real and then you know from there.

B

We start to launch more and more features on gke. So.

A

We launched regional high availability clusters. Sto was announced. We also experienced some security incidents. We launched GPU support on G, key and private clusters and.

B

Just ajust a pause right there, you know GPUs for me, are like a big deal because a about a year ago the talk I gave at this conference was about mini, go, which was we were building a alphago kind of inspired, go AI, and so we built that using kubernetes, and we couldn't have done that without having GPUs, and it was like super easy. Given the fact that the GPU integration in kubernetes is great and.

A

Then, just recently, CN CF conducted a survey that shows that 73% of the industry deploy containers. That's pretty amazing, then recently we announced anthos. So the reason why we are thinking about these events- and there are other milestones obviously, but we highlighted this specifically and thinking about this- we grouped them into three themes and those themes really impact the way we release versions into gke.

A

The first one is market adoption, if you think about a time line that just just described back in 2015, probably the technology kubernetes technology was mainly used by very tech-savvy company startups, but nowadays we see the amazing adoption of kubernetes and we see other industries, financial services, healthcare providers and large enterprises using urban areas in production. For us, it means that we need to rethink the way we release versions into Jiki. We need to take into consideration internal processes for this larger organization.

A

We need to consider compliance, regulation and other issues that may impact the cadence and the nature of the releases that we we are releasing. The other thing is we: are we supporting variety of application, we're supporting stateful application, stateless application, and with that we also support the growing ecosystem, so we have more add-ons more partners that we are integrating with and and finally, we are also enabling cloud infrastructure on kubernetes. So if you want to use GPU, TPU or networking features, we are doing the integration between the cloud and kubernetes.

A

That really requires us to carefully think about our releases and balance between velocity and stability.

B

Great, so I'm gonna talk, give it a high-level overview of the release process. Of course, the release process is very complicated, so I'm gonna talk about several different ways. We can think about the way that we do releases, so the first is kind of like kind of a process view like when I think about the release process, I kind of break it down as a five, very core steps. So one is develop. You know people are working on various parts of the kubernetes and the add ons and then we build, which is for me.

B

That means we're building docker images and we're pushing those to say docker hub or we're pushing those to Google container registry and then there's a packaging step and I see these as different, because for me now that there is a docker image now that hat now, that goes and say some kubernetes yamo files and from the kubernetes release and will take the kubernetes tarballs and we have to group those together and then we qualify them and the qualification is both like testing. It's also a soak time. So we have a test environment where we qualify.

B

It for quite a long time and at some point could take weeks, sometimes even months, we'll decide to cut a release candidate and then we'll roll it out to production, and all of this happens with again. You know that kubernetes version number, because really it's our ID. That says all of the things that work bundling together and then just to go back to roll out. Our roll out happens over four days so that you know if something breaks and hopefully it doesn't, but that will catch it on say day one or at worst day.

B

Two and that day three and day 4 are hopefully not affected by the rollout.

B

So what are we rolling out? If you look at our release, notes which we talked about two different sets of versions, and we talked about two different actions that we're taking so the two different versions are the master version of the note version. Now the master version is the version for all the software that we're releasing on the master, but also, as Kobe mentioned, the add-ons right, because those are are managed by us and the newid version turns out not to be very much stuff.

B

It's, but it's it is important, and that is the OS image version and the cubelet version. And then, if you going back to the actions, the actions we can take are either here. Are these available versions for you or you know here- are some upgrades so for upgrading the master. Here's kind of what I just described in picture form, a master upgrade, will upgrade everything on the master, OS image, all those components. It's also gonna upgrade those add-ons and a note upgrade is gonna upgrade just the OS image and the cubelet.

B

But this is important because we have to stay ensure that the the cubelet stays within some range of the masters. Now the management of the kubernetes nose is kind of a shared responsibility. If you opt into node, auto upgrades will do it and if not, then it's kind of your responsibility, but it's important that these things kind of stay closer together. So over time, we've tried to encourage, be more and more people to go with node auto upgrades, because the skew can really be a problem as kubernetes is a fast-moving product.

B

Ok, so here's another lens that I like to think about when I think about the life cycle of releases and that's the version life cycle itself. If you look at at gke- and you look at the say, our release notes- we have 3 different version, minor versions at any, given point that we offer there's say anyway, an NN +, 1, +, +, 2 +, and that's because of the life cycle of their versions.

B

When we release a new version on gke, it's a what we call like a new version, and that means you can create clusters at it. But you know we think it's GA, but there might be bugs right. But again you know these are the new features that you can use are in this new version. After about a quarter, we're gonna move over to making that the default.

B

We think that this had enough soak time to be the default version, and that's where we recommend most customers be is on the default version and then, after that, we're in the phase-out period and it's in the phase-out period is where we do our master upgrades. First, we do the master upgrades and if you've opted into no top grip, not a node, auto upgrades, we upgrade the nodes. It's important to do it this way, because if the master, if the nodes are ahead of the master, things get pretty funky.

B

So you have to ensure that you do the master upgrades first and then you do the node upgrades patches in gke and those these could be either. You know kubernetes patches or the the gke specific patches have a very short life cycle, but they very similar to the minor versions again just shorter. So this is on the order of weeks rather than months or quarters. So a new patch will come out.

B

Will you know see how it's doing and then, after about a period couple weeks, we'll upgrade people within that minor version to the new patch? So just to pause here? This is kind of a summary of where we are with the GK release. We've talked a little about kind of. Like upgrades, we've talked about kind of like process at a high level. We've talked about this version life cycle, but everything is tied right now to this kind of version. Identifier- and you know, as Kobe mentioned, like we're.

B

Moving towards a world where, like more add-ons, are moving on to gke more partners are moving you're using gke, and so gke is becoming this platform for applications, not just sort of for customers, but for us, and so what are we going? What are we doing about it, and so one of the things that I've been working on is component based releases and so component based releases. Is the idea that, let's, instead of having a whole lifecycle around, say the whole for the version number?

B

Let's have a life cycle around components, and so one of the things that I've been working on over the past six months is this project called the cluster bundle, which is a project that aims to so open source, the tooling and infrastructure for building and managing components? And it's something that we're using both on gke and on GK on GCP and on Antho. So you can you can check it out here. So what is the cluster bundle? All just gives like a very brief overview of what it is and why we use it.

B

So what it is is really just a very, very lightweight packaging solution, so we stick a name diversion on to some kubernetes objects and when you look at the Yambol file, that's that's. This is kind of all it is, and you might be thinking like. Why is this interesting and I think it comes down to the philosophy of what this means? Really.

B

Everything in this world is a kubernetes resource, and so this is kind of a powerful way to think so. Yes, we're gonna, have say our Etsy D, we're gonna, have our controller manager or whatever our stos. As you know, the deployments are back rules and so on, but we can also have CR DS that represent an application metadata, see RDS that represent runtime patches, CR DS, that represent dependency requirements, and then everything is bundled and versioned into this component.

B

So if we take a look at that lifecycle diagram before now, it's much simple it's the same, but instead of qualifying that vig version thing now we're qualifying components so we're taking that we're packaging into components and then we qualify and then we roll out components.

B

So what does this mean for the end user? You might not see very much except for us, it means granular qualification. It means improved release, velocity for components and also something I'm very passionate about is it means common infrastructure for releasing components so I've been working on this. For Jiki on JCP and anthos, I've also been talking with people in cig cluster lifecycle.

B

I know that Justin Santa Barbara has been working on add-ons and so he's looking he's been using this for add-ons and we've been talking with the cops folks as well, so super exciting, but this you know for us, this is only part of the story and I want to turn it over to Coby to complete the story about how we're doing how we're trying to solve releases.

B

A

You so I'm not sure how many of you are up-to-date with Game of Thrones, but this is not a spoiler, but in one of the recent episode, many people noticed a completely misplaced Starbucks cup in one of the scene, evidently and mechanism for managing risks and controlling disruption is important. So so far we spoke about velocity, but we also want to make sure that, with the growing market adoption that I mentioned before we cater to different customers- and we spoke with many customers and we realized that one size doesn't necessarily fit all.

A

We have different customers, different market segments and different needs. So just to give you some example, customer a may have a multi-week certification cycles for every release, they're looking to align it with their internal processes. Customer B doesn't use really auto upgrade, but they upgrade manually to get a better control. They are very excited about the ability to test early version of kubernetes where customers see they are very self-sufficient.

A

They know how to use kubernetes and they want to adopt feature quickly or quicker and roll them out to multiple clusters, and they want to also maybe validate the features and their releases on staging environment before they roll them out into the production.

A

So with that customer perspective in mind, we realize that we need to provide a more simple and streamlined process for customer to choose their version. So, as we explained before right now, you can choose the specific version. We are actually moving away from that structure to a concept called released channels, so we are going to offer three release channels. Actually, just two weeks ago, we launched alpha release channels with release channels. You can subscribe your cluster to one of these release channels.

A

So let's say you are keen to try out the new features or the new releases from kubernetes. You can use the rapid channel to get the most up-to-date and fresh releases coming from the open source community. If you want to run your production workloads, you can choose either the regular or the stable. The main div friends is really about the soak time that Josh mentioned before. So it's the stability and the stability and freshness of this version. We also think about the way we mature the version, so the more soak time, the more time.

A

The more time is, it actually accumulate. We will promote a feature from the rapid. All the way to stable another way to manage risk is transparency and transparency is key for us. We provide visibility into our release, scope on the gke side and you can see more than just a change log on our release notes. You can actually see the four days roll out that Josh mentioned before so for every version that we are making available you'll see in each geographical location. It's going to be available.

A

You also can see when we are going to duplicate a version and what feature are included in each version with the introduction of release channels. We change the structure of the release. Note so, as you can see here on in the screenshots, we have this tab structure and you can choose which channel you want to see the release notes for, as we explained before there, you have the option to choose whether or not to order upgrade your notes. If you opt into an order upgrade, you can do it through the UI.

A

You can see it on the right side or through the command line. We highly recommend you to turn on order. Upgrade the open source, the kubernetes, open source policy required a master to be within minor version from the nodes. So that's really important, but then again we spoke about the balance right velocity disruption. We want to make sure that, as we encourage you to keep your cluster up-to-date, we also provide you with ways to reduce disruption.

A

We just recently launched in as alpha node surge upgrade. That means that we change the way we sequence the node upgrade workflow from delete, then create to create then delete what I mean by that is at the time of new version availability. We will create a new node in your cluster with the new version, then we'll gracefully drain, the old node and just that remove it and that that ensure one successful completion of the upgrade.

A

It also secured the resources before we even start an upgrade and it reduces the disruption and the downtime for for your specific cluster, we ran some tests and it's pretty amazing to see the differences between the disruption period in the current sequence and the disruption period with surge, node upgrades.

A

We also recognize that there are different businesses and different type of business needs and different customers may have different peak seasons. So if you are retailer, for example, you might want to avoid upgrade doing Black Friday, Cyber, Monday or if you are a gaming company, maybe you want to avoid maintenance events during weekends, because you get most of the traffic during weekends, with maintenance window.

A

Today, you can specify for hours, maintenance windows which will respect and we'll we'll make sure that we initiate maintenance events such as upgrade only in these four windows, a four hours window and then, in addition, we are expanding the capability of maintenance windows to cover other use cases, for example, if you're trying to avoid upgrading specific days or if you are trying to block out specific range of dates. Let's say the holiday season or Black Friday, Cyber, Monday and the last thing is: it can also snooze and upgrades.

A

Let's say you know about an upcoming upgrade, but you need some time to get prepared for that upgrade. You will be able to snooze an upgrade for a short period of time.

A

And then the last feature that we aren't introducing for managing disruption and reducing reducing risk is canary clusters, so I mentioned before release channels, that's great, but we also recognize that you have risk mitigation plan and we want to help you with with those plans with canary cluster. You can create a cluster designated for canary usage or staging and then link it with one or more of your production cluster. What it means for us is, we will instruct the release server to roll out new versions to your canary cluster.

A

First, we'll give you the opportunity to test out new version either through running your automated test, or we can trigger your automated test for you and then once you greenlight, the upgrade. You made sure that nothing breaks the release server will roll out the upgrade to your production cluster as well.

A

So with this changes that we just described, which we spoke about going market adoption, growing ecosystem, evolving cloud infrastructure, we want to make sure that we keep this balance between velocity and and and low disruption. We will still have to face with the quarterly releases of kubernetes, but we also want to make sure that we meet customer needs. We are evolving basically from a cluster creation cloud service to a platform, a platform that supports both the extension of the platform through add-ons and partners, and also support different type of customers and users.

A