Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building, Managing and Automating Clusters at Scale With Prow - Michael Splain, Sonos, Inc.

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Building, Managing and Automating Clusters at Scale With Prow - Michael Splain, Sonos, Inc.

Whether building your first or 100th Kubernetes cluster, it eventually becomes clear- this must be automated. These days, building and customizing a cluster is pretty straight forward based on your required workloads, infrastructure and tooling. The real hard work comes when answering the question, how do we manage this long term? In this talk you’ll learn how we accelerated our automation by moving our cluster management to the CI system built and trusted by Kubernetes itself, Prow. We will start at the basics of Prow and work up to why it was a great fit for us. Finally we’ll show how it all works and discuss how to approach your own cluster automation at scale.

https://sched.co/ekDL

A

Hi, thank you for coming to my talk, I'm mike splene and we're going to be talking about building, managing and automating clusters at scale with pratt.

A

So a little about me, real quick, I can be found online at my explain. I work as a cloud infrastructure lead at sonos. Previously I've worked at barclay and paypal where my container and kubernetes work first began. I've been using kubernetes before the 1.0 days in various forms, such as the good old kubernetes, mesos project and multiple homegrown solutions. Eventually, this led to working with cops and becoming a cops maintainer.

A

I also founded the boston, kubernetes meetup and hope to continue that when this is all over so at sonos, I want to talk a little bit about our journey into kubernetes. So, let's see how we got here, application teams at sonos build manage and deploy their services in production.

A

These teams often deploy multiple times per day now, with over 29 million products around the world. This leads to some very interesting cloud infrastructure.

A

Unfortunately, over the years our teams have had to duplicate each other's code, which we really want to fix and speed up deploy times as well, and that's where kubernetes comes in, so I often get this question, so I start by building a cluster right.

A

Not quite now. Let me explain: there's quite a few things to consider when you're building your kubernetes clusters, let's start with cider blocks and networking to other resources, you may need to define them based on your company and your culture's set up. So you may need to make a lot of adjustments and changes there. Dns is pretty critical for any service that wants to be externally accessible, as well as security configuration for users and automation access you may want to enable alpha and beta at kubernetes api features as well.

A

That requires certain flags that you'll need to understand and version as well many times to have a successful cluster. You require things like cluster monitoring, logging and alerting, auto scaling and node removal is pretty critical in my mind. For kubernetes. That way, my team doesn't have to manage individual instances and we can focus on managing the clusters and the automation behind it.

A

The problems that I've heard run it people run into when they build clusters. First, is clusters become their source of truth, rather than putting it in code like a regular developer, would we try to focus on putting everything in the cluster? Well, that doesn't quite work. Cloud providers often sometimes also become the source of truth. When you use tags or certain automation that relies on their history.

A

You really then can't repeat that through the rest of your environments, this relies on humans to decouple and manage the infrastructure, which generally doesn't scale very well. So let's get back to the basics. Here are our requirements when we went to go build our kubernetes clusters.

A

First of all, we wanted everything to live and get all the configuration can be checked in managed via source control and pull requests. We wanted everything to be idempotent. You can run it as many times as you need to to make sure that everything is up to date. Almost every cluster should be as in identical as possible. There will obviously be differences between some clusters here and there, which allows us to try out things like different networking providers or other cluster resources. However, they need to be configured and checked in the exact same way.

A

That leads me to the configuration we really want to be able to check in and manage individual configurations of clusters as we need to in general. We should be able to abstract this across multiple clusters of the same type say: production clusters versus test clusters. However, there are many times where you're going to want to automate and test specific configurations on specific clusters over time, we really wanted to embrace fast iteration much like modern application teams.

A

Devops needs that as well, and our devops teams here at sonos, really focus on building kubernetes in a way that we can move quickly.

A

First, we wanted to work our scripts to work the same locally as they work in ci. That way, we can always come back locally and work to solve a problem that we may run into in the cloud.

A

All of our tools are versioned and all of our changes are deployed with code, even api changes that may be difficult to manage. We try to write scripts and code to manage their upgrades across our environment environments.

A

So let's talk about our tools. First, we have cops. We did consider many other options at the time. However, cops has a large community and is multi-cloud, so it was a pretty easy pick for us.

A

We then went on to terraform to build all the rest of our cloud resources. Things like route, 53 zones, s3 buckets, iam rules for things like loki and thanos that you may want in your cluster, complex networking, changes and other custom, tooling things that we might want to add on the fly.

A

Maybe we want to be able to schedule actions on our auto scaling groups so that all of them spin down on the weekends things that we don't really have to over engineer, and we know how to do it in terraform, so we can easily just add that whenever we would like the next bit, we added is helm, file and helm, which drive a lot of our cluster components and how they run helm is used to deploy two of the clusters, while helm file describes how we deploy to those clusters.

A

This really gives us the flexibility to decouple in the future and deploy individual components as we see fit. We can quickly implement workarounds for things like api changes or breaking helm, chart versions. That would be difficult unless we had this automation in place.

A

We also use yq and a bunch of custom scripts to help manage our different values that drive our clusters. These values grab specific bootstrap components that allow us to figure out which components we should run in what cluster.

A

On top of that, all of these tools are versioned. This allows us to upgrade individual components on individual clusters. As we want, tools are pulled down and executed dynamically within the repo similar to tools like asdf more on that in a little bit.

A

So let's talk about bringing up a cluster.

A

This is what bringing up a cluster at sonos looks like on the left. You see where we run the source command. This sets up our environment, aws credentials and secrets on the right. You see the actual workflow that is involved in bringing up and managing our clusters.

A

First, we run make cluster, which you could break down into smaller pieces like make cluster up or make helm file sync, but one command will also suffice. First, it templates and and manages the configuration. Then we spin up the cluster using cops, create or cops update depending on the type of workload. Then we terraform apply to add our additional changes to the cluster.

A

Once the cluster is up and running, we apply our helm file changes using help file. Sync. Now the cool thing with all this is: if a user doesn't have a copy of a specific version tool, we will go download that tool and cache it locally when the tool is needed. This way we ensure every human or ci user has the exact tool versions they need. This is very useful when utilizing tools like cops or terraform that can contain significant changes between versions.

A

This can also easily be cached in ci between runs.

A

This is what it looks like in code. So simply you declare which cluster you would like to operate on. You run the source command. That way we get our credentials and everything, and then we run the make cluster command.

A

This is what that same command looks like when we run it in ci, we'll get back to this a second if you're not familiar, but you may notice. This looks much like a kubernetes pod spec. If you look in the center, there you'll see the command make cluster.

A

Now you may say what about day two? What happens after that? You have to worry about, updates and upgrades things like cops, kubernetes itself, terraform, helm, helm, charts, cube, cto, that's a lot of things to manage right. What about adding new features? Things like metrics logging and security?

A

How are we going to test all this and make sure all this stuff works? Are you feeling overwhelmed, because we were too. The best thing we figured we could do is add as much structure to this as humanly possible.

A

That's where gitops comes in. It allows us to drive automation through pull, requests and easily manage many small changes over time. We can run thousands of tests to make sure our code works to the best of our ability.

A

That's where proud comes in prow is the kubernetes ci system built and maintained by the kubernetes sig testing team built on similar principles to kubernetes itself.

A

Now this is a screenshot of what proud looks like this is a simple cluster. I spun up locally, but you can also see that at prow.kates.io up top, you see the jobs that have run recently and the the length of time they've taken to run and down below you can click into specific jobs. If you're not familiar.

A

Let's dive a little bit into pro itself, prowl has job execution very similar to kubernetes. It allows you to define pod specs that it adds specific components to to make it much easier to automate. It allows you to have plugins and slash bot commands in prs, which makes it really easy to add additional tooling. Whenever you see fit everything for prow lives in source control, you can automate your your prowl updates, as well as the jobs themselves via changes in source control.

A

We really also like that you can have jobs defined inside the prowl repo configuration itself, as well as in the application repos that allows us to configure jobs that can make sure those other jobs that live outside the prowl repo itself are properly defined.

A

We also find that prow is very fast and scalable. This allows us to easily upgrade proud whenever we see fit and it runs in high availability, which is very useful in the cloud these days. As everyone would know, we also really appreciate the merge management feature which allows multiple pull requests to be approved at the same time tested together and merged all at once.

A

Below you can see a couple links to prowl itself and the github link to the pro section under the test infrarepo.

A

Now, let's talk about the types of jobs that you can run in prow, the main types of jobs you can run are pre-submits post, submits and periodics pre-submits run on a pull request. When that pull request is open, those pre-submits get kicked off. You can also define that certain pre-submits are required before the pr. The pr can merge versus, not post, submits, run on a branch or after a merge in general, you'll define the branch that you want.

A

Those post submits to run on and they will run as soon as code is pushed to those branches in github. Periodics are just what they sound like they're periodic jobs that run on a crime much like cron jobs.

A

So let's take a look at a pre-submit example. This is an example that we took for from the test infrarepo, but we use it to test our yaml as well. You can simply see here it is set to always run and it pulls down a yaml lint tool. Then we define a config for that yemelin and we pass in a couple values files as well as folders for it to check.

A

This is another example of a pre-submit job, this one's a little bit more complex. In this case, we have a script that we use to build and test our cluster configuration in a kind cluster. If you're not familiar with kind, I would check it out. It's very great. It stands for kubernetes and docker and we use it very heavily to test our cluster configurations.

A

This allows us to quickly build and deploy clusters and their configurations and get test results before waiting for a full ec2 cluster to spin up.

A

Other pre-submit jobs that we run, we lint all of our helm files. We lint all of our config files themselves.

A

We also run a helm file, upgrade test which, at the end of it, it'll, actually output a diff of what the difference is between the before and the after of the change have that way, we can easily see what the changes are.

A

We also have terraform validation and we can really add jobs very quickly whenever we want now, pull requests are tested in about five to ten minutes and once they're approved they're batch merged using tide, which is a proud component. Now we're not going to get into some of the components but I'll send I'll give you some links at the very end of my talk, pointing to some some great talks that we've had at previous kubecons.

A

Now, let's take a look at a post submit job you can see it has a maximum currency of one, which means it'll, wait for the first job. To finish, you also can see that there's a run of changed. This allows us to check if a given, folder or specific file has changed and only run in those cases you can see. It also runs on our develop branch and runs the exact same commands as before.

A

This is an example job that we run whenever we merge code into our kubernetes automation, repo from our from any branch into develop.

A

Now, you're, probably saying that's great and all, but how does that scale? Do you really want to upgrade all clusters when you're merging to a branch? Wouldn't you want to gate it in some way? Let's talk about a real world example.

A

This pattern led us down the path of separating our configuration from our automation code. So let me explain. First of all, we have a repo for automation and tooling of our scripts. This follows: git flow has releases with the develop branch for development and a main branch representing our major releases.

A

We also have a repo for our cluster configs that can be managed with a separate cadence that main branch represents. What is running in our clusters now. Those cluster configurations now contain a git sha that represents the code from our automation, repo that is running in the cluster. This allows us to manually deploy specific code and specific branches for testing.

A

It also allows us to hold off risky changes from reaching production and specific environments until our teams are ready similar to earlier. This is what one of our post submit pro jobs looks like. First, you can see that we have a run if changed here for specific files under a specific cluster. This allows us to trigger jobs only based on values, changes and shop file changes over here you see the branch, our main branch, that this would run on.

A

Of course, we'll need somewhere to send any of our errors in slack when they may occur now down here, you'll see that we have our magic deploy hash. This file path is passed into our configuration job. This job will read that file. It will then take the hash stored in that file and check out our automation, repo, then, as normal, we will run make cluster, and then we will also rolling update our cluster if it needs it.

A

So that sounds great and all, but there's one more piece to this puzzle. Let's talk about the auto bumper, the auto bumper takes care of the pesky updates to the get shaw's in our config repo and will open pr's to update the cluster from time to time. Let's take a look now, if you're familiar with the test infrarepo under the kubernetes orc, you may have seen jobs like this. This runs a similar auto bumper to what we run.

A

The auto bumper runs to update specific values in specific files. So we extended the auto bumper to look for successful projob deploys to our clusters. Once we have a successful deploy, we open a pr with a new git shop if it's changed.

A

These are the jobs associated with our config repos pull requests these first three jobs run helm lint on our configuration validate our terraform and yaml in our values to make sure everything is set before we upgrade a cluster.

A

These are all defined in, what's called in repo config in a dotpro.yaml file in the future, we hope to add additional jobs that will run cops update without applying changes and helm. Upgrades as a dry run that way we can ensure those changes are prepped and ready to go as well.

A

This job is defined in our centralized prowl repo and we'll confirm that our dotpraw.yaml file in this config repo is formatted and configured properly.

A

Sometimes, if you misconfigure a job, this may be the only job that successfully runs, which will give you a heads up, that you may have a typo or have broken something.

A

Finally, we have tied, which is a proud component tide, will wait for lgtm and approved labels from your teammates.

A

This can be useful to let anyone in your org lgtm a release if they feel it benefits them, but must be approved by a defined approver from the owner's file in order to auto merge if changes happen in the upstream branch between the time this branch was created and merged tide will ensure your tests are rerun with the post, merge config prior to actually merging these auto buffer jobs run as periodics a few times a day.

A

Those jobs look for other successful kubernetes cluster deploys once they are successful, prs are opened and git shadows are updated automatically humans will go. Take a look at those pull requests. Take a look at the diff of the kubernetes automation code that has changed between the version updates. Then we will approve those pull requests.

A

The merge will occur and at that time the cluster will be updated automatically. This workflow has worked out really well for our teams at sonos. So let's talk about some of our future goals.

A

First, we want to improve and contribute our auto bumper changes upstream into test infra. They apply to a various number of use cases and we think that other teams could really benefit.

A

We also want to use testgrid to view our test results and deploy configurations through our clusters.

A

We also want additional slack integrations, such as announcing before deploys are going to occur.

A

We also want to in set up a job template that way we can automatically generate all the proud jobs we need for different use cases.

A

We also want better content of our pull requests. That way, it's really easy for us to review changes that automation drives now here are a few resources that I think could be useful for you. The first is a link to the publicly accessible, proud dashboard for kubernetes.

A

Next you'll find the code test info repo, where you'll find the actual crowd code. Third, I've open source a tool that'll make it really easy to spin up a proud cluster using kind locally.

A

Finally, there are some really great talks that have been given at previous cube cons about prow. I encourage you to take a look.

A

Thank you for your time, specifically at sonos, I'd like to thank our team, scott mccalla, david muckle, dan miller medicine and chris cilevi. Thank you for having me and coming to my talk now. Let's take some questions.