Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Testing Kubernetes Clusters - Building Confidence in Your Changes - Guy Templeton & Matteo Ruina

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Testing Kubernetes Clusters - Building Confidence in Your Changes - Guy Templeton & Matteo Ruina, Skyscanner

Speakers: Matteo Ruina, Guy Templeton
Operating Kubernetes clusters involves a large number of components, each of which can impact the functionality of clusters. As cluster operators, building confidence in the changes being made is key to allowing frequent updates without compromising on reliability. Running the full Kubernetes conformance test suite every time is generally too slow, and doesn't always cover the functionality users of clusters care about. In this talk, you will learn how Skyscanner uses Kubernetes' existing test codebase along with Sonobuoy and Spinnaker to build a custom test suite. This allows cluster operators to build confidence in changes made to their clusters and their components, focussing on the functionality developers care about. This case study will enable you to make a start in developing your own tests for Kubernetes clusters, understanding the benefits this can bring, as well as the drawbacks you need to beware of.

A

Hello, everyone and welcome to our talk on testing kubernetes clusters and building confidence in your changes um in terms of what we're going to cover today. uh We'll cover a little about us, um then we'll go over the problems that we faced and that prompted us to go down this investigation rate and building our own solution and then give an overview of the potential solutions that we looked at.

A

uh Give you a deep dive into our solution, how it works and the the components that we brought together to make it work and then finally cover some dutch's tips and other options for you.

A

So in terms of who we are, um I'm a guy, I'm a principal software engineer at scott scanner. um I work um on the teams responsible for uh our shared container platform so and for the most part, that's kubernetes um and enabling the rest of the business to build applications on top of that and that operate reliably at scale. um In addition to that, I'm also a co-chair of kubernetes, auto scaling and it could be found on most places on the internet under the handle gj templeton.

B

Hello, everyone thank you for joining us. My name is mattel. I'm a senior software engineer here at stress camera, where I work with guy on keeping our kubernetes infrastructure up and running in the production platform tribe. uh You can find me technically on twitter, but yeah. Please look for me on github.

A

So um we both write for skyscanner, um who are the sky scanner, um so we're the travel company that puts you first, um we help millions of people, millions of people in 52 countries over 30 languages find best travel options for not only flights but also hotels and carhart each month and work with over 1200 travel partners to enable that.

A

In terms of kubernetes at skyscannerville, um we've been running the case since um 1.6 release, and we over that time we have built up the the number of clusters we have to more than 35 production clusters at this point and spread across four different aws regions, and then on those clusters, we've got 475 and more services and most of them running in multiple clusters: multiple regions for improved resilience um to a number of different uh potential scenarios um and in terms of scaling the clusters underneath them we've.

A

Currently, we currently run about 40, 000 plus cpu cores across the clusters, with about 150 terabytes of ram at any one time as well.

B

So what do we want to talk about today? Well, uh if you run kubernetes you're, probably aware that it's it's a complex platform, let's be honest, there are like multiple microservices api servers distributed database and, if you're running a production workload on top of kubernetes, you probably also want to install a few other components. For example, uh you have your, you might have your cni, uh maybe you have a service mesh or maybe you have an ingress controller.

B

uh You probably wanted to do something about the metrics and logging, so you have a set of observability components and.

B

While kubernetes has a good set of unit and integration tests on its own and I'm pretty sure that all the others that you're running they have their own set of tests, the very difficult thing is here is when you glue everything together. So how do you test your kubernetes setup with all the components you care about, with all your configuration inside either your uh cloud environment? Maybe you're running in aws, like we do or you're running on premises?

B

Why do you care about making sure that everything is up there? It works just better. Why? Why do you want to test that? Well, the obvious things is that you want to reduce the disruption. When you make a change into the clusters, you want to keep your customer happy with users happy. You also want to do something for the people responsible for the cluster.

B

So maybe your squad, you want to push out features for your users and then you want to keep like a decent velocity, so you want to be able to reliable uh test and making sure that achieve something that you're pushing out is working, and it's not breaking anything.

B

This gives you like confidence that uh the users are getting the behavior they're respecting. It also gives you a baseline. So if you make a change, then you can pinpoint like when the behavior changed or when something broke and maybe helps you like in better troubleshooting.

A

So um we've covered now the the problem, the reasons we wanted to look at this, and so we we started to exploring what our potential solutions to enable us then to do this work. um So the first, um which I assume many of you are familiar with, is it was manual acceptance testing.

A

So the idea of we'd stick a manual approval gate in our continuous deployment pipelines, and that would give us the opportunity, as operators, to go and run some manual acceptance tests against clusters or against individual nodes, whatever it may be before manually, approving the the pipeline to continue um the problem, obviously with that is, as we've scaled up to 35 clusters that this this solution doesn't really scale, as the number of clusters uh increases, so increa increase increases.

A

Every new cluster adds to your toil burden and, unlike with an automated solution- and it's also easy to make mistakes and it's inconsistent even if you've made it into run books if you're still relying on engineers correctly running the run books with the correct options, for instance, or copying and pasting commands in from runebooks.

A

You you've you've still got a human factor in there and a risk that things won't quite actually be what you thought you had run um so for that reason, we sort of rolled out manual acceptance sites pretty quickly, and the next option we looked at was a custom service, and we ran with this for for actually quite a while- and in this case we spun up a microservice on our cluster um and that microservice hud had our back permissions to create codes, create and modify some some objects inside the cluster um and that that was constantly on a running these tests and that that had a good benefit and that it made it easier to catch any degradation.

A

So if we made changes and say uh it worked fine initially, but six hours later, something could say bogged down the api server to the point that things weren't working. The way you thought they should anymore this. This would actually catch that it exposed metrics. So we could alert on those that that was a real benefit of this approach.

A

The the downsides, however, where that the way we implemented at least it was harder to block our update pipelines on so we we couldn't, we couldn't as easily basically say I have a conditional in our continuous deployment test of. If these tests fail the next time the loop evaluates, then after we've made these changes, then block um and the other was it's not.

A

There was no easy ability for us from a talk test when, when where we wanted, we were at the mercy of the the loop that was evaluating these things um and we could have. We could have done work to fix that, but we took the opportunity to look at other options to see if there were a better battery.

A

um So the next option was building on top of existing frameworks, and so there's existing infrastructure and kubernetes testing frameworks, and this would have minimized the custom work they had to do, and these different frameworks have a number of different different benefits. Some are written in gold, some are in python. um Some are very, very cognitive, focused some are way more generic and have their origins from before kubernetes.

A

um So there's there are a number of options: we've looked at there um and then we looked at more specific, like provided and supported by the kubernetes core kubernetes community um options. um So the first option we looked at was um cube test. uh This is a stored in the kubernetes test center repo and there's a framework that allows for the creation and teardown of test clusters.

A

um It supports the addition of custom bingo tests and it's supported by the kubernetes project, but it does come with extra overhead and because, obviously, if you're tearing up spinning up and tearing down clusters, you've got to wait for that creation and teardown to happen anytime, you're running these tests. However, the biggest drawback of this for us was it's not necessarily representative of our production clusters say: we've had a production cluster running for six months.

A

um We've we've had many changes um applied over time, spinning up a fresh new cluster and applying all the components you think are in your cluster is not mess. It is almost certainly not actually going to be representative of that cluster. That's had accreted changes on it, um so that that was the main drawback here for us.

A

Then we started looking at the conformance suite, and so you might be familiar with this if you've ever had a look at the certification of managed kubernetes offerings, these are the tests that um cloud providers or any provider of a certified kubernetes offering has to validate that they've run against their offering, and all these tests passed they test a huge range of core kubernetes functionality, make sure it all works with an acceptable limits, etc, etc, um so that that would have made it really easy for us to use the community's existing tests.

A

um However, the downsides that are obviously because of the range of the tests in there it's really time consuming to run that entire suite um there's there's some disruptive tests. If you're running the full suite, although by default it doesn't run the disruptive test um and also you're, not necessarily testing the behavior that your users care about our users, don't care about. You know some of the low level stuff. What they care about is is my application skill.

A

The thing that conformance suite did lead us to, though, was the tool that's used to run the conformance suite, which is a tool called sona boy. uh It's an open source tool for vmware. um It's it's very active community, constantly improving project, and but the killer feature for us was the ability to run custom tests by the creation of plugins. So that meant that we could build our own tests, only testing the functionality we could care about into into binary and use sonaboy to run that.

B

So what did we choose uh to iterate from our from our custom solution? So uh before taking a decision like we, we sat down and then we look at what we wanted from from our next generation tool, and uh uh the very thing was like something easy to run that the previous tool was like the by this as a demo set across the whole kubernetes. So it was not very easy uh if you have a cumbersome test tool, you're not going to use it a lot.

B

So I wanted something simple that you could run from a laptop when, while writing a new test or even if you wanted to quickly test like a live cluster, but then the sim 2, we wanted to be able to embed it in our cd pipeline.

B

uh We wanted all the the things that you probably expect in a test suite so run test is here or in parallel uh able to get in store and archive test results. So you you can do the com, the you can look at them over time and see. If you have like you, can spot them trends.

B

Another important thing is alerting on on a filling test. So uh you might take a decision like in our case we we page one of the cluster operator if, if the tests are not passing because uh maybe means that uh one changes, one change that we're making that made into production, uh despite of the tests is, you know, probably breaking the functionality. You might want to even add, like an automatic rollback.

B

uh This on the test street. On the actual writing writing the code. We wanted to find a solution that was allowed us to write the less vulnerable code as possible, so to rely a lot on other other people best practices. The community uh maybe uh have like a few examples of well-written tests that we we could took inspiration from, and so uh our current test solution is based on writing a custom soluble plugin.

B

So, as guy said, uh soluboy is the tool that runs the conformance test suite, but we wrote a plugin and then soloboy now is driving in our case only our skyscanner internal smoke tests, and then we had a look at the kubernetes for a native repository and there is like a huge section around tests and then to intestine. So there is all framework that gives you like a function and utilities to reduce the amount of code that you have to write.

B

Unfortunately, our repository is not open source yet because there are a lot of sketch scanner custom custom tester, so we haven't done it yet, but we- hopefully uh we are going to do that in the next few few months, but I hope to give you like an offer, so you can uh go away and try to replicate our own setup. So, as we said, we run the solo boy uh image to drive the tests. uh We made a little changes so that we can easily retrieve the field test.

B

Then we can differentiate the exit code when we look it into our cd pipeline, so we can bail out if, if any of the tests are failing and then we build the actual end to end image, we based these on top of the upstream kubernetes conformance image, because uh that image was already added jingo and everything that we need to run our test.

B

As you can see here, we make a couple of changes. In particular, we change the running station and I will tell you more about that later. At the end to end the doctor test, the binary is the angular binary uh produced by when you compile code, and then we pass uh some yammer configuration because we mirror internally, all of the ex all the docker images. So here we're changing the configuration of solo boy to to point to our internal docker registry.

B

uh In terms of the uh how the repository looks like uh we basically shift and lift the the end-to-end uh go files that are the entry point for the street from cumulative kubernetes and then we we brought like uh our uh smog test here. We have a networking file because we have a lot of network uh testers, so we move them in a separate file and the common dot go that als all the share the function between between the smoke tests.

B

So when what do we do? When we push a change to kubernetes itself or to a component, we have, as I said, we, we change like a little the soluble images. So we can, we can write. So we have the job that is actually responsible for triggering uh uh soluble itself and then retrieving the the result and the and the exit code.

B

We also have a job to delete and make sure that we start with the uh from scratch every time because robot doesn't like if there is like already any space, at least in our version, if there is already the namespace around them uh so inside the solar board in space, there is a there called uh slowpoint binary, uh driving driving the the test and these spin-ups like our actual uh end-to-end image and that creates uh or- uh and that runs all the tests.

B

So the the good thing about these is that every test lives inside its own namespace. So we run those tests alongside uh production workload, because then we have the confidence that we're not polluting other namespaces or we live like exterior resources around.

B

So what kind of tests do we do? We write? We try to really think about what our users care about, uh and so networking, of course, is a big thing. You have a bother. You want to talk to other pods. You want to talk to the internet.

B

We we use easter as a service mesh, and uh we have like some uh internal configuration uh custom configuration to to have a multi-cluster setup, so we want to make sure that uh that setup is still running every time with achistio.

B

We have a few tests around dns because, let's be honest, it's always dns. So you want to make sure that that is working internal means. uh We make sure that kubernetes dns internal resolution works. We run in aws, so we have a bunch of specific tests against aws, endpoints and also uh other tests around external domains. A lot of our users rely on auto scaling.

B

So, of course, we want the hpa to be up and running and also scaling on custom metrics and since we're running in nws.

B

Some of you might be familiar with the concept that they have for for giving permission or improves. So we want of our users to be able to assign uh the role that allows the report to talk to the their market or their database and that functionality needs to be to be working.

B

Moreover, there are like a few cases or when you are not changing anything at the kubernetes level, so at those all the same combinations version all the same and no flags anything but maybe you're changing them.

B

I or you are changing like some uh uh something uh you're to tuning the kernel, so you want to make sure that all the nodes of your cluster are in a known state, and this is an example like how how does a test look like so uh kubernetes kubernetes, a test framework use jinko as a as a as a framework.

B

So here you can see that there is like uh um the protopod uh and the pop-top test, so a communication between a source powder to the destination pod, and so we create the uh the source ball, the name space and the interesting thing is in the last line. You can see that with just one line we can make sure that the framework spread that doesn't uh doesn't happen.

B

Any error: when we try to create the namespace and the pod and have the pod running, then we can create a destination service with the destination pod again same function, to make sure that the destination pod is up running and waiting for and ready to to set the connection.

B

And then, in the last step, we create a service that we uh that is gonna forward, the the call to the initiation pod, and then we use uh other framework functionality to basically iterate and make the same uh uh call from the source folder to the destination pod. And we we make an assertion about it that we are expecting uh into under status code. If we don't, if we don't get uh that code within uh a certain timeout, then we we know that the pod uh is broken and- and we raised an alert.

A

So um matteo's taking you over how we implemented our solution there. um However, this didn't come without a few drawbacks and we want to take over like what we've problems.

A

We ran into some tips as well um as well as other options, if you're looking at this and think I you know this, this seems not the right fit for me, but I still want to test my kubernetes clusters um so in terms of the gotchas with our approach, the the first one and the main one is that, as matteo mentioned, we're using um code from the main kubernetes kubernetes repo to avoid having to write a lot boilerplate.

A

um However, the the main problem with that is, as you can see from this comment on an issue on the um on the repo- it's not intended to be consumed, this module this. This isn't his intended use case.

A

um So we need to use a workaround to get get this code to be in a usable state in our our project um and thankfully, in the same thread, someone has written a script to enable us to do that, um so you can use the script is what we're using successfully we've used it to get us through multiple minor version, upgrades of kubernetes and vendor in the new code successfully um that that works for us it works well, and so yes, it's certainly something I I wouldn't have any hesitation in recommending it's a solution to this problem.

A

If, if this is what you need to do is used in multiple different projects across the community as well for different parts of the kubernetes code base as well, um so the the other um problem with versioning the et code, however, is that the the other part of it being not designed for this means that there's no api stability guarantees like there is with you know, public facing parts of kubernetes, and these these are. This code is written with the intent being it's explicitly tied to a given minor or a given release of kubernetes.

A

um So there's there's no hesitation in and renaming removing um completely replacing methods, and that means that when you do a minor version update of kubernetes you, you have an extra cost here of potentially needing to update functions. So one of the methods that mattel highlighted in terms of names creating namespaces waiting for pods and that that method has changed previously completely changed name and we needed to do some investigation as to what the correct method to use was now.

A

um That's that's only a one-time and cost, though effectively when you're doing a minor, kubernetes version update. So it becomes another item on your list of we're going to upgrade kubernetes we're going to need to upgrade these components, we'll just we'll need to update the vendor code and figure out what what um what methods we need to use now.

A

Potentially um the other thing is the um the code um provides test images like there's a static, effectively a static mapping within the code of test images and tags and including the repositories there they're pulled from um if you're, potentially running an air gap cluster or one where, for security reasons, you can only pull images from from certain registries or because you don't want a third-party dependency, you may need to dive into the code find out what those images are, mirror them and also um use that use the custom um config file that mattel mentioned to effectively configure your test to pull them from different registers, rather than the default registration um there's also occasionally flaky tests.

A

um So if you have very dynamic clusters, where you're scaling up and down a lot or you're running on, for instance, aws spots, instances where your your instant skins can be lost, um sony boy or the version we run is not the greatest recovery uh dealing with tests being interrupted. Mid-Run. um So it might. You might end up with flaky tests due to the dynamic nature of the clusters you're running, then these in or something else um and finally, in terms of blockchains, you might need to run um the run.s8 script.

A

So this is a script that we pull in from upstream and modify slightly um because we want to basically make it more configurable for our use case. In our case, we want the ability to allow more unready nodes, and so that's basically an argument that we need to pass into the invocation of gingko and that that means we have to modify the script.

A

But it allows us to run this on clusters that are scaling up and down, because some roads might be unready and normally that would talk, because these tests fail and very quickly and in terms of some tips that I'd encourage you to think about.

A

Even if you don't take this approach like um capture the behavior that your users care about, but that it's the same as writing any tests, a sort of end time text, but put yourself in the mindset of your users and figure out what behavior you're actually trying to test is working, is still working, hasn't degraded. Whatever is you're looking for um leverage the community's efforts, like not just the code, also, the images that have been built and one of the tests natal showed used an agn host image.

A

That's something that's built by kubernetes test infra, and it ensures that, no matter what architecture you're running on or whatever the commands that you're running, whether it's a deck, whether it's a network call whatever it is they're, consistent and they're, they're, they're types of response, etc. So those those things there's a lot of thought and care put into them, make use of them.

A

Don't don't go around building your own images to do things that might fail in weird ways on different architectures, etc and finally, figure out which of your tests can be running safely in parallel, and obviously, if you're running these for every change against your cluster, which I would encourage you to do, you want them to run as fast as possible. So you get that fast feedback loop, so that your your tests don't become a pain point. They become an enabler rather than pinpoint.

A

um If, if you've listened to all that and you're still thinking, um I'm not sure that this this approach will work for us, whether it's too much effort for for diversion upgrades or whatever it is, or you don't want to write ginkgo tests and there are a couple of other options. um The first is um I I already mentioned the cube test framework. um Since we did the evaluation cube test two has become the the solution.

A

That's that's in a separate repo um and a number of different providers have got implementations so that I've I've linked to the aws implementation here as we run on aws, but um that enables you to spin up um an eks cluster quickly run tests against it to do some teardown. um That's that's! um Potentially, if you're scared about running tests on a live cluster with your production uh traffic on it, that might be the better option for you and and then there's there's some other community supported frameworks effectively. Then the first is test emperor.

A

So this is um a bit older. It's um not very! It's not specifically kubernetes focus but allows you to test your infrastructure, whether it's provisioned by ansible or salt, or a kubernetes for a kubernetes cluster, um and that's that's python based um the next two are a bit more kubernetes focused, um so the first is cube test, but a different cube test. Not the kubernetes cube test, which I've previously mentioned- and this is this- is um to enable you to use pi test uh in kubernetes.

A

So it makes makes use of um allows you to make use of the kubernetes python client, but in a pi test framework. So if you're really used to writing pi test test rather than big test, maybe that's the the approach you would want to go down um and finally, a slightly different approach is kuber healthy. So this is actually an operator um that runs in cluster.

A

It has some custom resource definitions and you basically interact with it by creating custom resources that define your tests and it takes care of spinning up the pods and gathering results and exposing metrics about what's happened. So, if that, if you want something a bit less involved in terms of you having to write everything in code and building binaries, that might be the the best option for you.

A

That has been our talk. um Thank you very much for attending.