Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive: Testing SIG - Cole Wagner & Katharine Berry, Google

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Deep Dive: Testing SIG - Cole Wagner & Katharine Berry, Google

This presentation is intended for contributors who want to know more about how testing is done for the project or want to write new test jobs themselves. We will briefly explain that open source Kubernetes builds and tests are executed by Prow as ProwJobs, then we’ll walk through the life of a ProwJob, demoing job triggering, execution, reporting, and cleanup. After that we’ll describe some pains we had withProwJobs and showcase some recent improvements and additions from SIG-Testing that ease them by making it easier to create job images, configure jobs to run, and display job artifacts: - Prow’s Pod Utilities transparently provide source code checkout and log/metadata upload to job containers. - Splitting Prow’s job config into multiple files to granularly specify ownership and ease review. - Using reasonable defaults for ProwJob fields to reduce the size and repetition of our job configs.

To learn more: https://sched.co/GrdF

A

All right, hello, everyone you're at the sig testing, deep-dive I'm, Cole Wagner. This is Catherine berry. We're gonna talk about a couple different things today: first one being prowl and proud jobs in particular, and then we're gonna talk about a kind and go fridge.

A

So browse the kubernetes see icd system, it's pretty modular in the sense that we can trigger jobs, of different types and width for different purposes, so, for we can trigger jobs for testing publishing building and we can trigger them in different ways. Have them run on different platforms and report to different sources? So it's a very, very flexible for running jobs and in addition to that, it also provides some github automation. So we have, we can implement, slash, commands, merge, automation and policy enforcement and proud today is used by a number of different companies.

A

You can see you there, so you might not know what proud we're proud jobs are, but if you've contributed the kubernetes project, you probably have interacted with them in some way or another. So you might have seen comments like these on some of your PRS. These are used to trigger jobs.

A

You may have also seen some these statuses at the bottom of your PR. These are the results of those jobs being posted, and you may have also clicked through some links in some comments to see the actual job results. This is goober nadir, so it's actually slightly different than proud, but it's showing proud job results here.

A

So today we're gonna be talking about the life of a prowl job so from the instance, it is born to the time that it is goes back into the ground. If you know if that were to happen.

A

So this is a useful way to learn about the different, proud components and kind of see the flow of how al prowl works and how we store state in prowl, and it will also kind of explain some of the magic that you see on your community's pr's.

A

So, as I mentioned before, prowl has many different job types, many different ways you can trigger these jobs. Many places that they can run in many ways to report. If I have talked about them all today, we'd be here forever. So instead I'll be talking about one of the more common cases, which is probably what you're familiar with on your PRS, and this is a pre submit job that runs on a PR and it's triggered by a test ball command. It will run as a kubernetes pod and report its status to github.

A

So this is the like. I said this is the one in more common cases that you probably interact with and I've got these little tabs at the bottom, so you can see kind of the steps that we go through.

A

So the first step is when the event happens on github, so here a test, all command has happened and github sends us a web hook and it's got some basic data in it and tells us who sent it. What the comment was what the issue is that kind of thing that web hook that has to make it to our cluster. Here you can see the ingress rule, the service and the deployment file. The details are not important. It's really just that it's getting piped through to a hook deployment.

A

So all of our web hooks go to a slash, hook, end point and we pipe them down to a hook binary and here's a little bit of a diagram for that. So you'll see that the hook binary is composed of a bunch of plugins, and these plugins provide different bits of functionality for different use cases, and we can enable them individually.

A

They don't even have to live inside the binary, so the the trigger plug-in is probably one of the most common plugins and it reacts to github events that indicate that a proud job needs to be created and run so yeah. In this case, we would react to a slash test test all command here. I'd like to point out. This is I'll, go on a little tangent here to mention that this is a point of integration for a lot of other crowd functionality.

A

We can handle any sort of github events here, so here's the github event for a PR being merged that would go to the update config plug-in and we automatically update config Maps associated with files touched by that PR and here's a slash, meow command, where we find a cat image on the internet for you and give that to your PR and really anything can be done with this kind of logic.

A

Plugins get this client here, so you can see they can interact with github kubernetes slack the owner's files that we have in the repo, so they don't have to provision most of their own tokens or anything. They kind of just get a toolbox that they can do whatever they need to do with and if you're interested in more plugins that we have available that that link goes to a bunch of the information about that we have about plugins that are currently available, so the trigger plugin.

A

That was the one that's important for this, that's the one that starts the jobs. The first thing it does is it looks at your job config and finds all the jobs that are supposed to trigger there's kind of a number of mechanisms that are indicate if a job should be triggered or not, but that's triggers responsibility to evaluate which ones should be triggered once it finds jobs that need to be triggered. It creates proud jobs, CR DS, to represent these.

A

So this is how we represent a job that is that exists in this running, so we create the we create this CR g CR D and it starts in the triggered state. You can see there at the bottom and we stick that in the API server and that lets this is pretty nice because it means that to represent our job state, we don't need a database or anything. We can just use that CD server there. The sed database that's already inside the API server.

A

So that's pretty handy so now trigger has created these proud job resources and stuck them in the API server and planked you can come along and planked is the it's a controller, that's responsible for the main lifecycle of the job. So this is the thing that will actually have the job executed.

A

So the way that this works is plank runs inside of the prowl service cluster and because all those prowl jobs are in the API server, it's syncing them from the API server and comparing them to the pods that it is seeing in other build clusters.

A

So the other build clusters are where we actually, those are where we're gonna run the jobs, and you see that the pods for those jobs are running in the clusters and we are essentially trying to synchronize the Proud jobs that we've identified with the pods that exist in the real world and there's a number of steps that we do to do that do to do this. These are kind of the the common handling mechanisms. So when we find a prowl job that doesn't have a pod, we of course need to create the pod for it.

A

When we do that, we might some knit containers to do. Source-Code, checkout or some sidecar containers to do log and artifact upload, but the the core thing is that we are creating a pod to actually execute the job.

A

Then, once the when we see a proud job, that's running and has a a pod that is completed either in the past or failed state, the job is completed, so we can mark the prow job resource as such. And finally, if we see a proud job, that's completed plank doesn't really care because the job is done. So we don't have anything more to do.

A

But if we just stopped there, all these proud jobs that are completed and all these pods that are completed would just pile up forever. So, of course, we need some garbage collection. We clean up completely proud jobs after 48 hours and we clean up complete deposits after 30 minutes. These are much or these uh garbage collection limits that we have are pretty strict because we run such a high load. But that being said, jobs upload their results to GCS. So this isn't like you're, losing all of your job history. When we do our garbage collection.

A

This is just the the recent history that where I was working with, but if you've just uploaded your results into GCS, you haven't really got any notification that your job is completed or anything, and you don't want- have to sit there and hit refresh on GCS to find out when your job results are there.

A

So we have mechanisms for reporting, job status and completion, and that is mainly achieved through the Cryer, so Criers, a component kind of like plank where it sits in the cluster and it's going to listen to proud jobs and it listens for changes in any of those proud job resources and when it finds them, it sends them to reporting clients that are responsible for reporting that status to an external source.

A

So one example of this is the github reporter. So when prior to tech, sprout job changes, it notifies the github reporter and the dead reporter can leave these kind of statuses on your PR, and this isn't just for job completion. So we're not just reporting pass or fail here, but we can also report when the jobs changes other states. So when it goes from triggered to running that kind of thing, if it errors or is aborted, we can also display all of that, but this is also a.

A

This is one of the branching points where we get more functionality. In particular, this is we get reporting to different sources, so, in addition to dis reporting to a place like github, we could instead report to Garrett with a comment or report to a pub sub with the pub sub message. So we can and we can have it potentially do all of these.

A

If we want it to excuse me so that that provides a lot of ways for proud jobs to report their status to the outside world, so that's kind of a the the overview there. You can really do a lot more with proud jobs, because there is so many ways to trigger them. You can trigger the based on time or based on PRS being merged, there's a lot of ways that they can execute as well. So you don't have to necessarily run your job as a kubernetes prop pod.

A

You could use the the que native build CR DS 4.

A

If you have some terrible reason to you, could run your job on Jenkins using prowl, and you could also actually I'm not sure if we have any more execution platforms right now, but we are adding more soon. So this is a very modular system, and here, just to finish off is a fun. Diagram of prowl may not be to scale but uh yeah. There was just the various proud components: kind of acted out with Gophers thanks Ben elder for yes, congratulations on making it.

A

Yeah so Catherine's gonna take over now.

B

B

So we can use proud to run our tests, and yet somehow we always still end up with bugs. What are we missing in an effort to find out I'm going to look at how we can measure the code coverage from our end-to-end tests? What we're running this work was originally motivated as part of an effort to figure out what conformance tests are actually accomplishing.

B

However, they can also be more broadly interesting and could show that what code is really executed over any given workload, whether that be conformance tests, all the tests or even running an application.

B

So the end result looks something like this: varus fees are specifically for results of running v1 11:112 and current conformance tests, at least as of when that screenshot was taken against a master build of kubernetes. This effect removes the effects have changed them. The underlying code I'm trying to see how test coverage is improved over time and, as you can see, it is improving slightly ish.

C

B

Can also send for coverage results to test crew it where you can use all your favorite tests. Great features like seeing infinite history, ish and also being able to have graphs. You might notice that everything here is passing even when it says zero. We tried not making everything pass, but this is what happens when we let our lack of coverage count as a failure. It's effectively rendered our triage totally useless by drowning out everything else. We turned this off in a fairly short order, so.

B

Support for generating these coverage reports landed in kubernetes a few months ago, and that means that you can use it yourself, because it's enabled at compile time, though, and no released binaries are built with coverage enabled because of a minor performance hit, and also because there's no reason you would do that.

B

You must build your own kubernetes to use it in order to do so, you must use the make based build system and pass it v-cube build with coverage variable with any value at all. This also works for other build types, including the ones for build in containers, make or make a quick release or similar will also work. Fine, once you instrumented cluster is running, it will periodically right coverage files to each services. Local disk.

B

There are a couple of knobs you can tweak here where the files are written, using cube coverage file and how often they are written using cube coverage interval which defaults to 5 seconds, which you might think would have a significant performance hit, but actually doesn't surprisingly.

B

Finally, you need to actually run some workloads and then collect your results, given that some components are in containers, some arms and the containers may have access restrictions which even them can be something of a pain. Fortunately, we have some pre-built solutions.

B

If you're happy running tests on a transient, GC cluster, you can use cube tests dump flag which will light locks to disk and include for coverage logs as part of that, your local disk avetis, if you don't- want to wait, 20 minutes for a shell script to construct a cluster, another option is to use kinds.

B

A new project by Ben VL de verte allows you to run a complete cluster in a container just build a kind node using the same built with coverage variable and using the kind coverage dump scripts you can find in the kubernetes test info repo. Creating the cluster takes under a minute retrieving four results is instant and you don't need to pay for any of the ends, because there are none.

B

Another problem vertically sorts of coverage measurements can encounter is noise, aside from barcode, can't directly executed by all tests, we're also going to pick up whatever other activity the cluster having to perform. At the same time, we can help mitigate this by running our tests more than once, but running for tests it can take a while, so man actually doing this might be kind of expensive in both time and money fortunately kind can help us here by letting us run in many clusters on a single machine.

B

Yes, and then we can run our tests against them here, for instance, I have six kind clusters running on my desktop. I can even run a complete conformance suite against all of them simultaneously in this time when it would have taken for start-up and teardown of a GCE kubernetes cluster.

B

So once you have your coverage files you'll want to do something with them. We built a tool called Goodridge to make for coverage files useful, particularly focusing on manipulation necessary to deal with kubernetes coverage files here, and we have this lovely logo, which was also drawn by Ben viola. So congratulations for having so many parts of his presentation.

B

For first of these, subtasks for go for it solves, is merging the coverage files, which is useful when you need to merge files from multiple instances of services such as, for instance, a multi node cost of it. You just tested, and also to merge coverage from different services into a single coverage file, so that you can see the overall coverage, including any shared code they might have.

B

We have diff functionality, which calculates the difference between two coverage files taken from the same binary or merged set of binaries, and shows you only what was covered between them, which is useful when you need to arm.

B

Which is useful in order to find out what was actually covered between two dumps taken. So if you take one before your workload and one after your workload, then you don't actually need to bother. Then you can exclude the code executed during cluster up.

B

For instance, um we have aggregation which is used when you're running a lot of these same files, nope when you run the same tests repeatedly and want to see how many of those actually covered the same code, which enables you to reduce the noise as like, using that six kind set up I have before and we have go forage a unit and a go forage HTML, which will generate a j-unit file, which is useful if you want to dump it into tests, grant or some other tool that consumes those and if auto, generates a browsable version of your coverage directly.

B

All these are using standard, go coverage files, so any other coverage to length ago will work as well, whether that be something like a an online service or indeed be built in go tool, cover command and finally, if you're interested in results, but don't actually want to run it yourself. We women see ourselves on our cow instance cluster, so you can see here we have feed conformance and all the tests with the coverage from the conformance tests being at 29%, and we covered from all of our tests, of which farm anymore being 35 percent.

B

So those may be some room for some improvement here.

B

So how does all of this work? Well, girl has some built-in support for generating coverage reports for unit tests. So what? If I told you that we could just make kubernetes a distributed system of unit tests.

B

So if a backbone of this approach is, as with all things, kubernetes a terrifying and yet unsightly unavoidable pile of Bash, so let's take a look at exactly what what Bash does to pull this off.

B

So the first thing we need to do is create some unit tests. What we're going to run in this case are unit tests are unusual in, but they don't contain any tests and also in fact, they immediately invoke the original entry point, as you might guess, from the name we're passing into a net coverage. There they're also code generated by a bash script for best way to write your go code.

B

When coverage is enabled, instead of building for normal way, the build system generates one of these fake unit tests and then uses go tests to generate an executable for this unit test. What was formerly a kubernetes program without immediately running it, because when your build cert process would take forever and then before we invoke the original entry points. Our fake unit tests first passes a fake command line in order to configure the go coverage, support and then sets up the coverage dumps.

B

If it's sort of command line hackery is not unusual in kubernetes, we actually do the same thing everywhere to deal with glogs behavior, so this now happens repeatedly in these setups and then finally, we have this periodic flushing function, which gets called at the ends and periodically in the middle. This features even more hacks. We first create an empty suite of unit tests. Then we construct a unit testing platform, but does nothing? Then we call a pseudo private function from the go testing framework to run our non-existent tests.

B

The end result of all of this is a current coverage being written to disk ready to be collected by some other process. Amazingly, this all actually works.

B

So to summarize, this works via unavoidably complicated bash code generation of unit tests that don't test anything also in bash, forcing a go to pass fake command lines in order to convince that we did something and then creating and running an empty unit test suite every 5 seconds. It's all fine, it all works.

B

Everything here was totally reasonable and that is about it. So yeah you have some closing remarks. I believe.

A

Yeah, if anybody wants to get involved with sync testing or prowl, pretty much, almost all the tools that we've talked about, have a folder under the kubernetes testing for repo and the cig. Testing slack channel is a great place to reach out. If you don't know how you want to get started, yep any questions.

C

B

The question was how long it took to make a graphic of spiff XP. The answer is about half an hour because we're really bad at using slides.

B

It was time well spent.

B

D

A

So currently it is a kind of a subset of seasick testing the because currently only Googlers have access to the cluster. We are in the process of switching that over to CN CF funded architecture, so that any contributor can be part of the team that is managing that if you're interested in that I believe there is a kate's, infra team working group. So there's a slack channel for that.

C

A

We'd love to open-source test Grid. If you want to open, if you want test grid open sourced, there is a github issue that you should post on and say that you want test grid, open source and help with that too. Yes, we very much would like to do that, though. That's kind of the last piece of our ecosystem that isn't quite open source, yet.

B

Kine can be used for end-to-end testing and indeed we do run some end-to-end tests on it and they all work. So it is a design goal and it's a good idea to do that. Just run your tests against it, like any other cluster.

A

Yeah so I think I was one of the early slides that I had the second slide.

A

Yeah no worries yeah, so there's some logos but yeah there's a number of other organizations that use proud, I, think some of the biggest ones are OpenShift and jet stack and probably is do and native after that. Can you.

A

A

The Linux kernel mailing list is actually starting to.

C

A

Yeah there's other people out there that are using it. There's yeah, that's also kind of kubernetes, though right so yeah I mean if you use kubernetes prowl is kind of an obvious choice because it runs on kubernetes and you're familiar with it. So it's easy for you to deploy so there's kind of a lot of synergy there, but there are some other people outside of the communities community using it. Yes,.

B

I do not have combined numbers fewness integration tests, but I do know that the unit test coverage, which we don't formally measure or collect anywhere, is currently 53 percent for kubernetes.

B

um So we are attempting to use the end-to-end integration test coverage in order to try and determine what what sorts of behaviors our end-to-end tests are actually exercising. So, while we aren't trying to get those to 100%, which would be basically impossible for an integration test, we are trying to see which areas are completely omitted, but probably shouldn't be I.

A

Think a lot of the utility is when you dial down into the specific packages and when you're looking at that scale,.