Kubernetes SIG Testing, 14 Nov 2015

Previous Meeting

⏯

youtube image

►

From YouTube: SIG Testing Deep Dive – Sen Lu, Google & Cole Wagner (Intermediate Skill Level)

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

SIG Testing Deep Dive – Sen Lu, Google & Cole Wagner (Intermediate Skill Level)

If you are a k8s contributor, and curious how k8s testing infrastructure works, come and join us! This session is dedicated to a deep dive into k8s CI/CD system - Prow.We will be showing how each sub-controller works in Prow, what happens after you submit a new Pull Request on Github, and how Tide automatically merges your PR. We will also go over the roadmap for the year! After the session attendees should gain better understanding of the Prow architecture and best practices in getting PRs merged.

About Sen
Sen started to contribute to Kubernetes Project since 2016. Sen is mainly focus on Kubernetes Testing Infrastructure, i.e. Prow, Kubetest, Gubernator, Boskos, and Testgrid...

About Cole
I've been working on GKE Engprod for a little over a year now. I primarily work on testing infrastructure, process automation, and automated PR merging for the Kubernetes Github org. I specifically focus on Prow and Tide.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hi everybody hi, I'm aaron, kirk berger and welcome to the sig testing deep dive. Today, san liu and cole wagner are going to be presenting to you. I meant to wear my left shark t-shirt to represent the team spirit here, but then I realized today is May the fourth, so I had to wear my Star Wars shirt instead.

A

So if you want to flip forward to the agenda sand today, you're going to be hearing about Prowse architecture from sin and Cole is going to walk you through a part of prowl called tide and sort of our merge, automation and merge. Where we're thinking about that and then sounds gonna give you a little bit more about the roadmap for tests infra. If we have time questions thanks, sin yep test hope.

B

Yep, okay, let's start so. The first question you might have is: what is prowl assume you'll? Have our contribute to communities? Prowl is something that after you open PR in communities, you might have a lot of like part interactions, someone like /lg TN or such we test and run a bunch of priests image for you, and also your PR gets magically other merged into the head of the kubernetes branch.

B

So that's all done by prowl, so pretend Osada, community ICD and as a part of the stick test environment, a nurse our job is to maintain proud and also a bunch of other tools for communities, develop developments. So another question you might ask is like hmm I remember a year or two years ago, kubernetes the ICD runs on Jenkins right yeah, that's right, and this is fine. Then why do you build another system?

B

And no, that is not fine. During the past few years we had a lot of problems with Jenkins. So, for example, Jenkins is very stateful. Whenever we want to update the Jenkins version or darker version, we to restart the entire rankings, and that would cause us to last. Our running jobs, that's kind of not acceptable, especially around the release, time and also don't need to mention the github plugins is really hard to maintain and there are tons of security issues. My teammate XP, like Jeff. He was like our Jenkins master.

B

So whenever we had Jenkins issues, we will talk to him. Ask him to fix it. He had a record of like spending in entire week on Jenkins without doing anything else. So um after more campaigning, we decide to build a new system and you can see the dog is very unhappy here and like firefighting everywhere.

B

That's the motivation. Why we have prowl proud, is a project start by SP xtr on github around 2016, and we have been fully migrated to prowl from Jenkins by January this year. So we are testing kubernetes right. Why do we testing kubernetes on Jenkins? We and that doesn't make sense, so we want to test kubernetes on top of culinary's. That will make it like really easy to maintain and deploy. You can define your job basically as a pops back.

B

We will just get you that part for you and proud respond to github web hooks directly, rather than we were using something called mont, github, basically loops through other open PRS and trying to do something for you and conclusion is. We know kubernetes much better as a community developers than jenkins. So whenever we have some prowess use, we just turn around grab something from the sick team and hey. Please fix this community sees you for us to make our testing for matter that's a much better like psychosis.

B

Ok, next question is sounds like prowl is awesome right. What does Pro really do? As I mentioned, when I was a contributor to kubernetes when I, for example, when I'm reviewing someone else, PR I? Do you / l, GP n? There is this thing called Kate. Ci bot will come in and assign you and add a label, that's part of Pro.

B

Also, when you open a PR, you have like no idea who should review my PR, so probably fixing to suggest a reviewer for your approver, for you also on the bottom of your PR, there's tons of prima jobs running like some failing, some are succeeded.

B

That's also triggered and run by Pro and last but not least, proud, like after your PR merge magic matches are the merge requirement, probably will automatically merge a PR for you and also, if you change a config file, for example, you want to add a new, proud, proud, so understands that and update the contact max for you in the prod caster. So you so there's like no menu deployment happens there.

B

Ok, this is the meat of the talk I'm going to walk through, like how actually proud works, how your PR is getting tested. So, let's assume here we have a github. Basically, it's like everything in github and we have a case case. Cluster has prowl deployed.

B

Everything starts with you open a PR or their thing called Fela bod that randomly drops a slash three tests on your PR. If your PR has a flaky test, github will send out a web hook and basically that's a github event through through hook. So just a legend reminder, everything was a blue square in the profits. Cluster is a proud controller. What does hook do you upon receiving the web hook hook hook will make a proud job, which is a customer resource definition represent your job.

B

Basically, it contains certain a certain amount of metadata like the status of each job and also a pots back on the bottom. Basically, like you define how your proud job needs to look like, there is also a few controllers called tied and splice, which manages like batch merges. Basically merge automations make sure your PR is good to be merged. Co will cover those on the merged. Automation. Part there's also her Logan, that's a great name by made by Ryan. Basically, that's a controller schedules, a schedule job on a periodic basis.

B

For example, we do kubernetes the IBEW hourly build like every hour. Her Logan will create a proud job every hour, okay, there's a component called plank in the middle. What does that do? Bank is a controller. It's like an operator type, so it loops through all the created, proud jobs based on the current status to the proud job. It will create a part based on your job definition. Basically, a pass back like your job can be like any pass back.

B

We use subscrive api types, so in this case this this is a example sa kubernetes prison. It we pose a cubic in CD image, start up a communities pod. It checks out our PR rounds, cube test. It build your beauty, fresh kubernetes from your PR. It will create a LED cluster. Using your company used, coolant is built from your PR and last.

B

The cube test will trigger a ginkgo e to iran. You are probably familiar with that and that will talk to the LED cluster lives in somewhere else.

B

So after everything, the example in this case, the idiot has to will exit 0 and succeed and one on failure. Just a reminder. It don't need to be that complicated. It just need, if you're, using probably just need to be a simple part that returns x is zero on succeed and one on failure, and we have a UI callback.

B

Basically, that's the FRA, a proud front-end that shows up all the current running status of the job like yellow means, the job is still running. Red means the job has failed and green means. The job succeeded and.

B

Saying plank has been watching like the status of proud job whenever it sees the project, it will talk back to the github, basically update the github contacts on your PR. To show that hey your PR is great. Your PR is good to go.

B

We also have a sinker controller, that's basically responsible for garbage collection like cleaning up pot and proud you're out south to rest of the month of time. Hope that makes it clear. That's like the basic architecture of proud like how everything works from your a github webhook to a kubernetes pod job, and it refers back to github.

B

Okay. The next part I want to talk about is the plugin for prowl. So that's like the bada automation part. How like, if you slash LGD and about applies the house, you team label, for you so upon receiving a github event, for example it even it's a comment event. The hook will dispatch the event into different plugins based on Avery, either a rejects match or a vent type. In this case, the upon receiving a /l GTM comment, a comment. The hook will forward the comment to the LGG unplugging and the LG Chem plug-in will add.

B

Al did here labeled to a PR the same thing as complicated plugin. When we receive a merge event, the convict plug config update plug-in will check if, if anyone has changed the pro config files, for example, I added a new crowd and that PR is merged. The config update plug-in will automatically deploy the map into the broadcaster and asked we made some like silly plugins like it doesn't we have a lot of those we have like me? Oh you can move.

B

You can also slash joke in this case, it will find a random cat image like from a line and append to your PR yeah, and we also support all kinds of customized plugins. If you want, you can always contribute to write your own plugins for proud to use by the case community, ok, a probably awesome prowess currently adopted by a few open-source projects and namely sto just AK, kubernetes and OpenShift.

B

For the prawn kubernetes. We have hundred eighty-one notes and more than southern course. We have been running three four points: three million jobs last year, stats from Arin, and we have seven hundred seven, seventy different jobs and we have set up resubmit for thirty-one goodness repos, it's my last fight. So, hey, you might think hey I want to try out prowl. Is this. This looks like super complicated from my diagram. Don't be scared out. What you really need to get pro running is anywhere.

B

You have a kubernetes cluster, you need your github token and you need a starter llamo. The start of the ammo is a basic, manifest manifest file for some basic product component. You can find it in our testing for a slash pro, so that's getting started on Andy.

B

He basically keep could apply the starter diamo to a cluster and you should get everything running and then you should be able to fold your webhook from your github repo to your prow and that should get your boss start run start running yep, okay, I love to go for how we managing other marriages for goodnights PRS.

C

So our goals are goals with automated PR merging are to merge PRS that meet specific requirements. Typically, that's determined based on labels on github and other PR properties on github, like what milestone it's in whether or not the status contexts are passing, and we one of the reasons that we want to be able to do this instead of having people click. The merge button is because github does not have very granular permissions, so we have to.

C

If we give someone access to clicking the merge button, they have access to modifying labels and they have access to click. The merge button on any PR, where we'd much prefer to use our own code ownership process with the the approval plugin.

C

Another thing that we want to do is ensure that a PR is not merged until it's tested against the most recent based branch commit. So I'll go over that a little bit more in another slide, but basically we want to make sure that the status context that you see on your PR are actually up-to-date with the master branch.

C

If it's been changing since your PR was tested- and importantly, it also needs to be super scalable, because the kubernetes org is enormous and we need to be able to process thousands of open PRS in the kubernetes repo alone, and we also want to be able to process the the dozens of repos that are in the kubernetes org.

C

So here's here's why we need to test against the the most recent base branch commit. So here you can see that there's a commit on master and there's two PRS that are based off of that commit. There's a read PR and a blue PR, and they each have a emerge commit. That is what we're actually testing so before we would merge the PR we would generate this merge, commit and test that to make sure that it's passing.

C

But the problem is that if we then merge PR one after its test is passed, this merge commit for the bloop. Er is no longer actually testing against the head of the head of master, so it's possible that there's a conflict between the PRS. Essentially, so what we really need to do is test this commit. We want to test the the one that's based off of the most recent base.

C

Branch commit and github doesn't really allow us to do this super nicely, because a job reports, its context for the PR and it's tied to the PRS commit, but that doesn't tell us if the base branch has been changing or not. So we need to be able to keep track of this ourselves.

C

Another big thing is that, with the the scale of the the kubernetes repo ins, specifically, we often have 50 P hours per day and when testing takes 45 minutes to run, that would limit us to testing about 35 PRS per day. If we need to do this serial testing approach, where we test one PR emerge and then test the next PR based on that, so our Q would essentially back up and we wouldn't be able to merge all the PRS that we need to do it just wouldn't be possible.

C

So instead we batch PRS together we test them together and then we can merge them together if all of the PRS in the badge succeed- and we have a bunch of logic to handle this intelligently, namely if we are testing a single PR and a batch at the same time, if the single PR succeeds, even though it could be merged, we wait until we find out the results of the batch, because if we can merge a batch, it's much more valuable because we can merge many PRS at once and clear our queue much more quickly, so batches are prioritized and serial tests see really tested.

C

Prs that are tested individually won't merge until a batch has actually failed.

C

So our previous solution to all of this merge automation was the submit you if you've contributed to kubernetes. You might be familiar with this UI. It's currently still used for the main kubernetes repo, but has been deprecated everywhere else.

C

So there's some major downsides with the submit queue that we have run into one is that each submit queue instance has a single binary and config / repo. So that means that, for every repo that we have, that we want to do merge, automation in you need to spin up a new deployment, a new configuration which is a significant maintenance burden.

C

It's also me: it is also scoped to the repo it's not scope to branches, so that means that things in different branches are still all in the same queue, even though they could be merged separately, because different branches don't affect each other, and this also kind of makes deters people from adopting the submit queue because they have to spin up a new deployment every time they can't just add some configuration to a config file.

C

One of the the other major problems was that it pulls four PRS and then filters through the PRS locally, and this is extremely expensive, because polling for the PRS is not actually too bad, but once we get the PRS, we need to get a bunch of additional information about them, like the labels that are on them.

C

Sometimes the issue comments things like that, and that uses a lot of API tokens, there's also complicated, prioritizing logic, which is it can be useful at times, but ultimately it's more complicated than it's worth and it confuses contributors when they're PR is not merging and something else is jumping ahead of it in the queue this omit queue does have batch testing like I was describing, but it was added as an afterthought.

C

It's an extension of this to make you and it requires an extra prowl component to be able to coordinate with proud to achieve all of this, and the final thing that is is bad about the submit queue. Is that, because it is not part of prowl, it does not know which base branch commit. The PR has been tested against for all of our test results. It has to go off of github, which, as we mentioned, doesn't doesn't know which base branch commit our tests were tested against.

C

So, in order to solve that, every time we want to merge a PR, we have to retest it right, then to make sure that it actually has recent tests done against it, and that means that if the PR actually was up, if its tests were up to date, we are doing an unnecessary test which is going to take 45 minutes.

C

So our new solution to all of these problems is tied and it uses a github search query to define all of its merge requirements. That's the the major important thing about tied. So instead of listing all the PRS and filtering through them, we just ask github for all the PRS that match our merger bowl query. So github will just return. The PRS that are mirja balandin even need to know why?

C

So, that's that makes the the logic simpler. It makes it more friendly and for github users since search query. Logic is more familiar for people who use github and it saves us API tokens. Another good thing about tied is that it's got batch merging built into it as the core component.

C

So once you set up tied and have it working there's zero configuration for batch merging, it will just happen automatically whenever it's possible, because we're able to use so many fewer API tokens, our single instance is actually able to handle much greater scale, so we're currently merging pr's and 45 repos from a single deployment. That's running in our cluster, that's compared to a max of 6 repos that we ever managed with the submit queue and that required six different deployments.

C

This is actually also using a the same github token that the rest of our proud appointment is so we're only using one github token for all of these, whereas the submit key would frequently a single deployment could run through all the tokens in an hour.

C

So this is much more efficient and we've replaced all of these to make you instances, except for the main kubernetes instance, which we will hopefully be replacing soon and because tied runs in the the proud cluster it actually has access to all the test. Data itself and part of that data is what base branch commits. We were testing against, so tied actually has enough information to know if our tests are up-to-date.

C

So therefore we don't have to blindly retest like this and make you does so here's what some of the UI for tide looks like this is a tight still pretty new, so the UI is still being worked on and still has some some room for improvement.

C

This image doesn't have a whole lot of action going on in the queue right now, but I just wanted to show what it looks like when there's a multiple branches being tested against here's a more interesting one. So this will show that these these PRS are have been tested against the most recent commit and are both passing and it shows that tied just merged one of them.

C

Another useful UI that we have is actually on the PR dashboard. This isn't specific suit ID, but it includes tied data, and this dashboard shows a bunch of cards like this one. So it will show you all of your PRS that you have and I'll show you that tests are running for them and also it will show a diff of the merge requirements based on.

C

What's your PR has so we'll show you what labels you're missing, what labels you need to get rid of, or any other things that are preventing your PR from being merged and finally tie.

C

It also has a status context that appears at the bottom of your PR, just like any other job would, and it will show you whether or not your PR is mirja ball and in the pool being tested for merge or whether or not there's actually something you need to do to make it mirja ball, and if there is something that needs to be done, it will detail what that thing is in the comment. So in this case it says that your PR is not mirja, becuz you're, missing the LG TM label.

C

So, as I said, tied is brand new and there's still lots of room for improvement. So some of the things that we're planning to improve in the future are how batches are done. Specifically, we only have we have a fixed batch size right now, we're only batching five PRS together at a time, and we only run one batch at a time.

C

We think this could be improved by running batches of size, one two, four, eight all the way up to the log of the the number of PRS that are trying to be tested and we can run all of those batches concurrently and that's a beneficial because one of the harder things about batching is what happens when a batch fails.

C

We don't know which PR in the batch was actually broken, so we have to have a process for kicking things out of the batch and if we have batches of varying sizes we'll be able to find a successful batch faster so that that's one of the improvements we want to make. We also want to make it so that you don't actually have to have all of your tests passing sometimes there's a status context that reports to your PR. That is not actually important.

C

Maybe it's a review status that is optional or a job that is optional and you want to get information on it on PRS, but you don't want to require it for merge, so we want to be able to change it so that your PR, so you can specify that certain contexts are not actually required and additionally there's a couple. Ui changes we'd like to make, notably on this diagram.

C

You can see that there's some, it shows jobs that have failing tests, which is a little confusing, because this is supposed to show things that are emerging and if your PR is failing tests, it's not going to be merciful. So there's a couple of improvements that we want to do there. But ultimately it's been very useful for us so far and it's been growing very quickly. It's quickly expanded to a lot of the repos that we've used and because it doesn't require a separate deployment.

C

All that it takes to configure your repo to run with our instance is just adding a configuration for your query that you want to confine mirja by RS. So it's it's dead, simple yep.

B

So that's tied, okay, so last, not least I'm trying to how trying to go over the roadmap for testing FRA with Aaron's beautiful picture talk. So this is the same slide. I present in the sick update on Tuesday.

B

Basically, what we want to do is first tide is awesome from Koh, so we want to switch tight for the main communities repo and we can retire our last living some acute instance, and we want to split up the prow configuration for each org or sick so that each stick can own their own trial jobs like configurations and I, don't have to review all the project chain. Dprs also have once we have a policy to promote or demote like merge blocking jobs. For example, we had instance cops arrest.

B

Job was like failing on every single PR because of there is a resource leak on a dress, so I have to like harass just in Santa Barbara, to fixing it. Instead of doing that, we can have some policies say if a pre-image job was like consistently failing for a certain number of time. We can demote demoted to be num blocking for a while. Until it's stabilized again that might help our developing velocity we're also doing a lot of compliments drive.

B

So we want to support both sonic running conformance test use, Sunna, Bowie and Q test, and we have a task where dashboard I probably can just show it basically collects all the data from were from different cloud providers.

B

Well, you're welcome to contributing your test result their last. We want to create a problem, though we want to open sourced a screw. It I mean everyone. Is you probably using test grid, but currently it's still closed source in Google? We want to open source it so that we can get open source contributors to improve that. We also want to integrating logging artifacts viewing pro, which we are currently using. Something called Google ADA, which you click on detailed link from your PR status, so that we can have native support supporting Pro. We.

B

We also want to create a bundle between Pro and test grid so that we can making more people to adopt Pro last time to say thanks to our testing for our computers, and we also welcome everyone to contribute into testing for us. So next time you can you're outta your github avatar can show up on this page question time.

D

So I've been interested in writing some prowl plugins for a while, but the problem is that I don't have any way to test them and I'm not like testing them against crashing, but testing them against updating. You know PRS in the wrong way.

C

There is actually way that we can test them. We have a tool called phony that basically just lets. You provide a web hook, payload that you've gotten or have generated yourself, and you can send it to hook. So that would allow you to once you have your plug-in written, you build it and compile it and you run hook, and then you can just send a web hook straight to it, so you can run it all locally and test it that way and there's instructions on how to do that in the in the beaver pro.

E

So we have been discussing two at logic: I want labels for Klein, go developer change, locks, for example, and we never made anything in this direction. What is the process to introduce something? You need some new label process and maybe a probe in.

C

Asking what the process for adding a label is.

C

For a command that controls a label.

C

So to make it PR blocking for merging, that's that's controlled at the tide level, so you would say in the tied career, you would say that it can't have this label. So that's a blockage.

F

C

To actually apply the label, you typically, if you want to you, have a command that controls the label or if the labels automatically applied based on the properties of the PR. You'd have a plugin that reacts to github events for the PR, so either commands that control the control, the label or the events for the PR, so that you can inspect the state of the PR and see if the label needs to be applied and then the plug-in can determine whether or not to apply or remove the label.

C

You just have to make a plug in and for brow. Well,.

A

So if it's something you're interested in introducing to the community, I sort of like to say the test in fair responsible for sort of implementing the process and providing insights from the boots on the ground level, but we also really appreciate review and insight from the contributor experience sig. So we generally like to make sure if it's something that's going to be affecting the kubernetes project as a whole, that it's run through both of those groups and we often have attendees from both SIG's at either SIG's meetings.

A

So you'll generally get feedback, but sometimes you might want a more thorough review, but yeah generally I would say. The process is just open up an issue if it's purely like a technical change in an existing thing, maybe test in front. If you're talking about how to change this for the kubernetes projects as a whole, you probably want an issue in the community repos since you're changing how the community does things and we'd have an issue in testing for the track. The technical implementation of that.

F

So what does happen when the test batch fails? What's how you recover from that.

C

So whenever we run a batch, we are also running a single PR from the batch. So basically, whenever a we need to trigger something, we trigger a batch containing a bat PR and we individually test that PR. That way. If the batch fails, we can have a signal on one of the PRS in the batch on to kick it out or not. Having multiple batch sizes will help us with that.

C

Yes, so when the batch fails in that PR succeeds, we, since we know that PR is okay. We end up, we merged that PR, and then we run another batch with the the remaining yeah. Yes and that's because uh if something passes serially, we know that it can be merged so yeah if we have to run a new batch anyways after that, we might as well run it against the newest commits on master.

C

The batch testing is triggered whenever we start a serial test. They happen simultaneously or, if there's not a batch running out. Yes, so if, if something is somehow gotten out of sync, it will start up a batch anyways yeah.

G

I'm trying to find those dashboards that you showed here, the dashboard of the test result, but I tried to google it and I'm trying to find it in the comments of merged pull request. I have no idea how to get there proud.

B

Arcade style, that's.

G

Like the dashboard shows odd jobs, Ronny and people that haven't been at its talk, how do they find it.

A

There's a couple things: there's like a contributor cheat sheet for common pieces of testing infrastructure, we're also working on improving, like our testing documentation in general and I, know that we have. We provide links to the deck plug-in if the job is currently running, I think but I'll. Let Cole answer that for sure. So.

B

If you also, if you're, going to kubernetes testing for our main page, if you scroll down there, are the links to all the useful dashboard you can find just go to github.com. So that's kubernetes. That's testing for also.

C

To answer your question more directly, since it was about the knowing about the tied dashboards, those are all linked from the tied status, so on your PR. We're tied is telling you what your state in the queue is. If you click on the link there, it will take you directly to the dashboard.

H

Yeah thanks, hey I, was so excited to contribute to this project and I run this on my cluster and then I found out. You guys need cluster admin to run a regular micro service because you see Rd for that, would you be open to changing that.

C

No CRTs have a bunch of advantages.

C

We can do validation, yeah, the there's, there's not uh I mean if you're gonna be setting up prowl in a cluster like you probably should have admin permissions I would think, but it's also just a one like you're administering the cluster. So it's also a one-time setup and you just have to you just have to create the CRT and then that's the only step that requires admin. Permissions.

C

Yeah well, there's I mean using config Maps or something like that is not really a viable alternative, or at least it has a lot of downside. We'd lose a lot of the functionality of validation and other stuff like that that we get with the CRTs.

D

Either so some there was a proposal around making a change to PR labeling to deal with the changeover to tide. What's the status of that.

C

So I actually left a comment in the dock about that just recently, if we the it what it seems to be boiling down to, is whether or not there's any possibility for PRS to be made to the master branch but be assigned previous milestones, if we're not sure, if that's actually something that occurs or if that should be allowed, if that's not allowed, then we don't actually have to write any additional code. We can just like change the tide, query and we're done if that is the case.

C

If there are PRS in the master branch for previous releases that we need to allow to merge, in addition to the current release, then we will need some sort of mechanism to control the labeling.

C

I think that's all the time we have yep.

B

And thanks everyone- and you can find us on suggesting on slack or acoustic testing at Google Groups comm, and welcome to make a contribution to testing by sending a PR to testing for repo thanks. Everyone Thanks.