Kubernetes SIG Testing, 22 Aug 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG Testing 2017-08-22

Description

Meeting notes: https://docs.google.com/document/d/1z8MQpr_jTwhmjLMUaqQyBk1EYG_Y_3D4y4YdMJ7V1Kk

A

Okay, hi everybody today is Tuesday August 22nd. If you are at Cigna texting weekly news on the agenda today, I want to run through a couple quick things that came up as a result of the discussion on kubernetes staff. I wanted to highlight some improvements that will tighten :, make you super native recently and I. Wasn't the rest the time off to somebody to talk about well, first, he can teach me how to pronounce Bosco's weekly recover.

A

However, um so, first off real quick, there was a discussion in kubernetes that or the threat of leaking in chat here and is in the meeting notes where somebody suggested the idea of maybe removing implicit self approval or self approval, and it's sort of mothballed into this. Much larger discussion about many people having opinions on how the automation should work on this project, including things like. Maybe we should always have self approvals. Maybe we should never have self approvals. Why is it that people who are Gina donors files can /lg m/l GTM things?

A

We should do that. You shouldn't do that and maybe we should auto label TRS that have with. Maybe we should make those not merciful and I think that sort of towards the end there Eric kind of came in and rightly suggested that so storage doing that sort of thing. This is a fantastic place to discuss those kinds of issues. A lot of people here who are familiar with these code bases and work on them actively and daily. So I think. Maybe everybody wanted to express an opinion on that.

A

Stuff has already chimed in on that thread and um thank you Steve for sort of taking the initiative to push ahead with yeah. Maybe it is a good idea to pave the cow path of four pull requests that are works in progress. Maybe we should you know people sort of naturally put within the front end of pull requests title. Why don't?

A

We have a bot, recognize that and apply label, and you know not merge those things do that in a way where, like is control to pull request offers, you mean necessarily be organization, numbers fit kind of empowers people. We have any other strong opinions on that stuff. We can chat about it briefly here or we can do those.

B

Yeah the only comment I had that I was like that brought me to actually make some of these changes was just thinking through, like we ran into once. We deployed the munchers just some distances where it was easy for developers to make mistakes and working with with the program, because it wasn't quite clear how I was working and so I guess just continuing to focus on like making sure that also the decisions are made by asking keep it at ease.

B

It does I understand the point that Eric was trying to make, but I think the intent of asking the community at large is just that. You know it's something. That's going to impact the workflow of everybody. So doesn't it you know you want that input from everybody, yeah.

A

Absolutely- and you know, I think that was the correct thing to do: Caden we're thinking about making a change that impacts the workflow kubernetes development community. That is absolutely the right place to ask that question and unfortunately, multi-cat was a bunch of people talking over each other about a variety of subjects. Nobody actually exclusively answered your initial question and page.

A

Should we turn this on or off um figuring out how to steer that International things just a certain thing, but I think we got a couple concrete items and the next steps are kh2, then that here's, the Rico's work on this stuff, please open, is here and we'll track that in this particular state, moving forward in the Mysterium committee, parlance of like lazy consensus, I, don't think I've got Latian sentences on any of the suggestions that were raised in that friend when you're actually working on a yet assume like adding a label for work in progress TRS with natural evolution, so I'm interested in that place.

A

But I don't want to hijack the rest of this meeting. For that. So we have further thoughts. um You can follow up on that thread. One real thing: if we can totally come back to that extend long decline. One other really quick thing I wanted to highlight. Maybe everybody here already knows about it, but claiming one fact finally picked up some work. That's been taking up there for a while to have here the ladies and slay standard out in standard air in urinator, so like that nickel PR that I liked.

A

If you look at that to see that it says um it actually says the tests with have here, I guess I could like share my screen, so this shows up in the video look like boom, so you can see that you burn a turn out. Sets tests there for the field there 300 that succeeded, that's really cool and for this proxy version be one Tesla field. I can click this little link here see stator there and or standard out as relevant to this test.

A

This was also I could have like click through to the raw dough, blog and draft and found it. But it's like here in more place and it's just a super awesome neat useful and improvement that I like to jump up and down on this on. So thank you so much like um okay and everybody who actually got this deployed. I'm sure it was a team effort but I know. There's you know it's not an easy picking up a PR that you wrote back in February and marching for that.

A

You can add to page a bunch of stuff back in your brain and remember what the problem really was and what the state of the world is now and whether it still make sense to solve this problem in this life. It's cool to see everybody like pushes forward. I'm sure there was some self interest and mutual alignment of interests, but thanks.

A

Okay, um with that I will hand off to set to talk about. Kids me tell me how to grant a stock. You know I think.

C

The proper pronoun patience like Costco, because it's from hungry templated help me bow that from the spanning yeah like it should be vodka, so the matter I'm, not a great Greek speaker, so I I'll give present just your swing.

C

Or we can check out, wait, okay, so, okay, this is my original design doc. So introduce some background. So the initiative or creating vodkas is for better managing our our testing projects. We currently have like over I think over 500 different testing projects, the Prine point before it was. Whenever you create a new job, you need to create a specific ECT project for it and and UK meet some like special treatment and all that had to be done through, like our Google internally.

C

So it's um it's really blocks like from open source world to add a new job or if they want to add a new job. I always have to ask us to like hey: can you create a project for us and whenever.

D

C

The jobs, what for example, when we don't support like really least 1/3 after we were reading odd ones, three Reese jobs, we don't delete out of creative projects and we kind of keep creating more projects but like leaking all the other choosing projects. So I wrote this: it's as a general resource management tool that, instead of ping with Pacific key to key project, a community CDI job, the job will grab a free project from the project pool from buskers and after it's finished running you know it has a janitor component.

C

Will white house will clean up the news project and return Reata to the pool for the next job to use, so it as flexibility to click and current you to disk is for easier project management, but later on, we might can have more single thing out of more units or this bucket system.

C

So the components of bacchus include a focus, a server that can that contains they pull like four or five hundred GCP projects and have a client that can.

C

Initiative this way, so a vodka can contain a pool resource objects. A resource object is just a generic object. You just give it a name, a type and from they check up, you can change a state like from any string. You want to give a example how it currently works is in in crawl, say we have multiple, unique jobs at the beginning.

C

At the beginning of the video it will look, it will ask the basket server for a 3d project and the set the speed to busy, and if you look at the I guess, I can grab a running.

D

C

Now started yet by this GP up down.

C

So, at the beginning of the wrong each job, it will cry acquire a GQ project from Bacchus, and that project is like is case, ganking CBG, a prod, that's and everyone will pick up a different project.

C

And while it's running you need to update, you need to give the Bacchus over a heartbeat to note by that alarms. Theory I am still using this project and whenever you finish using it, you just return the project back to boxes over but said the state to be dirty and.

C

The second, the janitor janitor, is always pulling coding other dirty projects from the Bosco server and around the internal gender script. It's Simic similar as the currency, I, dentists, say, I'd enter, but that's running once per day and the Buccos janitor runs right after the job finishes and cleans up the classics. The cleanest cleans up cleans up the auto resourcing the project and return the project back to vodkas and the third component deployed is called Reaper.

C

Reaper is responsible for kill any inactive like inactive resources, for example, if a job checks out a project, but the prop hog just exploded and the Reaper the Reaper will finds like find out. They just props. This project is checked out, but it's being active for like an hour, so I'm going to assume that there's something wrong with the job, so it will return the project back to the server but marquette 30, so the janitor will pick it up and recycle project also recently added a grass on the bathroom, so that marks.

C

So basically, this is like the force. They've bought our projects, one rope or GP project and we'll roll board UK projects like what UK is pretty stable. We always have like hundred something free projects and assembly, like the number of Billy project just means currently running jobs. We always like to have 7080 ish running jobs. The GC seems has we project and it's a EVP monitor and we can, whenever people adding like wine, more jobs and we find out. Oh, we are kind of short on projects.

C

We can just create couple projects and fill up the closet pool, and so this make think the main merit is vodkas, made it easier for people to creating new jobs to accept, especially for from open source like if you want to make a specific Kunichika job. You can just set a set of the jobs in the create a job in the prowl at the Contra JSON, and you don't have to worry about like project like create a specific key key project for it.

C

C

Think a future future work. This roadmap I want to address and later on, I want to make so currently vodkas had a pool of TCP projects later on. We want to make backup to manage it full of G, C or D K clusters, so there can be a like gardener component. Just keeps going up being up like new clusters and EGP job.

C

If it's not care about like cuz the life cycle, it can just check out a rest Kuster rather than a fresh project, and that that can eliminate like odd up down time from a fear on and each job and yeah I can avoid. Also can't avoid the flakes in the town square so that beefy job can focus on the passing yourself. Oh.

C

Yeah I have any questions like discussion ease this.

A

C

A

Cool I think Murray's got a question in the chat about design. Doc looks really cool I couldn't find a link to is there any chance? You can put a plaque. Yeah I need.

C

To share it externally, I think I'll do that shortly to make this happen so.

A

Like I've been watching me work on this over the past couple of months and it's been really cool to see it like gradually evolved and they get to the point which was just sort of used and it wasn't like I didn't. Nobody really noticed, I think that it started getting used almost everywhere and then I saw you start like becoming this guy for money, Python asking people to like hey. Could you give me your unused projects?

A

I'll, add them to the pool and like we got today and so I now you have this huge pool of Google projects that are names, what are their names, but they can be used for any jobs. So it's been this great way of transparently adding value to make easier, formal project. So that's that's it your goals. Let's say like this. This looks awesome. Okay,.

D

A

Two questions: first, the the gardener component, the maintaining of like a cool Koster, sounds really cool. Have you looked at and if Deus have some kind of cluster cool thankful there GTE stuff at some point, I'm I'm guessing you decided to look at the project level force because some clusters create additional Google cloud resources. How do you feel the gardener component and sniff is in that model or like in grass controllers and slash, create additional rules.

C

Like okay, what is very-- market like ingress control because um more feel like I, want to make the mmm the project so the gardener just running the existing up/down script? It's only responsible for bring up and tear down or clean up the cluster. hmm It just spit out the current up down logic from the current past logic,.

A

Okay, that make sense. My other question, like graphs, are always fun. They tell stories I noticed around 10:00 to 11:00 this morning. You can see that date. Yes, right there I'm guessing that's what happens when bucks basta southern run right like those dirty hands, ether I, just.

C

This morning, yeah I checked the graph and it prints out the janitor pod cannot contact the service anymore off a sudden, mmm I simply redeployed it, and that seems like everything back to the normal I need to like looking to mmm maybe to improve the robustness there other than that spike. Oh there's another spike for GC project around like last week and that, like self healed itself, I didn't bother looking to that yeah.

A

Okay, cool, like I, think these resources, graphite I, was planning to be super awesome test. Like you can see the system at work. You can see. The stuff is actually happening and yeah I would expect. There was a problem with the janitor some part of Oscars but yeah the dirty projects to go up. Then, when we fix it, security projects go down again, it's like merge, queue is blocked. Pending PRF goes up or excuse pit expending PR, metrics and graphs KITT. Thank you for making your dashboard for this done.

C

We might want to add more metrics like say how many projects are checked out like blah. Let some thing like the last hour kind of metric. Maybe that's helpful like to let us know like how often does our job runs?

C

Maybe they cannot all be a halt metrics as well, but I yeah, that's fun. To make my first graph on Meldrum yummy and, like oh.

A

My guest one of the random question I had it can be also many questions would be like. So this is all very useful, especially like if I wanted to use this on my own. This is really useful. If I use Google compute engine, how would you that you're feeling like for the level of effort it would take to use this or other files, so the major strengths, another five specific resources on AWS epcs or something.

C

Yeah it can manage any sort of project and I K cost of a the best like resource pool the resources. A is generic enough. It can even be like some Federation token for life's so talk with Eric. It can get dollar to server something.

D

I would like to eventually lose Eric right away and something I would be interested in working out at some point in the future is not every cluster necessarily needs to do Cuba so, like potentially we could use. We could use Bosco's to maintain a pool prod projects that already have a running cluster, and so, when I, you know, add a like recent, uh you know commit on each branch or something and then so. The testing could sort of.

D

We could decoupled the Uptown logic and put those in specific cluster lifecycle jobs, and then you know, sig apps could just grab a already running cluster and then run tests which could potentially you know really uh rapidly increase the you know, speed for running tests.

A

Yeah they don't, they have absolutely in the direction. This kid ahead, like I, said that I've seen um I'm not sure where it's at today, but I know that sort of the model that Deus used when they were a dentist and we're close they used can take the name off the top my head.

A

Thank you, some component that release cluster out of a pool of gke clusters and then using that clean cluster, they would install a whole bunch of helm, charts to set up a dais workflow as and then run into n test against that, and then they would move done. They would hand it off and put it back into the pool. So it's that sort of model and like I'm not sure how actively is being user maintained today, but that might be a place I'd like learn some lessons from.

D

That's of cool.

B

Would that be useful for poor quest testing or just post submit some periodic stuff? Yes,.

C

So definitely for think. The base priests limit, fatal job is already using vodka, like each Rallo just grab project and the other job is not using it because they are still live in Jenkins when we migrated us everything from Jenkins to proud and basically.

D

The about the service is leaving.

C

The processor and once we migrate more things off from Jenkins into Crowell & Moring, to use the resource from box bucket, and you can still have the ability to ping a specific project to your job. Like example, the GPU tests and skip good, really test that requires the quota you use do want to paint a custom-made project to a job. They still have the ability to do so.

B

When you ask for an get a new project, does it just bus goes to the reader credentials to add substitute.

C

We have other projects set up using the given the same service. Account I mean the service account we are using for like Tata like uploading, logs and stuff is also has the right role or other projects in Buster's.

B

So when, if you were to make this also lease you a cue Brunetti server, you would expect the queue config to exist already before you can just pick up text.

D

So we actually queue tests already has the at what there's a save flag. You can send to cube tests which will allow you to upload queue config to a GCS bucket, which you can then download later. So we use that for the soak testing, because, like once a week, will deploy the soak cluster and you know, obviously it runs a test and then finishes, and then throughout the week we just reuse that existing cluster we go.

D

The whole point of the so test is to continue using the same cluster, so the next run will then just download the acute config and then run that on yeah.

D

So that would probably be the way you know if we wanted to have a pre spun cluster. We could do that. Another fun thing. If we had really you know if we got if it got really fancy. You know this would take a ton of work but like we could identify.

B

D

Get like the challenge with pull requests is we obviously need to deploy a cluster that has your changes in it, but like we have identified that, like oh I'm, only changing the replica set controller or the you know, whatever the the stateful set controller, we could potentially use a pre-existing cluster and then just build the build that image that runs the stateful set controller and then reload the stateful set job in the existing cluster and then run ad tests, but and basil you know, may help with some of that.

D

But that's you know that's that would not be. That would be a very challenging thing to identify which components of kubernetes we need to reload. Each time we make a pull request.

B

So I understand it also like this is not part of the scope right now, but I was thinking about as a person that might be diagnosing problems or test infrastructure. As you move towards this job, that's leasing a cluster. How do you thread through the information to let somebody quickly identify okay, this job may be failing because the cluster didn't get cleaned up correctly. Let me figure out what ran there before, how the cleanup happened, etc, etc.

B

Good, like it good, oh.

B

So I want to say something good like today.

D

B

Queue up and keep down that get to go. Look at a different job on your PR to figure out how that staging happens. First right right.

C

I think before I mean after you spawn up the cluster I might be, there might be some. We can add some like verification check like say. Oh, this cluster is healthy enough. It's like all the components are functional before it extends before they add itself to the pool that the running job wants to use.

A

Yeah I mean I, think that is that's sort of like the those sort of tricky use. Cases will require some thought, which I think is why it made sense to start at the project level, where you can kind of nuke everything from orbit, because it's the only way to be sure that you're you're, starting with a completely blank slate, and then we can get into how we pass along log files and stuff and how long we should retain those sorts of things to carry along through kitten run.

A

I think maybe another thing to consider in that context might be gifts. You find that a job is sailing for whatever reason and it's sort of long-running jobs like scalability, although I don't know, maybe like scalability, maybe taken care of, because they're already pinned to a specific project, but the sort of scenario where you have a job that runs and it fails, and you want to be able turns if we go in and debug.

A

Why is that happens some way to be able to mark that project or that cluster has something to stick around for a little bit anyway. I think we're out of time yep cool, but much free time does lots of pacification except replay for your time today and we'll see you all next week, happy Tuesday.