Kubernetes SIG K8s Infra, 28 Sep 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes WG K8s Infra 20180928

Description

k8s-infra-team meeting Sept 28, 2018

A

Okay, this is Friday. This is the meeting for their kids in trotting soon to be work group. Hopefully we talked a little bit before reviewing their action items. I forgot to push the recording button. So, let's start back up again so Aaron Tim, please yeah.

B

So I don't know just to kind of recap on the working group thing a little fully but brought it up to steering committee, it's keeping non-controversial by saying, whoever, whichever saying it seems like we have prior art for them. Opening that code. That's the sig that owns that code, but you can revisit as need be so a subdirectory of cakes that I over DNS seems reasonable.

B

If we need to read other stuff other places. That's that's fine, too. I am happy to work on their charter or work with somebody on a charter. I think the main thing is I want to get a majority representation rule and there try to keep things loose and informal and do just enough to make sure we have the working group pieces of structure like assume, license and recurring meetings on a calendar and blah blah blah label.

B

We can apply to things and yeah and maybe at that point we can revisit whether or not we want to reschedule the meeting for something that's a little more world friendly and maybe avoiding Friday yeah.

A

Definitely so any other takers, otherwise I can work with Aaron.

A

Okay, going once going twice: okay, it's you and me: okay, so I! Let you start and I'll follow along and contribute. This does a sound okay sounds.

B

A

I'm not familiar with all the processes within the steering committee. Don't.

B

Worry we're making it up as you go along. Okay,.

A

B

Of the meeting we're trying to discuss that process and we're gonna see if we can, alongside that okay sounds good, so.

A

Let's do that I'd.

B

Like to tag along for that ride, so I can learn the process as well sure.

A

Sounds good, hippie, hacker? Okay, then what was the next one? So working group? We already talked about that. So one idea that came up during the release cycle in 112 was we had a few things that needed a home where we could run those. For example, the publish bot is right now in in running inside Red, Hat somewhere, and then there was this idea for a cherry-pick bot that we've been debating on a PR whether it's gonna suck up tokens.

A

You know number of times we can use, and things like that. So does it make sense to create a utility cluster as a place where we can start experimenting with ACLs and running these bought some of these BOTS and trying to get a handle on how we would give permissions for people to try things out, pull logs and things like that. I think.

C

That's the next logical step here and I. It would be fine to let that proceed in parallel, I'm also happy to sort of dump state from the utility clusters that we have and the experiences we've had around running it. If somebody wants to take up that mantle I.

D

Think a big problem with this is that we one of the first things that we don't have is like a credential store right.

A

Right the reason for picking on this one was that we are not in anybody's path. So to say, we are not like this is like an independent effort just to stand up the infrastructure and try to see how things how we can make things work outside of the Google infrastructure, but yeah. What do you use for credential sharing right now, right.

E

C

E

C

D

C

Have our we have our internal tool for sharing stuff internally for utility cluster, actually clusters? Plural, that we have running right now. We upload the credential to the cluster and we use our back and whatever to make sure that appropriate people have access to the cluster and that's it.

A

So would simple private github repository work in this case for now or I.

C

Mean do we do we need a repository to check in credentials or is it sufficient to say you know, for each service account that we're creating we upload the credential to the cluster and leave it running in live State, and if we need, if there's a problem, we recreate the credential I I mean so there's.

D

Like you need like, like an email address and probably like two-factor typed in the account I, think those things may not be best tracked in a cait secret, like the actual token for the robot that works great, indicate secret, but all of the other things you need for the account and not quite so much.

C

Which accounts are we talking about here like well.

D

Like, for example, for Kate CI robot, there is the github account. There is a email address that it backs the github account all those things have to factor on them. Sure.

C

I was thinking about the Google cloud service, account credentials, but not thinking about github. Of course, yeah.

D

I think James was getting at like, for example, an account for a cherry-pick robot, and to do that I think we probably want to find some kind of like password sharing tool or something we.

C

Were involved in our cluster.

C

Like this is an opportunity off for us honestly to eat our dog food a little bit more and look at what people are doing. We have cloud key store, I haven't used it myself, but maybe that's an option. I.

B

Mean that was gonna, be my google hat on suggestion. My Google office ounds, like this, is what everybody uses Bob for yeah agree with Dennis person. This is an opportunity to experiment, not necessarily find the one true way, but so it's like the the cherry picking stuff does require a full-on prowl set up cherry picking is basically another crowd. Looking and Christophe he's not here has some specific concerns around the way the cherry picking bot works in Red Hat repo. While it may not work here in kubernetes, but.

A

I think focusing.

B

On publishing BOTS, like totally 100%, agree.

A

Right, the cherry pick what was more like the Festa bot, which is run outside of Prague right, and you know it works like a regular bot, not as a pro plugin. You know it subscribes for notifications, but it is not part of prowess that subscribes to them.

D

Through crowd, I'm, not sure if it works without proud, I see.

A

D

A

D

B

This is a different pot entirely, but I thought I saw Anna. She can open up recently asking for slash, cherry-pick food to be able to carry a concurrent rage, food and that's there is there's code being merged into prowl for that right. How is that code enabled upstream we.

A

Do not correct when the in the same discussion there was like a sub discussion: where should we run it as part of prowl or run it as a standalone thing? We can handle one thing without.

D

Going in a different sheet hell, it runs, is an external plugin but I believe those still depend on like they hook up to the standard component and they told them they exist in what their commands are and what kind of events they needed. That sort of thing so.

D

He's the cherry picker or is the publishing robot something we can move? Yes, Bob machine robot. We can move right now and we can we do that and can we start experimenting with like credential management and things I talked to them about this briefly, and they said they used like a CNC F last pass, but they had just sent a password once it wasn't really a it wasn't really in use. Okay, that sounds perfect. So, let's start with publish.

A

C

So I'm, all for starting to do this work scrolling back to the very top of this dock. I think we have a few things we need to do before. We do that. Okay, specifically, we need to put a little bit of thinking into how we're going to do. Spend reports like we will need to be able to document where we're spending all of the money and that needs somebody to sit down and think about.

C

I, don't actually know the billing subsystem that well to figure out what reports are easy to generate and how we want to isolate like we have a community's public project. Do we throw a gke cluster in there and use that or do we create a new project and create a cluster in that one I.

D

Would really highly encourage at least splitting up projects? It not only makes it easier to pin down billing. It also helps with just isolation, even beyond something like our Beck I.

C

I mean I have a sense that that's true. Somebody needs to at least like look at those options and write down a page about why we should do it. Like. Are you signing up for that? I I.

D

Can at least write a page on why using a project for all of the things has been awful for tests and fraud, and we don't.

C

I would that would be great.

C

And then, in so doing, we can also address the if we're going to use multiple projects, let's figure out what the scope of those is and what the naming convention should be. So we don't end up with a mishmash like we have with open source is able to help in this then.

D

We've been discussing some more about the projects. We should be able to do that I, guess technically, the quarter is also not over. Yet we still have an objective for this quarter. No worries.

C

And then the other thing that we need to think about is criteria rules for adding volunteers, like is anybody who shows up to this call going to get access to everything, or are we gonna get very limited access? We may want to create secondary Google Groups. That I mean we've started this already with DNS right. We need to create a bunch of individual groups for each individual topic and grant those groups their role access those sorts of things. So we need some criteria for like who's who gets credentials to help I.

D

Would also add to that that we've recently had some issues again and we've switched to having a group with right and a group with read. Okay, the group with the read is a superset and they can look at things and maybe help debug, but they can't actually write to the cluster. Is.

C

That specific to testing, or do you think that that actually is going to apply everywhere, I.

D

Think we might want it in more places because it's also a way to sort of like start onboarding people like let them take a look at it, but not necessarily trust them with actually poking the buttons. Yet right. Okay,.

C

So again, we need, if I need a volunteer, is write, a page about the existing set up and how we should replicate it. Like we've already created a DNS group, we could delete it and create a different one with the you know, read and write names or whatever. What I don't want to do is start ten efforts and then having to go back and retool ten different times. I.

D

Don't think the reading right is a big or as interesting of a deal as how we decide who's going to have right, I.

C

Agree with that, so the the biggest task I think in front of this group is actually making time to make progress on these things and like putting aside we're putting aside an hour a week to meet. But if, if we can't get traction on these other action items we'll be in the same spot a year from now right right,.

A

Yeah, we don't have to wait for the work group stuff to be done before we can make these kinds of decisions right. Agree.

C

With that, like I, don't I don't mean to volunteer people but I. You know I. Think then, for signing up for writing a page on the project. Stuff I need somebody to sit and write. How you know make a proposal for how do we add volunteers like what are the rules yeah.

A

C

Every time man well.

B

I feel, like that's almost part of the Charter. In some sense, one of the options I was was the product security team is another option for another model for this group, because they have some level of betting in requirements right before people are allowed to I would.

D

Also, have we might actually want to include some of those same requirements like something that's come up in the past, for us is that there is some amount of CI for the security folks, and we want to keep that private for obvious reasons, but someone still has to run it, so they effectively have like the same level of access to. What's going on with that team, yeah and.

B

So I have experience running this sort of commingled requirement thing through in the context of the kit of admin team. So I feel like there's some level of overlap with the Charter, but I don't want us to get hung up on nailing down all the details of Group Policy and whatnot I'm interested more, and we should have some sort of likely pot processed event, responsible people right.

C

B

I think what I've been hearing then volunteer for is like we have some experience doing things one way in intestine truck in the past, so we can recommend why we wouldn't suggest doing it that way. But I don't know if we're signing up for a full detail-oriented proposal on groups and credentials, but like we can throw.

C

Out a strum in and have.

B

This group iterate on it I, don't know I'm.

C

Happy to iterate, like honestly, we just need a starting point. Well, I, don't know, what's been done in detail with sig West or with security groups. So if somebody knows those and tie them together, then that's that's fantastic, but I don't want to do is get stuck where we say you know: hey we're gonna, create this utility cluster, but there's only three people that I trust and the only reason I trust them is because I've known them for four years through the project and new people are showing up that. You know in particularly you know.

C

Tim's you introduced a couple of new people this morning might be worthwhile for them to say hello, introduce themselves, tell us sort of who they work for. Why are they interested in this sort of work? What they hope to achieve those sorts of things.

C

No and I say only that's. The point is like I would like this to be a recognised process where it's not a bunch of people who know each other, inviting a bunch of people that they know I listed.

A

Yeah, the main reason for inviting Manjunath was his work around the industry and multiple architectures manifest stuff and he's been working on images for for a while. So it would be good to have as input on how we can do the import process and things like that right.

A

So that's where I wanted to bring him into the discussion here in in terms of we do so right now, if you look at our Google containers, it has everything in there. How do we make sure and the other flip side to that is the cluster API flocks and the cloud provider? Folks don't have a home to push their images to so I want to get to a point where we have some kind of rules around okay. This is the registry that can be used for XYZ purpose. These are the images this.

A

These are the criteria for adding the images, and these are the people who could be pushing images to the specific so.

C

I would love to see somebody again driving a sort of a specification for how we want that to be I, know, you've looked at the promoter proposal from from our side and I think it's met with overall general approval. That's another big effort for us to start the migration of that. If we want to try to impose some new structure as we sort of make the transition I don't know if we should do it at the same time right, we really were thuggin all things right, all right.

A

And so you touched on one other important, but I want to spin off multiple efforts at the same time and get people going and not have to like okay, you're gonna, finish DNS and then we're gonna start something else. You know.

C

So I agree with.

A

That I want to.

C

Make sure we establish the norms and the rules and conventions before we spin up too many parallel there's the second part, which is like spinning up a new utility cluster and learning all that stuff, doesn't require any access to the existing clusters, correct, which means anybody here could start doing that and come up with the rules for which projects we put them in and how are we gonna set up our back and how we're going to do namespaces and etc, etc, and how we're gonna do billing reports to start the migration of the GCR stuff actually does require a level of access that only a handful of people have I, don't even have access to it anymore.

C

So we'll have to engage that carefully. I'm happy to hear about what structure people would like to see in there I'm all for structure. That said, we can't actually abandon the previous Convention, which was throw everything in the route all right, so we'll have to migrate over everything that we currently have and start to impose new structure. Do we do that at the same time? Do we do that as a second step? Yeah? That's.

B

A very good question, so, from my perspective, this is also why I think DNS is one of the more important things to get done first, so that migrating different services to different clusters, or whatever is a lot easier once it's really easy to flip DNS from one place to another. If we find out you put a bunch of things in one cluster that actually belongs in different clusters, or vice versa. Right, that's a great point.

A

I'm under that, can you speak a little bit about the work you're doing correctly.

F

So myself, so I, basically work for IBM India, so basically in covered, is community. What I spent time mostly is so reading all the test images and then from ages, multi, architectural, supporting all different platforms like I'm 66 to morality and history 90. So that's that's what the target we did and, in fact well we released all the K tests. You may just its supporting all these architectures that manifest so yeah, so the the future goal. What I'm trying to look at it is like so love tests in quality is at least in it.

F

We know or whatever right so this they are using their own repositories and then we don't really have any sort of a control where those images are and then you don't even know the source of those images. If something goes wrong, it's really difficult to fix. Yes, so what I'm trying to look at it is like knowing any any of this. Just what we have in our repository should have proper images supporting all the architectures so and then we should have at least the I mean right now. We should know how to fix.

F

If there are any issues or maintain those images properly and then and then control them.

F

That's what I'm thinking of and then in d1 what I was looking at recently is all these incredible posits what we have. They are beginning their work that passages right. Even that is not really good great way of doing it. So I think if you have some sort of major repository is created in this year with certain needs, I mean sudden naming convention, and then you know have some of a proper back model to know, allow them to push or modify their own images. To be great right, so can be a lot.

F

Not a lot of people have any sort of access to push those images of managed. So all the time we are going to of people and then.

F

He's helping us a lot but I can like now. We should have a streamlined process there and.

C

Are you volunteering or tell me if I'm overstepping? Are you volunteering to write a proposal for what you think the structure of the target repository should be yeah yeah, idealized, state, yeah?

C

Okay, because one of the things that we don't use we don't take advantage of is the fact that the repository within GCR can actually have an arbitrary number of directories right. We don't need to have kate's that IRG Kate's dodgy CR that IO /foo.

C

You can have slash foo bar Baz quacks, it's a big long path and it won't work just fine. We can establish a structure for those paths that represent our project structure. Better is sorry, go ahead.

F

G

The one issue with using the folders I don't think you can control access at a folder level. I think the access is only for the entire DC art project. So if you.

C

Want true, but if we use the motor, if we use the promoter, then it doesn't really matter because humans don't have access and we can through the promoter manifest, we can actually include owners metadata, perhaps and say like hey only you know, XT is the owner for the test subpar, but not for the DNS sub repository. So if you try to push something to DNS somebody, who's reviewing that can say hey, why do you have access to this.

G

Also add like a year or so ago and continuing internally at Google, we've done some auditing of the Google Tanners group or the Google containers at UCR repo, just because there's so many images in there and trying to figure out like where does some of these come from? You know what are their like. This is whatever and I think there's no reason you can't share that so I'll see if we can. You know at least get a copy of that shared, so that can least be something to use.

G

I know like I, you know, pulled I tried to pull out all the images so, like you know, there's old versions, all the e2b images in Tanner's, but like the ones we currently use or incriminate easy to be test images which is lighter access, and you know, sort of I've been at least trying to pull things out of Google and Tanner's that don't belong there, but yeah I. Think I think this would be great, but.

C

I would like to go I think I'd like to go the other direction and move everything into the kate's. That GCI are Kate's GCR, that I Oh umbrella, because we've set up the Auto regionalization of it, and so that sort of gonna be optimal for performance and cost, and we can just use the structure and the promoter to govern access to it. Yeah.

G

And I think I think yeah if we have proper access that way, that would be great and as long as we also make sure that we, like I, think you know, have a policy of somehow documenting where these images come from or somehow automation to tell that, because that's the other problem with a lot of stuff in goal. Containers right now is yeah. Is that her, like three years old and I, have no idea who built them who uses so.

C

We won't be able to retrofit all that, but we'll be able to say at least with the promoter we'll have a track record of saying who promoted what image and hopefully the PRS. We can establish the structure around the PR is there to say: where did this come from? What is the expected, checksum, etc right.

F

I think like supports a lot of this metadata, so we can feed those meta data looking at it. So probably if you have certain proper metadata into images, so that can be done like and give it to commit or any of those and then author and many of those levels are supported. So if you deal with proper metadata into it, so I think we okay.

C

Awesome, thank you for volunteering. Hope I did wasted on you.

A

You should work with the person who is doing the promoter staff who's that again, I forgot, Javier's.

G

Just thing yeah: are you in the kids testing for or not kissing, for the kids, infra slack yeah.

H

M'kay um ATG yeah, okay,.

G

I'll sign that you don't know you.

C

Dogen bug do I think so many times on the topic of getting DNS done as a gateway to other things. There are still, some AI is needed to get DNS done. I still need some help chasing down owners for every name.

C

I've spent some time this morning and this week, retooling ma I personal work in progress, repo to be ready to move under Kate's that IO, hopefully I'll, send a PR out this morning, but there's a bunch of to dues and those two dues include finding owners for all of the all of the URL on the URLs, all of the names that we have listed and investigating whether we need to actually port all of them forward and, ideally writing a script that would do the dry run to a test domain and then, if that passes, push it to the real domain, I've included instructions in the readme now on how to run octo, DNS sort of as you with your own credentials.

C

So anybody who's in the Kate's DNS Google group should be able to run it themselves. Now against the config. That's checked in there, I tested it as my own Gmail I added myself and removed myself, and that should work fine. It would be great if somebody would confirm that until we have a cluster up, that we can run the automation and I. Think it's acceptable to just say here are the instructions sort of like we do today with the website. Jeff like people push a PR, and then we manually update it.

C

We could you limp along with that for a while I think the instructions work, but it would be nice the same way for the website. We have a test test at SH, which actually runs through all the URLs and make sure that they still work. I would really like a like first do a dry run then push to a canary run a test against all of the name resolution against that canary and then push to reality and I. It's on my plate.

C

I'll do it eventually, but if we're blocking behind DNS I could use help I think you know walleteer hippie, hacker I. You know where the repo is github duck work in progress. It's linked in the dock somewhere yeah.

A

It's dead in the dock under Friday September 2014, you good to keep the ekor okay. Thank you.

C

And I'll push a I'll push a new push this morning with an updated readme and I'll change it from text to a dot MD. With this push, I want to then make a PR against Kate's that I owe to introduce the new subdirectory.

A

Awesome and then that brings us to not there hi another. Do you want to introduce yourself? Yes,.

E

Yeah I work for the Scout factory, with a a SS partner in the UK and currently working with on cluster API work and I. Think we'll be looking for a home for the artifacts that we produce also from asset class. The life cycle perspective. We, for you want to improve way that we're doing rpm in depth generation, particularly separating out the repositories where the least candidates going Rachel. That people who do automatic updates on systems can reliably trust that it's not going to break near clusters.

E

Essentially, I'm I'm, going to I have started a proposal that might sort out for the repository structure and then also using my best practice specs for rpm and the equivalent for Debian. We have it ready for next week. To be honest, so.

A

Now, did you go through the specs that are in Kay release and the ones in the kk repository? I need. I'm looked.

E

At them, I mean last time, I looked at the most about six months ago, so I need to revisit them. Yeah.

A

I think one of the first items that we've been talking about is like try to figure out a way to consolidate those two. There are spec files in both and the release infrastructure uses the ones and K release to actually make the release. But then people are always. You know forgetting to sync between the two.

E

Yeah I did that myself, but, like the last time, I tried to use it. I I checked up on that as well. So yeah.

A

Yeah new people right so that would be on on the side of the specs right, but then on the side of where we need to upload them or the directory structure that we need to upload it to. We had one problem during this relay cycle where we pushed what was it RC one and that actually went to the stable, a directory yeah so and we broke a bunch of people for a day. So do you have like best practices of how the X table experimental nightly? That kind of repository structure should be set up? Yeah.

E

I'm gonna put that in and I think. Probably we probably want the perversion we place, so people can do their Yama takes doing a press release and get upgraded. But then I have something as part of their cube: idml cluster API lifecycle to change inversions and do.

A

You work with both young and death, or just one of the two, because I am like lost and young. We.

E

Are so we're technically I sent our shop, but some people in our team I believe my even being Debian maintenance with some sort say we should. We should be able to cover both.

A

Interesting awesome, thank you so who would be able to give some information Tim about how we have our battery structure for the images? I know that I can build the images by myself but I, don't know how it gets pushed out or where it gets pushed out to or how the directory structure looks.

C

This is for a for Communities release in particular. Yes,.

A

C

I have not touched the release process in like 8 releases. There really team process has been great for me, so I would actually defer to somebody on cig release. This.

B

Is usually where I wouldn't Caleb miles name but I, don't he's got bandwidth for this or not Jeff? Do you know of anybody who would be a more suitable candidate.

G

That's probably the best, especially for Deb's and rpm stuff I feel, like Caleb, probably has the most context here, but yeah. He also yeah yeah yeah.

B

Your you might get punted to somebody else, but Caleb would probably be the person to start with. Okay.

A

So, let's do this, then, for this one: let's do the write-up that you are doing now there and then we'll try to set up some time with Caleb and get his thoughts, and you know trade-offs that he has seen before and go from there. Yeah sounds good. Okay,.

G

Think the other person who is sort of owned, some of the rpm and Deb stuff in the distant past, was Mike Denise. Although I think he set up stuff and maintained it for a while and then has not worked on in a long time. And so yes.

C

Interested forever I think.

G

That's one of the reasons why ik, for example, like all of the Deb's you know, Frew bun, we haven't updated to Bionic because basically nobody has touched it or maintained it in years. So exactly he.

C

G

Know he might have some context on how things were originally set up, maybe if he remembers but I, don't he's not a cart maintainer by any means? Okay, let's.

A

Start with Caleb then, and then we'll I'm sure, he'll delegate after giving us some initial idea. What else do we have anybody wanted to talk about anything else.

C

Oh yeah me I'd like to I'd like to regroup, ideally when we regroup in two weeks to have a real DNS repo up and be having a conversation about how and when to actually make the flip okay.

A

And everybody has something to do between now and then right. We don't have to have one more meeting in between now. Do you have something? But you not? You, have something: I have some action items and he PP hacker has one at least okay. I think we are all set here. Unless anybody else has anything else, just.

E

Have one thing so: we've been talking about Google related stuff. We will also need a EWS, it's specially for sick plus y, so quick cycle, because we can't do a Kelowna AWS provided stuff, it's not about that container images, right, yeah and even CI and all that stuff. Is this the way the venue with this donor? Oh, yes,.

B

But no I would plant on that for now and come talk to sick testing. If you have specific concerns, they're loosely speaking, we have AWS credentials that are given to us by CNCs so like. If you want to file, if you want to create a new job that does anything in AWS, you can totally just file PR. That does that today, I've had people from the US ask if that's how they can create more jobs, and so did where it's yeah.

A

Okay: okay, thanks. Everyone thanks. Everyone like.

B

Be Friday / Saturday, for you future dwellers.

G

Beautiful day billion I.