Kubernetes SIG Cluster Lifecycle, 7 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Cluster Addons meeting: 2020-07-07

Description

Meeting Notes: https://docs.google.com/document/d/10_tl_SXcFGb-2109QpcFVrdrfnVEuQ05MBrXtasB0vk/edit#heading=h.ddrn7k8vehon

A

Hey all if you're joining us from YouTube in the future, this is the cluster ions call and it is July 7th which can chat about some things. Gonna be smaller, meaning I think, because we are just coming off of the holiday weekend and Happy Holidays to all of you. So we just kind of came up with two impromptu agenda items. One is to just quickly discuss some merged work. Justin and I have been working closely with some touchy who's.

A

Our google Summer of Code participant- and she has some cool things, we'll talk about that, and then we figured that we could use some of the time today to do a little bit of overview over the cops PR. Maybe get it to a little bit of code, reviewer discussion thanks for joining. Does anybody else have any other agenda items feel free to pop them on?

A

As always, it's the first recording you watch. We have our agenda posted in Google Doc that is accessible via the Google Group go ahead and join that if you'd, let sweet so the main one that I wanted to discuss.

A

Just think that this is a pretty cool I've got a link to it.

A

So pull 68 for cluster add on some Sochi open this patch just and reviewed it go ahead and link this here like to take a look. We ran into with this augustus problem on this call before, which is that, when you're building operators that are intending to take some ownership over other objects inside of kubernetes, that the permissions can be wide and sweeping one of the most naive ways that you can deal with that problem is you can give the operator cluster edge triggered permissions?

A

But if you want to do more fine-grained access control, then you can generate our back rolls specific to that operators use case and the manifests that it needs to apply, and this is finicky and specific to each package that you want to have a channel for for installing the manifests.

A

So we looked around and we thought that it would be a solved problem that you would be able to figure out what the minimum our back is for a particular set of objects, and there was no easy way to do that. So some touchy built this are back generator in yeah. You can go ahead and look at this pull request if you're curious, it's all sharing it's this patch right here, which is pull request 68.

A

Sometimes we built a tool that is, we merged it to tools our Beck gem and it's just a quick little piece of code that loops through manifests and generates the proper rules necessary to work with those, and there are some things here like if you have a cluster roll inside of the manifests that you need to generate rules that are superset of those things Justin I can't remember, is that something that's handled by this tool already or did we merge it before I.

B

Think it might have made it in I camera with it. Essentially, this I think the initial. The initial genesis of this was just that when you're creating an add-on, if you want to add those permissions, you have to sort of do it manually and figure them at Miami. So it's this tool started off as here's a manifest. What are the additional permissions I need to apply to that manifest, like sometimes she basically discovered through trying to build one that that was like one of the points of friction of building an add-on operator and I.

B

Think what's interesting is then shed into fide that you know the requirement for a role. You know what you wanna create a role. You need the superset of that and then I think what it's very interesting is we took it from there to this idea of I. Guess it's already. If I like the idea of mmm splitting out these permissions into separate files and applying them judiciously and perhaps in different phases, I can't remem if I recall correctly.

B

Auerbach aggregation is in this PR, perhaps by the flag, but we decided to merge it as it was because we could see that there was gonna, be a stream of these ideas coming and we didn't want to like grow this PR infinitely. So we sort of cut it and merged it. As at a given point and I, imagine there'll be more PRS in this stream yeah building this tool, one.

A

Other comments on here is: we did talk about how this our back generation being able to reduce the minimum permissions necessary to work with a particular set of manifests or set of objects. It's not just useful for add-on operators, but potentially for all sorts of automation, or least privileged access.

A

So if this is something you're struggling with- or you want to tell people about this, here's an are back generator that can produce minimum rules pretty cool stuff.

A

The next bit that I just wanted to give just a little bit of floor space to talk about. Oh.

B

Yeah as I say like it's something, she also did another PR, which was getting removing our requirement for coop cuddle to be in the context.

C

B

Ya know one hopeful that actually cool.

A

We merged that as well. Yes, it might have been in an.

B

A

Building this to the agenda.

B

Making it not required anymore, there's an alternative, I think I was still there if you want to. If you want, according to it better.

B

A

Native coops ETL executives and.

B

It's supposed to use these the same code, as is called by the coop cuddle CLI, when you call apply so by default, client side apply and it just removes the requirement to bundle group cuddle into that container image and it may the image I think she I think she said it's about half the size, which is great. Oh.

A

Wow, it reduces the image by half, yes, I'd, be I, guess an operator images.

A

Binary- and this is something that we mentioned- could be a great example of vendor. Encode cuttle apply logic into another code base as an alternative for something like server-side apply, which is currently still behaving differently from coop cuddle apply. So this does the exact same behavior more or less. We did find one.

B

A

Case, which is that this currently does not deconstruct a kind list which is pretty easy Tareq. We ran into that with we've, we've not seen I.

B

I'm not wondering how it's different, given we're, calling it to the same code, I'm wondering what we missed, but yes,.

A

Yeah, it's like there's, probably some generators stuff before you.

A

Yeah, that's because we didn't grab all of the flag logic right so I! Imagine that's part of it.

A

A

A

Said I already said that great read, for example, for the nine coop cuddle apply.

A

A

Yeah cool stuff, very cool stuff I should probably put the agenda in the shared screen as well.

A

Yeah those were kind of the big items right Justin. As far as like things that we got in I believe.

B

So I see sometimes she has not joined up. Sometimes she was there anything else. You wanted to call attention to I think those were the two biggest things article.

C

Yeah I think you guys.

A

Yeah I moving on from that just wanted to talk a little bit about this first operator integration into cops Justin. You said that since you've opened this, you haven't come back to it. That's probably because I have not oh yeah, you got a review already from some cops folks, yeah.

B

We got, we got a review, it was actually interesting because um I think the it's around the the different behavioral models of operators versus the way K Ops has traditionally done it, where the manifests are pre-rendered and by the client-side tool, and so people concerned around the idea that the server now guess of additional permissions. So for some people, that's that's not what they desire. So some people want the flexibility, the dynamic like live, updating some people want or more on their side of control and like very methodical, they.

A

Don't want to change their operational model of like wait. We already have cops as part of our cluster management pipeline. We only want you know that glass to be broken open from a security perspective during.

B

That time, exactly right and I and I think like some of the like, sometimes she's done great work in the first month already and like, but in particular like the the are back work like that. That I think addresses some of these concerns.

B

It opens up the idea that we can deploy operators in different configurations, including, like moving the, are back permissions into that pre phase so into the so we can sort of move move. The the biggest concern is, of course, like this. The super powerful or one of the concerns is the super powerful operator permissions.

B

So even people that want control want, like the supervision and the self diagnosis, and you know self management they just don't sorry want to give all those are by Commission's and buy more tightly or by generating the arbor Commission's were able to then split them, and we could then run the our back or pre configure the are back permissions in the first phase and then the operator itself might not need, as many are by permissions so like I, need to update my PR or a follow on PR, I guess to one way or the other, to try to sort of expose that functionality, but I think it's gonna be I.

B

Think it's gonna be very powerful.

A

Fascinating so yeah we could. We can I we're actually talking about envisioning a third model of operations right over like a modified version of the operator where it receives temporal privilege. So current look like basically only operators have superuser or their add-on.

A

We can envision phase.

A

Temporal are back managed by tool.

A

You reduce operator privileges.

A

After apply and then there's also the one-shot approach which is or like the early exit operator,.

B

Running a job I mean I think there are a lot of these configurations that are that are possible and I think I think there's a easy win, which is the thing which is applying or which is creating the operator.

B

It can create the cluster role on the cluster role binding, which means that the operator itself does not need those permissions.

B

So that's a that's a it's a it's a! This is a continuum, and this is a step. It's a small step. It's it's not as far as one shot, but it does remove. It does remove a lot of the permissions now all the permissions, but it removes a lot of the permissions that the operator would otherwise need. So it's a it's a good first step along that continuum that might make some people happy and also show that we can do this sort of thing.

A

Great, that's a I'm glad that we've been able to encode some of that discussion and I'll play some folks take interest in my area and add to it as we worked on these technologies yeah. This.

D

Is a very ever thought of like a lazy evaluation for our back, where there is something proxying, the API server, taking requests and kind of generating a list of filled requests because the our back has not available into something that is like approvable later on.

D

This is just another really shot in the dark, yet.

B

We were trying to remember like. Was it? Wouldn't you that someone from read that I think it was? You showed like some Doc's around this and it looked really good, but we couldn't we couldn't find it again. Is that I.

D

Think it's more along the lines of like the Arabic generation, mixed with something extraneous to the operator or outside the operator that is applying something. This is a little. This is different in that that was a trump on operator installation, and this is more like you- don't explicitly have to write down what you need in your specification right, because it's going by API requests now this approach to describing now is kind of like we can't know if an operator is gonna explode and back off, so it might not be great.

B

We can certainly add that, as as an additional additional one I think you know like I think where it came from. Was this like when you're building the operator you reply, you know we have to play a manifest and we don't even need like.

B

We know that meant that manifest is generally relatively static and we can generally just scan it and produce the our vac that is required, including in diverse scenarios like where objects exist already, which is nice, I, think so I think I, don't think we're precluding some form of dynamic discovery through the audit log or a like a proxy or a kind or something but I. Think there's like there's the easier or more.

B

This static analysis has as a as a place, I think as well. Yes,.

A

I think the it's not fundamentally different from what Nick is suggesting. He was like just pointing out that, like you, could use an API server proxy to catch these things at runtime and return errors that interface, ultimately, that the operator would be using an urbanized client, so that proxy would need to talk like a kubernetes api server, which means that the operators errors feel kind of opaque.

A

But what Justin's point I think that there's like a lot of opportunity here for the operator to have something like a preflight check, which we already have in the declarative pattern.

A

That simply can no, you know what it's our Beck rules are and report that it's not compliant or that's not going to be able to apply these things or it could even be at runtime right, like you, can determine at runtime. Hey I didn't have the permission to apply these rules. I need this list of things and an example of where you could put that information might be on the custom resource that you're trying to reconcile or in a config map for the operator that could be applied by a cluster administrator.

A

Later you convert it to a role.

A

Really really interesting idea that I think would make the UX of like, during these operators, much easier and potentially even allow for some machine run mechanisms right to automate. The management of these are back rules like if you were I really like a Nick. You were you were talking about how there was a little bit of inspiration from like the Android user experience of permissions right for.

D

The thing that Justin wrote yes.

A

Yeah, what is that thing called? We didn't? We could not remember I'll.

D

Try to I'll go get the the docking posted here later, but yeah.

A

We tried sifting through the docs for things like permissions and capabilities and Android, and we could not find it for like through our reading notes and some hilarious, Google searches. So we're really I'll put it.

D

There it'd be great to have a link here. Yeah. We were hanging it up, something called the install API which in itself is like in flux right now, because we're going through like a lot of just questioning, designs and design turn in early on no one's you so we're trying to make that process less effective, but I'll find it and I will post it here, but with respect to lazily evaluating this stuff, what you and Justin said, or totally totally fair, yeah I just wanted to throw the idea there.

D

Do you know if anybody said it before no.

A

I think that this is super powerful right because in the fork workflow right, let's say a user wants to add a Prometheus service operator or sorry a Prometheus service monitor. You know like an object that they want to like add it to the manifest list or something you could argue that, maybe that's not the right thing to do, but a user wants to support their operator manifests and then point the operator to a new bundle. Then now it's not going to have our back right.

A

So if you have some sort of lazy, workflow right where you don't just get errors right, there's actually an action that can be taken by the administrator. Beyond that point, to you know just copy and paste this thing, which is basically the kubernetes animal, are all you know, equivalent of pressing okay on a permission dialog on the phone or something, and that would be really powerful kind of workflow.

D

Yeah, this is yeah up the alley of someone's or beckon reader and very I think was the align to getting a diff of the our Beck that is available, verse what you require and then generating new roles and role endings to fill that gap.

A

And this idea of composing role bindings together, is not something that we see a lot with the third-party style of add-on. I. Think a lot of people are just putting all of their things into one big cluster role binding or a cluster role and cluster role binding.

A

So we we did comment that we'd like to see a wider use of multiple role bindings for particular service account to get their namespace and cluster level permissions. Accurate.

A

Great thanks for contributing that, if yeah, if you end up stumbling on the install API Docs, please do post them here appreciate. It will do great, well great, quick, meaning Jane awesome to hear from everyone, assuming that we have nothing else, that we want to discuss. It's cool to call this good and we'll reconvene on some of these topics in two weeks or in the slack channel. In the meantime, thanks everyone.

D