Kubernetes API Machinery Special Interest Group, 17 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG API Machinery 20200617

Description

SIG API Machinery meeting, discussing a separate healthz/readyz endpoint for load balancers. Also discussed CRD sub resource configuration. Discussion primarily focused on projection webhook.

A

Hello, everybody welcome to the June 17th 2020 edition of API machinery. Sig meeting fatty can't join us today so off to put up with me, there's three items on our agenda and I think at least the second one could take a fair amount of time. So, let's get started, Mike I see you on the call. Do you want to introduce the topic.

B

Yeah, so this was opened by somebody else, but this is something I wanted to bring up for discussion here. That I think we'd like to see basically it'd be great. If there was a listener on the API server that just served health seek that didn't have offended on CI, the main thinking just being. If you're operating a you know, a load, balancer cloud load balancer in front of API server, and you don't you don't want to rely on like you, don't want what you wanted me to get the fact.

B

If, because, if someone removes the like, the public in code view role, role binding, that that helps the you know can always be hit by the cloud load, balancer or some other process like on on API server that needs to validate the API servers health, it's more just like we. Yes, we could, but you know you could put a proxy process that always has authorization and authentication to call health C, but that requires running one more process and everyone's gonna. Do it a different way? So this is.

B

This is yeah, that's sort of something what I wanted to bring it. Yeah.

A

Some some background context is: we have this insecure port that serves out HTTP and we are in the process or have gotten rid I think we're getting rid of it very soon if it has, if it's still in I still.

C

I tried hack, the lock cluster was in yeah.

A

D

A

A

So after that goes away, you'll need to go through the HTTP port, so the request is a separate port specifically and only for serving health information.

A

Sort of argument apply to metrics.

D

So people have tried I mean the idea. It's questionable whether someone who expressed an intent to enforce security policy for their endpoints would appreciate us providing a way to bypass that there are mechanisms today where it is possible. If you need HTTP to just create a process that does serve it on HTTP and always be sure that you have access the complication in having an additional port is real.

D

We've had problems in the past with our HTTP port, not matching what the HTTP one does. So it's not a zero cost thing for us to bear. Yeah.

A

I, also wonder: are there load balancers that are willing to get the health information from a different port from the one that they're distributing to? There are actually throws.

D

Okay, yeah because could be built-in right. Yes,.

A

Okay, good yeah I know there's at least one I think I moved to the AWS one there's at least one that and a TCP request and then hangs up after it gets a response and that causes committees to print a log warning and people complain about that.

B

Yeah, that's that's the older little bouncer. If you just do TCP health check, but you can configure it to. You know alternate reports if you want to make it just GC GTL about like health check, you can, if you want to make it like an HTTP health check, specify response codes that.

A

Yeah I I mean I can see why you'd want to do this. It does seem like it makes it simpler. On the other hand, I can also see the other approach, which is like okay, there's some like. If we're gonna operate a cluster for somebody, there's there's some stuff, we need to know like we need to be able to skim the metrics. We need to be able to check for health right. We need to I, don't know.

A

Maybe we need a management web poker or something like, but there's some things that we need to do, and so you're in a you're in a situation where, like you're promising something about the cluster, but the the cluster admin or the user is also got to make some return. Promises like if you're going to work together so I.

A

If from that from that perspective, if somebody does delete the public info, viewer rule or whatever like they kind of broke something about their cluster, and maybe they want to know that I, don't know that that's that's kind of my feeling. I could honestly go either way.

A

E

D

Would rather not take on maintenance an additional alternate pain, especially when there are alternatives.

A

Yeah, you know I, wonder if I wonder if we could make it make the user experience better by providing another name for the public, Reese or info viewer role like what, if we called it, a health health checker or like load, balancer mastered the control, plane, load, balancer provider, or something like that.

A

It seems reason I, think people I, think people like I would have this instinct to people look at their cluster and they see public info view or like I, don't want any public in the thing viewing info about. My cluster I'm gonna get rid of that thing, but if you look at it- and it says like hey, this thing is making your load balancer work like people are like okay. Well, maybe I need that I.

B

Guess, in a managed environment that you want to give someone the ability to not make their manage load, balancer work.

A

Yeah I mean I, mean honestly.

A

We have not this exact problem and in my cloud environment, but very very similar ones and like yeah people can delete the robot account or whatever and it's it's it's annoying and we've salutely need things to add things to make it better. So.

A

Yeah I mean I I. Do I think it's kind of good for people to understand what parts of the system the the manage the provider is managing for them, and the fact that there's like, like a security account, it's really kind of a it's. Almost a contract between the like the consumer of the service and the provider of the service right like like you're, saying I, need this I need this account to do the thing that I'm doing for you and the user. It gets to understand.

A

Like oh, hey, you've got this account and you can do the things that this account says you can do and you're not going to do the other things that the account says you can't do so from that perspective. I think it's kind of useful to have that that that boundary at least like just helping customers understand what parts of the system you're doing for them and what parts you're not I I can see people wanting to make different choices, though.

B

Yeah I do agree that, like you know a different, you know role or yeah like a different role. Role. Binding name would help like yeah the customer who sees public info fewer. It's like I, don't want any public info of use by anyone right like that that the person, yet the naming, helps with the perception, but is it I guess like the difference here with like a I guess with the proposal of an alternate port would be I'm? Not you know, I is the provider I'm not gonna make that port public at all?

B

It's not about even it being public. It's about like just operating like the server yeah.

C

B

Get like the contract like and I. Do you think like that? On the like, the you know, the robot account contract side of things like that totally makes sense if I want to prevent.

B

You know we're talking about access controls to data in my cluster, but but I guess on a more like if you're talking about fundamentally just operating the things so that it anyone can access it, what that's the provider or the customer like that, that's kind of where I'm thinking like to me, the motivations just it like an operator of the cluster, makes sense to have a separate port, but.

A

B

A

Using the are using the insecure part right now for this, no.

B

Act, we are using the secure port. Oh I, see okay, but yeah I mean people. You can lock yourself out of the box, which requires like operator intervention to to go unlock the box yeah.

A

That's yeah that's unfortunate. We have taken some rather extreme measures to to keep people from locking themselves out with web hooks and and yeah web hooks, especially.

A

Yeah I don't know, does that? Does that movie at all David.

F

So can I ask you a question relate to just I know we are talking about porch and I will admit. I have not thought this through exactly, but would a flag that makes said so that helps is available no matter what like, even it's like. You were always somehow authenticated. If you try to hit slash, helps and you're also out authorized sure, okay, I.

D

Guess I like that, even less because expectation I I do appreciate, like as a user I'd be pissed. If.

D

The solution was hard like if it were actually hard to go and say, like hey use this cute config and on you know this port exposed an HTTP server reflecting red, easy I. I would feel a lot differently right like if that work hard to do.

D

But it's it's not and it keeps everything completely separate. I.

A

Don't know if I understand what you're saying you're saying you run out. You put this into the practice of like some sort of proxy, but.

D

We just run a separate server right, like in order to support that GCP load balancer. We just run a separate server that can connect to read easy and it exposes ready to be only oh. How does that? How does that server have permission to talk to read easy? Well, we support anonymous off. If you choose not to support anonymous off, you could pass acute config right.

D

A

A

D

A

Account which has that towel yeah, you still have to give it an account. Yeah yeah.

D

Yeah, it's the same.

F

D

The power to scrape metrics and to and well to do, metrics for us right, so it's not harder to write and keep completely separate. So the.

A

User could still, the user could still like delete the robot account or or block it or stop remove the rule. Okay, it's a permission right, yeah good, so that doesn't actually solve the problem here. So.

F

I mean also, if up sorry, I was just gonna say in the mass provider case, like sort of the way. I would expect someone to do this and it's a lot of work but or whatever like in. If we talk about ETS specifically, is they would not use like a service account? They were using AWS identity that is backed by their own Authenticator, that you can turn off and they were also authorizing using their own web host. That you also can turn off and then there's no way to lock them out.

F

They would always have access, because their authorizer would always say they have access and their Authenticator will always say that this proxy is authenticated. Sorry, not the proxy, but this um this load balancer. Okay, it's just a lot of work in that sense, but that one is fully guaranteed unbreakable, because it's all just CLI flag based there's nothing in the managed provider case that you can do to break that.

F

A

F

That seems like a really just.

A

That's nice, like a better solution. Sorry go ahead. Yeah.

G

I was just think that managing another server for like health is II would also become an overhead to operate and, like manage I mean a long. It's.

D

Not realistically a significant one right like it's a server with nearly static config, we do this for every open ship cluster in existence. It's not on their Hardware.

G

I'm not disagreeing with that I mean like you, you're already managing like a bunch of thing, then one more needs to be managed, whether you just editing a golang or like if you were just written, the golang, updating, sdk or like anything, it's it's. It's just an overhead from my understanding.

D

For the reasons ago, I asked remove. This is because the HTTP port has had distinct bugs, and it's had CVE had CDs that are.

D

Worse in their grading because of its present yeah.

A

It's a pretty dangerous thing: your your one exploit away from from having a big problem, but actually I want to go back to Mo's proposed solution which is just use the use, the authorizer and authentication Web books to make sure that you're always authenticated and authorized that actually doesn't sound too bad to me, and uh that gives a property which I like, which is that you, you know you're going through the whole stack and nothing is broken.

A

Alright, like you could imagine like if people are relying on some part of the stack and the health checking code, doesn't exercise that, then you could think it's healthy when it's actually not.

D

Yes, it also guarantees you're able to fix your system just good.

G

A

Question I had.

G

On like a different port, when we are talking like just for the healthiest, so how would I mean like I, don't understand fully and that's why I'm asking like how would like, so today, the ok status comes from a different components or different controllers of like API server, as well as like at CD. So in that case, I was saying that we will basically implement all of that.

G

If it is a different code to communicate, how would the access you allow from two different port to the other components whether you're like it depends on like how configuration and settings are, but let me allow from basically inbound from only from this port from this that, like my inbound rules, so we will have to basically apply different, of course, always I.

B

Think if you have to the provider to determine the access control to that, whatever is talking of indicated, authorized LZ right, that's kind of proposal, that's kind of yeah. What we're talking about now, it's just saying: yeah, whoever controls the API server, also control access to this proxy health seat. Checker.

E

E

B

I mean I, think I guess.

A

I would for.

B

A

B

I think so I mean I think like not having implemented an authorizing webhook today and.

B

Hat like I guess just I'm, just thinking out loud like you could, if, if you even if you weren't like if you weren't eks, if you were cops or if you were someone else right like you could go, did you have to do the same thing? You just have to implement it yourself, whether that's like add a certificate authority. That's unique to that machine to the trusted, see a list and have a system masters identity so that it doesn't get like.

B

You can't get revoke the our back and like you're, saying David I'll give it a cute can fake that that uses that to talk to the API so or something I'd.

D

Even be willing to help right, yeah.

B

D

Like that's, that is something very self-contained. It's separate from the queue API server. It eliminates some of the concerns that that are leading to us removing the HTTP port. If that would help make it easier to like make a transition, I'm happy to open a PR to make that server. Iran.

B

Yeah I mean I think like if I mean, if there were the other thing I'm just thinking here is there's fragmentation right, like everyone could do it their own way, which is there's flexibility but like if there was even just a simple but you're saying like that's. Oh, you know a minimal implementation reference implementation to say: here's, here's, how you can have a proxy, a proxy sidecar to proxy health, C, that's authenticated and authorized or whatever I think that that would work.

A

All right, we're 22 minutes in I, think we should go on to our next topic. If that's okay, thanks for the discussion, everyone think I'm glad we came up with with something that seems like it might be: okay, okay! uh Scott! Do you want to talk to us about C or D customs of resources, yeah.

H

Howdy so I made a little presentation: I, don't know if you want to go through it now or if you've already looked at it, but I.

A

Think I looked at it I, don't know if you've changed it since, since didn't maybe.

H

I can cut he's a brain.

A

H

A

Just present it, and- and we can talk after that.

H

Okay: okay, if you don't know who I am I am Scott Nichols I work on the K native project right now, and we've been doing a lot of pioneering in the world of duck typing inside of kubernetes for CRTs. We do this to be able to extend our our resources and lucely couple. The projects and extensions together with without actual concrete API type knowledge, and we do this with a combination of aggregated- are back and some clever tricks around projections into common schema. So we can do things like.

H

If you look at a king native serving service, it has the same shape as a deployment, some bits- and this is intentional, because we can treat them as the same object. And so let's talk about the UC Vegas. Let me know if this still works presented: fullscreen, tweet, okay, so.

C

This is a quick.

H

Re-Education for me to just confirm that I. Remember all the bits and piece of the API server, and so this is an API recap around kubernetes api conventions and extensions and that sort of stuff. So you know the API server is very well structured around how resource crud is, is structured and then the API endpoints that get assembled through the resources, and this applies for both the built-in types and custom resources.

H

And so you know I just run through an example kind of assuming that the viewer of this presentation does nothing but I assume. This group knows a little bit about the API server, alright. So here's some more examples like where pieces of the resource get injected into the path, and there are some weird points in the API that we've worked in like log and proxy and I'm sure. There's others where the thing that you get back from this endpoint is not necessarily as any sort of sub resource of the object but a whole new resource.

H

That's associated to the object. You.

A

Know, or a completely different, completely different protocol entirely, like yeah I. Think of these, as like model breaking servers.

H

Yeah so I called them non-conformance of resources, but I don't really want to talk about this in my my top of it like the my proposal, it doesn't really address the problem of non conformance of resources, I'm more talking about API projections of the the top-level resource, and then then this presentation goes around and says like okay. What how do you extend things?

H

Well, you could use a CR D and where the bits get assembled into the rest, a paths come from the actual definition and it has some trivial support for sub resources, and this lets you do things like are back for patching and posting and getting so that you could maybe expose just your status and not your entire spec to some reader or writer, and this is exactly the use case that I'm trying to get with with some resources for arbitrary types- and you can see here there's an example of the the conversion API conventions that have been set up so there's so in alright.

H

The point of this is that in CR, DS there's two ways to extend the API.

H

There is the the explicit versions, storage and conversion webhook piece where the CR Dino's explicitly the paths that there's some service running on your cluster somewhere and it's gonna go and invoke some web book, and it's gonna do some other conversion and the type is going to be like converted to the new type based on the schema that was asked for, but there's a bunch of mechanics behind the scene that needs to happen to actually make that realize because of storage concerns, and things like that and so CR DS have two different ways of extending that API and I'm I'm, saying that the first way is the conversion webhooks, that's part like baked into the CR D.

H

The second way is validating and mutating webhook configurations, which is late bound after the fact and it's another webhook that registers. In the same way the conversion of a book runs and it could be the same service, but it has a different responsibility, but it's still doing new things for your API and then etc. This is what I mutating. What both configuration looks like ok, cool. So that's that's the the newest way that we do this in in the API server. There also is, and there has been the aggregated API server.

H

This is the you got a free puppy way. You don't get storage, you don't get you get off, but that's basically it and your. You have to speak the magic protocol, that's not super well defined, and it's basically, you have to consume the API machinery starter pack to get this up and running and there's a registration process that that has to happen and for for sake of completeness I show. This is how the registration happens. Then in some go code. This is how the path gets assembled based on. What's actually running so you can.

H

You can see here the quran tags example from CR DS. You could do it with the aggregated server in the same way, except now. Instead of all definitions, it's partial definition and partial go code running and there's a handshake process in the beginning.

H

So, in my opinion, API server resource this is for aggregated api's seems dated in comparison to see rdz. It doesn't have the declarative robustness that we kind of expect with kubernetes today in extensions, because there's this magic bit that has to run and that you can't know what's gonna happen until you install it so I feel, like that's a downside of a granade.

D

Api's I think it's worth pointing out. That view isn't universal and, for instance, things like metrics, which use a different storage back-end in order to drive your HP ladies, would not be successful using crts, so just because it doesn't work for you does not mean that the method itself is dated and does not mean that we would not recommend it for other people to solve their use cases. Sure.

H

Sure sure sure, but I'm I'm scoping my problem down to conflict the normal resources in kubernetes that are doing API extensions, not necessarily like the metric server. That's it's trying to expose a different API front.

A

H

A

Metrics API server method doesn't doesn't work for you, but it is useful for other things.

H

So my proposed solution is gonna key off of what already exists in CDs and we allow support for custom sub resource projections via projection configuration. You know bike shed names and in this example you you could continue to get extended new custom sub resources, based on the current way that you would register a web hook today. So this the advantage here, is that it's late bound and I as a third party vendor to some other resource I, could extend and decorate so I.

A

I have a question about this. One specifically um you're saying here that I can just randomly declare. Well, that's not random to me, but I can I can declare a sub resource. I can add. I can bring my own projection web book. I can make it source any type in the system and make its target sub resource any any place in the system or I like disconnected from the C or D type. I can add on tack on a sub resource. That's what you're! That's what you're suggesting here! That's.

H

That's right, I mean the same. Our back rules apply and authorization, and the cluster still applies and the this thing this web hook, that has to be able to go and reach and do something has to be authorized. So there's definitely an auth story that needs to make sure that this isn't work around to get access to resources that you're not allowed to see or look at or mutate. But yeah I am suggesting that anyone any vendor any implement hitter could go and extend a cluster and add a new sub resource to any resource.

H

D

Say that strikes me as a little unusual. We we had these things, be namespace scoped, with a domain with the intention that they would be logically owned by that entity right.

H

But I can add a web hook to anything that I care to in my cluster.

A

It's true I mean the admission of my book validating a mutating one's. The attention there is is policy and the idea is like whoever setting policy for the cluster gets the set policy over arbitrary types, but whoever is providing a type. It doesn't necessarily get access to anything special other than other than just that type. Yeah.

H

That's that's exactly the same situation I'm proposing here except the end result is not defaults and and it's projection.

A

So it's so the downside I see with this is like like in your example, there's someone provides cron tabs and then sometimes someone provides this. That's one book in the progress sub resource and if I'm, using this sub resource and cron tabs goes from v1 to v2. Now, I have a problem because my progress well-put may not have updated to the latest right.

A

So so, I I'm I could update API server and then find myself in a situation where I can't serve the progress of resource, because I have a version of the the contact that isn't compatible with the web. Bug yeah.

H

In that case, the this projection web book configuration have a status of I, am broken.

A

Yeah yeah that would yeah.

A

My my thought isn't so much that we it would be difficult to handle the situation because, like yeah, we would do that. But it's difficult for somebody to upgrade to her the cluster where they, it adds another gate to upgrading the cluster.

D

I'm still interested in continuing on I just wanted to make note that each anyone doesn't necessarily mean that I agree with everything here. Sure.

H

Yeah, it's I mean this is a first pass and definitely like a let's talk about this and see where we could go not. This is exactly what I would like to so.

H

Okay, so and then I indicate that the the existence of that projection web book could go in and do a status update and you could see in the status of the CRD. Potentially the sub resources have been registered for that particular CRT. So itself is an aggregate and it says like this is what I available and it's not necessarily everything that's in the spec, because maybe there's something that's non-conforming or the versions moved on or it's incompatible or whatever I'd like.

H

But this idea that we add some more status to the CR DS and says the CRT itself can say: hey I registered that I can now be served by the status and the progress and the whatever's some resources. Okay, so that's that's the lake band version and.

D

One question about the configuration you described: did you envision this being allowed to add a sub resource to a built in kubernetes to like deployment prisons, I.

H

Hadn't really thought about that, but I I think I. Don't see a reason why it wouldn't work.

H

But they're the one gotcha is that there is no way to go and ask the status of like the deployment resource.

A

Like the the kind you you mean, the equipment to the CR D status, yeah exactly it does, there's no I think you can. You can still figure this information out from the book of the homegrown and the open, API spec. So oh yeah discover this.

H

Yeah and okay I haven't really thought about that: the open API and what this would do, but yeah, that's that's you would. This would also trickle there too, okay cool. So then, let's see I'm just saying it's dynamic and late bound.

H

You can also do I'm also proposing that maybe we could do custom sub-resources projection inside of the the sub resources. We could do an extension, you know by Chad name. We have projections and this would be a simple JSON mapping, similar to how scale works today like so. It's schema maybe needs to be mutated, but that there isn't anything complicated there. Then this is the sorry wait, I, don't.

D

Understand what this slide is telling me.

H

This is the I, have a single custom resource definition for the Chron tabs again, so we're starting fresh I want to add the progress for, and it has a status fat percent in its it's resource status and I'm. Saying I would like to project just that field into the status result. When I asked, for you know, blabbity blah, slash progress.

A

H

This is saying from from.

A

The main resources status, top percent to the progress of resources type does data step, resent yeah.

H

You can think of this as like a filter, so you'd get all the metadata around the type, but nothing else that's not projected, so it wouldn't anything else in the status would be blocked here. All the spec would be blocked. You would just get a status object that has dot percent and then the value and.

D

The idea some description here that says this is only good for a get or.

H

Yeah we would that would have to get defined. Okay,.

I

So would you see this video that's being added in the projection in the open API for the status of resource.

A

No, you would see it for the progress of resource. That's right, what would be name of the type be sorry.

D

That's what I meant.

H

Then name yeah.

D

H

A good question well.

D

Yeah it's a scale but I don't see it specified here. That's why having trouble understanding what it's telling me! Oh.

H

Well so yeah this is some resource. Name is called progress, and so you can see on the other side in status. The computed sub resources for v1 are status and progress, and so those end up to be cruddy endpoints that we can go and hit.

A

D

Sorry go ahead, David was it doesn't end up being like a progress stable about example.com like when you hit scale, it's a scaled, auto scaling, Kate's eye. Oh right, we get status, it's the same object, but what would it be here? Roll over the actual kind me yeah.

H

Mm-Hmm, you know I didn't really thought about that, but yeah.

A

H

A

So I think this is actually pretty hard to do so if you set it up like it is the kinds are exactly parallel and it's just a filter, then you actually don't need a from. You just needs like a list of fields that are permitted to go through the filter right and it's fine to do rights in one place and propagate them back the other way. If you want to include renaming, then you can also kind of do what you're doing here.

A

If you want to do any moving, then it gets very confusing, because what, if you try to move something out of it or out of a field? That's in a list then so. Well, when we talked about I forget what the context was. Maybe it was a convertible notebooks.

A

We had a bunch of thoughts on yeah, I'm yeah, we're pretty sure it was the conversion wet books and we were originally going to permit like field moves and stuff between versions, but eventually we decided that that the sorts of things that people wanted to do to enable each one just required a huge amount of thought, make sure that the API made sense and and to make it understandable to people like, as you can see, from David's questions this. This API isn't immediately understandable, which isn't isn't a dig on the on the API.

A

Like the actual thing. You're trying to do is really complicated. Yeah.

C

A

That's kind of what we're like. Okay, it's it's just provided a web book to do it. Yeah.

H

Personally, for me, a web hook is more understandable and I can actually understand that code and write it and produce it, and in that case the web hook could define what the type that comes out of it is supposed to be, and it might be just like. There's a new metadata filter flag. That says this has been projected. It might not represent the whole thing all right. When you get status, you get the status of the object with the original kind, but the spec is missing.

H

This would be the same idea, okay, anyway, so I think anyway, I'm yeah, that's right for discussion, I, don't know- and that's that's the presentation so that the the hostage situation is you could implement. All of this thing by I can make a conversion web hook and do some very silly type, lookup stuff and actually like host a conversion that has no stored type that you can access.

H

Do you always have to go through conversion and then, through some name, lookup I can find the real kind you're looking for and I would give the same result with the projection web hooks. But I don't really want to do that. I'd like an official way.

D

So I can see your mouth moving.

A

So you can imagine today, you set up with version with a special name and you've got like one place for you stash all the fields that you don't want, the user to mess with and like then the place where the user sees you put the fields in the order you want them want them to see. So.

A

The only difference with a projection web book is, we wouldn't require it to roundtrip, so you don't have to store the fields that you don't want, the user to be able to change, and it would go to a different place in the URL path. So it's actually not too different from the conversion. Webhook as I see it the so. The place where the place where it gets strange is when you're registering this Web book separately from the CRD object itself.

A

I can understand why you want to do that, but that that's a little unusual I think it'd be more obvious to put it in the in the CR D, with the conversions that are already.

D

There about who owns what right, like, if you own your CRE, you.

B

D

H

Right, but the goal here is, is around being able to use controllers and operators and whatever you want to call them to be able to do things that are that again, like being able to assemble things that are loosely coupled, but they don't actually they're, not aware of the actual kind, it's using aggregated api's and maybe some magic labels or another resource to point to a set of kinds that adhere to some behavior that we would like to stitch together. If yeah I'd, like the use case I, could give one so.

A

I think there's some other considerations, which is if, if we make it, if the projection will book is a separate resource completely, we would have to make the targets like it'd have to be acceptable for you to call it will cook on to a built-in or an aggregated API, and those two have super different request paths, so it would be, it would be challenging um if, if your primary need is for CRTs like if, if the sections in the CRT, the custom resource definition were sufficiently separate, I can imagine you providing a manifest that somebody could apply over their existing CRT to add on your web hook, right like if the system administrator agrees that, yes, this is what they want to do so I think you could get that functionality or look you're looking forward.

A

Even if this is stored in the customer resource definition and.

H

Now, fine with me, the other benefit of having it in the CRT or at least some object in the in the cluster is that I can go and query for things that also implement that custom sub resource with some schema. The scheme would be important and I could say it for everything that implements the progress sub resource I would like to show the progress of their.

H

What whatever they're doing I don't need to know their kind I, just I hook up to the right, aggregated role and I can now show some really rich data around what's happening. My cluster, without knowing you.

A

Would be able to do that by going through the either the open, API or the custom discovery information yeah, if you wanted to, if you wanted to do that stuff, but for that to work you would want. You would want the the kind that David mentioned you'd want to make sure that everybody implementing the same thing use the same name for their kind.

A

Yeah, which, which is a detail that was missing from that.

H

So, okay, sorry, what we're running at a time what so the next step is or should I write another paper or like what should I do now? What's next I.

D

I, don't know that we have achieved consensus on whether we think it's a good idea to eat, no matter where this lives would we recommend that an agent that did not own itself add a projection onto it. I think that's like one set of things to decide on I didn't see an exploration. I I.

A

Sorry David, let me interrupt for a second I. Don't know that we have to agree on that like if, if we add it to the C or D resource I, think that's that's a clear implication from our part about who should be setting that. But if people want to compose C or DS from multiple sources like I think they should be able to do that.

H

It would so here's a bad idea what if the conversion was an extension and you actually converted to another CR D I'm.

D

Not I'm not trying to suggest that, like you do this in terms of conversions, I'm, not I'm, not trying to say that, but I just want to I'm, not comfortable with saying to people like you know, one one, agent controls the CID definition and a different age is attaching some resources to it. I'd be I'm, not comfortable recomme. At this moment, I'm, not comforting that someone do that I'm, also not clear on which end points we exist inside of our cube API. That would be satisfied by this right.

D

Would it be a replacement for something like like pod, pods binding and that one's notable? Because it's very frequent, it was one of the very first ones that we created. It is a projection of certain fields, but it's notable because it was exposed with a different set of access and has a different set of validation rules associated with it, which actually gives a pretty tremendous amount of power right.

D

You could say no one can change this field in spec, except someone who has permission to touch this and I'd be interested in knowing whether and how this would better address that, because it does seem like a pretty substantial one that still falls naturally into a crud projection.

D

A

Yeah yeah, that's a good point. I would expect this to handle the pods binding, and the point about differential validation seems like a really good one. Honestly, I'm gonna suggest that the next step is to like write up a more detailed design, doc. Maybe I think something along these lines can work so it might be. It might be time for a more detailed design, sound right, David. It.

D

Does and I think I would like that design to focus on how we would modify the creb resource unless you.

B

D

Can't exist and part of the CRD resource and then I'd want that very explicitly described, because that would be a way where I can look at it and say like okay, it's in this resource. If someone tries to compose it from to school, that's I can't stop them. Cuz I can't stop them today, but but the ownership signal there is fairly strong yeah.

A

Yeah exactly yeah I'd like to see a design. That's that's focused on adding this projection type web book to the CRT resource API this year, dapi, okay,.

H

Sounds good I'll try to draft something up, and maybe next time or the next time after that, all.

A

Right, thank you very much for writing up such detail and coming to tell us about it. I've, definitely like yeah, that's a great level of detail and and an example for other people to emulate when they have ideas for us. So thanks for that, let's see the next topic. I think unless fed a has, joined and he's here, he wanted.

D

The site of the 1.8.

A

Moved yeah a lot of snow, the dates have moved so I'm reading this email for the first time right now and sounds like Docs has gone up to the 19th, which I guess is the end of this week. Code freeze is July 9th. Now a deadline is August, 6 and August. 25 is when we're going to release P 1.19, so we've got more time.

A

Wait. 2020 is going, we might need even more, but it's good to have some and we have 8 minutes left and I see. Ken has suggested that we discuss something but I don't know exactly what yeah yeah.

C

I think so I hope it. We can start it and I understand the constraints now 7. So, ok with the release of 118, there was constraints put in place around validation in particularly around expertise, map type and, as we start to see, people using open operator, SDK and cube builder when they use types that are out of the core validation fails and it fails because the court types do not define defaults, many, maybe apps, maybe most or all I, don't know of the core types definite.

C

The definition of core defaults are part of comments and and not really defined as part of a specification. So it seems like the right place to fix. This would be to have defaults in the core and it seems like. Perhaps we need a kept to manage that I know that some of the defaults are.

C

mmm How do you say that they're dependent upon multiple sources, as opposed to one definition that is easily defined so with that I just want to open up, make sure that there was awareness and I and mostly I was interested is? Is there effort underway that I should be aware of that I could hook into.

A

So I would say this sounds like a great topic for the API expression working group, I'd.

F

A

A

I think our intention or this my intention was, if there's this: if the static default is possible, it should be programmatically propagated from the from some sort of commentary or tag in our in our in our comments should be propagated into the open, API spec yeah. So we should do that. Just that I think a.

E

A

E

Think we discussed that topic once or twice at a gay expression meeting and it's something we should.

D

E

I'll redirect to.

D

That Thanks I'm interested, if it yeah I'm interested in intent when you are encapsulating these types from core. Is it your intent to try to run and build like all the defaults out and store those defaults in your API, or is it more like you want to ferry a pod spec from A to B and when you actually fluffed up that pod? That would be the time where you want. The defaults realized for the current default settings for a pod.

C

Yeah so I'm one of the maintainer x' of the of libraries that are being used and I, don't always know the exact use cases, but the ones that I've been exposed to. They are generally using something like container like they're, defining an aunt in ER for their crv, something along those lines. They literally may not need all the defaults. To be honest, what what happens is they are generating a CR D off of a defined go structure and the CR d that gets created is not valid by default.

C

So it fails, and just trying to say applies. Er de to the trip Renee's cluster yeah.

D

So I'm wondering if that is actually a an indication that the type that they are using looks a lot like a container but doesn't have the same validation requirements as a container and at least when we questioned we did this for say workload types we actually want in in openshift. We have we skew right. It's not always the same open-shut api server that map's to a queue.

D

If you have server so things like a container definition change over time, and we actually want current defaults to be applied nah the defaults that happened to me in use when we froze the object. So what we realized is that our pipe is is schematically very similar, but it is not the same.

A

Yeah I have we've got like elements left. Let me just say I. We feel pretty strongly that the api server should be the thing applying the defaults, so we would not be exposing the defaults of the intention that users apply them themselves to the object, especially not if it's something that's going to be stored and submitted at a later time would not want people to do that. But.

D

But I gave you an expression seems like a good spot and in general the spots where we can actually correct what we have makes a lot of sense.

A

All right, so, let's take that topic up there. If there's more questions and I think we should call today thanks everybody for joining.