Kubernetes API Machinery Special Interest Group, 2 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes SIG API Machinery 20201202

Description

-[eddiezane] Working with OpenAPI data inside kubectl
Recommended to join the discussion as part of API Expression WG
-[dims] gogo proto is going to be gone
https://github.com/kubernetes/kubernetes/issues/96564
https://groups.google.com/g/kubernetes-sig-api-machinery/c/tcwFubV9Boo/m/-C3C3Aw-AgAJ
https://github.com/gogo/protobuf/issues/691
-[eddiezane] Documentation for `kind: List`?
https://github.com/kubernetes/kubectl/issues/837
-[jefftree] SSA GA Requirements:
https://docs.google.com/document/d/1y7bs8PI98gEnKvZi4GQXxQa1_OxPtBL4fQ0b4k48dwo
Targeting 1.21 for GA (yay!)
-[kwiesmueller] Stripping managedFields from API Responses
https://github.com/kubernetes/kubernetes/issues/90066
Idea to add a dump an object from a users perspective using managedFields
Accept header sounds good, allow a format that can represent different managedFields encoding versions in the future

A

Okay, we are live um hello. Everybody welcome to seek api machinery meeting today is december, 2nd 2020, probably our one of our last meetings in this wonderful year. So um I hope everybody is holding enough energy to go through the finish line of 2020..

A

uh We do have some topics on the agenda. I don't know if everybody is going to be able to make it, let's see if teams can make it today, uh but still is going to be an interesting meeting. So I would like to start uh in the order of the agenda. uh Let's start with eddie and the topic, I would open api and keep ctl.

B

Cool yeah thanks for having me um so I'm I'm trying to work on some open api related cube control commands um one of them. We started as like an experimental sub project of like a generate command that will use the open api uh example field as a way to provide a template, so your user can be like cube control, generate deployment or something, um and so right now we're trying to solve a problem of there's people going.

B

There's no good way to start with, like a yellow, manifest other than like, knowing what to write copy pasting one from somewhere um and it's so so to the point where inside of the the c cad and the other kubernetes developer exam that they tell people to do a cube, control, uh create deployment and then queue control, get deployment, oh yaml and pipe that out to a file um and then delete the the right.

B

You know the read-only fields and start with that um to save time um so we're exploring ways to like make that a little bit easier for users, and so I I threw together a proof of concept and a prototype.

B

um Basically, where I'm going with this is I'm having a hard time working with the open api libraries we have, especially um the gnostic library. That's the protobuf, backed open api library. um I've been saying for the past two years that I'm still really new to go and and this the protos are kind of really tripping me up. um Is there any good place that I can get some examples or talk to people or anyone want to take a look at what I've done and tell me it's terrible.

B

You're asking just about like how to navigate uh how to navigate and if I'm, using the right thing.

C

So, for example, stefan uh he hangs out in the uh the api machinery channel. Can you can you hear me now.

D

Yes, ah okay, we can hear you danielle, okay, I was gonna say: um oh you are on antoine. I was going to say that api expression working group is probably the best place to um go for this uh because uh yeah, I think, that's the best place to go.

E

D

Yeah, so I think uh specifically to get useful example data. We would have to populate some fields in open api that I don't think we really populate, or at least uh not.

D

You know authoritatively right like not in the tested and known good fashion, and I think that'd be super useful, um not just for giving letting people have examples, but for fuzz testing our api surface area, I think it'd be great if, uh as part of the open api spec, we listed like some example- random values that were valid because right now, it's really hard to know that.

B

D

Thing about this.

B

Approach is it works with crds too out of the box, as long as authors provide an example in their api spec, um so.

D

Right, but that's what I say like we don't like. I don't know if any authors do that at the moment, uh so right like it would be a great feature if that is populated, but I don't know that it is populated anywhere and I definitely don't think that it's populated uh consistently so um yeah uh api expression working group would be uh the place to go talk more about that awesome. Thank you, uh and also, I think uh uh antoine is a good uh person to ask about our open api.

D

um You know about the cube open api repo right. That's got uh most of the logic that we use.

B

Yeah yeah, I do like I said. The thing that's tripping me up is is working with the protos from gnostic, because that seems like when you grab a discovery, client and then you ask you grab the open api out of that. It gives you back a proto-based document.

D

There should be uh helper functions to parse that and and traverse it in the in cube, open api repo.

F

Yeah we're using portal because it's much faster for cubecode to decode uh on every single instance of running it.

D

Yeah got it uh okay. uh Should we should we go ahead and do your do these out of order and do your next issue? Next, maybe.

B

Sure so this is uh one. That's come up a few times. um There's a couple linked issue trees. um It seems that the the kind of list uh is confusing lots of people in different projects, because it shows up once in an example somewhere, but there is not actually any documentation really outside of this one example where it's used, so I don't know if who's the ones to own this or if I need to work with sick docs. um But I just I wanted to bring this up.

D

um So list is to group resources that are not of the same kind uh like usually, you would use a specific list.

D

I am a little surprised that anybody has a reference to this anywhere because I don't think it appears in our actual api surface area.

B

It I don't so if you click that helm issue in there, um there's a bunch of issues that are linked to this helm issue yeah that one um and the I can find the one example where it's linked to, um and I think when you, when you get back when you like cube control, get pods dash, oh yaml or json. It gives you back. The kind of list should give you back a pod list, not a list.

D

I think list like, like, I think, cube control would like use a list internally after when it parses a yaml file. That has uh just a bunch of uh you know how you can use the separator in yaml and get separate documents into the same file like I think cubecontrol would use a list to represent that in memory. But I am surprised that people- I I'm kind of surprised that that escaped our internal thing like I had no idea that people looked at this.

B

So I just tested cube control, get pods, oh yaml, and it gives kind list with an.

D

B

D

C

That um there have been bugs going back and forth on it in the past, although if it's been that way long enough, I don't know that it would still be a bug.

D

So uh a list it.

C

D

A kind list really did it.

D

Do like uh v equals nine and look at the actual response from api server.

C

When, when I accidentally broke it in the distant past, it was uh only on the client side uh and it was because cube control would just say: here's all the items put them into a list for rendering a v1 list is just a container for holding other resource manifests inside of it, and they don't all have to be of a uniform type right. So v9 is.

D

Podless coming back.

G

D

I think that's probably why it doesn't show up in our documentation, because that's not what the server returns and the thing that is documented, is what the server returns.

G

H

G

Of the reasons might.

H

Be that um if you do get parts you can easily do get pod services, which would return uh two or multiple different resources, in which case the list is the only way to represent that.

D

Right but that's a that's something: cubecontrol does right, the server doesn't yeah doesn't give you separate types there.

H

Yes, because you will be having two requests for each type and then keep cuddle uh aggregates that and presents a single list.

D

Yeah, um we might return an actual list for custom resources, but I don't know we.

C

Asked for a list type inside the okay and we were okay.

D

Yeah, okay yeah! So um as far as getting this in official documentation, I am not sure what we can do from api machinery perspective, because it isn't what we expose and all the like. The the api documentation that we generate is all generated off of the things that we return from the server so.

D

Maybe a topic for um sig cli.

D

I know that's not a good answer, but I don't have a better one off the top of my head.

B

Okay, yeah that works, I just I wanted to get a take on it um is so so so so that's the answer. So this is something we we don't actually expose or want to expose to users. Okay,.

D

I I mean it's very useful for cube control to be able to group uh objects of different types. um So, but that's a it's a uh cube control thing, not a uh api server thing. Okay, thank you.

D

Yeah I mean I, I think uh um api server would understand it if you sent it one, but there's actually no context in which you send api server. A list of things.

B

The other place this is exposed is in the crd example. So if you go back to that original issue, we show using kind if you go to top.

C

That one yeah that's a cube, control creation right.

D

Yeah, that's the alternative way of uh yeah. You can represent a list of items for cube control, a couple different ways by explicitly putting in a list like that or by listing multiple resources, one after the other, with the document separator in yaml.

D

If you do both, I guess cube control flattens it I'm not sure what happens. I think they flatten it.

D

um Yeah, so that's an interesting question. I I don't. um I don't know if cube control is a convenient place to document this. That would be ideal if, if possible, if not, I guess we could have a discussion about how we could document that I'm not sure how.

D

um Should we go on to uh dim's jim's issue if that is well, even if it's not acceptable, I don't know if we can do better right now,.

D

uh Dims, do you want to uh elaborate on this other than put the issue in with a catchy title? Hopefully.

I

So um uh the background is like: I have no skin in the game, I'm just trying to update dependencies right, but then, while researching uh updating some dependencies end up ended up in this weird thing like okay, we need to update grpc in hcd, because otherwise we can't do some stuff and continuity. Folks can't do some other stuff and ended up uh opening an issue in hcb asking for. uh Can you update grpc uh in the in the master and then when the release happens, we'll switch over to newer lcd?

I

So in that discussion I got asked this question: saying: uh go: go pro protobuf, um do you use it while we are using it? Are you going to do something else, or are you going to uh get rid of it, and then I promise to find that uh for somebody and then I ended up digging through mailing lists, and I found this thread where there was a discussion back in may saying: um oh, the gogo protobuf is losing maintainers and other people will need to step up.

I

So the question is like I don't know what the current situation is. uh Has it since gogo implies this? Is a google funded uh repository uh at that point, so I wanted to raise that question here and ask for like what is the current status?

I

uh Will we be able to depend on people taking care of this repository? Will we be able to update dependencies in this repository and keep it alive, and you know: are there enough ci jobs, uh that kind of questions.

D

Yeah, I guess um my question is well so, as far as I know nobody's working on switching over or anything like that, I think it's kind of a hard project to to do so. So my question is: what happens if we don't do anything, because that's what we're doing at the moment.

I

uh I think we'll survive uh for some time, um but it would be good if at least we identify somebody who has uh maintain our rights. If we run into trouble, we can lean on somebody to go. Do something, um that's the minimum that I would recommend trying to figure out. We did the same exercise for c advisor too. uh At one point uh you know uh david uh stepped down and we got one more david to fill in.

D

Okay, um yeah, I don't know who would be good.

G

F

G

Google project, okay,.

F

This is not a. This is not a google project. So, as far as I can tell.

C

We're worried about having maintainers in case there's some kind of a cve that we have to fix. I assume tim's.

I

Yes, yes, or if uh it holds back dependencies that we need to update rest of kubernetes too so.

C

Do we use this as just a generator or do we use it uh at run time as well, because it was just a generator?

C

Then we generate the code that marshals uh the protobuf objects.

I

I think that I maybe I'm wrong, but I saw some runtime thing too, but I I can't be sure about that.

D

The generator code probably has some runtime dependency correct.

I

C

Yeah yeah, if we get stuck the ugliest solution, is just to make a copy of this. In case, I owe somewhere yeah.

I

I'm trying to avoid that, because if there is nobody taking care of it outside, it's going to be harder to find somebody. You know among us who will.

D

Care right I mean: if we had a cve, then we'd, then I guess we'd have to fork it uh right, not the end of the do I uh I I should. I should just know this, but I don't do our. Is it strictly a dependency of kubernetes or is it a dependency of our dependencies? Like does does like etcd imported or something? uh Let me do a quick check, because that would that that's where a fork becomes very tricky is is if, if uh one of our dependencies uses it.

I

uh Yes, there are uh several things used: use it um container d, tt, rpc, uh sp3,.

C

Spf 13 something so hang on if they're using it as not maintained and we're using it as not maintained, are we all just staring at each other to figure out.

D

It's the uh maintainer chicken.

I

Yep, let's see who blinks first right.

G

I

D

Okay, uh I guess container d is uh dependent is, is a different part of the system, so um uh I see using it too. Oh lcd uses it okay, all right yeah, but we only need an scd client to work.

D

So that's true, but we've had problems in the past where, like the lcd, server and client have to be using exactly the same proto library.

I

And it's in our vendor tree right. If it's in our vendor tree, then we care.

D

C

So but like I'm thinking about.

D

C

Ended tree like if we made a fork because it's in a different spot, you would use our fork to marshal and unmarshal our kubernetes types. So.

I

We'll be doubling we'll have two copies of the same code. That's slightly.

D

Different well yeah best case. You end up with two copies and everything works worst case you have two copies and then somebody tries to pass the types to somebody else and it doesn't compile right.

C

Yes, but we wouldn't stick an etsy d upgrade on this, though I guess, because I assume that's part of the concern dims, that, like we, we get to a point where you can't upgrade some other library that depends on your version of this library, because you both depend on it right.

I

Yeah, at least, if you can find somebody who can update the go mod, go some and run the ci jobs and make sure nothing is nothing breaks. That's a minimum right.

D

um That's tricky because uh we basically have to use the library that the version that ncd is using yeah or prove that the one that we want to go to is safe for etcd, because that has bitten us at least.

I

Yeah and looking at the contributors to that triple uh clayton is like number four from top so.

G

um Is there like a communication channel shouldn't it be possible to negotiate with both projects? What to do with this.

I

uh They haven't made a single change since august first week, so I don't think the the project is healthy anymore.

D

Well, I wonder if um I wonder if we should ask our etcd experts to look at what it would take to move scd off, because I think that we should figure out what the what the leaf dependency is going to do. I don't know how to uh so.

D

The reason I don't mention container d is because I don't think those types interact anywhere, um so I guess that would just be a separate conversation, but we api machinery can't switch to something that cd isn't using. So that's going to be a problem. If there's a cve and we can't get a maintainer and like if we fork it, then we've got to get fcd pointed at the fork too, which could be weird. Maybe it wouldn't be too weird, but then so.

I

If we want, if we want to move a city off uh the, I can help uh with getting continuity off to the same thing as hcb. I see yeah. I know enough people there.

C

uh To help with uh that change now, as I recall the mailing list thread from long ago, there was a perf difference between gogo proto and the alternative right. Yes, that's absolutely right.

D

uh Is there still one I think antoine might have the most recent memory.

F

Although I would be very surprised, if does not okay, one one is based on generated code and the other one is based on reflection. So.

D

Oh okay, um given that maybe we need to fork it and start maintaining it ourselves, how would you recommend.

H

D

Danielle, uh I'm not excited about that at all. Maybe we should wait until there's an emergency yeah. That's fine! uh That's.

I

Perfectly reasonable, you know as long as we know that we are taking that risk and we say that uh when the time comes we will do the needful. I guess.

C

Yeah yeah sound a lot less weird: let's have two copies now, don't I daniel.

D

Yeah, maybe we should document that we thought about it, and this seems best to us at the moment. All right somebody's really excited about replacing this, then you know great, makes the performance difference.

I

Yeah, I'm not. I don't want to touch it. If I can help it.

D

Yeah, oh jason, mentions in in chat what, if we got a maintainer, uh could we could we get someone uh maintainer on the original repo right? Okay, uh maybe that's.

C

D

I don't have anyone who would be raising their hand to do it. I don't think it seems like clayton has contributed.

G

D

I'll I'll I'll take a note to to look at the look at the repo and see if I can think of anyone who um might be good.

J

Yeah, I was just thinking that it might be easier to find somebody to get added as a maintainer, rather than taking on a whole additional fork and all of the additional trouble that you have to go through of all of the interdependent projects. Adopting the fort versus the upstream project.

J

Yep still not ideal, but you know a lot less trouble from a um migration standpoint yeah. I agree. Okay,.

D

A

Usually comes to these meetings should we try to bring this to his attention to see what he thinks, like maybe.

D

uh I'll I'll take an action item and, and uh worst case I'll ping in about it. How about that? Okay, that's good.

A

Okay: um let's go to our next item. Jeffrey, are you still around? Yes,.

E

Oh hello, um so server side apply has been around for a while and we're looking to uh target ga for 121 the stock, basically tracks. What we believe is remaining to be blocking for ga and we're looking for feedback.

E

So for 121 we're mainly. We want to focus on still ability and we have an emphasis on bug fixes documentation, uh particularly for api users.

E

It might also be useful to have a entry controller um just to demonstrate something that uses server side apply, and we don't have that many features that we want, but the main feature that we do have is the type api client um for that.

E

um The other thing was about hiding manage fields which kevin will talk about later on for features, and the remaining items are just mostly bug fixes and pretty much like ed test and all the other small stuff that we need for ga, so yeah screw all the way to fine.

E

But yeah, if you have any feedback, just feel free to comment on the dock directly or like just print something weapon that you need.

D

um So we're hoping to get there for uh 121. Yes, that is our target.

B

C

Oh and that entry controller that you're talking about using you would use the client that you create right.

E

E

D

uh I just have one thought, which is: it might be um good to phrase this in the form of a checklist.

D

E

Yeah, um I'm also looking to update the cap original cap that we had so I can add that to that.

F

D

This will actually be ideal. Yeah.

F

The checklist is kind of the the table of content. To be honest,.

D

Okay, that seems like a lot of items is: is everything uh owned? Are we looking for volunteers.

F

D

F

I mean we can use some help, of course, but uh uh we have people having multiple bugs, but every bug is assigned to someone.

D

Okay, um should we maybe have a check in in a month or so and see if everything is making the progress that we want? Oh.

E

C

Can schedule that- and it might be useful to come back here when there's a pr that shows the controller trying to use the client, so people can see it in in action and provide feedback there early.

E

Okay sounds good.

A

uh Jeffrey and anand, the best thing, the best way to find um you and or you know, join the workforce is on on the working group meetings right for now.

G

A

G

A

Slice yeah, it's like, of course, yeah. I was thinking on. You know ways to let other people know that we are trying to accomplish this goal. Finally, in 121, so maybe people get excited and you can get some extra hands.

A

We'll see very good, thank you so much um and I think it's a great uh lead way to kevin.

G

Yeah um I've been reminding last meeting that people are not so happy with the amount of managed fields we put into objects and there's an issue I didn't know of, and we recently discussed this in our last meeting and uh thought about what we can do about it. The the issue has a lot of thumbs and a lot of comments, and what we thought about doing so far was to find a way.

G

So we do not allow users to prevent the api server from sending them the managed fields if they don't want them um the options we like we, uh we just wanted to bring it up antoine and I had a discussion about it earlier and how we can do it um so far.

G

It looks like we can add uh um something to the media types. So we just give an accept. You accept um yaml without managed fields, and we would try to tell the serializer to serialize it without managed fields, um but we wanted to bring it up and see what the opinions are in here and if anybody knows any any things we should think about before we try.

D

Yeah, I think that's okay, as long as like, uh if the user sends this object back to us, it doesn't cause some strange, unwanted change um and.

C

D

C

Request the client still, if you want to.

G

What did you say, sorry david? I couldn't.

C

Answer there's a way to get that content. If you want it right, you can imagine an editor that wants to highlight who made which changes.

G

Yeah, your current idea is to do not change any current behavior but to add an additional format that you can request and you get without managed fields. So it's not.

C

I mean I could see it defaults in either direction like we're still beta it's breaking, but it would be one that I could see being kind of reasonable yeah.

G

I think the discussion was about cube ctl get because ndo-yama, because people use it to dump full objects, and if we break that by a change that by not sending any uh not full object anymore, it would be could be weird because people use it.

D

G

D

The the default of cube control on the default of api server don't have to be the same. Yeah. Okay,.

G

D

uh So uh I guess my current thought is that we should leave api server, giving the full detail and you can suppress it if you want to, but I don't have any like reason for believing that I I think in parallel. We should.

D

We should make sure that there's user visible ways of making use of this so that people don't think we're just adding this for no reason and I've already seen, and we should probably have a demo or something in this meeting. Maybe next time of the q control uh analyze, was it analyze or blame.

G

D

Blame yeah, where it uh printed out the yaml object and said who, who put in each line. I think that's really cool yeah, the the other thing that I think we could do that'd be really nifty for users is a like. uh I don't know what to call it, but the the feature is, like you say, an object in what user you want and it just prints out you. It prints you out just the fields that that user set um so it'd be like a super powered version of cube. Cuddle export.

D

I think that'd be pretty cool. I don't know if anybody has worked on that, though,.

G

Yeah, that kind of would be like extracting the last applied object.

D

Yeah yeah but you'd be able to do it from anyone's perspective. Yeah.

G

D

That would be pretty nifty to see what the what hpa.

G

D

It's doing, for example, let.

G

We write that down.

D

All right, so did you.

G

A

What you needed kevin.

G

uh Anton you've got any everything any questions.

D

G

Yeah uh uh yeah, I understand, um does do you have any input on the idea of sending an accept header instead of adding a request flag.

G

D

G

D

Yeah, I think the content type or accept accept header is the most natural fit. Okay.

F

Yeah, I think maybe a yaml plus manage fields or something like this as a way to say this is what I want.

D

um I I think you should think imagine in the distant future. We come up with another serialization format. For these whatever, uh uh like right now, we've got manage fields: v1 is the default. If you ask for managed fields uh not at all, that could be a version of managed fields and if we came up with the v2, that would also be a version of manage field, so whatever we do for this should work for a third, a hypothetical third um serialization format. Otherwise, we kind of designed ourselves into a corner.

F

Yeah, that makes it out of sense to me.

D

Yeah, um uh so I think it'd be, I I think the maybe the most important next step is to get this stuff checked into the cap or some other place where we can have an official sign off.

D

Does that sound right to you, david.

C

Yeah, I think so: okay.

A

All right, very good, that was our last item in the agenda. I think that covers what we wanted to discuss today. Is there anything else I'm missing? If not, we should be good.

A

Okay, I'm good very good, so probably we'll see you one more time this year, um I hope um everybody's doing well and staying safe and thank you for joining today. I will upload the meeting shortly after today. We'll let you know thank you, everybody for coming.

D

D