Kubernetes Kubernetes AWS Provider Subproject, 23 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes - AWS Provider - Meeting 20210723

Description

Recording of the AWS Provider subproject meeting held on 20210723

Issue triage

A

Hello, everybody and welcome to the bi-weekly provider aws meeting. I am your moderator facilitator for today. Just in santa barbara, I work at google a reminder. This meeting is being recorded and will be put on the internet. uh So please be mindful of our code of conduct, which boils down to any good person. uh I should have said today is friday july 23rd 2021..

A

um We don't have a lot on the agenda today. I am going to paste a link to the agenda in the chat so that people can have it if they would like to add their.

B

A

Which I see nick has done or would like to add any items, but otherwise we will just go through. um I guess, sort of the triage of our uh sort of open issues and um see how it goes from there. Let me try to share my screen.

A

A

Okay, can you see the provider aws window and about one two, three, four other tabs? Yes, perfect?

A

Okay, um so we had a couple of uh just to review what happened last time uh the ipv6 uh pr was ready for review. Thank you uh nick anyone else that reviewed and merged that um I have not yet looked at this uh pr for uh service account names other than what we talked about last time, um but I think.

C

Yeah, I think it's it's ready to merge um walter approved it, but it needs another lg tm, just because uh the most recent commits didn't have it. So um if anybody wants to look at it, um that would be. That would be very much appreciated.

A

Awesome I will try to after this meeting then um otherwise we're going to go through the issues in uh cloud provider, aws, repo and kubernetes kubernetes that are tagged, aws, um we'll start with a cloud provider aws uh which are intend to sort by recently updated.

A

uh There aren't as many of them. Obviously in here um I don't know if there are any that, let's see in the past two-ish weeks, it seems like it's this point up, um but I don't know if anything anyone wants to call out, otherwise you can sort of go through them in order.

A

All right: well, let's go through them in order uh creation of an anal nlb ignores the proxy protocol. Annotation.

A

uh When an nlb is created, its listeners target groups are created, but they aren't enabling proxy protocol.

B

This comes from the entry limitation, so in the entry code we did not uh support proxy protocol for nlb, uh but then, if we use the aws load balancer controller, we do support the proxy protocol and annotation. So that was the reason this issue might still there in the car provider.

C

Yeah, I think we should just comment that and um I don't think it's super high priority for us to add it to entry, since it's supported by load, monster, controller.

A

Yeah, I think, that's reasonable. I think uh it's not like a bug fix, so it seems like it's a motivator to get to the uh out of three one, so I think that's reasonable would would one of you, two nick or kishore, mind commenting on that then uh sure yeah, thank you and then we can keep going.

A

uh The next one is a request for a more flexible nlb health check with health check payload, uh ideally that their request is that in a community service, the nlb health check should be able to send a small payload to the target port or endpoint, which I assume is supported by nlb.

A

This isn't some effects been aws feature I don't know is: is that supported by nlp.

B

uh Not that I know of that, we can send a payload uh okay. I I've seen like tcp uh and http https, but not like any custom payload that customers can configure not to my knowledge so mostly like a nlp site, feature itself.

A

Yeah, I mean, I think, uh we've.

B

A

Something we had something similar in uh the chaops project, where we overcame a limitation of, I think it was classic, but whatever um by basically using a sidecar pod. So maybe that's something we could recommend here. um So in other words, sidecar, pod or sidecar container uh speaks in this case zookeeper and uh basically uh like when a health check an http health check comes in. It will do it in this case telnet health check to zookeeper.

A

Does that makes sense? I.

B

Mean they could they could also configure tcp based health check right, so it's only gonna look at the connection establishment uh and not worry about the internal protocol. Details in there.

A

Yes, that's true, I think the uh so, if I recall correctly, zookeeper has a bunch of, uh I think they call them four letter commands and one of them is: are you okay and it's like a it's? A deeper status check, it's more like a kubernetes readiness check so and it returns. I am okay.

D

Or something uh based on this and.

A

Exactly uh I cover what it says: if it's not okay, uh but um anyway the uh so I think I but I think it there is a not okay answer, so it's not just. I think it can be listening and not be happy, um but but I think also we can just I mean if we, if we assuming nlb, doesn't support this, which and it's a fairly least request as well. uh Then I would I can comment and recommend a uh a sidecar container to adapt the zookeeper protocol to http or something like that.

C

A

That sounds like a.

C

Reasonable recommendation.

B

I see the concern was because the zookeeper server throws an error if they don't receive the protocol. That's what I feel. Okay,.

A

All right, I was muting to type, but uh we can come back to that one. I will. I would come on in a minute um all right. Well, let's keep going uh publish credential provider consumables, some random person called nick turner.

C

Yeah uh this is this is uh on my to-do list um either me or somebody from my team. I have sad news. Actually, a burke left went to some strange team at google. Actually not gke, though I'm not sure what uh what team it is. It's uh something to do with.

C

Functions in the cloud I think, but so he was assigned this. um If you want to just assign me, I can take it from him.

A

Awesome, uh google has very imaginative naming so it'll probably be called cloud functions if it functions in the cloud.

D

A

Reassign to you, uh I think I could just do that.

C

But give him a warm welcome for me. Okay, I.

A

Will do uh all right yeah, but that would be wonderful to get those published um uh the next one is another, I presume tracking issue, which is a test nick. You know the one of your test framework for ede tests, yeah.

C

I have a pr for this, uh which I started, but one of my co-workers had some thoughts here so um working on it. I haven't had time in the last couple months, but getting back to it soon.

A

Very cool, um uh an issue with the helm chart uh relatively old one, but recently updated. Let's see what uh okay, so they got an image pulled back off when using the latest helm. Chart and comments are that it also happens with the cloud controller image.

C

Can you just assign me I'll test it and make sure um it.

A

Looks like the right it worked. Oh, I see this one. The 131.

B

C

Oh, that's because I don't think maybe.

A

It wasn't popular, it seems like it sounds just like the last issue or the last, but one where it was around publishing these artifacts. So.

C

I'll double check in the repo right now.

A

I was just credential provider I'll, be like.

A

Might also be a lack of the the the v before the 121 they they have v in here, but like sometimes we use v. Sometimes we don't right.

A

uh Okay or just there's, actually no tag. It's uh and martin stefani says uh that there are date-based tags, but no oh yeah.

C

I can confirm that.

C

Okay, let's see.

C

um So I think yeah, the the.

C

There, oh okay, there's a 121 alpha zero with the with the date prefix um and there's a 120 alpha zero without the date prefix.

C

Yeah, maybe we pushed the 120 alpha zero by hand. No, we don't think we would have done that this could have been like prior to and after a change to the automation. Maybe um we did make a change to the container, build automation so yeah I will. I will follow up on this one. I guess um I don't know do we do we care if there's a tag like without the day. It's convenient, certainly.

A

uh Yeah, it's it's it's certainly I mean it would be nice to have the tag it's sort of what everyone else does. So, let's have the clean tag: the cleantech yeah, okay,.

C

A

Right, I will work on that, but I mean if it's a massive pain, it's you're right. It doesn't really matter it's just more consistent yeah. I.

C

Think it's. I think it should be relatively easy to sort out.

A

Wonderful all right: um let's go back to the.

A

uh This one, I think the next one here is uh ebs storage class, not working and somehow get cloud provider returns, nil, which seems very unrelated. But let's see uh trying to use the out of tree provider. I guess with the with an rke cluster, it's a branch or red hat.

C

I would guess right.

A

um Let's see and then.

A

The cloud provider, the cloud managed deployment finished successfully no control and subscription, because we had a new ubs storage class and create a new pod with pvc using that storage class, which sounds right and then the pvc cannot create a new ebs.

A

The cloud and returns sales version volume with storage class, okay,.

A

C

They either need they either need csi or they need to pass um external cloud volume plug-in equals aws to the kcm right.

A

Right I mean this just sounds like it's not set up correctly right it doesn't. This doesn't sound like a bug, it sounds like uh or it.

D

A

At first, at first inspection, it sounds more like it just wasn't, set up correctly, yeah.

C

I agree with that.

A

But they are using the chart so.

A

B

Maybe the way to do this is.

A

Just make sure that make sure if we work towards having tests or something like that, then we can be uh confident that it does work. And if vendors want to make things work, then they should make things work.

C

Yeah but um okay, I guess the the way to do this is to look at this rancher deployment. um I'll also ask to see like what their settings are in terms of uh do they have external cloud volume plug-in set and do they have csi installed and then see what is expected based on their settings, so yeah.

A

I think it's reasonable to verify that it works with some configuration and highlight the flags you think might are most likely to be wrong or the like mismatched most likely to be wrong and then say, like here's, here's the working configuration. If we have such a thing right so yeah, I don't. um I don't yeah yeah.

C

Cool I'll self-assign here.

A

Okay, thank you, um node labeling, from aws tags. I think we've talked about this. Oh yeah, indeed a lot, because this is a june 2020 issue.

A

Okay, and uh this is the idea that goes around a lot saying that we want to label nodes from aws instance or ac2 instance, or ec2 instance, tags and.

A

It looks like we do. Okay, I didn't know that we did that in chaos, but all right, apparently we do do some of this. I think we do it because we have to for cluster auto scaler, but anyway,.

C

Yeah abrac started this one.

A

Okay, I mean, I think.

A

You said: abra started this.

C

uh He started looking into it yeah I don't know how far he got, um but so.

B

Is this one is.

C

The proposal to take like all tags or just under a specific prefix.

A

I mean that's where that's where it gets tricky right it uh I mean I'd, say this is not a proposal, uh but it's not a formal proposal but yeah.

A

uh I feel like it's appropriately triaged. uh We could unassign ever that might be yeah because I assume he's no longer working on it good actually, but other than that it is a feature request.

A

uh It is not assigned to a milestone. I think that's that's correct. It sounds like there's some progress going on on people trying to point out how other things in the ecosystem do it and what we should do. I am I'm very unsure why we do this, but that's have a look at. Why chaops does it.

A

All right when I commented on the diff, uh okay.

A

That's a pretty big diff all right! Well, it's not clear why we do it.

A

Let's see if it was clear in the initial.

A

A

A

All right, it's a little misleading to say that we do this. uh The reason is.

A

We were, um it is possible to configure some node labels that also get reflected into the tags, and then there was a race condition whereby, when we were getting the node labels for an instance, we might get the wrong ones based on the wrong version, and so the workaround was to source those instead from the tags, because those correctly follow the correct version of the auto scaling, launch configuration or launch template.

A

So I don't think it's quite accurate to say that we are reflecting the we.

D

Are we are only.

A

Doing it for the, or should we be doing it for the labels which in chaops we synchronize back and forth between the two. So it's not quite accurate to say that that is uh done in chaops.

C

And it wasn't intended as a feature of synchronization exactly.

A

It's not intended exactly it is. It is an artifact of the fact that we know that we set those in that way. But um yes,.

C

Well, this clearly is like a pretty popular ask: 27.

C

uh Thumbs up so uh I think it's definitely something we should look at.

A

Yes, it sounds like actually one of the uh looking at this. The specific issue that opened this is around the roll labels, which is blocked by cubelet, so that could be sort of the problem.

A

So that it's basically asking for a so chaos has a privileged controller, which is a borah controller that is able to apply those labels. um But if you're doing it like with cube adm, for example, I don't know how it you can set those labels.

C

A

D

That's the label that is also rendered by to control right like when you get nodes. It shows the label based on or shows the role based on those labels. Correct.

A

And it has a controversial history, because uh there was some confusion about whether kk, whether it's yeah kk would like the label not to exist, but kk has no proposal for how to do anything with kubernetes without that label. So.

D

A

It's uh it's an interesting. It's an interesting like position.

C

C

Eks's managed node groups feature is also blocked from applying cubelet forbidden labels to its nodes.

A

Yeah I mean I don't know how I don't know how.

A

It'd be interesting to know, like I mean I'm happy to share how chaos does it, but it'll be interesting to hear how other people do it and like.

D

Find if we can find some does apply the label at some point.

A

D

Yeah, like it joins a node and then it it runs. Some like post-join process that like applies the label, but I don't know how it bypasses the the cubit. um Then I rule for the label.

A

It might just be running, I mean you just have to run as a different user, so they might just you run as an admin.

D

ah That might be it.

C

A

C

I'm sorry going, uh I was gonna say um I don't know, I'm not familiar with the joining process, but doesn't the like bootstrapper role have something to do with that? It's like the node bootstrapper with its own permissions. I don't know yes.

A

I think this might be a good sig cloud provider topic, maybe, um which is essentially how do nodes securely join? um And I don't you know there is no. It would be a great like chaos has a relatively secure joiner or we intend it to be like secured by more secure design or intensive design.

A

uh You know, but it's never gone through an audit, so you know, uh but, for example, using the aws instance data is that right instance document the signed instance document or uh using um using a a call to the sign token service to sort of prove you are the node that you say you are to then get to start that bootstrap process and then to securely identify the node and identify which uh roles or labels which labels and taints it should have, because those are what um those are security, sensitive things.

A

That's why we stopped allowing the cubelet to do it, because the idea was, if you had a privileged workload, uh if any node could just self-assign that that label, you would have a good chance of getting that privileged workload.

A

So that's why it was blocked, but there was no proposal for how to actually securely deliver the labels instead.

C

C

That's an interesting.

C

Ask I don't know andrew, do you think this is worth bubbling up into sig, clock writer or if we were gonna look at doing something like this and.

C

I don't know an r cloud, node controller- I mean it, it probably would have to it would require changes to um the the node controller logic itself. Right, there's no.

D

C

Like there's no place to modify the node.

D

Yeah and potentially the the default surface account and role of the controller, I guess but um the like. So this request has come up multiple times across many providers where, like people want their um own way of kind of doing like no tools, so they use tags um so that then they can use those for note, selectors and whatnot.

D

um So you know if we're seeing it in enough places. I think it's worth um talking about doing this in a generic way where you can uh like, maybe not specifically with tags but like a mechanism to request some arbitrary labels that should be applied to node as it's registered similar to zones but like just more generalized.

C

Cool all right um I'll take a note to to bring it to uh the next six pop provider meeting.

A

Do you know if that is next week or the week after.

C

uh We had one this week, I believe so. Okay should be the week after.

A

Cool all right, thank you all right, uh another long runner we're up to 24 days ago. So I guess we actually didn't have a meeting last week. So why don't?

A

We do the next two, um because then we get to 28 days ago and then uh that's good yeah, at least for this repo we can see if we wanna, we have the appetite to do the next repo, um but uh so the next issue in in the top five aws repo is nlb services with external traffic policy, local route traffic to nodes that cannot cannot handle it for a short time when a node joins the cluster.

B

So this is like a known issue, like known limitation from the nlp side, where, like the health due to the health check complication, they are. They apparently are healthy for the short period of time and eventually like when the health check kicks in. That's when the true uh health check status gets reflected in the target, so for now, like uh one way is to uh use the ip targets so that this limit limitation isn't there, and we also follow up with the nlp team.

B

We are following up actively engaging them uh to see what would be a proper fix for these customers.

A

Wonderful, thank you uh that sounds that sounds great yeah, so that was the last update effectively that uh there isn't an issue and uh yeah. That sounds. That sounds great.

B

Yeah we are aware of this and we are actively pursuing this at this moment. I don't have any other detailed information, but as as and when we have publicly available information, uh we'll update the tickets.

D

So that's great, why didn't the health check note port, like mechanism not.

B

uh So it's because of the nlp right so when the nlb instance uh join the target group like when the instances are added like for the short period of time, nlps internal accounting uh treats as healthy. So it's actually trying to prove the target and because of how legacy and lb work uh like they do send some traffic uh to the instance. uh So that's where this thing comes into picture.

D

I see so like it's like the default health status prior to probing the health health check, notepart.

B

I got it correct. They do send some like, even though it's unhealthy, uh like initially nlp, does send some packets uh just to prove and make sure like things are in order. uh So that is the reason why this consisting confusion is so it's for the short period of time. Initially, when the node joins the target group.

D

A

All right uh it I mean it doesn't sound like there's a lot, we can do there and it sounds like it is under control. So I will, I think we can move on to the next issue, which is uh sounds similar, but a little different. uh First provision of nl oops first provision of nlp ignores logging and cross zone. Oh no, sorry! This is different. This is kubernetes. I guess first provision first provisioning ignores.

B

I fix this entry. Sorry to interject, I fixed this entry. We probably need to get the fix ported to this repo as well, because it sounded like similar thing that I worked in the entry code. I.

D

Think they have.

B

Mentioned the pr as well here uh yeah.

C

I did a cherry pick of everything that had changed from entry up to a point but yeah, I think maybe I didn't. uh Maybe this came after.

C

So I can go back and look and see what what, if there's any other fixes that I need to import.

B

Is it a manual process like you have to look at it time to time or yeah yeah.

C

It's uh unfortunate.

B

Sure, if, if you need help, do let me know, I can also like whenever there's any fixes or any issues that I know of, I can also see if I can cherry pick.

C

Yeah I mean you could ping me or cherry pick whatever works, if you notice something, but um it's not super difficult. So it's not a big deal.

B

And I don't have a testing framework for this cloudtrader v2. That was the reason why I wasn't doing a cherry picker.

C

A

Okay, uh thank you so yeah, that's, hopefully that can get cherry-picked and then published. That'd be awesome. um That was the 28 days, which um I don't know. If there's anything else, anyone anyone wants to call out below the line the line is here.

A

I don't know if you can see where my mouse is, but the line is around here. um Otherwise, we'll I propose to take a quicker look at kk.

A

That one looks weird.

A

Okay, it sounds like we need to do more documentation, I think, would be the request there, all right, otherwise I'll have a look at or we'll flip over to kk.

A

All right flipping over to kk.

A

uh These are issues labeled with area provider, aws sorted by most recently updated um and we have looks like we have three that are updated in the last 28 days, uh which I propose. We have a look at and then we can do a call out for any others. How about that?

A

So the cloud controller manager doesn't query cloud provider for the node name.

A

Okay, that sounds odd, uh causing.

C

The node to be removed yeah, we talked about this one and okay,.

C

C

uh So um when you switch over to external cloud provider, um there's kind of three cases, I guess that a new cubelet coming up um can configure themselves uh with regards to this uh or three different uh cases that were anyway. So cuba can start up fresh.

C

You can have an existing cubelet when you upgrade or you can modify an existing node and upgrade the cubelet on the node.

C

So when we're talking about ec2 instances and when you modify a cubelet running on a node and change uh uh so so when when cubelet starts up and uh that the hostname doesn't uh agree with the node name, cubelet then can't find its node object and uh so you'll run into issues. um So I guess, if cubic can't find its node object, it's to stop posting updates and then eventually the node will probably get deleted and pods will get evicted.

C

But andrew mentioned that if you use the hostname override flag or potentially the provider id flag, then at the same time that you upgrade the cubelet, then um or I guess uh yeah same time as you upgrade the cubelet, then cubelet will be able to find the node object and everything will work. Fine when you create a new cubelet. um It should also that case should also work and when you have uh just an existing cubelet, um which already has uh it's no no name figured out that that should also work.

C

So this was the only case and I think there's a workaround. That is fine, um so I don't know there was some other discussion about potential changes to the cubelet, um but I don't think it. uh I don't think we got super far so.

A

And if I, if I do, have a new node, I do not need to pass the any flags other than top right or external.

C

A

C

Andrew can correct me if I'm wrong.

D

Yep, I think so yeah.

A

So that I mean that that seems reasonable, like it sounds like in the long term we're in a good place. It sounds like in the expected use case. There's there's no workaround, which is you know. We don't really expect people to keep nodes, upgrade nodes in place in general in kubernetes, especially on a cloud right so and there is a workaround, so that seems reasonable.

A

uh That sounds like you've put excellent comments on there, so that that looks great um next number. Two of our three external traffic policy, local on aws, does not work if the dhcp or the vpc is not set exactly to region.compute.internal.

A

Well, we definitely have seen that sort of issue and it sounds like it affects external traffic policy, local uh gossip cluster, I'm guessing they're running chaops. I don't know why it would matter to be honest.

B

But the comment the recent comment was like: it works on a certain version, so that was what the recent update was. I haven't had a chance to uh like recreate this exactly and verify. That's why it's still open.

A

Okay and it look, it sounds like it's around passing the hostname override to uh eks, I'm guessing um sorry passing the hostname override to cubeproxy, at least on eks.

A

It's long runner.

A

Okay, so it sounds like I. I don't know why the dns name is used, that's a little weird, but um must be a reason.

B

That's a default version default option right if we don't specify the override, that's what gets used so exactly.

A

B

A

B

So this is the queue proxy, as I understand so right.

A

Maybe it's finding the node somehow okay! Well, I'm gonna have a look at this. It sounds like people are looking at as well.

A

Sounds like it's common to eks and uh five years. Is it five years old? I don't think wait.

C

So when they say uh external traffic policy doesn't uh set to local doesn't work. What does that mean? All targets in that target group were unhealthy, even though the pod I.

A

Mean be fair, this is uh this is three years ago, so it may be that there are different symptoms over time, but.

B

Yeah, it could also be combined with the local traffic policy issues that we have seen. But again I haven't had a chance to like exactly replicate this.

A

You and I are already assigned to it. So that's I think, that's the right set of assignees, so that's perfect, uh but yes, I will also try to take a look at this one.

D

Generally uh so like there's a host name past the cubelet and then there's also the host you proxy, and so if q proxy is given a different host name, then it doesn't render local endpoints like it doesn't think an endpoint for some node is its own endpoint and that's where the local traffic policy would fail. Well, that would be my guess for this. One.

A

So it's about resolving the node or finding the node.

D

Yeah, like cubelet's understanding of the node name, needs to be exactly what the um name on the cubelet is, which on aws needs to be the private dns.

A

Yes, I mean the nice. This is like this is a good workaround as well. The using the down word api. I think it's called to pass in the um the real node name. That's nice.

A

um All right uh sounds like we should probably look at. That sounds like there's a reasonable explanation for what is likely going on, I'm just needing to figure out what, whether it's still happening, and whether our respective tools should just bake in effectively this, which actually feels very reasonable.

A

Okay, uh next and the the final one which we talked about another another similar one.

A

Health checks failed outside of ingress controller aws database nlb exact same issue as a previous issue check, which the same title: okay, uh but can't get it to work. I give exactly the same content, another customer. It does work general nginx.

B

A

A

A

I wonder if this is the same as no does he have external traffic policy, local.

B

Doesn't look like I did try this, but again I wasn't able to reproduce or get to the bottom of it. Yet. Okay, still in my list.

A

Okay, yes, that makes sense. I don't think that's all that we're just doing a triage, so we're just sort of like trying to make sure that we are keeping aware of these things and it sounds like we are.

A

uh Oh that's! So there is another one party. Actually, oh.

A

ah So it might actually be related to excellent traffic policy. Local okay. um Is this the same person jt weaver, who filed this originally now rice, bowl, junior.

A

Network moon, fish.

A

Oops yeah, it sounds like it might actually be related to the other one which would be nice obviously, but because external quality logo, certainly original traffic policy, has definitely come up a lot recently. ah That's interesting all right.

A

That's a good, uh oh!

A

I read that as a huge salary.

B

uh Okay, oh so, this might also be related to the vpc uh like.

D

A

Yeah it links back yes,.

A

The the external traffic policy with the node name so.

B

That would be nice.

B

I see okay, more incentive to get to the bottom of these two issues.

A

Exactly all right, I think that and that that that would actually sort of make sense and that that might um because I don't think the original reporter in this issue called out external traffic policy, uh but they may have had it anyway.

A

I'm not sure is this a home chart. Oh.

D

It's a helm chart. Oh.

A

B

What do you have there? It is.

A

Good news all right, so why don't we mark as a likely dupe.

A

A

Punk is that right, 6146, yes, got it all right.

D

That's good news.

A

All right, um those were the issues that we wanted to do, based on our 28 day, rule 28 day rule. um I don't know if there's any others that people would like to call out I'm going to move the line there. We are, that is the line.

A

Otherwise, we can give people a couple minutes back.

C

I think we made some progress.

A

I think we actually did.

C

A

Like that's like they're, in reasonable shape, it feels like there's a couple of issues which are uh like tractable as well. I feel like we have a good grasp on them or beginnings of a grasp on them all right. Well, I, if there's nothing else, then I will stop the recording and wish everyone a very happy weekend.

B

C

Everybody thanks.

B

Everyone bye-bye bye.