Kubernetes AWS Provider, 29 May 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes - AWS Provider - Meeting 20200529

Description

Recording of the AWS Provider subproject meeting held on 20200529

A

Hello, everybody today is friday, may 29th. uh This is the cloud provider aws bi-weekly meeting. I am your moderator facilitator. Just in santa barbara, I work at google a reminder. This meeting is being recorded and will be put on the internet on youtube shortly uh and to therefore please be mindful of our code of conduct, which essentially is to be a good person. uh We don't have too many people here today, so we probably don't need it, but uh please do raise your hand if it gets contentious.

A

I pasted, or I am pasting a link to the agenda in our chat- um feel free to add items. There feel free to add your names on there. Although I think almost everyone already has. We have just three items on the agenda, so it might not be a full length meeting.

A

um The first one is one I added which is actually about.

A

Two weeks ago we also had a discussion about labels um and it's another. One came up in sort of cops, but uh it's really uh an atom from pusher that handles spot termination, notifications and it expects a label on the nodes which it uh should apply to. I'm not sure this is actually needed in that. There are also other notifications that apply to non-spot instances, but I think the challenge was that the pusher annotation was a node role uh which is controversial.

A

It isn't scoped to kubernetes, sorry, it doesn't scope to aws in any way, um and so I was wondering whether we wanted to try to figure out a better uh or a more aws specific uh label that that we could sort of agree on. I feel like this might be a better topic for a cloud provider, the state cloud provider like overall meeting, because then we could try to establish some hierarchy and say: aws gets this and gcp gets this type thing.

A

But um I don't know if anyone had any initial thoughts, but otherwise I was going to go to sync cloud provider on wednesday and uh propose some something along those lines.

B

Yeah we had our meeting this week, so it'd be the ones after, um but my initial I mean I I don't I didn't. I don't know how pusho works and how small termination happens. So I I don't, um I can't say much, but my initial gut feeling would be. um If it's a aws specific behavior, then it should be an aws, specific um label or annotation um unless we're, unless, like all the other providers, are saying like. Oh, we have the same kind of use case, so maybe we do need.

B

We there's a use case for like a kubernetes um space label,.

A

Yeah and two weeks ago it came up with, I think, machine type and machine family, where it's even more like so spot is you know most people have some most clouds have some notion of spot or preemptable instances, but uh the machine types, everyone everyone seems to have a notion of machine type, but the values aren't uh comparable. So it's it is it's going to be a fun one. I think, to discuss there.

B

Yeah and like the contentious thing about like node rules, is that, like that specific label is useful because cube ctl will read it and it'll it'll show and like in the get nodes output like spot worker and so like, that's obviously helpful, and if you used an aws specific label, you wouldn't see that yes and.

A

I think I think users like that, but I think the people that dislike snowdroll in general would presumably dislike us sneaking more in there.

B

A

So I mean it sounds like we in aws feel like a uh some sort of aws. Specific prefix is probably a good idea. Is that reading the room and we should take it to cloud provider and hash out some of the details.

B

Yeah sounds good, I mean like my feeling is like if you don't want to be blocked, then or like, if you don't want to go through, like um people like fighting with people to allow this kind of thing, then probably just that, but I do see like a really valid use case for using the node role label. So I mean, like I think it's worth having that discussion for sure.

A

Yeah I mean I I feel like we, uh we don't have to block people from setting the node role I feel like. We should avoid requiring people to set the node role so and pusher doesn't even the pusher termination handler doesn't even require it. It's just like it's the default value so yeah.

A

uh If there's nothing else on that, one andrew you have the next topic around the dark side.

B

Yes, um so at the bottom, I linked to the actual doc site that um nick and I set up. um It is literally like it's blank like it just says.

B

Like the you know, kubernetes aws provider has no content, um and so I had opened a pr um that kind of outlines the most basic um kind of layout of the doc site that we can start with, and then we can maybe create an issue or do a call for help um to see if folks are willing to like take parts of exceptions of the doc site and and open prs to help us put that together. um So I think yeah it'd be great.

B

If folks can provide feedback on what kind of sections that they would expect from our doc site.

B

I had a few ideas, but I'm sure there's people have opinions on what a doc set should look like and I'm hoping we can solidify like the layout before the next meeting, and then we can start having confiscation about how to get folks to help us here.

A

Yeah that looks really good and uh the other one which occurred to me is the cloud config. I think it's cloud configurable or cufflink config ini. I don't even know what it's an image.

A

That's that uh I hate that I just don't know. If we, I don't even know we should document it, but I mean I feel like we should document it, but like carefully describe that it is advanced and like yeah sort of. But yes, uh but I think this looks really good that I imagine the service is gonna, be the biggest one.

A

uh The service load, balancers.

B

Yeah and definitely like documenting some of the gotchas, with like private host names and all that stuff, so yeah, okay, so yeah, please take a look at the pr and um provide feedback on just like yeah, again, like mainly just asking for like what what this the headers of the website should be.

B

Okay- and I have the next item too, so I can go over that quickly. um I had merged a pr the other week um on the staging project to so that we can have an official um gcr image repo for the external cloud provider. So I'm going to work with nick to set up a pipeline that pushes a tagged release on the external cloud provider repository to an image and then we'll update um the sample manifest. We have to deploy the auditory provider, I'm using that.

A

That sounds really great. Is it? Is it going to be a? uh Is it going to be a uh a pro or gcb type job, or what is what do you think you have in terms of the mechanics.

B

It'll probably be a proud job, um just because you know it works and we all like. We have references to cetera, but we may um I'm working with nick, because there may be a case where we want to push both the gcr image. And you see our image that the folks at eks manage. So um we need to figure out what that might look like.

A

That would be awesome. Yeah I mean, I think we uh wearing the work group kits in I've been sort of involved in some of that, and um I think that the intention was always that at least the promotion should be able to target.

C

A

I don't know whether it's important um that isn't implemented, but we need like. I think it was certainly like a designed in the back people's minds as they were designing it, and so, if it's uh that the promotion at least would be able to promote to ecr and other repos and registries and similarly on binary artifacts that would be able to promote to gcs and s3 and other equipment storage. That's really good.

A

It's just that no one has yet to implement it. I don't know whether we, I don't know whether anyone cares about having staging on ecr gotcha.

A

And dear, did you want to mention you put in chat about uh the importance of documenting the indie file? Sounds like a battle of scar.

D

Yeah, it's just because uh people can set passwords with different characters; it turns out in uh yeah, so you can't encode that correctly bad things happen, at least on the vsphere provider.

B

Yeah gets interesting because, like in the in the aws provider, like you, don't technically need the ini file like you can get things working without it um as long as you configure like the tags and everything correctly, whereas with vsphere like it's a hard requirement because of the credentials so that'll be interesting to.

E

Talking about that sorry andre, I missed the beginning of this.

B

Conversation but.

A

E

The credentials for gcr at this point not yet not yet: okay,.

A

You you have them you just they you, don't you don't ever.

B

Get I just set it up.

A

To whatever yeah.

B

But like I don't personally.

A

Have the credentials.

B

Because and and that's that's intentional with this design.

A

I actually think you do uh you, you are able, if you, if you're as you are one of the owners, I think you can push to staging using your she's. Your gmail account. Oh okay, if you log in that's my.

D

Understanding and the so.

A

The intention is, staging is not supposed to be a security boundary. The promotion is the security bound chocolate. That's my understanding anyway now, but we should push using.

A

uh Do we have anything else on the agenda? I'm just checking switching tabs here we have nothing else on the agenda. I don't know if anyone else has any other topics they want to bring up or whether everyone gets a whopping 45 minutes back.

D

A

D

A

Let's see, uh let's see what we got, that's not a bad idea. uh Let me I will try to present.

F

Let's see all right.

A

I believe you can see uh the kk repo.

A

Can you see that kkk, yes, yep, okay, good, all right, uh we, I don't know what query we normally use, uh but I think we normally use something like this nick. You can keep me honest here or I don't know. If nick you would rather drive this as you've done this in the past.

E

Go ahead, um yeah, either recently updated or um kind of switch off created like the most common stock was recently updated, but I think recently there's.

B

E

There's an area.

B

Provider, aws label.

A

Yes, I just wasn't sure whether they all actually got tagged with that. So I.

A

A

Was it label area provider aws? Otherwise, I have to use the drop down there. We go.

A

That doesn't feel like they are all getting labeled. If the most recent one was 22 days ago, okay, I will flick back to the uh the overly broad one, so we can sort of try to make sure that they get um okay. uh I guess we should go from okay, actually there's a gap, so here we go so uh if we go from, I guess anything newer than a let's look at this one, because it looks interesting and then we can work our way up.

A

I guess aws error creating security policy on load balancer when using custom, ssl negotiation policy. This sounds like another interaction between multiple of our.

A

Multiple of our various annotations.

A

And I just don't know if anyone else can at least tag this with aws, because this is clearly aws.

A

uh It certainly doesn't it looks legitimate on scanning and it looks non-trivial, so I will leave it open. I don't know if we want to do anything other than that, I think labeling is a good start. Okay, uh next one up, node port service creation e to e test relies on insecure class configuration. That sounds more like a testing issue. So that's fine or not really an aws thing. Aws doesn't support local zone availability zones.

A

Is this like outposts? Does someone know.

D

Nick, what's you can create zones which are like city specific? uh We we need to bump the sdk at the very least, then there's probably some other bits and pieces. I don't know.

F

A

uh Down for everyone- or just me, I wouldn't imagine this- is that contentious that it's gonna have thousands of comments, but we will see uh exit.

F

Try again all right.

A

It's very clearly aws, so at least make sure that it's labeled with aws.

F

uh Why is that.

A

F

A

Would already use all right.

A

uh Let's try a different one and come back to that one: okay, just a bad one. Let's just try reopening it again.

A

Okay, there we are here, it comes all right, uh advance releases, local zones and we probably just need to update the sdk and easy to region, oh because they have a different format.

E

Okay, yeah, actually so jeff wan is my co-worker he's working on this. I believe ah that's great.

A

Yeah, yes, I think that'd be that'd, be great to fix uh um yeah, I'm just trying to think whether there's anything other we anything more. We need to do beyond um like updating the the region inference I mean it is still a zone. It's just a more localized zone. Is that fair.

A

All right that seems like a a good one. It looks like it's been triaged. I think we could put it in the milestone. Is that a good thing to do.

E

Yeah, I think um it's a reasonable task. 119 yeah. We can always.

A

Bump it, um but it's since it's actively worked on. Oh in fact, there is a pull request. Yes, perfect, all right! Well, I will have a look at that pull request I'll make sure I'm assigned to it.

A

Everyone's assigned to it, you're assigned to it as well: nick, uh okay, uh the next one on the list was this uh read-only nfs based pv pvc allowing writes to the file system?

A

uh Okay, I don't know if this is data specific in any way, it doesn't look like it. So that sounds fine. I don't know where even it was appeared, oh it's on either s okay, so that is not our plugin, that it's correctly labeled and I will close it out or close the tab.

A

uh Next, one pod, with huge pages, get stuck in state terminating state on deletion, doesn't sound like an aws issue, again, probably just running on aws yeah or maybe not even running an address. So that's not even an issue not able to use the desired security group when can when creating load balancer from kubernetes.

A

That feels like an aws issue. It is an aws issue, so this labeled and again our annotations, created the load balancer.

A

Install tool is blank. Let's, I cannot read that there we go uh the error revoking.

A

Okay, that's a little weird! uh It might well be a tagging issue or something um I suspect, oh because it's internal.

A

That's interesting, I don't know if anyone has any ideas on what this is right off, the bat can.

E

You go back to the air, I couldn't see it sure.

E

E

A

It feels like it is appropriately.

E

Labeled, sorry, I was gonna say: can you sign me this one I'll take a closer look thanks.

F

Nick all right.

A

A

Actually wasn't that was that, uh and that was just an internal limit, so that does feel surprising. All right uh services should be able to preserve udp traffic when something this sounds like pure networking. Nothing to do with aws, probably.

A

uh Yes, I don't think this is gonna have anything to do with aws.

A

Yeah uh or not, on the surface abs, multiple low bouncers in the same cluster compete with each other.

A

Doesn't necessarily feel like an aws issue, maybe a cloud provider issue. It is labeled as such. It's definitely a network issue.

A

uh Can't access service port that doesn't sound like an aws issue. That sounds like a docs issue.

A

9080, but can't access, 80 90., it's making me read very carefully. uh Let's see they are.

A

Okay, I don't think this is a native issue and it isn't good.

A

uh Can't access sort of support, hpa with external metrics scales incorrectly, doesn't sound like adidas again, but we will just have a quick little look. Yes,.

A

A

Nothing really to do with aws per se as far as we can tell based on this on the surface uh and it's a scientist skating. So that's good.

A

Okay, sorry, my machine is being a little slow at times. I should beef up my uh streaming machine uh pods on unreachable cubelets are deleted, not evicted really doesn't sound like aws, but we can click through.

A

Yeah just appears to be only that it's uh running on aws, but not an aws issue.

A

uh Cubelet should handle node shutdown, ah that's amusing, so actually I'm going to uh so. This is related very related to the initial uh topic that we had around these labels and hooks, and I know there are- I was actually looking at this uh with a cops contributor this week. uh There are a large number of these um uh hook providers. I guess you'd call them.

A

We should probably think about doing something here. I'm not sure I would call it a.

A

Andrew, what do you think about labeling this cloud provider and having the discussion there?

A

He dropped ah then I'll. Do it yeah just go for it. I mean, I think I think there are two things right there's um when we get a shutdown request, we should do the best thing we can, which would be a node issue and there's this sort of behavior when, uh like apparently gce or aws, give us notifications that things are going to like a 30 second or two minute warning.

A

Then we can do something smarter there and that one, I think, would be different and I think that one would be cloud provider.

A

I guess and I'm going to write a comment as to why.

F

Shut down signals still a note.

A

Okay, priority based preemption can easily violate pdbs, even when necessary, due to multiple issues with the implementation doesn't sound like us in any way, we'll just click through and double check.

F

I should probably close some of these tabs.

A

Okay, uh yeah, it's on eight of us.

A

Okay and finally, in our most recently created issues that mentioned the word aws that are not necessarily so tagged, cubelet neglect to mount all, but one of a pods csi volumes of the same volume handle, even though they have different volume attributes this one's going to be trickier.

A

But it looks like a storage issue and it is tagged as such, so at least on the surface, although yeah it doesn't seem to be, there's no accusation that it is aws so uh specific. So uh we can leave that as it is he's also on my team, so I think he would have tagged it with provider aws if it was awesome, great and so now, if we want to, we can have a quick look at uh labeled issues, labeled aws that are recently updated and see.

A

If there's anything, we think is problematic, uh skipping the one two three that we just went through aws friendly elb names. I feel like we discussed that a lot I don't know, there's lots. Do we want.

E

To remove life cycle rotten on that one if it's still relevant.

F

A

Sorry, I have to sort of sneak in under the.

A

uh How do we mark it as.

A

Is there a way to mark it as like perpetual, like? Never don't rot frozen right.

B

Life cycle life.

E

Cycle frozen, I thought, give it a try.

A

Looks good! Thank you.

A

Okay, uh the external traffic policy. Local on aws, does not work if the dhcp of the vh of vpc is not set exactly to this, that is surprising, uh like in a triage. We don't answer, we need to address them all. It feels like that's progressing.

A

Are there any people want to call out, I guess, as things we should take a look.

C

C

Good but it's been open for a year uh and uh code freeze is coming up according to a recent comment, so I figured it'd be worth bumping or something.

A

Yeah last time we looked at this, I think there was it was uh ah well. I was gonna say last time we looked at it, it was maybe not ready to merge and that we thought there might be a conflict, but it looks like that might have been fixed, so um I will yeah that that map is what was missing before. I think uh I will certainly have a look at this uh nick. Do you want to look at this as well.

E

Looks like we responded to the bottom and and got a response, so we should.

E

Double check what the latest is.

A

I'll certainly add myself to the assignees here. This is one which we get to get in. Can you.

E

Add me as well.

E

A

Five assignees: that's a lot uh nick nick, were you uh yeah thanks peter for calling attention that one nick were you about to say something about the a lot of them being about nlb labels or what we gonna say.

E

I was just gonna say that I recognize um like these few here from last time, but um that doesn't mean we shouldn't take another look at them. Looks like they're, recently updated in the last few days. A lot of them yeah. Is that one we should start with you think or.

E

Looks like there's a couple feature: requests there and then yeah. We can take a look at these like host name, overwriting snored. That.

A

I think we know about.

E

That this one, I just thought was interesting.

A

um The leads notes before they become ready, sorry for the.

F

F

A

A

Oh yeah, this is very old and there's a suggestion so yeah, I think andrew's, asked a question and that's probably leading to a close, I'm guessing yeah.

A

Okay, so yeah, that's that's uh waiting on input. So that's the right place.

A

um uh Anything else: what should we hit next.

E

Can you check um elb security group not cleaned up when extra specified? Yes, I remember that one.

A

All right, so I think we've talked about this one before so. Let's go to the bottom and see why it uh I've got a me too uh who's assigned this moon fish is assigned this.

A

Sounds like it might be related to the other one as.

A

E

Can you sign me as well and I'll just take a look after this.

A

Yeah I'm also interested in this, but I'm not trying to have time in these two weeks. So, but if you see anything interesting in there, let me uh let me know if you find, if you get time to have a look at it cool yeah. Thank you.

A

uh Let's see any other ones that we think are why don't we focus on the bugs? How about that? uh That's a label kind of.

A

uh These are the ones we're seeing our own shadow here. Okay, so we have this external traffic policy local on dhcp. Should we see why that got pinged, because that.

A

Sounds particularly like a particularly tricky combination.

A

ah Okay, so there is a workaround.

A

Okay, so I feel like that's in a good state right now. um I can have a look at this. uh It sounds like there is a resolution, so.

E

Is there a resolution in 116 and above and or what is that? Where is the so there's this workaround, uh which.

A

Is up in the same way there's a fix apparently, but.

E

I'm wondering what version that fix was in.

E

Let's see, thank you.

A

uh It sounds like the the fix is the that the fixes that the work around I think I see thank you, I think they're the same backlink, so it sounds like.

A

Yeah, I I'm happy to have a look at this and then it sounds like you know. We can maybe do something in cops like maybe we should to the docs, but we can at least figure out what what the step is.

A

Seems like, and then we can like get into eks and kappa, maybe yeah, it's a pretty. It's a pretty rare combination. So actually is this: what is this? uh What cluster is this? This is a uh ah it is a cop cluster, see I'll take a look or the rig. The op was a cops cluster. That sounds like it's on other other providers as well um hostname override ignored when cloud provider is specified. We are pretty sure that I'm pretty sure we've seen this a lot. It's just true.

A

Yeah I'd like I'd love to fix this at some stage. This is the whole like you're not allowed to. uh The name of the node has to be the private dns name which we'd love to fix, but got stuck on storage. In my vertical correctly last time, uh the storage controller uses node name instead of using the node.

A

uh So it was actually happened there. Oh yeah, someone removed the label, so that's good. We could freeze it now. You've taught me the trick.

E

So what it, what is your understanding of the state of this? It's something that people want, but there's no. Are there any proposed solutions.

A

Or yeah, so my understanding is, um if you try to pass a hostname up, this fly called hostname override, actually should be known, node name overwrite. uh You can pass it to cubelet and elsewhere. You can change the name of the node that cubelet will create uh if you're specifying a cloud provider. Apparently this doesn't work and in particular it doesn't work on cloudwriter aws, uh the reason it doesn't work on cloud provider aws.

A

Well, the we can't support it on cloud provider aws, because unless you uh register with the correct your node with the correct uh name, um it's not gonna, like the aws cloud provider, is gonna not be able to reconcile your nodes to your ac2 instances because it matches them on private dns name. um Therefore, yes, we do not support it.

A

It is ignored because if we were to accept to listen, it would be bad um if, if we can change the aws cloud provider to take as input always a v1 node instead of a string, node name, uh then on the on the node object. First of all, that's a good change anyway, because the node name is not a real identifier um like it can be recycled uh and it is.

E

Recycled on so so take in a node in the inner in like the functions, but also be able to query using more than just private dns.

A

Yes, uh yes, um taking in the note on on the note object, there is the instance id anyway, so it it also means we don't have to do a translation, um because we we have the node the instance id right there.

A

um We could try to maintain our own mapping, but the fear is that the like, if our mapping got out of sync, it would be bad, so uh yeah, the the the short answer is change all the cloud provider interfaces to take uh v1 nodes and then we can no longer require the private dns name, and then we can allow you to set whatever name you want for the nodes, every whatever different name. You want for the nodes, and then we can acknowledge the hostname overwrite flag.

A

It's less less bad than it sounds like it's actually fairly mechanical. We just haven't done it. I think it's because it crosses six. I guess that's another one to bring up in cloud provider in two weeks is like. I saw an email around a proposal for an instances, v2 interface, that might be a good one to put in there.

A

Is that a fair reason to keep it frozen.

A

Okay, should we keep going or do people want some time back.

E

uh I'm gonna have to drop in about three minutes.

A

E

Right, let's, let's.

A

Do one more and then we can, uh let's you want to pick a a doozy.

F

A

Nadir do any of these cause. You problems, that's not a great one, but not that I know peter any of these.

A

Nick uh this one sounds really bad. Actually uh incorrect could be deletes the wrong security group, so, let's find out how incorrect it is, and this can be a good one to end on. I think, if my poor little laptop.

A

Someone did it with a uh ipad the other day, so maybe I should try that.

A

A

uh Running a service load balancer type aws. After deleting this service, the cleanup process enabled us to remove the wrong ingress input rule which provoked that all traffic between nodes was unable to reach out destination.

E

I actually did a bunch of investigation into this one and I was able to reproduce a very similar issue, but not their exact um issue, but it looks like this person says they have a patch for it. So we might need to look at that.

A

Yes, ah okay, oh and there's a pr now from uh what I presume is alberto, but.

E

Okay, I think I should already be assigned to this so I'll just have to look at this. Do you want to also review the oh cool yeah? You want to add me and then also review it.

A

Okay, yeah: we should definitely have a look at that.

A

Yeah we actually, we should probably look at the uh prs that are open. uh uh It's not tagged with aws, that's okay,.

A

Yeah, I think I'll go through some prs as well see if any are as we're coming out to future free see if any error um tagged aws, but not uh sorry, I'm not tagged out of us, but you know try to find a name with prs that should go in for feature-free, so that we can hopefully close out some of those issues. That way. That would be an easier way to do. It.

E

Awesome I'm going to have to take off um but see you guys great singing.

A

Nick uh should we call it time at this point cool all right. Well, thank you. Everyone! uh Everyone gets a bit of time back and I wish everyone a very happy friday and a very happy weekend.

D

A

C