Kubernetes AWS Provider, 26 Jun 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes - AWS Provider - Meeting 20200626

Description

Recording of the AWS Provider subproject meeting held on 20200626

A

Hello, everybody today is friday june 26th. This is the aws cloud provider uh meeting by the team meeting. I am your moderator, associator justin sung oprah. I work with google. Okay reminder: this meeting is being recorded on the internet that I I am pasting a link in the chat uh to our agenda, uh which uh is currently relatively empty. So please do feel free to add your name or any items you want to get to uh on that and we'll be sure to do that.

A

um Otherwise, I think the uh broad plan is to do a demo from the deer and then.

B

A

Through some issue triage, um if there's nothing else, then why don't we handle.

B

Foreign, hello hi, um so I was supposed to demo uh class api aws, but um done that a few times now in previous, so one of them repeat over old material um for one of the things we've been talking about is uh ways to test the external provider um and maybe using cluster api to test that. So um we were having uh conformance errors in uh the cluster api provided aws repos.

B

So I started um rewriting some of that conformance to use a new test framework that was developed by mainly fabrizio into cluster lifecycle, um and it wraps a lot of the sort of things, the most common sort of test scenarios and I sort of extended that to also handle cube tests so I'll just go through how that works briefly and then show you what the output is, and we can um I'm going to move this to cluster api properly. um So it's reusable by the other providers.

B

um The plan is like we could start using this for to test csi cpi um and it just wraps sort of cube tests and gin coats. So it's a it's nested jinko, but it works mostly. um Oh can you let me share a screen.

A

Sorry about that, just one second, uh let's see, I will make you a co-host. uh Yes, I do, you should be a co-host. I think you should share a screen. Is that true? Yes, perfect.

B

Yeah, so in uh our class api repo, we have a uh test and it's sitting in this e2e new. um First of all, the main thing you do is um you provide a configuration file, and um this says what images are going to be used to bring up the cluster, um so we're going to use cluster api 2a36 um for the purposes of testing class api provider aws. We actually generate an image and then inject it. So um the way the test framework works is it brings up a kind.

B

Cluster uh injects images into that kind cluster. So if you've got any local images that you haven't really pushed to an external repo, you can sort of rewrite them on the fly. um So that's useful when, when you're just making changes code, you just want to see what the impact is. Then the test framework can load them in. It does string replacements on the manifest for the deployments. um So we turned off the leader election just to save a bit of time um changed the metrics, so it's um globally readable.

B

So you don't need to get used to our back proxy um and then there are a bunch of variables. uh Some of these are hacks right now, because of bugs in cluster api and the.

C

Way that works.

B

Is um each provider provides templates and the templates uh basically allow you to do string substitution in them? So this defines what the cluster is going to basically look like and then um we add some extra variables in the test suite to um say how many machines it's gonna use, so we're gonna use five machines.

B

um This bit here is saying we're gonna start with a 118 to aws ami um and we also add some of the scripts. The injection that's been used to inject the latest ci release of kubernetes. So that's where it looks up the latest, ci artifacts and then downloads from the um google bucket the that version and make sure it's present before um plus the api brings up the cluster, which is using cube adm underneath um and in terms of the um test suite itself.

B

We have some glue in the cluster api repo that sort of brings up um brings up. The cl brings up the aw scout, so we uh check out an account from boscos, which is the testing for bits that hands out aws accounts.

B

um We there's uh there's utilities to run up the cloud formation templates that you need to use that support the cloud provider. Integration and cluster api- um I we've added some stuff. We need some changes in infrastructure, but we. What we have right now is the ability to use aws session manager to open up a shell on the machines and run arbitrary commands that we can output to files.

B

um So just as a starter, we've just got like getting the kernel log system d logs and like what is the virgin container d. I executes pretty quick quickly, it's about three seconds per machine and we just have that running every minute.

B

So just getting that all the time um and then to actually do the test itself, um we have a function in the um test framework which allows you to apply a manifest and that will bring up the class that essentially then just waits for that cluster to be ready, installs, the cni um which you can swap out as well.

B

You can say where that cni path where, where that many manifest should be that gets loaded in you, can instruct it how long you're going to wait for the cluster to come up and then um finally, we can't remember: oh yes, um this is the bit that I'm going to put into cluster. Is this cube test framework? So um you start off by just giving it a new configuration.

B

It will look up, add stuff to the command line flags as well, um for your test for cinco, and then um one of the things that you pass in is the cube test.

B

Configuration so um cube test will accept the yaml file, so we just pass that in as well. So you can use different configurations if you want to do different bits of testing um and then it essentially just wrapper around it. It constructs the command line to run cubetest and creates a report directory in terms of what you get back. um You get this artifacts directory in it. You have all of the controller logs from cluster api and metrics from them.

B

uh You get the logs for each command. That was run on every ec2 instance. um This is where, if you have ssm session manager, support on your account, um if it doesn't it's just not going to appear or it will uh write an error file and that's currently what's happening in power at the moment- a list of all the resources on the kind cluster um and the cloud formation template that was applied, the e2e config that was created. So uh one of the things that the cube test bit does.

B

Is it um if you specify to use the ci artifacts it actually modifies all these kubernetes versions as well, um and then finally just moves the cube test output into the white place for proud to pick it up and send to test grid.

B

um So I'm hoping that's going to be an easier way to start testing things um if, when um using cluster api, and we can use it elsewhere in the communities project, so um yeah, that's what I'm on this show.

A

uh Super cool, thank you dear. uh I don't know if anyone else has some questions. If not, I have a bunch of questions I'll kick us off. How about that? uh Are there any? uh So, first of all like this looks great where it, where is this code going today and where is it going to go type thing.

B

It's commonly an open pr in cluster api provider, aws um and you'll be able to consume it directly from there and then gonna make a more I'm gonna move the cube test wrapper to the cluster api repo itself.

B

um So it's usable by azure and everyone else.

A

Very cool, the other question I had is: are you doing any testing uh other than the uh like kubernetes e to e testing like? Have you figured out a way? This is what I keep struggling with. Have you figured out a way to run your own tests like I'm, it's very difficult, or it's not even easy to do within group tests, but maybe because of the way you're wrapping things you have a way to do so.

B

um We do already not using the text so in our old tests right now. We do things like um create too many vpcs and then test the behavior. What happens um what well? How does kappa respond? Does it ever report? Does it do errors correctly?

B

So we do have a bunch of tests around that as well, and it's just metal constructing you can get a the test framework, can give you a proxy to the created cluster, and so that will give you a client set for you to go and do it gives you a cube config essentially, so it gives you the config of the class that's created, and then you can do whatever you like, um and you can use the aws api separately within your your test to do test other bits and pieces. So yeah, that's the idea.

A

Awesome I'll take a look at how you did that. That's great! Thank you. I don't know if anyone else had any other uh questions. Feedback just rounds of applause.

A

All right well, thank you. Thank you that was hi nick. I didn't see you, there switch views all right. Well, thank you. Dear. uh Let's see, if we have anything else on the agenda, it doesn't okay, anyone's added anything to the agenda. So uh what I propose we do is we go through a little bit of the um backlog of what has happened over the last two weeks and any issues that are sort of on the back burner?

A

I guess we could look at previous issues as well uh our previous topics, um but let's see how we're doing I will share what I hope is a window. Others will make a mess with everyone's screens.

A

Okay, let's see if this is the right one. Am I sharing a uh github issues view which is scrolling?

A

Yes, yes, good all right, so uh I thought we would start with our normal, uh like looking for issues raised in the past, I'm not sure with this two weeks ago, so we should do like any issues like raising the last 28 days that are uh potentially aws, and so all I've done is just do the broadest possible search which is aws, and then we can go through and make sure that they are tagged appropriately.

A

um This one looks like it might be: aws specific that one definitely looks like it's idiot specific. I bet he's tagged already, so actually that's fine. How is that one again.

C

The one about the credential provider- um it's probably something we definitely want to talk about. I know it came up problematic.

A

This this one, nine, two, nine two, one, six, two cool: let's, uh that one seems a.

C

Little urgent, maybe we need to.

A

Yeah, okay, uh all right, we don't necessarily need to let's, let's go that way, then, let's uh we can do this pre-triage, we'll just scan through them. uh Definitely this aws credential provider causes twenty tens. uh Ten to twenty seconds. Hang up is definitely interesting uh that one, I think, is well known and not even output flooded, that seems fine or like not uh critical and then uh yeah.

A

Ooh that one looks nasty so we'll open that one. But anyway did you want to talk about this andrew.

C

Yeah I mean I, I don't have a lot of context on it. I know that it came up in some channels, um but basically because the cubelet does more like the dynamic loading of the credential provider. um So like it doesn't matter what you said as a cloud provider or anything um um it it loads, the credential provider based on.

C

If it has, if it sees well it loads it always, but then it it doesn't enable it until it sees um the aws like uh certificate or credentials file on this on the system um and so like that could be nice in a way where, like, if you're pulling from ecr from like another cloud like that, could be useful, um but because it does that, I there are certain scenarios where like um like it.

C

Could it could it could freeze the cube from loading um because it automatically tries to load the credentials, uh the credential provider for ecr? That's as much as I know, um but it sounds like if we're if we're freezing the cubelet on startup for when you're, not using ecr, that seems problematic.

A

Yes, uh so it looks like from this logs. It looks like it's it's in in between three and four here or oh yeah, three in between three and four here sorry, I was like what happened to four uh in between three and four so specifically on this getting the credentials which I assume is actually like pulling the metadata service. I presume.

C

I think it was trying.

A

C

Yeah, I think it checks like the local keys first and then it tries the metadata service. If it doesn't, I don't know exactly yes,.

C

But like, why does it, why does it freeze the cubelet like it should? Just um I mean it should just asynchronously frees up right or it's a asynchronous error when it tries to do that. um I think maybe maybe somewhere there's something we shouldn't be doing.

A

Yeah, it sounds like we might have changed it to like, being the somehow be synchronous right.

C

Yeah, like awesome, there was a big refactor of the quidditch provider in 116, so it's possible that changed yeah. That's that's the pr! Okay.

D

So wait: do we know that if this is a regression or not.

D

C

um Yeah, I I I didn't dig into it to check, but it seems like it's a regression based on that big refactor. We did so. I don't think it existed before the before the um the refactor. We did there.

A

uh In terms of the I haven't followed that cloud provider extraction effort here. What are we doing around uh extracting these.

C

D

Great questions.

C

D

Go ahead nick, I was just gonna say so, uh like the the credential, the providers themselves aren't changing much, it's just um how they get called. That's changing. So there's like a you know. It allows you to actually separate that code out have a config file which tells you which providers you have which binaries you have locally.

D

So unless we explicitly go and fix this, it would be the same.

A

Cool. Thank you. Yeah.

C

Yeah, I think someone just needs to take this and just figure out like uh why it's why it's hanging up like I don't think we need to figure out why the credentials don't work, because that probably an environment, specific problem, but someone should follow up and see like okay, like if it's erroring their um does the cubic code need to change or there's a provider code. You can change so that it's not hanging up the entire keyboard. I think that's the thing we need to resolve. First.

D

Yeah, I can just do some investigation and figure out where it's what's taking like what's taking time and then just I'll comment that, on the.

A

Issue, well, I mean yeah, presumably it's not even in this. uh It's that we want enabled to be a called asynchronously right.

A

That's not trivial, because I don't I don't know yeah it's difficult. We almost. We almost only want to call it. We want to call it lazily. I guess, like only when we first see our first uh ecr image, which itself isn't an easy thing to now. I guess, but.

C

I thought it does that, like. I thought that keyring does only call it if it's if it sees the first image, so I'm maybe enabled as a different code path, but.

A

Yeah, okay! Well, we can have a look at that. So yeah, nick you're gonna have a look and I'll I'll try to take a look as well, but yeah yeah.

D

That's good awesome.

A

All right, uh other than that, uh I did pull up this one.

A

uh That looks just like a csi potential issue, rather than anything aws specific other than it mentions bfs as their particular problem.

A

Okay, uh all right uh are there any other issues anyone wants to call out. Otherwise we can just sort of go through like prs and stuff uh in sort of the more normal order.

C

Okay, so there was one. Actually um I don't know how new or old it is, but um I'm getting pinged a lot about like udp support in the aws nlp yeah.

A

We have a is there. Is this a pr? There is a there's, a pr right.

C

D

Yeah I had uh pinged my co-worker to take a look at this one. um Is there.

A

Any action on it they're, actually I I see there are actually two there's this older one. Let's see if I presume.

A

I presume we're actually talking about the newer one, because I thought this one needed a rebase, for example, uh and some other stuff, uh so.

C

So the problem is that I guess, like the provider code just is doing like it does a protocol check and it only allows tcp. No, we just just you just have to you know, allow both happening.

A

We have to elaborate, and we have to make the map uh some of the port maps service aware. That was. That was a if I recall correctly, this previous one, uh at least initially, but at least initially fell uh on that hurdle. I guess this looks better in that it has the uh yeah the mapping protocol yeah.

C

There's also another there's also a cap around, like so service type lb. um You can't specify mixed protocol port list, so it has to be all tcp or all udp. um Then there's a cap to allow both so does does nlb allow you to. I imagine it does, but just double checking like this yeah yeah, okay,.

A

I'm pretty sure it does the the restriction that you mentioned, who, where does that restriction, come from.

C

uh It's in the um api server validation for service type loan, bouncer yeah. It turns out it's much harder than we thought to just like. Remove that validation check, um but.

A

Yeah, I'm surprised, that's there. I thought I thought google's one's also. I don't know who knows, I guess google's one so supported either. Then, who knows all right.

C

That's surprising all right yeah, it was surprising it was. It surprised me too, but it's there.

A

Okay, so uh are you saying that, even if we do this there's still more to be done or.

C

So if we, if we merge this, it means that you can create an all udp alb, but you can't create a mix nlp that is api service stuff. You have to do got it.

A

All right I mean I, this is certainly short enough. I can have a look at this and uh nick it doesn't see if anyone on your side has looked at this uh yesterday. Oh oh.

A

It may have merged.

A

Someone commented superseded by this one which has merged. So maybe this is. Is it the same person? Sorry, I will flick back this. There are different people, uh so it looked like pj ryan submitted the first one and.

A

This kish orj, I don't know where to put the line there uh ah and he contributed or patrick ryan, contributed to pj ryan contributed to this one as well. So that's great and this merged.

C

Okay, cool great yeah. I missed that. Okay, sorry about that. No.

A

Worries, let's see uh yeah great.

A

ah In fact, it was.

B

A

So nick, if you asked mike to look at it, michael micah looked at it yep I did cool there. We go. That's good news of course. Now apa machinery is gonna, get a lot of weight, but yes, you can now just redirect them to api machinery, uh and I will I will see whether we can close this previous one because it looks like we probably can.

C

Yeah- and I I think that's why now that I think about it, I think this is why I might be getting like more pings about the removing the validation for the mixed protocol, because this pr merged and so people want to use nlbs with both protocols. But then the api server is blocking that. So that explains it.

A

Yeah, I guess this means no one runs a dns server on kubernetes.

C

Yeah, that was that was pretty much the number one use case for that actually like they want to run a dns server through nlp and we're not allowing it.

A

Okay, well, live and learn. I did not know that. uh Okay, um so that's good news. Should we have a look, I guess, as we come to feature freeze, maybe we should look at other uh prs that are uh uh like maybe ready to merge and would otherwise miss freeze, I'm just trying to find the oh. I guess we don't tag them for aws. So let's do this.

A

Pull requests sort recently updated anyone who's most. uh Does this make sense to look at these and see whether any of these are critical, as it were.

A

Am I the right repo, okay,.

A

uh I don't know what this one is.

A

Okay, I don't. I think this is just a okay. I think that's just a test, so uh I can have a look at that. uh The new event api has nothing to do with us. I don't think uh empty file. Pr triggering uh ah aws remove volume, iops limit.

A

I will open that one. We have the udp, we have udp uh api.

C

Just close the two.

A

Udp ones, because yeah yeah.

C

A

Open them and we can go through and say like. Are you happy to close type thing? um That's nothing to do with aws. I don't think support private ips for aws and obs. That sounds like is, let me guess, that's just another annotation, but we'll see implement lifecycle, tcp hook. I don't think that's us external registry cloud, credential provider extraction. It sounds like we're less keen on that. Oh wait! Nick. What uh did you this? Is you nick.

D

Yep, um that's me, uh but I I I was actually just looking at the release. Schedule did. Did the code freeze get pushed back.

A

It's it was pushed back from the original about a week ago, I think, or maybe two weeks ago it was pushed back by two weeks. So it's now july 7th is that right july, 9th yeah.

D

9Th, thank you um so I was. I didn't know that, but um I was thinking that there was no way that I was going to be able to get this uh by the original date. um Had I known that it had been, I might.

B

D

Reconsidered that um I still am skeptical, um I'm finishing up some some stuff uh that will probably take a little bit longer so anyway, I don't. I don't know um me and andrew had talked about it and we had said, like you know, if either one of us has time to pick it up before our code freeze than we would um otherwise I'm going to definitely pick it up at the start of next cycle.

A

Yeah I mean to be honest that almost feels like one that might be better to introduce earlier in the cycle rather than a late in the cycle thing.

D

A

D

What we we or I learned a lesson last time that uh people get pretty angry when you drop that late in the cycle, so um I don't really want to go through that again.

A

It's not really. I I mean it's more just like just like you know, you, like. The pressure is on to fix things faster right. I hope no one actually gets angry um yeah. Okay, that was one on the second page. Now it looks like they are not, as uh I opened a couple that might be relevant uh okay, so let us see if we can burn some of these down.

C

Oh actually, can you can you also open up the um the ipv6 one.

A

uh Yes, if you give me a little bit more of a hint, oh is it the bottom here sorry, uh I can just search for ipv6.

A

A

Thanks, oh speaking of things, we should not introduce at the last minute.

C

Oh, you know, I wasn't saying that we should merge this, um but I think this is something that we should start noodling in our brains and thinking about um it's one of those things where, like once, you emerge and you release it, you're stuck with it and maybe we don't wanna. You wanna, really think about what we want. The node address semantics to be for single stack, v6 and tool stack.

D

Can you add, uh I don't know what is his uh github name is, but gus angus. I think angus yeah. I miss.

A

Ccm or a sign himmer.

D

uh You can assign him as.

D

A

Interest to you.

A

A

Yeah, it sounds like what this is saying is if it's, if it's only ipv6, that's available like to use that. That seems pretty reason. If, if that's what he's saying what uh our phillips is saying, what uh right yeah ryan, uh what what are phillips is saying, then um that seems reasonable, but yeah. We should certainly think about it, because it then opens up the question of of dual stack right.

C

Yeah like, in particular, if you want v6, will need to dual stack like with this pr work. um Yeah.

A

Okay sounds like we: don't we're not keen on merging this in this window,.

C

I don't think so I mean it's up to you not to be but yeah.

A

C

A

Okay, uh this removing the volume iops limit, the ah okay. I like that idea. uh The idea is basically to like abs raises the limit and uh we shouldn't hard code, a limit that eight of us can raise, particularly in yeah. That seems reasonable.

A

No, I think we should probably merge this. What do people think I mean I'll, have a look at like why we introduced them in the first place, but it doesn't seem like the sort of thing we should be enforcing. Maybe the minimum.

D

Yeah, I don't I don't know the reasoning for having the limits in the first place. Could.

A

Just be the error messages clearer, but yeah if there's no strong reason to do this, this seems like a solid uh thing to merge.

A

uh This one we think we can uh hopefully detube or close uh private ips for aws nlbs uh prs but specific static, private private ipv4 address from creating a service background data nlp. In addition to the support existing support for elastic ips. But for internal and hobbies, and it's an annotation which probably we could have all guessed and.

A

That seems reasonable.

E

Anyone think this is karen. Sorry, I read through the pr- and I wasn't sure like how is this different than the pr like for 116 that added support for static ips with the other annotation?

E

Did I just like totally miss it.

A

It sounds like it has to go into a different field like maybe it's a do. You think it has to go into this field. Maybe that's what I was sort of imagining was the sort of problem, I'm not sure, okay, but you mean that we previously added a pr to map uh arbitrary elastic ips via a similar annotation, which I.

E

A

E

Yeah, it was in 116 for uh nlbs. I think it's 69 263.

E

A

Yeah, I always, I only know.

E

This because, like I, was waiting on 116 to use this feature, so that's really really.

B

I think those elastic ips are going into the allocation ids fields, not the private ip one, so it is different.

E

B

E

Oh yeah, the uh the allocations.

A

Right there, too, is it this this yeah yeah. I mean it's a great question like I'm, assuming that, like you, have to specify the one in the one field and the other in the other field, which is why this person did the pr, but we should at least verify that is true got it, because it is possible that uh that that uh this person, d.a cohen, wrote oh.

E

Sorry, I am I like totally blank on this. Sorry, no, I see why this is going on. I'm done, I'm gonna write, the the eips are public. Sorry.

A

Okay, cool and it's been a long week yeah. I know that feeling all right uh I'll take a look at this. I think this one probably looks reasonable to maybe go in. uh It would be nice to have tests. I don't know if that's gonna happen so, but we can have a look and see.

A

I'm also not sure if it gets reconciled correctly, so we'll have to have a look at that like if you change them, what happens all right? Let's keep going the uh well. We talked about this one nick right, uh you probably are not going to put it in now. I don't know if there's any thing we want to discuss on this, one looks like it's probably being discussed in the cloud provider extraction.

D

A

Don't know andrew anything, you.

D

Want to discuss sir.

C

Yeah no it'd be yeah, just it'd be good to get this in, because I know it's it's an alpha feature and we're. You know we have a future gate to block it. um So I think if the pr.

B

Was ready to merge.

C

In the next week, I think it would be acceptable, um but yeah, like I understand, like you and I both have other things um internally to deal with, so so we may not get to this, but I think we should try to do it if we can.

B

So yeah better effort.

C

We'll do our best if we, if we don't, then whatever we'll get to it in 120.,.

D

Yeah, um given that I now know that the the deadline was pushed back, I will see uh if they're yeah, best effort um but we'll see.

A

There's a there's a bunch of holidays in between or like four of those days or holidays, or at least for googlers. I don't know how many other people get, but yes, there's a there's, a big holiday that knocks out a bunch of those days uh which might actually be an opportunity, depending on how you feel about holidays.

D

Anyway, yes, yeah, just just take uh how how many you get and divide by two, and that's probably how many we get.

D

I'm not gonna go.

A

There, anyway, there are some days that may or may not be holidays.

A

uh I was counting the weekend by the way in our particular in that form anyway,.

A

The uh the okay, the next one on that I sort of pulled up on the hot list, was uh fixing an extra security group removal from the aws load balancers. I think we talked about some before. I think I promised have a look at it before and I will endeavor to have looked at it again. It feels like this is a bug and we should do that uh this one I opened just because I thought it was weird, um but it looks like it's going to be gce specific. So it's fine.

A

I just didn't know that we had the same problem like multiple interface ips. I don't know I will I'll leave that open for now. Just for me to go back and look at okay.

A

This is a pr to disable a short routing path.

C

A

C

This doesn't affect um aws because we don't set the um low balancer ip in the status we just set the host name, so the short path isn't actually enabled.

A

ah That's right, this, this person is openstack, okay or works on openstack, okay,.

A

Okay, thank you. uh Should we upgrade our aws sdk? um We probably should uh there are probably at this was may 27th, so I suspect we should probably do another update even after this um yeah, it's on one foot, 1.32 dot, something no yeah. Well, it's just the it's the regions that I think is sort of the biggest challenge. It'd be nice to not have that dependency it's, but we're not going to fix it in this release. So.

C

Yeah every once in a while, I see a um yeah like a pr to update the sdk, because the new region um is available is.

B

C

Plans on aws to support like a api to fetch the regions or the zones, or something like that.

A

There I mean there is.

D

There is nick, I was just gonna say like we have an internal api, that's sort of like that, um but I don't know of anything external.

A

There very much is a described regions and describe availability sounds call on aws uh and you recently added describe instance. I don't remember the word but like describe instance types, uh and uh I know it's always awkward to know whether something's internal or external, so yes, that was recently added, so that's also there, which is also great.

A

um I think the problem might be around like which region do you query to find out the regions which is a little bit of a mess so, uh but we might be able to square that by like asking the metadata service what the region is and going from there, but I think the problem is like I. I don't think you can go from a region to to the end point purely like if you've.

A

Never, if you don't know about the region, because some regions- maybe don't follow the rules that used to be the case in the aws sdk, I don't know if that's still the.

A

Case I mean so like the the ec2 endpoint is regional and it doesn't. It isn't always the case that it is always just like ec2.region.amazonaws.com or whatever. It is yeah.

D

I think usually it's just the ied endpoint is different, because that was the first region, um but we can take a closer look and see so if we could just have like one exception and then the rest, if the rest I'll follow the rule.

A

Yeah anyway, uh so we can probably update this one. uh We can probably merge this one and then uh I appreciate very much that this person has given us the script to do this in future, uh because we will likely we may want to go to 132. But if someone has done the work to do this, then we should probably uh like get this in.

A

I'm not quite sure we haven't get it in, got it in yet, but we'll take a look: uh removing a stale load balancer for edge cases in aws, when user changes from a classic load balancer to a network load balancer. Oh yes, we don't delete one. Yes,.

A

This is slightly risky. I guess.

A

Yeah, that's a tough one.

A

Because are you really here, I see some shaking events. Are we really sure that it is the right load balancer? It is a uuid generated load balancer, so it feels like it should be. Okay, don't do it?

A

D

Why are you so skeptical andrew.

C

um Yeah no, like it just sounds like a bad idea. I mean like we, we don't re-reconcile um like if you delete a service. um Well, I guess we added finalizers to services, so we do catch. The duct case.

C

I don't know it sounds like a bad idea. It sounds like there's just a lot of like you know, um errors along the way that happen, and you know you don't want to be accidentally deleting people's little bouncers yeah. It's it's.

A

Tricky because look for my first glance at this, at least it looks like we just see that there are two little bouncers with the right name, and then we delete the one which is the wrong one, and that strikes me as like that would have been tolerated before and now we're going to delete the note balancer. So, although it seems incredibly unlikely, it seems risky, the alternative might be to observe the delete event and do a sort of best effort deletion or to observe the change and do a best effort tradition at that time.

A

I don't know if that's better, because it just makes it like less likely to leak but not impossible, and so like we're sort of giving people a full sense of security that, like we're gonna clean it up when we're not.

C

Yeah and in general we don't watch events for the low balancer and service. We it's like a one-way reconcile right so, like I don't think we should try to be smart about like reconciling the the cap. The cloud state back into the surface state should just be like service state to lb and keep it that way. For now,.

A

And I guess the workaround here would be to uh change your service from type. Oh, I see, you're working on would be to change your service to from like type load, balancer to type uh node port or something and then go back and then like then switch. But I guess the point is: if you do it, if you do it just by changing the annotation, maybe you don't have down time, maybe or certainly you're- certainly you're guaranteed downtime by changing it to no port, where you're not guaranteed down time. In this regard,.

D

I mean if we do like, if you, if you can get around it by doing that, you know if we're already deleting it. If you're deleting the service, then shouldn't we be trying to solve these edge.

C

A

Saying we shouldn't have documentation.

C

To say, create a new one. Here are the steps to do.

A

It yeah that is true. You can always create a new service, I'm not saying we shouldn't, we shouldn't try to fix them. I'm saying we should make sure that we don't really hurt people along the way.

D

Yeah I mean if, if like we had load balancers with normal names, then this might not be a good idea, because you'd be creating the monsters with the same name over and over again probably, and you could delete the wrong one right. Wasn't there a pr for that or.

A

An issue there have been a lot of issues that proposed it. I don't think we've ever uh got close uh because and for the same and for the same.

C

A

Yeah, like what happens when you change it.

C

And this is why I would like to work. I think it'd be great, just throwing ideas up there. We should start. We should just cut a new version of the cloud provider and start from scratch and, like just rebuild all these assumptions um on a like a youtube provider. That's at a tree but topic for a different day.

D

That's indeed a topic yeah. I will second that idea, um but yeah we don't need to get into it.

A

Now, okay, I'll look at the this one as well. I mean it seems like a very valid and important thing that if we could magically fix it, we should fix it, but if we have to make sure that it's worth the risk as it were, I I don't. I can't think of any way to surface this information other than like. Could we somehow like give people a hint like it looks like you've done this? Would you like to do this.

A

And we could raise an event. I just don't know if people would actually see it they're more likely to see an event than a log message right, but like still.

C

I mean they'll see the event once it's broken but like if it's broken, then.

A

Yeah I mean I was suggesting that we wouldn't delete the old we wouldn't manually. We wouldn't automatically delete the wrong one, we would just say: hey, there's a dangling elb or I guess nlb. I guess it can go both ways, but there's a dangling elbv1 and you might want to delete it once the traffic has flowed over or whatever whatever it is.

A

All right, I suspect, we're not get that in before the future. Freeze how about that uh this one we talked about earlier. Oh, I think we've actually gone back to my earlier tabs, all right. So this one again uh again, this one was a test which I think is probably a dupe. uh The hangout which we talked about earlier and uh the the pr which may or may not have introduced that and a probably unrelated issue in csi. Okay.

A

Would anyone like in particular to so I'll just flip through these? If anyone wants to and then we'll call it time, because it's more or less done but like if anyone wants to like take ownership of any of these, I'm going to flip through them one more time, and please shout when not on mute, and I will stop so adding ipv6 addresses for node addresses.

A

Removing the volume iops limit, I can look at that one. Oh presumably, I should look at that. One! Okay, uh that one is we're just gonna, do a dupe detection like is this okay. To close, I can look at those two, uh this one private ips for aws nlp,.

A

Not hearing a lot of voices here uh I mean I I haven't seen it it's quite difficult.

D

Seems like a uh easy uh like if it's just a matter of it's the same as that uh eip feature and and it's supported, and it's just an annotation that adds that it seems like an easy merge.

A

Yeah well, when we can, let's uh nick you are already on the ball for that one, uh this one I think I was going to have a look at. I am going to have a look at. I think, maybe nick you were also going to have a look at it, but um this one is just interest to me. uh I will do this one. I don't think any of these people.

A

I can have a look at that one and we might need a 132 if there were new regions.

A

Since may 27th, I can't honestly, I don't know I feel like there were some loca and whatever uh we can have a look. um Removing the stale lube answer for edge cases in aws is.

A

I think we should probably have a look at it, but I don't think I I think we're unlikely to get it in by then, but we oh, we should tell jeff juan why that is oh, like it's. uh I call it yours, I guess.

D

Yeah, I can, I can't wait. We can have a conversation with him. Yeah um I'll get him to come to the next.

A

Meeting actually cool yeah I mean, I think it's I I hope we've been fair to the uh to him. I guess uh in that um I think everyone would like it. We just have to make sure it's we're, not gonna hurt anyone, it does seem unlikely, given the uid uh and that's.

A

uh What is this? I can have a look at this you're you're you're. You are marked as a reviewer nick, but I can also have a look at this and then um nick were you gonna have a look at this or is jay gonna have a look at it.

D

Yeah, this is I'll. Have this one be my top priority? So if I do nothing but.

A

Yeah that makes sense.

A

Too many nicks look they're, not even on the top page of nick's.

A

Oh you shouldn't have dropped the I you're too web web 2.0, uh okay, cool, that's it all right! I don't know. If anyone else has any uh other topics they want to uh bring up or discuss. I don't have the agenda open right now, so I don't know if anyone else has that anything, but otherwise uh let me stop sharing. Otherwise. Thank you very much to everybody and we will see everyone in two weeks: yeah. It is not a holiday. So yes, happy friday. Thank.

A