Kubernetes Kubernetes AWS Provider Subproject, 30 Sep 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes - AWS Provider - Meeting 20220930

Description

Recording of the AWS Provider subproject meeting held on 20220930

Agenda: https://docs.google.com/document/d/1-i0xQidlXnFEP9fXHWkBxqySkXwJnrGJP9OGyP2_P14

* Discussed: Support multiple route tables (https://github.com/kubernetes/cloud-provider-aws/issues/432)

A

Thank you all right, welcome to provider, AWS I, don't know the date. It is September, 30th um yeah, so we're just gonna go ahead and get started. um Doesn't look like there's too many people here today, um so I I'm going to assume there aren't any significant sub-project updates.

A

However, if you have anything, please go ahead and add it to the um the list and I will and let me know- and we can talk about it. um I think uh I did have one agenda item which I haven't added to the agenda. Yet so I'm gonna go ahead and throw that on um I. Basically just wanted um kishore's opinion on something um so I'll just well. We fill in sub-project updates. I'll. Add that to the agenda.

A

All right uh looks like you: have a patch release and the load balancer controller. Anything you wanted to mention about that.

B

uh Yeah, so it's a minor release or sorry, not a like a minor bug fixes- uh and this is to address some CVS that were found in the recent uh 243 release and the release notes are in the link here.

A

Got it? Thank you all right um yeah. So we have this issue um AWS um to support multiple route tables. uh Did we ever get a chance to discussion? Discuss this one key short, I can't remember.

B

uh I, don't remember okay, so.

A

It's essentially I mean this. The request sounds reasonable to me: it's um apparently uh they need multiple route tables in order to um create a cluster with multiple zones. When routing is done without an overlay, um so I guess that would be cubenet, uh but.

B

It seems like this is PPC native networking or.

A

No I think it would be I mean I, guess it would be cubenet, which is.

C

Yeah route, the uh technical doesn't have to use this, but I think this is referring to when we turned off Source desk IP checking in with route a slash, 24 or whatever it was to each uh to each VM um in the round table. So custom routes would be what I would call it, but I I, actually thought I, don't think it's I, don't think it's multiple zones, I thought we had I'm trying to remember the sequencing I thought I thought that this functionality worked for a long time.

C

I mean I, think it still works today and I think it works in multiple zones, so I'm a little confused, yeah.

A

Wrong, we need to understand what their setup is, so we can understand what part isn't working for them.

B

um Yeah, because if we talk about Right, Round Table, it has to be the VPC native networking right, otherwise, uh like overlay would use it differently. So.

C

So, uh uh in the early days of kubernetes, we discovered that it does not have to be native networking you can you can route a slash 24 to a VM just in a custom as a custom route, you turn off Source desk IP checking and it appears and I'm sure I have one from AWS loves this feature.

A

So what do you mean I.

C

Have here so it works uh so like like. Basically, you can program, uh so they they, uh the IP addresses, are not assigned. So uh on your VPC. You have like a big like a slash 16. Let's say um you get some uh IP addresses assigned by AWS in that slash that slash 16. Excuse me, uh but you define essentially slash 24s from some other IP range, and you can then say that, uh like those slash, 24s should go to these VMS, regardless of AWS IP assignment.

A

And that you only make that configuration in the route table so interesting, correct.

C

Correct uh and that's how we did it for a long time that was the seemingly the best way to uh to do uh custom routes. I think the real to do multiple IP addresses. The real issue, of course, was that um for a long time there was a limit of 50 55-0 routes per uh route table, custom routes for Round, Table, I, think and I believe that has now been raised to either 500 or a thousand or some some much bigger number.

B

But- and in this case like uh would we have to disable the source station check as well on the.

C

Hbm correct each VM has to enable has to disable sorry. I can't speak today has to disable Source desk IP checking.

B

Because we don't do by default for ecas and other like kubernetes, that we use.

C

Yeah I mean I I, don't know uh so the uh this I think gave way to um it seemed like it was easier to to either uh use IPv6 or to just use an overlay, um because an overlay wasn't that heavy right like a VX slan overlay, is pretty comparable. So, okay.

A

So, just to make sure it's clear and I understand like so what what would be used in a setup where they don't have a bccni and they do have wrap tables like what cni we're.

C

Using it's, it's cubenet with the AWS cloud provider, an AWS cloud provider or your the old AWS cloud provider had a routes, interface and the routes interface would set up a route centrally Coupe controller manager would set up this route per per node. What I'm right I'm, actually wondering like chaos, is probably I'm looking at chaos, because I think chaos we have the configuration for um I think it still supports it. So I'm trying to remember like whether we need multiple route tables.

C

How that works uh like whether route tables are zonal or not,.

A

Okay, so, and- and so you were thinking that multiple route tables would work, slash one round table per Zone- that was kind of your.

C

I couldn't remember that I was trying to remember the details. I think I feel like it works and I'm trying to like it works with cubenet and you can have multiple zones, but I can't remember whether you have to set up multiple route tables or not. So that's what I'm trying to that's trying to track down.

C

Approaching to fire up a multi-zonal route table cluster and see what happens all.

A

Right, thanks for clearing that up.

C

um Yeah so much clearer.

A

So somebody says we're using Gardener just to create the kubernetes Clusters. It uses the gardener extension provider to set up the infrastructure, there's a cluster Global route table per cluster, which includes the kubernetes node cider as local and internet gateway of civil route per availability. Zone. There is another route table.

A

Which includes the coverage note cider, as local and map gateways default gateway is calico or psyllium cnis Calico being used in the majority of scenarios.

A

As of now, we use overlay networks.

A

Ipnip.Govian would like to get rid of them. Therefore, our goal is to use the cloud control manager to create corresponding routes for the per node, pod, ciders and infrastructure works for all other infrastructures. We work on Azure, GBC nodes.

C

I.E, this is a very complicated setup I. My my suggestion is this doesn't belong in the AWS cloud provider and is there some reason why it has to live there? It doesn't sound like it has I, don't see why it has to like something anything can watch the nodes and do this configuration right. Adios, Cloud, control manager from that perspective isn't special, um and so, if it's this unusual, a setup, I would say it doesn't belong in the Cloud controller manager, but I I, don't know, I mean.

A

Yeah I can ask them.

C

More information.

A

Well, I guess the argument would be like: why does this apply to all users and and if they have a good argument there.

C

Then exactly or even even a substantial number of users right like why, like is this something where like this, doesn't it doesn't sound like this has to be in the AWS Cloud controller manager? It doesn't sound like it applies to many users. Can we do it externally, prove that it works? And then, like graduated or something like that, you know yeah.

A

Exactly yeah I mean I would be fine with implementing this. If, if it really did apply to um you know multiple setups, uh because.

B

We don't hear about.

A

B

A lot I mean this is the first time that I'm seeing something like this right.

C

And I was trying to drag up like it from my memory.

A

Let's see that's good.

C

B

C

Have a code in all right so.

C

uh Let's see if I can just paste the link to this, this.

B

C

Is the network network Builder code in uh helps isn't necessarily going to help, but so there's a VPC.

C

We get a list of the subnets.

C

We create an internet gateway.

C

We create a route table.

C

And we map it to the internet gateway.

C

And we I'm pretty sure we're gonna yeah. We associate the subnets of the route table.

C

So, which I think that's basically a pretty standard setup but I think it's one round table I, don't know why they have multiple route tables. To be honest, maybe it's the net Gateway thing.

C

Yeah I, I, I, I, think I. Don't think we should be supporting this immediately without any additional information, and my gut feel is that we're we'll probably say, hey like this doesn't belong in and it was Cloud controller manager.

A

A

Yeah I'll I'll have a response after after the meeting um on the issue, but thanks for um remembering, as best you can uh at the uh original implementation.

A

A

A

Table per cluster, which includes the node cider as local and then so. Do you understand the script Justin that they said? There's a cluster Global route table Round Table per cluster, which includes the node cider as local.

C

Local means that uh those VMS are those IPS are on the, so we're basically setting up a router right and we have a table with IP ranges and we're saying that those particular IP ranges. We expect to be like directly plugged into the switch as it were like. If you imagine a switch right like I know, there's no switch physically switch, but.

A

Is this is saying the entire node like the entire subnet subnet, that includes the nodes? Yeah? It's not it's not like the Pod cider per node, or anything like that. It's just good.

C

So, that's that's, not clear. I think you can also I think that's a good question like what on Earth is going on here like. Why do they have multiple route tables like I? Don't understand that like so then then they set up the second round table per AZ and do a different like a different, different Gateway and I. Don't really understand. What's going on there.

A

C

That's exactly that's my.

A

This is a same note cider, but then there's this different. There's a Nat Gateway versus the inner internet gateway.

C

Emily says when they say the kubernetes nodes, cider I, don't know whether they they mean the per node kubernetes pod cider.

A

Right, that's what my original question or that's what I assumed at first, um but then they have it on the global and the local. So that doesn't really make sense. I guess it has to be but cider.

B

That includes, unless.

A

Unless they made different things for each one but well I, imagine it's it's a cider that includes like the the cubelet or the the like default.

C

I think you want to map the I, don't understand why? So, maybe you can answer like why you would have multiple route tables in AWS like I? Don't get it uh like I I can understand it for isolation right, but I understand why you would have them sort of linked, okay, sure.

B

I, don't know if you we can have multiple it's for eni right. That's my understanding, like you can have power, subnet or network interface, that you can have a route tables, but I I, don't know like if we can attach multiple route tables on the same entity.

C

I think each one is attached to one route table but I'm guessing what they're doing is they're routing between route tables. Somehow, but yeah I, don't I. We need to okay, it doesn't sound like we have a clear understand. This isn't like uh there's, not a dock being like yeah. This is why you smell from rats that I'm just missing right, like so.

B

C

If there's multiple ones.

B

Like there's only going to be one effective route table that comes into play, that's what I get uh so not sure like whether they use the internet gateway or the NAT, uh because in our configuration it's like either or right, we use Nat for private Network and then internet gateway. If there is like uh igw, if there is a public facing address as well, igw is one on one that is more General, but uh this kind of uh conflicting here, the two requirements as far as I understand.

C

I think we can ask them to explain what is going on in terms of network topology here and like maybe some pictures or like maybe they have a dock somewhere but I I short, the short answer is is probably is likely to be like it's a high bar to get into the AWS Cloud controller manager, and this is not it.

B

Doesn't it says everywhere else like Azure GCB.

B

A

B

Is where it doesn't work so.

A

But I think the question is: is there another way to do it um on AWS or are they? Is there really a missing like missing functionality here.

C

We don't know what they're trying to we don't know what they're trying to do right like what are they actually trying to do?

C

They tell us what they have done, but they've not they've, not explained. uh Why.

A

Yeah, just generic support for multiple tables.

A

And- and you believe this I mean this statement sounds wrong: multiple zones when they're running and stuff, but not related right, I.

C

I, don't think that's needed, I could I could be wrong. It almost also depends how they have done it, but no I, don't I, don't I thought we had that support already. um I might be wrong. There.

A

Cool all right, well, I think. Obviously we need more information from them um and I agree with with uh with our answer until they convince us otherwise, um so cool thanks for the help awesome.

C

And I, don't think we're I, don't think it's I think it's actually like. Also we're basically saying look, go and build it externally, and then we can like incorporate that code right. It isn't if it if it proves valuable to everyone, but like I I, think we're saying like there's. No there's nothing. It's not like uh it's not like controller manager where this is the only place.

C

You could do these things previously like if AWS Cloud control manager needs to switch off some functionality, we should make sure that is if that functionality is available, but they should be able to do this outside of AWS Cloud controller manager and coupon manager.

A

Right, yeah, no, that makes sense. Yeah I mean yeah it like, if they need I, mean assuming they're, not using eks, which I don't know right.

C

The gardeners uh Gardner is a alternative to chaos and eks and yeah.

A

So then um they would need to be able to disable the route controller of whatever they're using and then they could write their own controller. Really because, as you know, or if it's just disabled part of the rock controller, they can do that. We can I mean.

B

A

Be a lot easier to Implement them. That's what you're saying right exactly.

C

And I think like I, don't know if the flags to disable various controllers are in there I I. Imagine the route controllers only turned on in certain circumstances anyway. So.

B

C

B

C

B

The configuration like the global route table, they might be referring to the one that's configured in the kubernetes cluster and we might be using tags uh to filter the or get the route tables uh if Global is not configured. I think that's where they are going with this. As far as I understand just by looking at this error message here,.

B

So we try to list the route tables and then we did find multiple of them uh based on the cluster tag and that's probably what they want us to support, still not clear. But that's where I'm going with this.

C

Yeah so I mean the eks doesn't register node routes. Does it like you, don't register any node routes? Do you? No? We.

A

Don't we don't actually enable the route.

C

Controller right, so that's all I mean that's that functionality is like they just need. They need the same thing right. They need to turn off the route controller and do it in some other controller that unblocks them I think.

B

A

B

I'm I was searching the error message and then we do hit. We have a possibility to hit this, so this is from the route controller, of course, uh but yeah. Maybe we don't hit this uh case with the eks, because we don't enable the.

A

Yeah, we explicitly disable the right controller because we don't need it for um rcni ppcc.

A

C

A

That all right go.

C

Ahead, Justin well, I just want to say it also. uh It also does stuff. You don't want to do if you have a uh if you're doing another way like it will go and configure stuff in on the route table which you don't want, it will consider it will configure those pod ciders on the route table which I'm pretty sure you don't want.

B

Okay, it's from the route controller. Okay, I got it I'm not from the I, was thinking about like whether we discovered this from load balancer as well, but not the case.

A

um But yeah, were you wondering if, if it's really just a simple matter of allowing it to find the global route table,.

B

So there's like both Global configuration and if Global is not configured, it looks at the cluster tag uh for the route table and, if there's more than one it errors out. So that's what probably is happening here, so they might have tagged multiple route tables with the cluster tag and expected it to work.

A

B

Could be as easy as that uh in that case, of course, then they have to stick with the limitation. But again we need to understand what they're really trying to do with this.

A

Cool yeah well, I think we've uh business. So um does anyone have any other agenda items that they want to discuss.

A

Football I think we can probably leave it at that. Then.

B

Awesome thanks for listening. Thank you guys, yeah.

A

Thanks for joining bye, see you later bye.