Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Road to IPv6 Support in... John Gardiner Myers, Ciprian Hacman, Ole Markus, Justin Santa Barbara

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

The Road to IPv6 Support in kOps - John Gardiner Myers, Proofpoint; Ciprian Hacman, polypoly; Ole Markus, Sportradar; Justin Santa Barbara, Google

kOps has been adding support for IPv6 clusters. Learn about the design of and challenges faced in providing a turnkey IPv6 Kubernetes infrastructure, including on AWS and other clouds. kOps maintainers will describe the use cases they are targeting, the network architecture they chose, and how they are managing address allocation. They will give details on the components, both internal and in upstream projects, that needed changes to support IPv6 and the bugs and limitations they had to work around. They will also reveal trivia, such as why the kOps service network is fd00:5e4f:ce::/108.

A

Hello: everyone, my name is cyprian hackman, I'm a devops engineer, working for poly poly and today we'll be talking about the road to ipv6 support in chaos.

A

What is kops kops is a cloud aware: kubernetes installer provisions, the underwear cloud infrastructure and also installed the kubernetes core components, including add-ons since k-op zones, the life cycle of both the cloud resources and kubernetes it can handle all aspects of creating a native ipv6 kubernetes cluster focus so far has been on aws and gce, but adding other clouds should not be an issue.

A

Why ipv6 larger address space allows for simple flat network architecture, no need for not no need for complex calculations and doing magic with subnets easier to avoid ipv6 conflicts between vpcs accounts companies, native cloud, drought, native routing for pods allows direct targeting behind load balancers without going to cube proxy or notepad.

A

It's both performance and debugging improvement.

A

Cni's can also be much simpler if they are designed for ipv6.

A

This is not to say that current cnis are not ipv6 aware, but they are designed with ipv4 in mind and have to deal with all the limitations of it.

A

We decided early on that we would support single stack addressing for pod networking and do all stack at the edge this allowed to reduce the complexity of the development and very easy to assess the current status of ipv6 in kubernetes.

A

Also, support for dura stock is quite complex and negates many of the benefits of ipv6. Not many would be able to migrate. Ipv4 single stack clusters to dual stack anyway again, due to complexity and risks and downtime.

A

As at the time, instances were only allowed to get single ipv6 addresses, we decided to use unique local addresses.

A

This does not unlock the real benefits of ipv6, as nat is still required, but the assessment of kubernetes readiness can be done. It also requires overlay, networking and tunnels between nodes, but we were ready for that.

A

Well, ready, but not quite ipv6 must be configured in lots of places, and I really mean lots of places, so both control, plane and cni must be configured, and they must actually agree on that configuration.

A

After we found out the right settings with the help of the kubernetes developers, we hit a challenge with aws cloud provider. It was not aware of ipv6 addresses at all, which was not really good for us as we needed it to assign the ipv6 addresses to node objects. We fixed that and again another challenge.

A

Ip addresses were not prioritized as we expected after many debates, the decision was that cloud controller should send the ip addresses in the order that it desired to be used, so it basically decided the priority we fixed that and tests were possible. Now we created periodic tests and we tried them for some time. Okay, during this time, together with other developers, we fixed all the needs that we found mostly testing issues, but there were some minor issues in kubernetes too, and we thought that this was it for now, but surprise.

B

My name I work at radar and a sportbreaker. We are really interested in those benefits that we mentioned earlier and last fall. Edwards released a number of ip6 features that allowed us to implement just that.

B

So we decided that local, unique, addressing, isn't really going to give us any of those benefits. So we decided to drop support for that and focused on global energy instead, so global investing required us to do quite a few changes in how we provision our networking.

B

So we had to enable not 64 dns 64 so that the cluster can reach things without an rp6 address. We had to do some changes to the route tables.

B

We had to do some changes to the launch contacts on instances. We also had to do things like disabling mosque grid and overlays, since they would be getting in the way when we are using global routing.

B

Launch templates also doesn't allow you to configure to have both an ip6 address and a prefix at instance, creation time. So in the launch dropbox, we said that instances are assigned automatically on a single rp6 address, while during the bootstrapping of the instance, we had to create our own logic for provisioning, an ip6 prefix for the instance.

B

We also had to do something that would take assigned physics prefix and give it to the corresponding node object so that the cluster knows what it addresses what's scheduled for. The given instance can receive when all this was put together. We had something that looked like this, uh as you can see, there's a mixture between like before and after six services, and the reason for that is that cuba was a little bit inconsistent with what address family it used for host networking.

B

So all the products that you see here with ip4 addresses are those running with network enabled.

B

And let's consider and also for about five minutes, so we were primarily testing on ubuntu and on ubuntu. There was a bug in systemd network b that when it tried to renew the dxp list, it would get a reply from aws containing both the single address and the prefix, and it will fail to properly parse those and just decide to remove everything.

B

So all the instances after about five minutes no longer had any addresses assigned.

B

uh On the other distros that we were testing this with, we had a few other minor issues like, for example, it was a bit tricky to configure the kernel to properly accept the router announcements since when you're using prefixes, rather, announcements are disabled by defaults, but eventually managed to either fix dimensional bugs around them, and we now support uh the antenna and new ubuntu lts version that was just released.

B

As mentioned, we had to create something that was aware of the prefixes that edwards would assign to the instances by default. There's a controller that looks at the whole cluster wide sider and takes the subnet from from that insider and assigns it to each of the new nodes and that wouldn't work since the node cider has to match the prefix assigned to the instance.

B

So we have to decide disable the controller that hangs this logic by default and create our own controller. That was aware of the prefix that edit was assigned, and that touched the corresponding old object, giving it that prefix as its point either.

B

The really cool thing about this is that cns are completely unaware of what's coming on them. So all the cloud specifics include prefixes and all that is hidden kind of behind that controller.

B

So that means any cmi can support global routing with ipv6 out of the box without doing any changes. As long as it's configured to look at the node object, spot slider.

B

The controller currently runs as part of the apps controller, but theoretically it could be a tiniest download or it can run as foreign.

C

Hello, my name is john myers and I work at proofpoint at proofpoint. We are particularly interested in how ipv6 can solve problems with local ip address allocation, k-ops initially implemented ipv6 using public topology, because it is simpler in public topology. All nodes are dual stack and are accessible from the public internet.

C

In order to allow non-host network pods to connect to ipv4 targets, we enable dns-64 and coordinates and the local dns. In order to do the latter, we submitted a pr to kubernetes dns, adding the dns-64 plug-in chaops creates a nat gateway in each zone in order to perform the nat 64.

C

After aws released support for ipv6 only subnets, we implemented private topology in private topology. We have three types of subnets. First, at the edge we have dual stack utility subnets. This is where chaos put nat gateways and public load. Balancers does not put any nodes in these subnets mercury nodes are placed in ipv6 only private subnets.

C

In order for the nodes to connect to ibv4 services, k-ops enables dns-64 in the subnet's awws dns servers it routes, the nat 64 range to the nat gateway in the zone's utility subnet and the rest of ipv6 to an egress, only internet gateway for the control plane. Nodes, k, ops, creates a new type of subnet called dual stack control. Plane nodes need to be dual stack because k, ops, uses instance, targets for the api server load, balancer and aws load balancers only support ipv4, for instance, targets.

C

This type of subnet is like public subnets except nodes, don't get public ipv4 addresses and it uses an egress only internet gateway.

C

K-Ops needs to make some arbitrary assignments for in-cluster resources, as these addresses are not routable from outside the cluster. They are made from the unique local address space, though they are not technically unique for the kubernetes service network. I attempted to spell service in hexadecimal, you kind of have to squint a lot for node local dns. I attempted to spell node dns.

C

There is additional work needed in upstream projects in order to get a practically functioning cluster. At the time the stock was recorded. We don't yet have the aws load balancer controller working. This is probably because chaos wasn't giving it sufficient. I am permissions.

C

The aws cloud, controller manager, service controller, doesn't work with ipv6, so we will need to disable that and use lbc for services of type load. Balancer. A lot of lvc's defaults are only appropriate for ipv4.

C

We hope to make it configurable to use a set of defaults appropriate for ipv6 when running on ipv6 clusters. External dns currently doesn't support quad a records. This is a work in progress.

D

Hi, so I'm justin, I work at google, and so I thought it would be good to tackle the question of how specific to aws this work is now that all this great work has been done by the kubernetes community for ipv6 and by the k-ops and aws teams, specifically for running a full cluster on aws.

D

Can we just run this on another cloud? Well, the short answer is that it's pretty straightforward. We have to enable ipv6 on the cloud resources like subnets and vms, but then most of it carries across to gcp and presumably to other clouds. Also there are, there are some minor differences like on gce vms automatically get a slash, 96 ipv6 range, and so we don't need to write code to request it for each vm, but in general the sequence is pretty cloud agnostic.

D

The vm is assigned an ipv6 range. Our ipam controller writes that into the nodes pod sider, the cni provider then assigns pods ips from that pod. Sider ipv6 siders are fully routable, and so we don't need to configure anything. If anything, we now have the inverse problem. We need to be very sure that we have firewalled off any access that we don't want and generally because this sequence is cloud agnostic.

D

The good news is that by building on the aws work, it almost just works on gce here's, a quick screenshot showing the classic guestbook example running with ipv6 pod ips across a few different nodes. These are in the 2600 range, so these are fully routable ipv6 ip addresses.

D

The gotcha, though, is that not everything quite works. When it comes to networking the ipv4 kubernetes, networking stack is pretty complicated, but for ipv6 we need much less. We don't need an overlay. We get a cider with enough ips. We don't need routing logic. The sliders we get are fully routable. Ipam is pretty simple.

D

Now there are plenty of ip addresses, we don't have ip reuse problems and we generally don't need to worry about running out of address space, but, as we've seen with aws, some of the assumptions made by the stack simply don't apply anymore right now we have a few snafus as with aws.

D

The classic way of exposing services with type load. Balancer doesn't really work. The load balancer is exposed on an ipv4 address and cubeproxy and iptables can't not that to our pods, because we're turning off a lot of layers like setting up cloud routing. Sometimes we now miss steps. For example, the network unavailable taint sticks around and we now need new code to remove it.

D

All of these are surmountable problems, but they are new problems, maybe hopefully, by the time you're watching this at kubecon. We'll have fixed all of these, but I think what's most interesting for the future direction of cni in ipv6. Is that these aren't the problems that cni providers addressed in ipv4.

D

Cni providers in ipv6 are going to be primarily about a small bit of configuration and generally getting out of the way. What's far more important in ipv6 is firewalling and generally preventing access and those are pretty different requirements to ipv4.