Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 22 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG-Network Intro & Deep-Dive - Tim Hockin, Google & Vallery Lancey, Lyft

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

SIG-Network Intro & Deep-Dive - Tim Hockin, Google & Vallery Lancey, Lyft

Networking is hard! This talk will start with some background on Kubernetes networking. Attendees who are not already comfortable with the "hows and whys" of basic networking in Kubernetes can get a bit of a primer before we dive deep on a few of the more recent developments and efforts in the networking space.

https://sched.co/UakP

A

This is probably the closest we're going to get to a signet work. The movie this conference, so we're gonna start off with a introductory session. So if you want to learn more about the network stack in kubernetes and how it fits together, this is a good place. I'm, going to step through a walkthrough of kind of the base components take maybe a couple quick questions around each one.

A

Then move on to the next we're around halfway through the session we're going to switch to a deep dive, which is more focused on some of the recent and ongoing work and we're going to have contributors speaking to what they've been doing and what we're kind of planning for the next year or so down the line.

A

So for a mile away, is Sigma Network, it's just responsible for the networking related things in the cluster. This translates into a whole lot of things. We own a bunch of components: individual controllers, we own api's, we own specs, so things like CNI drivers. We are trying to not maintain any I, think there's one that's being deprecated, but we maintain a specification of what is the CNI. How does it work? How should the drivers work and then we owned many fuzzier logistics such as just?

A

How does the traffic move around in the cluster, how the networking related things in the control, plane, work and so on?

A

So let's go through our walkthrough of the components, starting from kind of the base of the stack and moving our way up to the higher level things so, first off, if we just install a cluster, say we're doing something bare-metal. This is what we get. We get some pods on nodes and if you go and try to ping google.com or hit your other pod, you can't and that's because you're missing the CNI driver, because there's not one installed by default.

A

If you use a distro, there might be CNI driver is responsible for actually handling IP allocation and dealing with routing from thing to thing. So the driver is responsible for managing your IP space, pulling down some IP and giving it to a pod and then making sure that some kind of routing system and rules exist, be it like IP tables or eb PF or some kind of cloud system to actually get your packets from A to B.

A

So maybe do we have any quick questions on the cni and what it does.

A

No hands all right so now that we have the ability to just hit other things in our cluster, that's pretty manual. We want to be able to do more than that, but also we need to think about securities. So there's a lot of misconceptions.

A

I've talked a bit earlier this week about some of them around what can access what so here's the graph if you've got four pause in two different namespaces as to what can actually communicate, and the answer is literally everything can so, even though we expose nice little pros, like short service names within a namespace, every pot IP is accessible from every other pot and that's not ideal.

A

We use something called Network policy to help with this network policy is optional, spec that many CNI drivers support that lets. You essentially add firewall rules to your pod to pod interlinks the way network policies work is by default. It fails open. So if you have no policies, then everything still took me to talk to everything else. As soon as you add any policies, we use selectors to point out positively apply to and specify ingress and egress rules.

A

As soon as the pot is selected, it's completely shut down, except for any rules, so the selector wipes out access and then at that point any additional rules. You apply specifying that pod layer on things like whitelisting to let more specific things access. So in this case we have a denial example. All we're doing is we're taking some pods that match up a and because we're selecting them we're now denying all traffic, because we don't have anything explicitly letting drop again. For those who aren't familiar.

A

The label thing is a very, very common pattern: incriminated internals, essentially as a way of reflecting relationships or pattern. Matching we have labels on any given thing using just like app is something is a very common but doesn't have to be app. You can use multiple and then, whenever we want to reference this, such as like a hierarchical relations, we use those same labels and say all pods are all resources that match this.

A

We apply to it.

A

So here's a somewhat more practical example of ingress we're taking a namespace that matches a team selector we're taking a pod selector for a and we're saying that for B we can Inc us, so you can't ingress to B, except from this thing a in this namespace that looks like this. This is a much more realistic, well subset of what your service graph likely looks like, because they're, probably very few things in your cluster, you actually want to expose to everything in the cluster.

A

In reality, you have some kind of a complicated digraph of what is actually interacting. This is also something that a lot of service, mesh functionality and so on exposes in similar ways. So any questions briefly about network policy.

A

Quiet crowd all right, so next up is end points we want to make pot. Ap is more accessible if you've ever had to manually deal with I have a fleet of VMs. How do I connect to them all you'll understand very well. Why we want something more than just exposing raw poppies and points provide two sets of abstractions for us.

A

The first one is just that it's a list, so we have one object with a whole bunch of IPs. We don't actually have to list all the pods and scan through them. The other thing is, it includes the pod health checking and so on. So if a pod is not ready, it's not in the endpoints. The pot is ready. It is in this way we just have suddenly a list of everything, that's ready for traffic, that we can then easily consume in our service discovery or used to populate a load balancer or something else.

A

Questions about endpoints, ok from this point, yep.

B

C

Sorry, but that question was when the controller, which is observing, let's say an important creative end, sees the creative end. Does it mean that the port providing that endpoint is in the ready state? It's a guaranteed or controller needs to basically double-check that the port is in the ready state.

A

I'm pretty sure it's when the check is ready, but I would appreciate clarification on that right wrong. There.

B

There are actually two lists in the endpoints structure, one for ready one for not ready, so you don't need to worry about it.

A

B

Another question here.

D

I have a related question on the previous slide. We saw the policies yeah, so this kind of structure so who or where is it in force? Is it in force at the CNI level? So we see the podrace creditor it is passed on, but I believe, probably the controller doesn't do much about that. It just passes it on to whoever implements the network, so that is it handled at the CNI level, enforcing the policies or where are they implemented, which level this.

A

Is enforced by the CNI drivers but because the network policy is not a mandatory part of conformance, not every single controller does support it, but this would often translate into like not making a rule in Google cloud or not programming and iptables rules. If a certain path isn't allowed and.

B

There are also a bunch of third parties that can implement network policy independent of networking, so you really need to put together a comprehensive network plan.

A

All right so on to the next I guess: just wreak Lera fication on labels already kind of touched on that it's just the way that we are referring to the kind of hierarchy of things. So we have our set of pods. We also have a label specifier in our endpoint set, specifying how we actually pick, which pods were washing for their readiness or lack thereof.

A

So because, again, just consuming raw endpoints that it's better than having to watch all the pods, but that's still not super out of the box useful because you have to actually coordinate everything. So what all the time we just want a load balancer of some kind. We just want to hit one thing and not fuss about it at all.

A

So services are the answer to that services: program up a load balancer the default, the sorry, the behavior and logistics of which vary and point it at all the end points in doing so they also create an IP again, the origin of which various includes that in the service. So now you have one single IP to hit, and the service controller is now continuously making sure that all the ready endpoints are programmed on the load balancer. You know there's a number of ways that this works. First are.

A

There are four different kinds of services that all may have a bit differently. First, one I'll mention because it's very simple is type external name, and it's basically just a cname. If you want to use that mechanism to expose an alias, it can be seen them to anything, including some random domain outside your cluster. The next kind, and probably the most common kind, is a cluster IP.

A

Cluster IP allocates some IP within the kubernetes cluster and then has an internal load balancer. So I'll talk more about one implementation in a sec that we have for that built in next type is node port. It exposes a specific set of ports across all your hosts and it relies on direct investment there. So this would be, if you want to say, have some external load balancer or some external service discovery just pointing easily at your nodes, something usually from the outside, and the last kind is a combination essentially of coaster, IP and note port.

A

It's type load balancer, it's basically the most primitive form of ingress. You can get where you are, at least in terms of the constructs where you are opening new ports, and you have a cluster IP, so you hit a given node, pour it on some machine and then you get routed through the cluster IP load. Balancer to your pod, this does have some downsides, because you are routing around a little bit within your cluster. It was a better alternative that I'll talk about later on questions so far about services.

A

I'm making Tim work here.

A

E

So they'll awesome that you mentioned lowdown, sir I, don't know what it was. um Why would you ever use that instead of node poor, if it's just note port plus more like that, doesn't get you anything.

A

It gets you the more, so you don't necessarily have to rig up the load balancer yourself, because you can rely if you have like a cloud provider, integration or something on them to spin it up for you. So this would be a way to just get like RTC people are balanced or spun up for you. It gives you ingress into the cluster.

A

F

How much control does a user have over the load balancer for the cluster IP.

A

I'm not entirely sure I believe it mostly depends up to the implementation of whatever service controller. There is there's.

G

At least out of the box, the cluster IP is or the cluster IP is a range that you give to the cluster creation time and then it's either Auto allocated or you can set it and it will try to get it and if it can get that one that you wanted I'm sure, there's an error right, there's an event.

F

G

Actual you mean how the traffic is split, that one is not currently part of the API, except for external traffic local, which governs whether the traffic is going to be routed to your local node or across your nodes.

A

So it's not part of the API. You can control it by. If you can configure your load balancer itself like to proxy, which is a built in implementation. You have a lot of different knobs. You can tune around how it works and there's theoretically, some load balancers, that let you use annotations to add extra features.

H

Yeah thinks I'm.

I

H

We still like canonically I. Remember we used to have like the cloud providers all used to use like there'd, be everything would use note poured under the hood as like the glue like I, remember at least GCE or GK used to do that. Is that still normal to use note port under the hood for what your tcp load balancers directly talk to if you're in the major clouds so.

G

Actually GCE does not use note port because there's this it can balance the VIP directly onto the VM. I know AWS used to maybe not anymore, if you maybe not anymore,.

G

Right and then there's at least I can mostly speak to G CP g CP has this notion of network endpoint group, which is which basically for certain types of load balancers that can go directly, so it doesn't need to transit via the node port to get to the service yep.

G

So in some sense it's up to the implementation, whether or not it actually needs the node port. Some do some don't.

J

A

No port with type load balancer is common, but not strictly required by any spec. So if you expose pot at Pease directly on your network, like I, think that's a common Google behavior at lyft, we use a CNI that does that. Then you can directly program up individual pods on your load, balancer and that's also usually way faster because you're not translating through anything you're directly going to your instance.

K

Yeah this does the clustered provide any mechanism for the load balancer to provide a feedback about the other status of the endpoints, or it is the responsibility of the rope balancing implementation itself.

A

There's health provided about the little balancer itself, but I, don't think there's anything in the spec that directly provides endpoint level feedback. So.

G

Depends on what kind of feedback you want, but just earlier this year, there's a feature called pod ready. It's either called pol ready, plus + or pod readiness gates that basically lets the infrastructure. So when a pod is coming up, it first comes up and then it becomes ready.

G

The question might be: ok if your pod is participating in a load balancer. How do you tell the system that that pot actually hasn't been programmed yet and basically, pod readiness gates is a additional gate that the infrastructure can add to your pod, to say: ok only when the load balancer has programmed this pod then say flip this gate to say: ok, this pot is actually ready because it's not only just ready from a local level, but also from a network infrastructure level. Essentially.

A

It's a way to make sure that, like my pod is on the load, balancer is considered part of the readiness check so consider. If you didn't have this I want to go and rule my deployment and update my app. Ok, all the replicas are replaced, but up a program. Deliver balancer takes a minute and everything's swapped out so suddenly from the perspective of my load, balancer everything's down. So that's why the readiness gates are critically useful is because we can introduce that extra concept of.

A

Is it ready without needing to get in the way over the path of the health check, because otherwise you can have a can opener in a can situation, because if you try to block the health check, then you can't necessarily rep traffic to it. I.

B

Will just add to that that I think this is a place where we can do better been looking at a lot more higher level. Networking systems later, like load, balancers, that that really want their own direct endpoint health checking and our API is don't fit the for some of these yeah.

A

I think this is an area for growth. This year we provide the building blocks, but it could be more powerful and it could be a little bit less mechanical I saw.

B

A hand in the back there boy it's on you all.

G

L

Is it that's the first packet that hits a load balancer or is it up the sessions shakiness that every packet goes to just a single and destination like say you wondered 1.1.1.

A

Sorry you're asking if there's a single destination so.

L

You will have multiple pots running behind this load, balancer for a single application. Would there be any session stickiness at the load balancer end, or would we just as first packet that would be heating the load balancer and then, after that, the packet would be reaching directly to the port.

B

Again, I can take that one.

B

So I think there's two questions here, there's one of stickiness, so there is an option in service for stickiness that that is based on the five tuple right, so client IP port protocol to the server IP port. So you can establish that if you want it, the rest of the question is about I, think about performance and like whether the load balancer touches every packet that depends on the implementation.

B

So, if you're looking at the default like the IP tables implementation, whose based on IP tables NAT that will look at the first packet, establish the connection, make a connection tracking record and then ignore the rest. Other implementations might do something different.

A

All right, I think we're getting more and more questions as we get to more of.

L

These Vimal sorry! So what? If you have configured chases stickiness and the endpoint dies or the port dies? What happens in that case when.

B

We detect that endpoint has been removed from the endpoint set. We will remove any I think we remove any connection tracking records that were related to that endpoint, so that you'll have to pick a new new endpoint you're. Basically, your stickiness gets lost, at least that's the default implementation, yeah.

A

A lot of the specifics do wind up being, however, the load balancer works because the spec doesn't control the load balancer at a fine granularity. It just provides the basic programming details.

A

A

Services and endpoints are designed as a way to provide potentially custom solutions, but there's one that many many people use if they're not using a service mesh and that's cube proxy ku proxy is our built-in service implementation that programs things with its couple different modes, most people use iptables and it deals with assigning the cluster IP in terms of network rules and then forwarding to all the backends. So in iptables mode, it's actually kind of funny.

A

If you look at it, we basically create a redirect on the cluster IP to a whole set of IPs all with kind of different weights, getting more and more likely, and that's the very kind of interesting approach to doing random distribution. There is a PBS mode which lets you do like real round-robin routing and more advanced things.

A

Thank you Tim and boy for the running around so.

J

The question is around q, proxy and IB tables and I PBS mode. Let's say: I have some custom OVS kind of based implementation. Would it be possible to write a Q proxy plug-in to like program all these kind of iptables rules as connection tracking rules in the OVS? Would it will? Is there some something like a plugin to keep proxy that it that we could write in the future.

B

As the maintainer of Q proxy, no I I don't want any I, don't want to build it anymore. Monolithically than it currently is. I'd rather have people fork it and build a separate one. That does exactly what you need. Basically, there's way too many config parameters that need to come in to cube proxy to cover all the different proxy modes, and it's become very confusing for people and I would rather see people sort of vertical eyes and go go build their own thing, so I've basically put a halt on new proxy modes.

B

In theory, you can do what you're talking about, but I'm, encouraging you to do that in a separate code base. Oh.

B

And sorry, and by the way there was a new implementation of services launched this week from some of the folks at VMware, launched something called an Andrea I hope I get the name right. It's an OVS based implementation.

A

So DNS, if we have a pod, we can nicely route to one of our names, like my app or nginx Aretas, and there's a reason that all that magic works as well as a lot of custom internals and that's because we inject DNS reading information. So when the cubelet starts up container the cubelet being the component that actually manages containers on the host we program in some service endpoint, where DNS is supposed to go.

A

Those DNS requests are then routed to some DNS service normally running in the cluster, like the default. That shipped is accordion s deployment, but you can customize how that works, and you can also pick a different DNS setup if you so desire.

A

Derek dear I asked questions around DNS.

A

Alright, thank you for that.

A

So the one that I suspect everyone's going to have a question on is ingress. Ingress is our kind of top-level http F or not htv7 layer, seven HTTP way of having an composing cluster traffic, that's from an external source so going into the cluster the way ingress works is we define a set of HTTP backends using potentially rules like host names or paths, and then correspond them on to some service. So there's a couple of nice things about this. One is just that it's a way to actually do your layer 7 routing at config.

A

So instead of writing a custom, nginx deployment and making sure you have to update your nginx deployment to read up to your services, we provide a more carbonated native way to do that.

A

The functionality of ingress really really varies, depending on what control you're using so later in the session. We're gonna be hearing more about like what's wrong with ingress, how it's going to get cleaned up in the theoretical v2 future, but right now the exact rules that you can use very provider to provider and if you've ever especially had to use different providers for ingress you'll, see that there's all these custom annotations.

A

They, like you, defined things like TLS or a named IP from some pool, or algorithms or security settings or cache settings, and it's all different and that part is kind of unfortunate, because it's one of the more sticky pieces as soon as you go from cloud to cloud or system system, or even just if you want to like switch over to a cheap nginx in cluster stuff.

A

So questions about ingress, I'll, specifically defer future of ingress questions for the next talk.

A

All right, here's an example of us composing, some very simple rules. We have our main path, so we default with some particular back-end, and then we have an API back-end, where everything that matches that HTTP path gets read it there so much nicer at least than having to have a custom router in your cluster. For this.

A

So that's a overview of. What's in the networking stack I'll touch briefly on some of the stuff that's been going on lately.

A

Definitely biggest pieces of news we even heard about them in the keynote is ipv4 ipv6. Dual stack is now available and is continuing to progress. That's been a huge, huge effort for a long time and it's been a notable pieces of work in the sig since this spring I think almost everyone has chipped in in some way. With that effort, the big one is also the endpoint slash change where we enhance the ability to scale out clusters to have more nodes and or more pods, by reducing the traffic churn. That happens.

A

That's something also we'll hear about in the next talk.

A

As far as upcoming work goes. Future of ingress is a big one ingress with beta for a long long time. At this point it's effectively GA and we need to figure out how to polish it up officially GA and then kind of move on to building better systems, and we don't have very particular plans here yet, but we are looking more at the future of multi cluster networking and trying to figure out how to make that work better.

A

So if you want to get involved with the sig, it's relatively easy, dare I, say: I got involved first, a year ago, at last cubic on North America, there's a couple different ways: we communicate slack and probably mailing lists are the best ones. Didn't actually put a link here to the sig network info page. But if you just google criminate a sig network, you'll find us. We also have a biweekly hourly call on Thursdays at 2:00 p.m. where we kind of talk in person about anything that requires a bit more coordination or agreement.

A

So if you want to check things, I would recommend subscribing to those and maybe just shadowing a caller to and kind of seeing what's going on, promise we're a friendly Bunch you can see. Lockean is Halloween costume there. Oh.

A

Yes, that's the one of the Starfleet engineering uniforms.

J

A

So also, if you want to check out like just units at work and dive in really helpful, if you can help review PRS, especially if you're a pretty good coder viewer, helps to understand the stack very well, but there's a lot of things that are easily transferable or look at issues see if you can triage them and verify them, maybe even work on a solution. If you don't think it's gonna be too challenging.

A

So in github we have our carbonated kubernetes repo and everything that is for the cig is labeled Signet work in the issues and the PRS.

A

Also my advice is someone comparably new is do start small, because a lot of this stuff is much more complicated than you probably think, and because this is a big project, a lot of tech, debt and many many users. There are many things that are conceptually simple and in practice are not so yeah. If you want to find us, here's how you do so we'll probably have a pretty quick break and then bring on the next speakers to go into the deep dive. Thank you.

G

So this is the Signet work, deep dive on Bowie I'm, Rob, I'm Dan and today in Barcelona I think we did a signal, work deep dive and it was really deep into some of the topics that Valerie has covered in the first half. But today what we're going to do is deep dive into what is new in the networking space, and we have about five topics that we're going to be covering. One of them is endpoints slice.

G

The second one is the ingress API and service in ingress, API evolution, and we cover dual stack service, topology and, on everyone's mind, setp know it's very important protocol, so endpoint slice so.

M

Endpoint slice is a great new feature in kubernetes all about making endpoints more scalable, but to understand endpoint slice. We need to understand a bit of background of how we came to endpoint slice, how that was the solution.

N

M

M

So kubernetes clusters, just keep on getting bigger kubernetes is successful. It's great news, but, along with that success comes problems. That of scalability and endpoints became one of those bottlenecks. Each network endpoint for a service needs to fit in a single endpoints resource. Now this works great up to a certain point, but when you start to get services that have lots and lots of endpoints behind them, let's say you have thousands of pods. You start to run into some real scalability issues, because all of those endpoints stop stop fitting in a single resource.

M

There's a real size limit at CD default size limit that just prevents that from happening by default, that's somewhere between 5,000 and 10,000 Network endpoints, depending on a few different configuration variables. But at that point you just run out of space in your endpoints resource.

M

Now you could modify the configuration and let that get even bigger, but the problem is that doesn't really help anything, because we have a second problem that is very related to that first problem and that's that any change to the endpoints resource needs to go to every single node in the cluster and that's not great. So in a 5,000 node cluster, a single endpoint change could result in around 5 gigs worth of data being transferred, so just think about DVDs worth of data going out every single time of POD changes inside your cluster.

M

It's not great, admittedly, not that many people are running clusters. This big, because it's not great yet but endpoint slices- are set out to try and fix that. So the idea behind endpoint slices is really simple. We have this huge endpoints resource. Instead, we have smaller endpoint slice, resources that are really just slices of endpoints, really simple.

M

So here's an endpoint gamal because we all like yeah MLAT, cube Khan. Here's what an endpoints resource might look like. You probably haven't interacted with this very much because it just kind of happens, B's hot, behind the scenes.

M

You have a service, you have some pods matching that service and behind the scenes, there's an endpoints controller that runs through and tracks all these IPS for those pods and keeps track of them for you and, like I, think Tim mentioned earlier, there's an addresses field and there's a not ready addresses field, and these are just both lists of addresses that are either ready or not ready for your service. So it's a really simple concept that works really well to a certain point and then starts to fall apart.

M

So with endpoint slices, it's a similar thing, just with a few extras out of it. You can see that we've added an address tag now so ipv6 is supported, address type as an example.

M

Additionally, we've added some labels because with endpoints we had this really simple thing where you have one endpoints resource and you have one service, so the names are the same everything Maps directly. It's really easy, because now you have multiple endpoints slices for every service. You have to do a little bit more work to reference the service that an endpoint slice belongs to.

M

So in this case we have a label referencing the service name, and we also have an owner reference that I have not included here, because it would make it completely unreadable, but an owner reference pointing back to the surface as well and there's a few other differences. I'll cover shortly.

M

So one of the big additions with endpoint slices is, we specify and validate IP addresses. So we have address types that are ipv4, ipv6 and fqdn. Eventually, we may add more address types, but for right now, that's where we're starting and for a given endpoint slice. You know that addresses in that slice are going to be valid, ipv4, v6 or fqdn, depending on the address type on that slice. Ports also now have an app protocol field, so we already have a protocol field on all ports and kubernetes, but that's l4, TCP, UDP, etc.

M

Whereas with this a protocol field, you could actually specify something like HTTP, which hopefully will be useful for consumers of an end point slice, API, endpoint slice, also switches around what we're doing with conditions so before we had just a list of addresses that were ready and addresses that weren't ready. So if you wanted to add any additional contingent condition or information about an endpoint, it was actually pretty difficult. So we've switched that around to add a conditions field on every app on every endpoint.

M

That right now just includes a ready boolean, but could include additional information about the condition of a given endpoint in the future and additionally, endpoint slice. Endpoints contain topology fields, currently that's node name zone and region, but arbitrary information can be added here, very simply, as just like a labels field, just specific to topology information and finally, a key point here. This is not enabled by default in any kubernetes cluster and 1.69 1.17. You need to use the endpoint slice feature gate to enable this.

G

What it is beta, it.

M

Is beta? That is true. So let's talk about performance because the whole and we went through all this- was to try and make this skill, and the great news is the first implementation of the endpoint slice controller worked well at scale. It was fine, it actually performed a little bit better than the endpoints controller, but that wasn't actually ever the real bottleneck.

M

Here, the real bottleneck came with cou, proxy and bandwidth and and I'll say, data transfers dramatically lower, but cou proxy was still a bit of an issue because it's a little bit more complicated now so I needed to do some testing some scale testing to see what we could improve and my basic methodology here was to spin up a cluster with Kubb test, modify the KU proxy manifest just to enable profiling and then use coop, color port forwarding, along with p prof, to actually profile coup proxy, and this is actually really really helpful if you're ever trying to debug anything in kubernetes and understand why things aren't as fast as you might expect.

M

My colleague min Han has a really cool tool called coop script on github that allows you to push custom builds of qu proxy to specific nodes in your cluster really easily, if you're, debugging, qu proxy or other networking tools, I highly recommend that, and so what I would do is I have a whole bunch of different versions of coop proxy running in the same cluster on different nodes and compare their performance and the final results I'm going to show we're gathered on a hundred and fifty node cluster over a 15-minute window, scaling from 0 to 10,000 endpoints for a service and I used, four versions of coop proxy kubernetes, 1.16 1.16 with slices enabled and the same for kubernetes 1.17.

M

So before I get to the actual charts. Let me show you the three changes I made because they ended up being significant.

M

The first one was I was using endpoint IP for sorting and that ended up being a incredibly expensive operation. I had thought endpoint IP must just get the string and return it. The actual string in the endpoint object was a IP port combination, so it was calling net util to parse an IP out of that and then returning a string, so that was very slow and I was using it for sorting and not good. So 41 percent of total CPU time was spent.

M

Calling this single function not great so I changed that just to use the string that already existed, and this went away so on to the next problem.

M

So there's this cool thing an end point sliced cache that transforms a whole bunch of endpoint slices into a data structure called endpoints map, and this is the data structure that Kubb proxy uses behind the scenes to actually render proxy rules, whether that's IP tables or IP vs. They both use this data structure to actually render the proxy rules, and this was called on each endpoint slice update.

M

Now that's about how endpoints work before it, which is fine, but the problem is it's a lot more complicated to merge a whole bunch of endpoint slices together and then actually transition them, transform them into an endpoints map. So this really didn't need to have one on every single endpoint slice update this data structures actually only used when the proxy runs and at scale. There can be a whole lot of endpoint slices being updated between proxy or runs as an example. You can get many many endpoints, less updates.

M

Every second and proxy errs may only run every few seconds, so there's a significant difference here, changing this logic around, so that we could only compute this when it was needed, ended up being a significant savings and finally there's a detect stale connections function that was taking after these previous fixes, a whopping 82 percent of total CPU time, and it was being called on every single connection, even though it was only actually useful for UDP connections. So this was kind of a big deal and I imagined for most people.

M

They had at least some TCP connections in their cluster, if not most. So, if that's the case, we're wasting a lot of time on something that didn't need to be called now. We didn't actually have information about what protocol was here at the point, but just passing changing some things around, so we could have that information ended up being a very significant savings. All right. So that's that's the three things I changed.

M

Let's get into the effect that had so memory, it's an improvement kind of the the two low lines there are, the new 1.17 implementations and the two slightly higher lines are the 1.16 implementation. So maybe a 20% improvement, maybe, but it could also just be noise, so this isn't a huge sample size, but there's probably some improvement here now, CPU usage was a little bit more dramatic. My first endpoint slice implementation was not great.

M

It turns out that big, yellow line that continues three minutes after all, the other implementations are done is my initial endpoint slice implementation so not fast and the rest are significantly better, but you can see the line the the trajectory that initial endpoints implementation had it as its scale that kept on getting slower and slower and taking up more CPU utilization. So even our baseline was going up at a pretty rapid pace here.

M

The two lower lines are the 1.17 implementations, one representing endpoints and the other represent that the other slightly lower one representing endpoint slices. So this is a fairly significant improvement and where I think it really becomes more obvious, as if you look at the P prof numbers we're seeing here, which are related to CPU time spent over this entire window and endpoint slice.

M

The initial implementation is an absurd number there, but even our initial endpoints implementation is pretty slow compared to our newer implementations and, finally, to put this into a bit of perspective with some real numbers, we improve performance from the baseline in 1.16 around 20 times. This is only really I'll give a little bit of a caveat here. This is only really going to be noticeable when you're operating at scale when you've got a few hundred endpoints in your cluster.

M

It doesn't matter that much, but when you start to get into the thousands and tens of thousands of endpoints, this is going to be a huge difference. So that's what we found with some relatively simple to proxy performance improvements.

M

So let's talk about the endpoints I placed time line and when you might actually get to use this when it might be enabled by default, the design was actually proposed in Barcelona this year at PubCon the cat was created in june and approved in july. We got our first alpha in 1.16 and the it was actually released as beta and 1.17, but not enabled by default, we're hoping that we can get this enabled by default in 1.18 and if that all goes well, maybe we'll get GA sometime 1.19, but that's admittedly optimistic.

M

So there's lots. You can do to help us with endpoint slices first off. Does it work for you? Can you break it? Are we missing something we want to get this out? There there's some great improvements along with it, but to really get this enabled by default. We need more people using it. So please test it out and speaking of testing, if you have ideas for how we could test this better and make it more thorough, definitely interested in better tests for endpoint slices and then maybe the biggest thing of all. We want to migrate.

M

The use of endpoints API over to endpoint slices, so this includes things like the windows, implementations of coop, proxy ingress controllers, dns and probably other things that I haven't listed here. But with that I'll turn it over to Bowie to talk about ingress. So.

G

In terms of every API evolution that we want to do in the system, we first have to kind of look at the landscape and how it has changed from sort of when it was originally designed to what it is today. So, on the hopefully right-hand side of this diagram, we have kind of the landscape of proxy ELB providers and controllers, so I kind of loosely categorized, these into cloud, elby's, middle proxies and transparent proxies, and as we see that the functionality all of these is slowly converging in terms of what they can all support.

G

It's, but it's a slow process on the left-hand side. We see that as clusters become bigger and bigger, for example, in the case of endpoint slice, is that there are more complex roles and personas involved in kind of deployment of applications on a cluster. In the early days of kubernetes, we really focused on the empowered developer, who is going to be a single team? That's going to self service, most things when you get to bigger and bigger clusters that are shared among multiple teams.

G

Now these roles have been split up and as an example, some of these personas may be an infrastructure provider. This is someone who owns the cloud environment, for example, setting up the cluster environment itself. Then there is the cluster operator who operates the cluster and enforces policies globally across the cluster or maybe net ops who enforces policies across all of networking and then finally, we have the empowered application developer. So the key behind API design is to kind of think about how this evolving landscape then translates to api's.

G

Now talking about ingress ingress is actually a pretty popular API. It has very wide support among different implementations and for some of the cases it's actually good enough for a non-trivial number of users. So from a cig network standpoint, ingress is still a very valid API and the plan is to clean up the spec and take the type to GA. Did you know it was still beta?

G

So what are we going to do for the ingress API, so clean up the object model? So, as you saw the diagram before there's this new resource called ingress class, in fact, every single ingress implementation has an annotation that says: okay, which controller should be in charge of this ingress, while there is an engine X or it's a cloud provider and so forth, so actually make that a real resource.

G

The next thing is to tweak and fix the specification, so we have a list of sort of minor discrepancies between how it is specified or maybe under specified, currently in the API and what the actual behavior we see of all the different implementations out there. There are some theory names like back-end to default, back-end the path and host name matching.

G

Actually, the implementations we find do not sort of follow what is in the comments in the types that go so we're kind of cleaning that up and then finally we'd add a bit of flexibility that will be hard to change later in the GI api, which is to enable you to not just route from your ingress to a service, but also potentially targets and alternate back end. However, for the first cut we would expect controllers to only support the service type.

G

Now, let's talk about evolving this so, as I said before, we had a slide with kind of the changing landscape. So what does this mean for the goals of an ingress, api evolution and remember everything? I say here is super early proposal and the point of Q con is to get all of you, users plus implementers, to give us feedback. So we see at the high level three main goals. One of them is looking at. Those personas is to better model them with the resource model.

G

The second one is as the implementations, all convergent capability is to support more modern load balancing features, and then we did an ingress survey user survey in 2018 and we came back with two opposing viewpoints. One of them is that it must be portable and the other one is I must be able to use all my power features of my underlying implementation. So important is key, but maybe not just portability but sort of predict ability for the user.

G

Can a user predict how portable their thing is and then not limit them if they want to go out of that spec finally, sort of build into the API standard mechanisms, for extension, not just for API growth, but implementation, and maybe vendor specific behaviors, because, though one of the feedback that you could see from the survey is, if I am actually a power user, don't limit me I know what I'm doing so.

G

This is a sketch and this is sort of an evolving API. But what I want you to take away today is sort of look at this and get a general sense and work through your your use cases. Whether or not this setup makes sense. So, generally speaking, we have four resources here: there's a gateway class, a gateway, a route actually many types of routes for each protocol, and then your kubernetes service and I'll go through kind of describing what the role of each of these resources is in your cluster.

G

So at the very top we have gateway class gateway classes very analogous to ingress class. This is intended for the infrastructure provider to provide different kinds of load. Balancers that are available to the user of the cluster. Now in this example, we have two gateway classes.

G

One of them is internet lb and the other one is private LP, and in this cooked-up example, when you create a gateway of Internet lb class, that represents that the user intends to export that service as an internet available service and then, if they use the private lb class, that means that they are intending to perhaps export their service to their local VPC.

G

Now next comes gateway and route, so gateway is sort of the linchpin of this API gateway is the thing that when you instantiate a gateway, it kind of wires everything up now what gateway contains is sort of a concrete manifestation of an instance of load balancing, for example, if you were on a cloud provider and the gateway class was by the cloud provider, creating a gateway will then call a bunch of cloud api's to wire your low Belzer of.

G

Additionally, if you were on say using an in cluster implementation of ingress, but now gateway in route. It will then instantiate and do the configuration to sort of provision, your load, balancing and then. Finally, this green box is the route. This is the application description that will, you know, be controlled by the application team and because all these are three different resources we can kind of attach different roles to them. For example, infrastructure provider will get our back for a gateway class, the net off spur.

G

It helps want to restrict how gateways work, so they get the gateway and then the dev team could get the HTTP route. Oh, you can still have self-service. Basically, you would give the dev access to Gateway HP route and then maybe in the most extreme case, you could get gateway, class gateway in HP route, so kind of this is still early. We have an H. This is these links, don't work if you're looking at them, but when the document is shared, you won't be able to click on them all.

G

So these are also on the cig network, mailing list so sort of how to get involved. This is still very early stages, and really we really really really would appreciate feedback from users and implementers and there's a working group. That's going to be started it's coming soon in terms of the info. We intend to have bi-weekly meetings, use the cig network mailing list. For now, until people tell us we're spamming them too much and then also create a slack channel and there'll be a repository in which all the spec working issues will be held.

G

So really, this is feedback on the proposal we want to hear from users and implementers.

G

In fact, when we had the session that we on Tuesday I think we went into a deep dive and there was one user thank you for coming, but it was very helpful to hear their voice and, of course we want to see experimental implementations as well. So moving on to dual stack.

M

So dual stack is really exciting feature of kubernetes that is starting to see some life in the form of an alpha release. To understand what we mean by dual stack and kubernetes. We need to run through the goals for dual stack and kubernetes, and that is that pod should be able to support both ipv4 and ipv6 addresses. They should be able to communicate ipv4 to v4 or ipv6 to v6.

M

They should be able to access external servers on ipv6 or ipv4 addresses, and the same pod should be able to be targeted by either an ipv4 service or an ipv6 service. Node ports and external IPs should be support both ipv4 and ipv6, as a departure from that services and points and endpoint slices will all either be ipv4 or ipv6. They will all be an individual single family and, of course, with all of this, we want to ensure we're maintaining backwards, compatibility with ipv4 only and ipv6 only clusters.

M

So, as you can probably imagine, dual stack is a really complex feature to implement, there's a huge it's a huge project and it affects lots and lots of things across the kubernetes ecosystem. As an example, service, ingress controller, endpoints, endpoint, slices nodes, CRI CNI, runtimes cluster configs, and if you think about anything in kubernetes that touches an IP related shell scripts, whatever it happens to be all of these, things will have to change to support dual stack. So there's a lot of work here, and that means there's lots of opportunity to help.

M

But let's first talk about the alpha release of dual static, so first release. What for alpha was 1.16 and we got some updates in in 1.17. All functionality is hidden behind a feature: gate, ipv6, dual stack and the change is involved right now are that pods will have a new pod IPS attribute, so that pod IPS attribute will contain both an ipv4 and ipv6 addresses the services have a new IP family attribute that will either be ipv4 or ipv6.

M

Endpoints will contain addresses matching. Only the Service, IPS family and endpoint slices will behave similarly, but they also have ipv4 and ipv6 address types that add an extra layer of validation there.

M

So, let's take a look at a simple example: here: we've got a pod that has both v4 and v6 addresses and we've got a service that we created that matches that pod and it's assigned a cluster IP that is matching its IP family, so v4, and then because we have a service that matches the pod. We have an endpoint slice that gets created automatically for us and that pulls the ipv4 address out of that pot. Now. Similarly, if we want to have an ipv6 service matching the same pod, that's also possible.

M

So we've got a service with an IP family of ipv6 and it's got a v6 cluster IP and automatically created for us. We have an endpoint slice that pulls the ipv6 address out of the pod, and that represents that endpoint slice. So that's a very high-level overview of how dual stack is going to work inside kubernetes and I just used endpoint slices here. But of course the same is true for endpoints.

M

So a huge part of all of this is cout proxy needs to support dual stack and for any of this to work and I think it was Cal had the idea to use some kind of meta, proxy or design which basically involves running separate proxy cures for each IP family. Instead of making huge changes inside each proxy er just pass either fully ipv4 addresses into a proxy ER or fully ipv6 addresses and endpoints into a proxy ER.

M

So you have to running in parallel inside qu proxy, so two implementations of IP tables or I feel I, PBS or whatever it happens to be, and those endpoints that match that family are automatically sent to that proxy ER.

M

Currently, that implementation is limited to the IP vs proxy ER, but there's some great work underway for iptables support and although there's no support for Windows proxies I think we hope that can happen soon so have to cover some of the timeline of dual stack and thank Dan for actually filling all this out, because there's a lot of timeline when it comes to dual stack. This goes back way before I was involved in kubernetes 2014 for the very first bits of ipv6 support and the initial ipv6 single stack efforts began 2017 and 2018.

M

We hit a dual stack proposal that eventually turned into a cap and eventually resulted in the 1.16 release that included alpha support for dual stack, just this August, so we're making progress, we're hoping we can get to beta relatively soon, but there's a lot of work to do and there's lots of opportunities to help. So coup proxy still needs a dual stack: implementation implementation for Windows. Additionally, for what has released we'd love feedback testing? Does it work for you?

M

Can you break it or anything, we've missed and additionally, because endpoint slices are also new and not enabled by default? Does this all work? Well when we have endpoint slices and dual stack working together? And finally, is there anything in this implementation that we missed along the way and I'll hand it off to Dan.

O

Service topology: this is something that is also relatively new in kubernetes we've been talking about it for probably a better part of a year, and even just last week, some cool stuff happens so we're starting on that road, but at its base, service topology tries to answer the question: how can I talk to a pod that, as close to me or as local, to me as possible?

O

So the problem that we have there, though, is what does local really mean, and it turns out that local means many different things to many different people, and it definitely depends on your infrastructure, how you set your cluster up, possibly what your networking solution is all different kinds of things, so we really need to make sure that service, topology, a enhancement proposal and as a concept, was flexible enough to address all these different use cases- and you can see here that the kept for it was accepted back in December.

O

There was it took a little while to kind of discuss that and to finalize the cap- and there was a number of months after that, where there wasn't a ton of work that was going on in public, but it turns out that there were some PR was posted relatively recently and merged last week, just in time for 117 I, believe that adds some of the initial bits of service topology to the various parts of communities, including the proxy implementation. So what does service topology actually do?

O

First, we can back up a little bit and say that we do have a.

O

Kind of an implementation of service topology in cube proxy already, and that was a tag that allowed the proxy to direct traffic to a pod that was on that particular node. So that's kind of very basic, but obviously it doesn't address some of the needs that other contributors had for locality and service topology.

O

So the cap adds a key to services that are excuse me, a topology keys to services and each of these keys, they're fairly well defined, keys and there's only a couple right now, but I expect there'll be more added in the future, as we have better ideas of exactly what topology means in different cases.

O

The proxy implementation then takes a look at the service itself and tries to match the the labels on the service, with particular labels on the node. So at this point it's mostly node based, and it turns out that the excuse me yep excuse me topology labels on the node itself. It matches them in preference order. So if you have, you know, you want to try the hostname first and if the hostname doesn't actually have any matches, then it will also try to match things like availability zone and kind of go on down the list.

O

So it really is under the control of the person writing the service and to some degree of the cluster when it decides what keys to put on each particular node there's also in future enhancements, there will be a pod locator resource.

O

It turned out that there were some scalability issues with the current implementation, and so the contributors working on service topology will be adding the pod locator resource to help address some of those issues around how you actually do the matching in last I think we have SCTP, which is stream control transmission protocol, it's actually more of interest to I, think more telephony applications and telcos, because that protocol is very heavily used in some of those industries and if you're not familiar with SCTP, it is basically like a mashup of kind of TCP and UDP with some additional features for reliability and redundancy.

O

It kubernetes only supported up until a couple of releases ago, I believe TCP and UDP for a lot of the service and ingress and other operations, and it turns out that as kubernetes is trying to enter new worlds and enter new industries and businesses. That SCTP was something that quite a few users actually were interested in having support for in kubernetes. So the implementation sits alongside both the TCP and UDP bits as a core kubernetes protocol, and that shows up in a lot of different places in the code.

O

So the one place that it might show up that most you might be familiar with is in the service, and you can kind of say that this service has a particular protocol attached to it. And that is the protocol that I want to have the proxy pay attention to when it needs to remove the traffic through the cluster to a particular back-end. So SCTP is now one of those protocols that you can add to your service, but that also means that we need to take that and move it through from services to also endpoints.

O

We need to add support to host port for that and then, of course, network policy as well. So the actual protocol does touch a lot of different and kubernetes.

O

The supports actually been there for a while, and it was alpha in kubernetes, 112 and we've been finalizing some of the rest of the kept for that and we hope to possibly make it beta in kubernetes 118 sometime early next year. It needs a little bit more of a push over that finish line. The other thing to note is that if you are developing a network plugin that you know a CNI, essentially you will need to add support for SCTP to that network plug-in if you can to take advantage of some of these features.

G

Okay, we're standing between you and lunch. So in conclusion, what does this all mean so for endpoints slice we're basically enhancing the API to support massive scale services and endpoints in terms of the API work on ingress and service, it's going to be better l4, l7, modeling and then raising the bar in terms of features and extensibility dual stack. Basically, you can use ipv4 and ipv6 in your clusters. Service topology gives you control over Interzone region traffic.

G

Well, not just that it's very generic thing and then finally, a CP TP support, gives us protocol support, completeness and that's it so Q&A, and this is where you can find us. Valarie also had a slide like this. We'll keep this up here.

G

Over here, where are the oh, they are posted as a PDF, oh, no, that's! Maybe Tim can do it or Valerie, because we were not on the speaker thing.

G

Send it to who we can work it off offline.

G

N

Thanks for the deep dive gentlemen, scaffolding from ZDNet, a question about the IP family attribute.

N

Anyone here in the audience could tell the difference between an ipv4 address from an ipv6 address just from looking at it. So that's a obviously this the Scott colons in it. For heaven's sake, that's it's not IB before, and so my first thought is: do you really need an explicit attribute saying the following is an ipv6 address and then I remembered something that you were saying about the time things take in in parsing and the the performance attributes that you wanted to give to it?

N

Is that attribute explicit just to save the CPU the time of actually having to look at the first second third characters and say: is this a colon? If so, then ipv6 that's.

M

A great question, I'll say right off that I wasn't I'm, probably not be full expert on this, but I'll say one key reason to have IP via the family is just this, for the sake of validation, to actually ensure that and in say an end point slice or whatever that all addresses match the specific IP family, but Tim can give more background. So.

B

That's a really insightful remark, because the first version didn't have the distinction. The second version did and the reason we added the distinction was in fact what we noticed. As we were, adding dual stack support is we were going to ask every node in the cluster to run through every endpoint and parse it and make a decision about whether it was v4, v6 right, and that seemed like a really redundant silly thing to do when we know that it's cash inefficient to do that there isn't a there, isn't a simple API.

B

We guess we could write one, but the simple API was go parse this IP, so it's gonna walk through the whole thing for every IP address times five thousand nodes times. You know and it blows out really quickly and we can't do anything about it early, whereas if we go this way which I really wasn't a fan of, but I came around to now, I can validate it a priori on the API server and split those out with no work being delegated out to every node.

B

So we took a little bit of a hit on the API in order to save a bunch of work in the nodes.

D

Okay, so in the meantime, the related question is the IP family, so it has, as we see it, has to I piece. Can it be a case where I have both the IP as ipv4? What I am coming to is multi IP support for the pod? Can it be extended, or is there a thought that the same construct can be extended to support multi IP pods? You.

B

Guys are relentless.

B

The short answer is yes: we've made sure that the design accommodates this, knowing that this is a likely direction to go. I, don't know exactly what that means. Yet you know, there's been a bunch of work that that the Dan and others have done in the community around having multiple interfaces on a pod. You now have a way officially that you could represent it.

B

Although you're gonna, you would blow up a bunch of controllers, but the logical extension would be okay, let's make it possible to actually do this I think we don't yet have a broad definition of what that means. If I have two IP addresses in two contexts like for a controller, how do I interpret those one of the things an astute reader will have noticed is that we left room in the API for metadata on each IP address. So the IP address is not a list of strings.

B

It's a list of structs where one of the members is the string is the IP. So we have room for meta. We can use to decorate this, so we've left ourselves on a vector for doing this, but we haven't done it yet.

E

Maybe I missed it. What was like the purpose of the problem they were trying to solve with endpoints life like were their endpoints, that weren't needed they were being sent down or like. Why is it expected that it's helping you.

M

Can you say that one more time what.

E

Was the problem that you were trying to solve like any such scale but like if you still have to get all of the end points question.

P

E

Or are you were there unused endpoints that you were getting because it's like a bulk, API yeah.

M

That's a great question, so the the problem with the endpoints resource is that the full resource had to go across the wire when a single endpoint inside that resource changed. So a single IP changes in that endpoints resource the whole thing goes across, whereas an endpoint slice that whole thing is a whole lot. Smaller.

Q

G

R

Thanks, okay, so is there a reason? Servers can't be both v4 and v6 so that you could have a friendly name and if it's a v6 service or a v4 it'll be able to choose what it goes to and then a follow-up question is: is there any thought in making opt in for v4 so like by default? Every pod gets a v6 address, but it opts in to get a v4 address. Also.

B

So they on the first part, we did that mostly as a shortcut, we thought we looked at the use cases that we had in front of us and we didn't have a concrete use case for having a single service on two families that wasn't also solvable by having two services. The friendly name, like we talked to a few customers and users, and the friendly name didn't seem to be enough of a motivation to take on the complexity of doing both at the same time, we're still open to it.

B

If we find the use cases and the demand for it, the second question, I think, is not something we put a lot of thought into honestly. We left that at the network level for this United States Oh.

O

O

The second question.

O

For the second question, kubernetes supports ipv6 single stack, so I think a lot of the functionality is there to not get v4. If you don't want v4, but that said it's going to be single stack throughout your entire cluster. At this point we don't really have a way of seeing that some pods should only get some addresses or some you know, family and some pod gathers that you can't really necessarily mix it up, but you know I think we'd probably like to get there.

O

We just need to kind of keep going through all the bits and api's and components and make sure that they're, okay with that I, don't see any particular block for that kind of behavior. It's just that's, like you know, phase six I.

S

Just had a quick question about the importance last thing you mentioned that there was a condition on hm points lice is the sort of intended use there to be like replacing the old end point instead of having the like ready and not ready in a single condition, you'll end up with two endpoint slices, one that has a condition of ready and one that has a condition of not ready. Is that the no okay.

M

Scared of this thing so yeah, the question is great right now an endpoint is either at a single endpoint, referring to a an IP, is either ready or not ready and there's some D duping involved and I think we.

L

M

Towards if there's two that come in that are conflicting, we lean towards ready, instead of not ready, but I'd have to double check the code. But there is only one representation of an endpoint is the goal, so we don't have two different ones that are ready or not ready.

T

With the introduction of ipv6 and dual stack, when do you see their phasing out of all the strategies like V PCs and everything?

T

When do you see that coming so it's easier for the user trusts to have one IP and public key addressable at peak I.

B

Don't know that we've really made that a goal most of the users that we're talking to today see dual stack sort of for the indefinite future. There have been lots of conversations around NAT and NAT 4:6 and other encapsulations to make it less complicated on. Well, you just push the complexity somewhere else we haven't.

B

We haven't had a conversation really about like what is the ultimate end goal of v6, at least if there has been that conversation I haven't been part of it, but I'd like with all these questions, like they're, great we'd, love to hear requirements. If people think they have ideas for what we should do.

U

Thank you, I'm, not sure if the Osmonds joining the in the presentation, but is there an API backward compatibility strategy for moving to the wall, stack.

B

The api compatibility story, I'm actually pretty proud of, we spent a long time working on that. Mostly it works the way you would expect it to work and to do that, we had to sort of define what we thought. The various use cases were with old clients and new servers and converting from old to new, and we explicitly call that all the different edges. So if you have an old client- and you only know the singular IP address and you mutate that IP address will sink it into the zeroth entry of the list.

B

If you know the new new API and you write the list, but not the singular, then we'll sink that back out. We can tell which changed on an update and if you change them both and you change them in compatibly, then we'll actually fail that the API call. So mostly it's gonna. Do what you expect it to do. If you have a dumb client and if you have an enlightened client it'll be the same.

B

Oh and yes, I sort of as the tiebreaker. If we get stuck with both, we choose the v4, but that's a really corner case. So.

V

When one cares about that you're stuck so the odorous talk, Vanitas allows to consist the Craster with the ipv4 only host, with the ipv6 host or in case of the we. When we create the dearest talk at an all node should support it. Happy default on them. Ipv6 the.

B

Assumption now is that all nodes are dual stack right where we're willing to see where this goes now right. We, we defined the first three phases, we're still in the middle of implementing phase 3 I, assume that there's probably 10 more phases beyond that, as we figure out more and more of the user requirements.

K

Yeah, this is not an ipv6 question, so it's related to the to the changes to the gateways, route and services, so that yeah. Do you think that this change or this proposal may restrict extensibility to other use cases because, like I, feel that the ingress concept is it's abstract enough and you can introduce several other use cases, but with the Gateway routes it feels like you're restricting into a certain model that might not be extensible. I, don't know so.

G

That's a good question: you should definitely contribute in terms of your use cases and that's kind of we're at the stage where we're sort of sketching a framework and then collecting the use cases and making sure that the mapping is okay. One thing is that if you find that ingress is perfectly valid for your use case, then it still will exist and be supported as a ga API, so that thing will never be deprecated or removed in favor of the new API I hope.

B

That everything you can express an ingress you can express better in the new gateways plus routes API. The thing that got me excited about the new API is I, see it. The ability to represent the lower level like l4 things that service does like service is a terrible API. It's a bunch of different concepts that all got glued together into a single resource. I see this as a possible way of breaking that up, so that we can actually have a cleaner API to the l4 constructs also.

B

But if you, if you can think of a use case, that is representable in some other way that isn't in the Gateway system, then we want to know and and we're explicitly adding a bunch of extension points so that people can take their implementations and and do crazy things without resorting to annotation.

Q

Thank you. This is another question about that. Couldn't be the illusion of ingress that segues on from that, as.

R

Q

Better being able to express a separation between l, 4 and l 7 constructs, is there any thought to opening up ingress to a wider range of l 7 protocols? There are other l7 protocols in HTTP F curve and his concepts of stuff. You wanna do an ingress. Is there any thinking of that yeah.

G

So you should look at the sketch to see that that was one of the provisions we want to add in. Is that not just HTTP routing but different other routings?

G

There are a number of ideas floating around out there, the most important of which is you just simply are able to specify the protocol, which I think doesn't quite exist today in any official API.

B

But remember one of the extension points is in the routes resource, so you can actually like in theory. You could build your own traffic management system that routes, the you know some traffic from a gateway into a my sequel route or whatever, whatever crazy route you're thinking of in the backend, and that doesn't actually need to be standardized by us or by anybody. You can go. Do your own thing via extension, CR DS, that's one of the objectives! Oh.

W

Yeah, thanks for the talk, I just want to ask I guess in retrospect, if you could kind of go with this alternate reality with the kind of information you know, how would you go about doing that? Kubernetes, networking different like Tim, you mentioned kind of getting rid of services or kind of changing it up. Are there other components that you kind of get rid of or tweak or just not think about it? In general,.

B

Services is my biggest one yep. Yes, it's become really complicated to think about how to add anything to service because of the way it intersects with all of the other things in service, and there are known bugs, for example, in the rest logic, around service, where immutable fields become mutable.

B

If you do the right sort of ballet dance and they're really hard to fix because of compatibility reasons, so yes, I would love to burn service to the ground and start that one over ingress was originally designed sort of to be what Bowie is describing here, but we didn't have the follow-through or the sort of the vision to to see how to do it correctly. Also, we didn't have C Rd at the time, so that made it very difficult to think about extensibility that way so yeah, knowing what we do now.

B

I think those are the the two biggest things sort of orthogonal inputs to services and higher-level extensibility and and maybe more Omnibus Network driving like the network drivers right now are kind of spread all over the place. There's the cni driver a little bit of logic in cubelet. There's cube proxy there's your network policy and they all sort of do things that interact with each other and sort of poorly defined ways. If I could go back, I might up level that ones.

P

So for the new features we introduced any be impacted with the soviet mesh anthos, you guys doing such a features, an endpoint slice and increase ib affected to the soviet mesh. You guys offering.

B

The the gateway stuff is explicitly not about service mesh. A lot of the concepts overlap, but we're really focused on ingress sort of true what we traditionally call north-south. That's the feature set that we're focused on some of the service mesh implementers are participating now or or talking about participating and they're. Looking at whether this api can define sort of the the service mesh api and in fact I did a talk this weekend at the rejects conference on this topic a little bit but service mesh is explicitly not the goal.

G

With regard to endpoints slice, I do know that people are looking at it. Basically, as these primitives from kubernetes get a little bit more sophisticated, there might be a lot of reuse just then you don't have to maintain to api's that do kind of the same thing. Yeah.

B

That's a great point: the the thing I've been thinking a lot about in the last couple of months is incremental adoption, making it easier for people to sort of grow from basic use cases into more advanced use cases and, frankly, kubernetes has made it very difficult to sort of evolve your usage of kubernetes.

B

You know you hit the end of the runway on service and then you have to stop everything, you're doing and move into ingress and change the mechanism and change the controller and change the api and go retest and recall and benchmark, and then you can carry on sort of where you left off and then, when you run out of the ingress run map, you have to go into like a service mesh and you do the whole process over again. So I've been thinking a lot about how to make more incremental adoption and I think.

B

Actually, this the the gateway API is a path towards that, where you can start with your your gateway and when you hit the end of the capabilities there, you can change the class, and now your class becomes an l7 class and you suddenly are able to add more functionality through the extension points, but you never had to go back and rewrite your API right, and so this is sort of my interest in it and I'm really I'm keen to drive on that model and every day every time I talk to people, they sort of nod their heads and they get it I hope we can deliver that.

I

Currently, I'm a part of a container that interface feature team, the we currently we are watching on the and endpoint resources and we are creating based on that. We are creating endpoint files and based on that, we create flows. So we use ouvea switch to program the flows. So basically we watch on the endpoint resources now that endpoint slices are coming. So what are this I mean? Do we I mean it's still? The endpoints will be there the way. How do we watch for the resources? I mean just.

G

What's the transition point, Rob yeah.

M

Yeah, that's a great question: I meant to cover that in more detail in the actual talk, but endpoints will continue to exist for the foreseeable future. Of course, if you want improved performance, greater scalability use the endpoint slice API, if at all possible, but endpoints will continue to exist and provide that backwards. Compatibility for a long time.

L

What's up, thank.

B

You to everyone, if you have more questions like.