Cloud Native Computing Foundation Kubernetes Forum Bengaluru 2020, 27 Feb 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Extending Kubernetes - Lessons Learned From an L4 Ingress Implementation - Srinivas Brahmaroutu, IBM

Description

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects - Learn more at https://kubecon.io

Extending Kubernetes - Lessons Learned From an L4 Ingress Implementation - Srinivas Brahmaroutu, IBM

This talk will focus on how to connect your workloads running in Kubernetes using vendor provided Hardware LoadBalancer. When a service is created with type LoadBalancer, Kubernetes Service Controller automates the creation of an instance of LoadBalancer with external IP that can be reached by the clients of your application. The firewall rules are also properly configured by the cloud provider for your service object. The intent of our talk is to provide you with necessary information on how loadbalancers work and how to use them with Kubernetes workloads. Implementation for LB varies in their design, implementation and capabilities they expose. In this session we will firstly share our experiences working with loadbalancers on different cloud providers. We will also discuss how to share LoadBalancers among workloads to be cost effective and user friendly without vendor lock-in.

https://sched.co/YVxr

A

All right, hello, everyone thanks for joining my session on extending kubernetes our implementation on elf for ingress controller I am sinuous bra morale, I work as a software engineer at IBM.

A

Excited to come to Bangalore to give this talk and meet. You all I'm told that if you have questions on the session, we can meet outside after the session and discuss the original presentation of this implementation was given in cube corn 2018 in Seattle. We both developed the application we hung. He is now the chair of cig scheduling and I actually participate on kubernetes conformance program as well as sig testing and six storage I've been working on open source projects for four five years now and about last two three years on communities.

A

To give you a big background on this, how do we expose the communities workloads externally.

A

Especially for El for services and the background and motivation for why we ended up developing this solution and we introduced a concept called shared load balancer, which is a CRT and a custom controller, which is the way to go nowadays and I'll. Show you a few demos on different cloud providers. How this is done.

A

What words are nothing but set of containers running inside communities pods? They can communicate within the cluster using their internal network, but they cannot communicate to outside world and in order to facilitate the communication, we have a resource called service services are of two types right: internal services which are of service type, cluster IP and external services, which are of type node port or load balancer.

A

Cluster IP server service exposes the service on a cluster internal IP. Choosing this value makes the service only reachable within the cluster, but not outside. Our main focus today is to provide the connectivity for all our clients outside of the cluster.

A

So let's see what a node port service is. A node boat service opens a port on the nodes of the cluster and when the client wants to connect, it needs to know the public IP, along with the port of your service, to reach your back-end applications.

A

So the port range is 30,000 to 32,000, 767 is allocated for and all the workers should open. This should have a public, IP and open these ports for you, for, in order to communicate the other option is service type load balancer. When you create a service type load, balancer, the its it's.

A

Actually, the service provider, who knows the best of the network setup and whatnot and they'll, create a load balancer for you, which is external to the cluster, and that's a public IP. You used to connect to your node port service inside the cluster.

A

This is the best option and using this option you should be able to connect a workload running inside the cluster through the load balancer to your external client.

A

Again, this is very dedicated in the sense that each of your applications running inside the cluster will end up having a load balancer, that is, that cannot be shared by different applications inside the cluster, which causes some of the problems that I talked about today and briefly, I want to touch upon the ingress.

A

Ingress object is native to kubernetes and with the ingress object, we have shared connectivity at l7, HTTP headers have lot of information and using the path supports, we we can allow sharing and routing by setting up rules on the ingress object and have the English controller process. These rules and send your traffic to different applications are running inside the cluster.

A

Ingress is a deployment model. It can be run as a daemon set. Ideally, there is one external load balancer like I show here that connects to the ingress controller, which process the rules so that you can connect to multiple applications. The advantage with this is there is less configuration less resources to manage. There is only one load balancer. You can also have multiple ingress controllers running inside your cluster by defining various classes of ingress objects, so they can act upon certain set of classes.

A

Popular ingress control controllers are like traffic and genetics or envoy, and it's useful for l7, so based on my brief introduction to services in communities.

A

What you see here is a quadrant, a simple node, port or load balance of service for l, 4 or l 7. They are dedicated. Each application requires a service, and the service is one-to-one correspondence to an external IP with the with ingress.

A

You can share l7, but there is something missing here which even the upstream wants that is sharing on the l4 side. This is not a hypothetical problem. We ran into that's where our background and motivation is. We have an application that we wanted to borrow and the application had several individual workloads inside one for to date, JDBC connections we needed to connect to databases and a UDP to transfer data, and then they a web console.

A

What happens here is that when you, when you try to deploy this application, we end up creating multiple load balancers. That is not an efficient way for us to manage.

A

What I wanted to do is, as you can see here, if I deploy an application. For instance, four of the applications I want to have a single external IP with multiple each of the applications have their own port.

A

To achieve this, we wanted to share the load balancer that the cluster service provider will create for you. The way we can do that is if we can create a port for each of the services.

A

We can use that port to delegate the traffic to individual node port services running inside the cluster, such that we can talk to different applications.

A

This way it is cost effective. We are using less number of resources, it's also user friendly. You don't have to remember much. There is only one external IP of the load balancer and much better to operate on and we can use the well-known ports of the applications that we can expose to outside world and we are using the Kuban IDs assets. We are not reinventing anything. Only concept we have to introduce is a shared load balancer, which is a custom resource object. It is exactly almost similar to a service definition.

A

If you look at the spec of the service is nothing but s. Lv has to send that information back to create the services, so all we need is similar to a service pack. The CRT is similar to that.

A

If you break down the problem, the problem now as I described in the last two slides, you need to open additional ports on already deployed load balancer and, of course, you said all the security groups on Amazon or firewall rules. Forwarding rules was not and then associate this to the backend parts. So all these ports that you are creating on the load balancer has to be able to talk to the backend part, and how do we delegate that information back to the end-user, because he does not know what I did on the load balancer?

A

The way we implemented is using a custom resource which I call it shared lb. Sometimes s lb sometimes so bear with me on that a custom resource object when it is created. It creates a load, balancer type service, which means that a share a load balancer is created for you and to create the load balancer we create with a dummy 4333.

A

What not you see it in the demo and once we create that we read the spec of the custom resource object, use that to create a port like in this case for ta, and that for ta is associated with the node port service inside the cluster. Now we have the connectivity into the cluster to talk to the application we want.

A

The last step is whatever the node port services we created. We need to transfer the information of that back to the customer. Resource object store it in there so that when the user queries, he knows exactly what is the IP and the port. He needs to connect it. This way, you can share multiple L for services through a single load, balancer.

A

Let's see some demos that will probably give us a little more insight into.

A

So this is Google flower. There are three nodes that are running out. Sorry.

A

Give me one second.

A

All right, we are running a Google cloud inside a global flood with three notes running the version of the Google Google flouts server is 116, so we'd like dim, said in the morning. We have all the new CRD definition, validations and whatnot and plus the new controller runtime.

A

There are no services running and I deploy for applications on this cluster.

A

Well and when you say get CR is there is a shared LBC already already created and if you see the instances of shared shared lb, there are none. At this point we want to connect to all this for applications, so we create for instances of the Sherrill base. Each of them have the information how to connect to those applications.

A

This triggers us to create one load balancer only one, because our capacity of the lower back of this particular application is to run five connections on single load balancer by default. So if you look at the GCP console, there is only one load, balancer created at this point in time.

A

And on the front ends, if you see there are that's the dummy port I'm talking about when I created the load, balancer type and as the applications share deltas are created, they are getting all that loop is happening. The node port service is created in the cluster and the assignments are happening. So all the four services are having the same external public, IP and the ports are mapped. There is a four thousand one port that is mapped to a port on the load balancer.

A

Now, if you see there are four friend ports that are created for you, that will be able to talk to the backend node port service to connect to these four applications, and we have to create some firewall rules to open up the ports. So we by default, open up all the TCP and UDP ports from 30,000 to 32,000 767.

A

And there is only one external IP that we are buying, so the load balancer is said to use an external IP and set up forwarding rules so that the traffic can go through.

A

As you can query the shared load balancers, they all show you the name of the load balancer as well as a port. So now, if I do a net cat on these I'm able to access all the applications that are running in the back, let's see a different approach now on IBM cloud. I want to see if my TCP and UDP services work again like I said: load balancers are specific to individual flower providers. They have different.

A

Quirks or whatever you call it so again, our Nikias I'm, running 116 and I have three nodes.

A

There are no services running at this point and let me create two deployments: one is TCP and the other is UDP.

A

Let's see the CRD definition seared in a petition is already uploaded into the cluster, so we can create the CRD objects. I am going to create one. Like I said here. I show the service definition of the CRD object. The port is 4001 for the TCP connection and 5,000 won for the UDP connection, when once I create the TCP shared elbe at this point that triggers me to create a load balancer on IKS.

A

As you can see, get service shows you a load balancer created with a public IP and the same IP is associated with my shared lb object. Also, apart from the default port, 3 3 3, there is also 4,000 won, and if you look at the shared lb, the 4,000 won is associated with that, let's create the UDP service. Now, both the services are going to share the same load, balancer again and the same IP, but a different port.

A

If there is a port conflict, what we do is we create a new shared new node, back load, balancer and use that load balancer for that port. So no worries about poor conflicts. Let's do an echo test for the TCP and UDP with a minus? U option so we did a echo test onto the application. So, let's look at the application logs. The TCP log shows you the echo test. Work! Fine, as well as on the UDP we saw they could test was fine.

A

You can set the capacity of how many shared connections you want on the load balancer one other.

A

Cloud service provided I would like to show. You is Amazon.

A

Oh by the way, I'm also showing in this demo, another random port assignments. So if you don't specify the port here and up what is assigned so that so in this case, I am sitting the capacity to two on Amazon.

A

Let's run it on Amazon Amazon is still you can only deploy 114, so you don't have all the goodies that CR DS brings with 115 and 116.

A

There are no services running at this point. I want to do three applications since I said the capacity to two eventually I'll be triggering how many load balancers two, because we will run out of connections on the first load, balancer.

A

Shared LB definition is already in there. Three deployments are created.

A

By the way, the solution is namespace centric, so you can apply it to a multi-tenant environment, again I'm, showing you what the shirt shared lb port should look like. Four thousand one I created a shared lb. This triggers a creation of a load balancer on Amazon, as you can see in there, and there is a dummy listener, as well as port 4,000 won for the first application, and we also have to create on Amazon security groups.

A

So if you go into the details and click on the security group, there is a forwarding port on that Citigroup. That needs to be said, inbound rule 4001, so we're all good. So we can connect to the application Amazon. Of course, users see names to resolve, and that is my first shared load balancer that is created on Amazon, with a dummy port and then I have my shared lb, which has opened about four thousand one. If you look at the shared lb, it shows me the definition slightly differently, because we are using one fourteen.

A

We can do some magic with the with the output to see in a much more concise way, so shared lb 1 is connected to that load, balancer at four to four thousand one. Let's create the second one. What happens? Our capacity is two, so that means we can still share the first load balancer. So the second shared lb will still use the first load balancer and we will get an association pretty soon with the same load. Balancer.

A

As you can see, the cname remains the same. A new port is four thousand two. Now we create the third shared lb, and now we ran out of capacity which triggers the loop to create another load balancer on Amazon. For us honey cares.

A

There you see the second load balance is created with a dummy port. Three three three and the third shared load balancer will be associated with that.

A

The first two are two, the first one and the third one is so at this point: what happens if my post shared load balancer is deleted? I'll still have to load balancers, because I cannot rebalance the traffic. At this point we don't have any solution to rebalance. So, as you can see, the load balancers are updated, the listeners are updated, security groups are updated.

A

You can see the second load. Balancer is using port 4003. First one is using four thousand one and four thousand two to connect to all my three applications.

A

And if you go check the security groups, the security groups are also updated accordingly,.

A

With the inbound rule set for those two.

A

And s lookups on those C names will work because it's an external load balancer we can resolve it and the netcat on to the first application of works, fine tip, let's create a delete, one of the shade load balancers.

A

So at this point nothing else will be done. The the load balancer, both the load balances will remain because there is at least one shared application still connected to them. Only thing that changes is the rules, change.

A

Because the 4001 is application is deleted. We clean up those things.

A

In the interest of the time since I am running out, let me go back to.

A

Here to wrap up I would like to discuss more about the design and implementation. Essentially, we used existing resource objects from kubernetes only thing our controller does is manage the objects using the Cloud SDK to to add ports and security groups or firewall rules. What not n is configurable and we create load balancers on demand and we use well-known ports on the applications. So it's not easy to connect. There are some differences on the cloud providers that is listed here on the slide deck you can go through them all in all.

A

The most important thing I want to say is our code is on github. Please do check out our code and if you want to better our solution or make some changes, you wanted us to make some changes just communicate with us on github. Again, thanks for listening to me, any questions I'll take it outside. Thank you.