Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Extending Load Balancer Using CRD in Different Kubernetes Clou... - Wei Huang & Srinivas Brahmaroutu

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Extending Load Balancer Using CRD in Different Kubernetes Cloud Providers - Wei Huang & Srinivas Brahmaroutu, IBM

Major k8s cloud providers (IBM Cloud, EKS, GKE, AKS) provides in-tree implementation for LoadBalancer which varies in their design, implementation and capabilities they expose. In this session we will firstly share our experiences working with load balancers on different cloud providers. After that, we will demonstrate a common solution based on CRD to enable sharing the same LoadBalancer among different services to reduce cost. In some L4 service scenarios, there are multiple TCP/UDP services need to be exposed publicly. Here comes a question: is it possible to only use one LoadBalancer to expose those services (on different ports)? Instead of being given multiple LoadBalancers which is neither cost effective nor user friendly. Furthermore, is there a common way to adapt to different cloud providers while maintaining a simple interface? We will explore these details through our demos.

To learn more: https://sched.co/GrUd

A

Thanks for joining today's session, and today we are going to talk about extending camellias load balancer using CDs, some different providers, and let me introduce our self first I'm Way and in straining we both work on IBM and also in the kubernetes upstream I work, mostly on six scheduling and he works Muslim, Sikh testing and Cisco storage, and this is today's agenda. First, we will try to give you a whole picture of in our days in kubernetes.

A

How can you expose your internal workers externally and, secondly, we are trying to spot what's the missing piece in the whole picture and what's our motivation to solve this issue, and after that we were introduced concept called Cheryl balancers, try to solve these issues, and we will give some demos on different, copper, visors and the vanilla. We will do a wrap-up, ok, so as a kubernetes user, I think, sooner or later, you're came across problem, which is how can I expose my internal work clothes externally?

A

So, basically, basically it depends on how where you were closest in layer, 4 or unit 7. It has different answers. So if you are running on layer for your work clothes, basically, there are two ways: one is called notepad service for not pass service to work. You first, you have to ensure your public IP on your work, node accessible from the outside, so so there you can define your service spec, which type notepad so that you can use the public IP to access your internal group roles, but there are some limitation here.

A

First one is I'm. When I mean the public IP, actually you can specify any public IP associated with any worker node in your cluster. So this somehow means you have exposed your culture via the public IP to the outside world. So somehow has some security concerns. So if you try to run production, work level use this way as just to you to do some extra security reactions and the other limitation is that when you open the part on your public IP it actually, the part is open.

A

Now every woken up, and also it has some ranchy limitation by default. The range is from thirty thousand to thirty two thousand something. So this is for a node part. A more viable way to expose your layer for were closed nowadays. Incovenience called the balancer alabanza is a whole of the same. Your different service, spec with type load balancer, but keep in mind, there's no default load. Balancer implantation in the Vanina cabinets are offering in delegates the invitation to your underlying providers so well.

A

It makes sense because to make the layer for network routing efficient way, you have to leverage your underlying network infrastructures capability, so you can do the routing better. So this is four layer four, and if you were close working on layer, seven things are a little different, because if it's layer, 7 network packet, it has more information than layer four packet, because layer, four is just about IP plus part but layer. Seven, you have much more information.

A

You can leverage, for example, the HTTP header information, so that you can specify different rules like the same host name, slash different sub pairs, so that there should be a central component can recognize this characteristic of your a of seven traffic and get them around had populate your internal work house. So in Cavanaugh's the different the rules to define your incoming requests, the characteristic is called ingress rule. So it's a API object and the central component to get the traffic routed to internal path is called in, usually Skaar ingress controller.

A

Anglish controller is usually implemented in index or m'boy. So this is for seven as a wrap-up. So if you are REME only us for notepad our load balancer service works but keep in mind. I put. It were dedicated here in the right picture, so that Acadia means your request. One little balance of results, resource I gave you one and if you'll require the second one I'm sorry I cannot reuse the first one I give back to you, because the first one has been dedicated to the official request. No matter.

A

The first request is lat weight or someone so I call it dedicated. So this is kind of 1:1 mapping for layer 7 in grass, I call it a shared shared because you're different several ingress rules. Then the traffic goes into the same central in grass controller. So what you'll pay for is just the public IP of that in grass controller. So it's an to one map user account called share.

A

So if I look at the picture in the top right, you might be found that there's something missing here. So in other words, how can I expose my layer for internal work clothes in a shared way? So that's the problem definition and in upstream the usually called L for ingress.

A

Ok, so this is actually not a hypothetical requirement. Post upstream has this, and also we internally have this requirement so for our internal team there, so we have stream, they do have this requirement to. They have a they have to bundle their warehouse solution to the end user. So, basically, inside this bundle they have several layer for services. They have to TCP services. The user needs to use to DBC to connect to, and also one UDP service.

A

The user needs to use for data transferring so so if they use current a load, balancer solution, they give to service pack, create two services and they care to load balancer resource pack. This is not exactly what they want. What they want is the credit several services, and they get just one of the balance who responds back so that I can just use this Yannick Robbins ip+ different part to access.

A

The internal were close so little like this I request to, but I do want only one blood band, so resource back on different part, and one primary motivation for this is about the cost, because no matter you are running on which cloud provider, the more load balance of resource use, you use the more you pay and also we want to be user-friendly because, for example, you pack in this solution to the user and in the users perspective what I care is just your kill. You give me just one solution right.

A

Why I need to remember so many in the video, low variance IP and the connector right ideal way should be I just gave one, and it just used that IP to connect on different parts to internal different were close right and also if you want to manage different little balancer resources. Of course, you need some extra operation efforts.

A

Okay, so we try to resolve this issue and, if I jump to the solution, I want to spend more time on this to really understand this problem. So this problem actually has some sub problems a list here. First, one is I already have the load balancer resource right, I can I can create by using the cloud provider, but how can I open the part so that I can share the load balancer on different part? This is the first problem. I can open additional parts.

A

A second one is how can I associate that the additional part to the internal were closed, so this is a second problem. A third problem is if those two are possible, how can I give this accessing information back to the answer and user? You know friend, anyway. Okay, he is almost our design. So in this solution the service 11 service is no longer acting as the very fake head to the end user. Instead, the customer resource object, as acting as a verification to the user.

A

So I don't need to worry about that too much about the spec of the CR and our mission later so the CR is the spec is almost the same as service spec. So basically you have incoming pot target impart protocol and slack labels. So when user create see our object, firstly, we will check whether we have available load balancer resource or not.

A

If we don't have, we will try to create real load, balancer object, which is just create a service with type load balancer so that it do some post wrapping things and the get traffic works from outside and notice here, I didn't. I just opened a dummy part because right now, this load balancer is just a dummy one. I want to hop here and how to open additional part and, what's the specific additional part, the part comes from internal service.

A

I create a waste type node part and that part has specific specification from the incoming stream of attire, which carries, which is just some normal service pack. I mentioned earlier incoming power. The talking part protocol and the selector labels I put it into internal service, for example, is a node path service, and so that kubernetes gave me an odd part. So this node path is a specific part. Additional part I need to open on this load balancer so that it answer the first problem of sorry. Then we go back to the first problem.

A

How can I open this specific part? So we have to invoke specific cloud provider SDK to do things usually do two things. Why is open the part itself? The second some thing is to open the incoming. The core is security group inbound rule that is sort of firewall thing, make the firewall happy to let the traffic-cam see.

A

So this answer the first first problem, which the additional party should be open, then how to open- and secondly, is how to do the Association Association associations by using the part to do this, and because we create an internal service and the last mile will be guaranteed by kubernetes from the internal service to the backing path by compartir by the created the IP rules. So by this we answer the first problem: the traffic hits the load balancer on additional part.

A

First and the traffic goes to internal service and the in traffic goes to the takane path, so the traffic jam went through and let's go back to the third problem. How can I give this information back to the user?

A

So thanks to the design of ceoddi, we just put the information back to the CR object, so the user, your script control, gets the object and get the necessary information back to get connect to okay I were to a live demo on Amazon UK s so on the EP, because right now, I don't have any even resources.

A

So, firstly, I will try to switch the contacts to this environment and the latest version in case is one thing and I: don't have any other service and I'm trying to create some backing path.

A

And Ciardi definition has been there, but the CR objects is not there. Yet. The SLP is shot for shared a load balancer and in our program by default, the capacity is 5, but to demo some specific function here. I just sent it to to a capacity means how many slots I want shared. A lot of balancer can serve.

A

Okay, this is the specific definition of the C object. Everything has no difference than a normal service. Pack is part to keep our. If you don't specify protocol is by device to TCP and select label is try to find that the talking in parts okay, I'm created, so basically he s needs almost three seconds to Crane.

A

Ok, this is the one I mentioned that the load balancer I want to do. The sharing of this load balancer- and this part, is the Dhammapada I mentioned earlier and dks give us a hostname, and this is the internal service. Dimension is of type notepad, and this is the part we need to do the Association and also we stated back to the start, be object to see our object. But here you can see, there's nothing additional information printed, because only in 1.7 kubernetes it has a feature called print additional columns.

A

So for other other demos, this okay, but here we have to use some other cube control tricks to print the additional information. So here so external IP has been specified as the host name and this part. Okay, let's take a look at the universe, console okay, this the load, balancer object has been created. So two things we need to open. The palette itself in either base is called listener, so the 4,000 warrior has big.

A

The second thing is to file open the firewall rule, so you have to go to the security group and in security group here there is the inbound rule. So here 4001.

A

Let me create a second one, so, ideally we specify the capacity to write, so it's still available to solve another incoming request. Let's take a look, so I create the second one and you can see the little balance of resource is still 1 and the second internal service has been created and also can sob get the co object.

A

You can see the second line that instan I P is exactly the same as the first one. So that means we have succeeded to use the same rule and to serve to incoming requests, and as now we are running out of capacity right. We are creating the third one to see whether it will create another bloob answer for us.

A

You can see. Yes, there is another one created here.

A

Okay, the TCB stray. Does your object just using another danced hostname. You can check here.

A

Finally, let's use Netcare to check the connectivity because, unlike other call providers, the eks gives you hostname rather than an IP, so you need to ensure the local the answer. Lookup works so good. It works.

A

Let's check, okay, good, it succeeded. So that means the traffic really went through from the outside to inside part.

A

Ok, let's delete the first, a shared, not answer, object, a deleted and there are still two little bands on resource, because the first one are still serving. The second SEO object right and overcast. The first tier object has mentholated and let's check a little on-screen console yeah. They are still, you can see here the 4000 and why the first one has been delayed here. This is the firewall rule and go through the load balancer itself.

A

Yeah yeah, the first of them one less noise has me also deleted. So this is almost a demo for for eks and I know what you're singing thanks.

B

Ray I'm, not as brave as very so I'll, probably as or a card demo here or also in the interest of the time, so I would like to show you several scenarios for different clouds. Let's start with jjigae on jke again you set up the environment to to connect to GK cluster cluster has two nodes here. The version we are running is 111.

B

There are no other services running at this point, but we are creating four deployments on this cluster. So the idea is what, if happens, we are concurrently, creating multiple services. Would we be able to handle that? So there is, as he already definition already, for the shared lb. We haven't created any instance of the shaded, be that means we do not have any backing services. Yet that means we don't have any load balancers in the cluster. So, as you can see, the jke console has never load balancers in there.

B

So let's go ahead and concurrently create four shared elders. All these four objects now trigger has to trigger a load balancer to be created and, as you can see, there is one load. Balancer created it's a bit. Slow on gke to create a load balancer.

B

So let's roll forward by about a minute and then you will see all of a sudden a load balancer created with an external IP in there and like we pointed out, there is a dummy port created 33333, but then we also have created four other ports for for the four services that we created concurrently.

B

So in the end, if you see for all the four shared lbs which are backing the services, they all have the same public IP and each of them has a different port to connect the back-end application we have netcat shows the first application running on 30,000 151 works, fine and trying the second application. It works. Fine. So looking at the console right now, you can see that the load balancer is created and it has to target poles.

B

That's two nodes that we have in the cluster and if you look at the front-end, there are five forwarding rules, four of them a one is a dummy rule and four of them for the four back-end services that we have having different ports and also, as you can see, if you go to the network, there is one external IP that is created for the load balancer with all the fiber five folding rules out here, and we also create a firewall rule.

B

In this case, four, for the proof of concept we created one firewall rule for TCP, as well as you to be opening up all the ports from 30,000 to 32,000 767. So that's on on.

B

On, let me show you on IKS, in this case I'm trying to demonstrate you.

B

Using both TCP as well as UDP service, again we create the environment. We have three nodes in here. We are running version 111. There are no services, we create two deployments, one for TCP and one for UDP, and there is a CRD definition out there already created. So let's go ahead and create a CR object for TCP service on port 4000, one serving on port 4000. Once we create this, we also create a UDP service on port 5000, one target port 5000.

B

We now see that there is a load balancer created as soon as we created the first share lb and the idea of the load balancer is associated with that port 4000 one, and then we create the UDP service.

B

We we see that both the shared alby's are using the same load balancer here and but on different ports, of course, and the information about this Cheryl base is, is shown here, let's test it out, netcat to each of these services, TC echo test to go to them and when you look at the part, the part has that echo in there. So we are able to properly connect to the backend applications that are running behind this.

B

What happens if I create a random port now, so the port is commented out there for, and we will see what happens if you create this share there'll be again a service get created, and this service now, as you can see, has allocated a random port for that we will create all the requirements.

B

Firewall rules and forwarding rules for you around it quickly. I'll also want to show you a KS for it with a different purpose.

B

Again, in this case, I am setting up the environment for 8 years. There are two notes on a KS by default: that's what you get with a free account, running version, 11, no services running in the cluster and I'm, creating two deployments: TCP 1 TCP, there's this he already defined already. So we are going to create one service on port 4000 and once I create this service. There are no load balances at this point in the cluster, so once I start creating that service will have triggered a creation of a load balancer.

B

This load balancer takes a little bit time on a chaos once it's created that is associate. The public IP is available and that's associated with the service shared lb service and, as you can see, the shared lb services using photos and if I create a second service on the same 4,000. Also, you can see in the console that this particular load balancer is created and if you further look at it, there is a. There are two rules, one for the dummy port and one for the port of our service and.

B

Those are the two rules and out so you can see there is an ban security rules created for both the ports here. So let's say now: we create another service again on port 4000.

B

What do you think will happen because already we have a load balancer serving on that this will trigger a creation of the second load balancer for us we are sharing, but if there is a reason why we have to create one, we would create one for you and again we will will be able to associate that IP address with the new service that is created and you can see there are four inbound security rules created automatically for you on that port is accessible for the new service we created and if you try to create another random port service.

B

At this point we have to load balancers, we haven't received the capacity of the load balancers. We did not reach that, so we can reuse the load balancers. So the third service that is created with a random port is using the IP address of the first node balancer. So that's the three clusters. Three scenarios.

A

Okay, let's get back to the slice, so basically the demos we gave the functions actually acting works on every.

A

It works on every cloud providers we just okay, we just use the we just demo the different functions on different platforms to show you that we'd really have imprint them so foreigner. Let's give some design implementation considerations, so some decisions must be made when they desire to get it-get have implemented. So first wise, we choose TR the over service with a notation as a very front end with the user interaction. So because, if you use the service fees, annotation annotation is just stream.

A

Planes Trains, key values, they're just text instructor tests, so using that you can now enjoy the benefits of a CD, brings to you like defaulting Virginie, validation, admission control, etc, etc. And, second, why is we choose namespace there, the vs. custardy, because it's a more user friendly, Angeles Year B, so that we can also associate it with this year.

A

D to our back policies, you feel running on multiple tenancy environment and also we tries to create a real load balancer resource on demand, rather than maintain one poor for the rotor balancing resources and also how many requests. Okay, wild answer can serve. It's also configurable, and also we also adopt some best practice on CR the controller, and this is the summary our summary on different color, visors I.

A

Think most of them has been mentioned in the demo, and this can be a reference if you are interesting any specific platform, and also this is not specific to manage cabinet service. So it's actually because we we just distribution in space on the underlying cloud infrastructure, networking infrastructure right so for eks. The solution actually also work for OpenShift or rancher or happier because underlying the hell they are both used. Eds use is easy to use the Taliban some resources in a dress.

A

Okay, finally, I encourage you to thinking in Cuban atheists way when you do your in-house given and his solutions. So when I wants to try to pull the things like lower balancer, see our object, internal service and the live answer together in a sequential way. Actually I was inspired by the design in Cuban and his part, if you, even if your knowing communities that when you create a pod, actually the first thing could be nice did, for you is spin up a pause container. It's a very tiny container.

A

It does nothing, but do some things like setting up the Linux namespaces settings groups so that afterwards it adding the real containers to your new Linux namespaces. So in my design, though, the balance was resource. It actually acting the same row as the past container. It does nothing. It's a kind of dummy. One has dummy pod, but Inc does the post wrapping things to make sure the traffic can come see so that afterwards we do open the additional power that do the Association? So this is the first story.

A

Second story is when you think the problem, so we want to do sharing right. It's the problem becomes from one one mapping to any one mapping. That means you have less resources and you have more requests. So how do we do the mapping so at this time, so things are different. It's actually a classic scheduling problem. For example, if one load balancer has been occupied by incoming requests on the part of course, you can now use that specific broadband service for another incoming requests using requesting the same pop right this in the past conflicts.

A

It's also a classic problem in scheduling, also in the scheduler in common area, and also right now we just simply conveyed and says how many slots you can give for four incoming requests for wine love answer. But what, if the incoming request has weight means? Why is heavier than the other? How do they do this? So in cabinets? It's a resource limits model. So this is the thing you can also follow.

A

If you have this requirement and also how to do your schedule, scheduling strategy, whether you want your resources to be more educated, a balanced or more being being packed right, also, and also you want, for example, incoming requests wants to be more coexist with this load, balancer or I. Don't want to be coexisting with this relevant resource, it's vanity and anti vanity issue, so this posts have implementation and pattern in the upstream cabanas schedulers, so don't need to Reeve and will something has been implemented and there are already some patterns so yeah.

A

That's basically what I won't say: okay, that is almost all for today and any questions.

A

Yeah, that's just the scheduling problem I mentioned so if first one come see and there's no resource, of course we will. We will try to require wine from the underlying infrastructure right. If the second one comes saying, we were to try to see whether existing one can solve the incoming wine. Then you can, if you can serve wine, where we use there with no balance of resource.

C

A

You mean the waste created the customary sauce.

A

Yeah, depending on, if you want to use specific part, all you'll want a random part. So so it's a additional feature we provided. That is also one reason we choose here: the over service visit annotation, because if you want to provide this random part feature, if you use service, you might know that the incoming part is a required field. You know service pack right so that you have to use that specific part, and you cannot ignore it by using CID. Everything is in your control, so you can see you can do whatever you want.

A

So yeah, the question is: if all the associate the see our object has been delayed elated and how do we deal with the load variants resource? Is that okay? This is the question we haven't. We haven't fixed, not fixed. We have enough put code that don't fix it yet, so my ideal is to put a periodical goroutine to check whether we have additional totally available load balancer resource. If they, for example, we can set a Rachel.

A

For example, if we have fifty percent of the load, balancer resource is totally only used, we can delete them, but in current information there has only.

B

The load balancer resource, all the forwarding rules and everything gets.

A

No, so for you meet the load, balancer resource or the SIA object, it can have multiple Nationals yeah.

A

We were created to.

D

B

A create one, damn dummy one, and we know that because sometimes it's it cannot create load balancers without creating that unless you have one dummy one, there's implementation.

A

So the yeah, the motivation I mentioned in earlier so basically is about cost right. So if you want to you so basically, if you create more services, you'll get more load, balance or resource back and you pay more.

A

B

It's just the you're changing the paradigm at that point, I think I mean basically each deployment is necessary with the service services or said support. So that's we're not taking that away. I mean you're still, instead of creating a service object. Now you are creating shared and the object. That's the transference! We go if I understand your question correctly right.

A

So if you have any question, we can discuss offline in deep details.

E

A

Actually, this job are not implemented. I just want to yeah introduced the concept that, if it's a schedule part you might have this requirement. So this right now is hypothetical one, except for the part, because party is very easy to ask, then if the past, you can't have the party can't get conflict and the right now, if any infinities upstream is on the part or node level, rather instead on from the resource level, like we created this little venture resource, it's just a hypothetical requirement. I just put the example here.

A

No, it has a handsome.

A

E

A

So, basically, we reuse the implementation of load balance. Actually, so the resource you use is still the underlying global resource right. So we have the robust resource, it's created by creating a service with type lower bounds and it created underlying object in the provider and we just created a internal service. That is the two things we created and they are associated with on the additional part we created and the time the information was set back to Azure object. That is basically the thing and we didn't.

A

We don't need to do much on the libertà itself, except for opening part of the in part, mummy, open part and firewall rules. That is the only two things we used SDK to do things. Other things are all kubernetes concept: CR the load balancer in a new service and yeah.

A

It depends it depends on your underlying little balancer. So right now we just create a default one, but if, for example, your incoming requests were asked for such an affinity right, we will carry their information and create a little bantam with that specific annotation for you field, let's say a node session and we can do better scheduling. For example, only the requests asking for such an affinity can use this reversal for you, because if you create a default, see object means before you want a default service. You don't care quite about session friendly.

A

We can use other blob answer for your this user, well resource allocation and actually all the fields in the service. We can actually just carry them and created a little band so for you, yeah.

A

Sdk for CID control.

B

Our specific stick, yeah.

A

A

Yes, you have to because you have to up manipulate the part and firewall rule. So this list, some SDK authentication ways on different cloud providers.

B

A

You have to provide the credentials, of course,.

A

A

So, basically Kozma if he doesn't work because kizomba a SS, if you want to open the additional part and although the balancer that part should be open at the right click on the worker node. So how do you do that? The only way I can think of is just using that node park service because it can open. The part is open on the instance rather than inside the part. Sorry inside of the part and don't confuse with the security concern I mentioned, because the internal service is created here. It's just an internal service.

A

You don't even need to the work, the worker node to have a public IP, just previous total farm. We leverage the internal service with node parties by two things wise. It can give us notepad open down the working out and also it will carry out the rest of the things the routing iptables rules compulsory. We handle that so security go ahead.

A

So badiy far so the question why I choose CID where they modified the labonza function? So this is a good question, so, basically Lopes it doesn't have default implementation. You! If you want to change the implementation, you have to change every implementation of every cloud provider. So what we are trying to do is create the Union for solution, not Union for solution by Union for concept and uniform interface to try to solve this problem. If we go directly to modify the imputation, that doesn't quite make sense because it's just.

A

Yes, that's correct, yes, so you casting is, for in grass controller.

A

A

Yeah, that's doable because, basically, if the traffic, if your solution work for layer, four, of course they work for layer, seven yeah, but it's just that we already have more elegant solution for f7, because we have l7 packet has more information right. You can better distinguish the incoming requests and the actually Mondays I think contribute summit. Brian grant.

D

B

Any more questions, thank you very much. Oh.

C

A

Basically, they still work on the same cluster, but they can, of course Kansas can can be recreated in different namespace yeah. So.

A

So basically, this is just a sample, so we created everything in the default. Namespace, like you, see better more secure way. Is that, though, the balance of resource and an internal zero crystal should be transparent to the end user, it's better to put them in your name, spec order, for example, cube system only and the Soviets account your CI controller. Has that authority to access them and the sphere object, of course, is a user oriented that they can manage those yeah. This yeah good question yeah.

B

Cool. Thank you for.

A