Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Multi-Cluster Service Deployments with Operators and KubeCarrier - Rastislav Szabó, Kubermatic

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Multi-Cluster Service Deployments with Operators and KubeCarrier - Rastislav Szabó, Kubermatic

Complex application services consisting of multiple interconnected components can benefit from deployment in multiple Kubernetes clusters: Examples include for instance running the application core at a cloud provider and the database with sensitive data in a on-premises cluster or running computational-intensive tasks in a cluster with specialized hardware resources (e.g. GPUs) at the same time. This approach however brings several challenges. How can we interconnect the clusters so that the applications in different clusters can communicate with each other easily? How to allow for multi-tenancy and easily spin up multiple instances of such services in the same clusters? In this talk, you will learn how such a deployment may look like with the help of Kubernetes operators, the KubeCarrier service hub and the Submariner cross-cluster connectivity provider.

A

Hello and welcome to my talk, my name is rastisal and today I'll be speaking about multi-cluster service deployments with operators and keep carrier. So before we start, let me introduce myself. I am a software engineer at cobramatik, where I am working on cobramatic kubernetes platform and cubecarrier open source projects.

A

I am specialized on multi-cluster application management and multi-cluster networking and in the past I also contributed to several other other open source projects such as regatta. I o fdi go vp contvp, cni or sysripo.

A

So as the title of the talk suggests, it will be about multi-cluster service deployments. Today. During the talk I will go through different aspects of multi-cluster service deployments and I will also mention some community driven open source projects related to them. We will go to multi-cluster infrastructure management, then multi-cluster application management and finally multi-cluster networking.

A

At the end of the talk, I will also show a quick demo of multi-cluster application deployment with cube carrier and multi-cluster networking with submariner.

A

So before we go into any details, let me talk about use cases for deploying applications across multiple clusters. One of the reasons for doing that may be close user proximity.

A

For example, we would like to serve users from different parts of the world without high latency, another reason, maybe regional, high availability, where we may want to minimize the impact of regional outages. Another reason may be security and organizational separation.

A

For instance, we may have to use dedicated clusters for each organization or organization. You didn't the next one may be data locality, for example, databases with sensitive data may be only available on in on-premises clusters and last but not least, one of the biggest use cases is edge computing, where we usually run many smaller clusters distributed across multiple locations uh because of flow latency requirements.

A

So let's start our multi-cluster service deployment story with the necessary infrastructure for running and managing multiple clusters.

A

For that, each cloud provider usually provides their own solution, but if you wanted to automate operation of many clusters across multiple regions and different cloud providers, including on-premises infrastructure, and do that all by a single pane of glass or a single api endpoint, I really recommend you to take a look at the open source, kubernetes kubernetes platform, which can easily do it for you, okay, let's assume that we have our hundreds or thousands of clusters running now. Let's talk about distributing some workloads on them.

A

The kubernetes sigmatic cluster provides two possible solutions for that the other one is called cubefet. It aims to solve much more than just multi-cluster application deployment. For instance, it also aims to solve scheduling, dns policies, etc. It is widely used, but quite complex to use. The newer concept is called work api. It is a simpler approach for deploying workloads to clusters.

A

For instance, it does not cover cluster registrations or scheduling, but at this point it is just the api definition without any implementation.

A

It is based on the work custom resource definition, which can contain a list of resources that should be applied to a target cluster.

A

On the right side of the slide, you can see how the work api custom resource may look like. It refers to a specific cluster and, in the workload manifests section, it contains a config map. Both approaches share some common concepts. They contain a single source of true from where the workload manifests are propagated into managed clusters and a control loop, which applies resources and tracks their status.

A

Now we are getting to the cube, carry project which we created in kubermatic. It builds on similar concepts as the previously mentioned solutions. There is one management cluster which is our single source of true. We call it service hub and there is. There are multiple service clusters that can run application workloads.

A

The difference is that by design it only works with kubernetes operators, application operators run in the service clusters carrier, discovers their custom resource definitions and make them available for users in the service app and cubecad, then propagates the custom resources from the service app to service clusters to drive the operators running in them.

A

Finally, keep carrier has built in multi-tenancy, so multiple service provider and multiple service consumer accounts are supported.

A

The reason why we decided to use operators in cubecarrier to build our multi-cluster application management platform is that operators help us to automate full application lifecycle within a cluster that includes deployment upgrades, backups, etc and at the scale of hundreds of or thousands of clusters.

A

It is really necessary to have this automation in place. Managing individual kubernetes resources for each application in every single cluster just cannot work at this scale.

A

This picture illustrates how keepcare works from a higher level. The service users interact only with the management cluster like kubernetes apis. They deploy custom resources derived from the original crds provided by application operators running in the service clusters.

A

Gibb carrier then distributes those custom resources across clusters which drives the operators running in them, and operators in the service clusters deploy and manage the application instances based on the content of the custom resources deployed by cubecarrier.

A

This slide shows how multi-tenancy works with cubecarrier. As you can see, keepcare supports multiple service provider and service consumer accounts, which are separated by namespaces, also users within each account get proper kubernetes arbitrals set up and assign automatically. We have three personas illustrated on this picture.

A

The platform operator operates uh the management cluster and manages the keep, carry installation itself and also manages the carrier accounts.

A

The service provider manages service clusters, operators running in the service clusters and registers the services or their customer resource definitions in the service app. The service consumers interact only with the management cluster, where they request and manage their service instances.

A

Okay, so now we have our hundreds or thousands of clusters running and we can automate the deployment of applications into them via our central service app. Now, let's talk about how the applications can talk to each other across these clusters.

A

The kubernetes multi-cluster provides a solution for that as well, and that is called multi-cluster services api.

A

It extends the kubernetes service concept across multiple clusters in the provider, cluster services have to be explicitly exported using a service export customer resource. The multi-cluster services implementation that proper then propagates. The service export into service imports in all clusters in the cluster set and the exported service will then become accessible from each cluster. In the cluster set at the dns name, service name, dot, servicenamespace.sbc.clusterset,.

A

An open source project which implements this api is called submariner io. I will use this project in my demo. Okay. Now I would like to show you a demo of multi-cluster application management with cubecarrier, combined with multi-cluster networking with submariner.

A

The demo. Topology would be similar to the picture that I have already described. We'll have one management cluster that we'll use as our service app and three service clusters. Where we'll be deploying our workloads for the demo, we'll use the redis database as our managed application, we will run submariner across all four clusters to provide multi-cluster services connectivity, okay. So what you can see here is the kubernete kubernetes platform, where I have created four clusters: each one had a different cloud provider in a different region.

A

One of the clusters I'll be using is a management cluster that will be our service hub through which we'll manage the deployment of applications into three service clusters. So let me go into the console in the top left corner. You can see uh the management cluster and then we have three service clusters here. So it's cluster one two and three: the cube carrier has been already installed in the management cluster and the service clusters have been registered in the cubecarrier management cluster.

A

Also, the submariner have been installed in each one of the clusters, so the multicluster services should be working when we'll need them.

A

So let me start the demo with showing you the cubecarrier accounts that I have created so we'll have three keep clear accounts uh in this demo. uh One of them will be the service provider account, and then we will have two tenant accounts, so the service provider will provide a ready service to the tenants in each one of the user clusters in service clusters. I deployed the radius operator already, but so far nowhere these instances are running in any of those clusters.

A

At this point, we can check via cube ctl get catalog entry. That cubecarrier has already discovered the custom resource definitions in the service clusters. So uh so, in our catalog entries list, we can see redis in cluster. One red is in cluster two and release in cluster three, which are ready uh for for using by our tenants.

A

So at this point our tenants can request an instance of redis in any of those service clusters and the way how they can do it is uh by creating a custom resource which is derived from the original redis custom resource definition. uh But the api version refers to a particular cluster and a particular service provider.

A

So in this case, by using this custom resource, we will deploy a redis instance in service cluster one and the spec of the of this custom resource contains uh some some uh information uh needed to deploy already instance such as a password for accessing the database that we will use later.

A

So let me now use this deployment yaml file and deploy a redis instance in the service cluster, one as a tenant, a uh to make uh to make sure that we can see what happens I'll run. A watch commands here in the service clusters, one two and three uh they'll be watching for all ports and uh well.

A

We are correcting the the name redis in in across all the bots, so I'll now go ahead and fire uh this command uh so again, we'll apply this custom resource which refers to cluster one under tenant a uh and that will create an redis instance for our tenant.

A

So soon we should be able to see that the redis instance has been deployed for tenant a in the service cluster one. Similarly, we can. We can request a redis instance for the for the tenant b, so using the same custom resource uh under tenant b user.

A

We can request a redis instance for the tenant b and, as you can see, it has been just started very. Similarly, we can also deploy some radius instance uh in in the service cluster too. So the deployment manifest would look uh exactly same apart from the api version where we refer to service cluster 2.. So let me deploy that one as well and we should see in the service cluster 2 that our redis instance has been started.

A

Okay at this point, we have triggered these instances running into different clusters.

A

Now, let's try to connect to those random instances from a different cluster, so in the cluster 3 we are not running any radius instance.

A

And potentially we could we could try to connect to those randy's instances from this cluster. So in order to do that, we can rely on multi-cluster services api, which is which is implemented by submariner, which is already installed in our clusters, and so in order to export the service to a different cluster, we should first review what are the existing services of our radius instances.

A

So, as we can see, we have a radius instance service in the tenant, a namespace which is of type cluster ip and similarly, we have already instance service in the tenant b namespace, which is of cluster iptype. So in order to export them uh to other clusters in our cluster federation, I would use the command subctl. Export service substitute is a binary provided by submariner, but essentially what it does is that it deploys a service export custom resource with the name with the service name, read this instance and the namespace uh as as given here.

A

Similarly, we can also export the other service instance in the tenant b. Namespace.

A

And at this point we should be able to verify that service export exports have been properly created by showing keep cattle get service exports and yes, as you can see, our radish instances in tenant a and tenant b namespaces have been exported.

A

Similarly, we can do a service export in the cluster 2. For there this instance running in there and again we can verify that that service export customer resources have been properly created in this cluster.

A

At this point in the cluster 3, we should be able to see the service imports automatically created from the exported services in cluster one and cluster. Two and, as you can see, uh cubecat will get service. Imports gives us three service imports two in cluster one and one in cluster two.

A

So now we should be able to access any of those three services from this cluster. So let me exit into a client port running in this cluster and the way how we can connect to those services is using uh dns names uh that refer to original service, name original namespace, dot, service, dot, cluster set dot local, so this uh domain name should we should be able to resolve right now and it seems that resolving works.

A

So we can go ahead and try connecting to the transit redis instance uh with the redis cli uh tool. uh We are just referring to the exactly same domain name and we are using the password that we specified in our radius custom resource up here.

A

So this should work and if it works, we should be able to run a ping command and we should get the punk from the server right uh exactly same way. We can. We can also verify the connectivity to the redis instance in the cluster too, so it should just work uh as for the radius instances in the cluster one okay, that was it for the demo and for my talk as well.

A

If you have any questions, feel free to contact me at this email address or via slack right after the session, and thank you for watching my talk.