Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 22 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Multicluster Toolbox - Adrien Trouillaud, Admiralty

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

The Multicluster Toolbox - Adrien Trouillaud, Admiralty

The Kubernetes community has been talking about "multicluster" for several years. The topic is often associated with the Federation (v2) project. While Federation offers a solution to a problem (top-down resource declaration with templates, placements, and overrides), multicluster is about more than that. As Adrien was working on a different problem—peer-to-peer scheduling—he soon realized that some of the building blocks were missing: how to authorize requests from pods in a cluster to the Kubernetes APIs of other clusters; how to watch resources in a cluster to control resources in other clusters; how to garbage-collect objects in a cluster owned by objects in other clusters. Adrien started assembling an open-source "multicluster toolbox", composed of multicluster-service-account and multicluster-controller. This talk explains how to use them to solve unique multicluster problems.

https://sched.co/MPa8

A

And thank you everyone for being here, so there's been a switch in the room. So if you're looking for the security session, it's in g2 and well here in g1, it's the multi cluster toolbox, so multi cluster. Some of you may wonder why anyone would need multiple clusters dealing with one is hard enough, but if you're in this room you probably have an idea- and this question has been answered multiple times so today- I won't be focusing so much on the. Why?

A

Rather on the how how we solve multi cluster problems so just picture multiple teams working with multiple clouds, multiple data, centers and multiple regions, problems problems are bound to happen.

A

So the tools that we'll introduce in this talk our lessons learned for a months of research at Admiralty, a company I founded through research, this domain space and offer solutions, but I won't talk about those products in this talk, so why not just use cube, fed cube fed for those of you who don't know is the new name of the Federation v2 project they renamed the repository last week and in some situations it's a very valid response. It's a yes! If your views case you know matches. So what does Federation v2 solve?

A

It solves a problem where you have a central cluster and you want to do top-down resource declaration, so you want to define declaratively your infra, your resources across your Federation from one cluster, and you can do that by decomposing your problem into templates.

A

That is, basically you want to deploy, for example, deployment here into multiple clusters with like this basic template, and then you define placement, which is a list of cluster names or cluster selector, because clusters are customer resources, information and over eyes. If you want some variations, so in this example, there are just more replicas in close to true than cluster one, but not all multi cluster problems are can be solved with Federation.

A

There are plenty of application examples of multi cluster. We don't have time to. You, know, go through this list and discuss it but feel free to you give it a look, read them and I'm sure some of these will resonate with some of you.

A

A

But download off-the-shelf solutions to solve these problems. The tools are missing, but what kind of tools? What are we trying to build? So just like to build a shelf?

A

You know you need some tools and well you know you want to build a shelf, but you do you know what you want to build when you think about multi cluster and what I think we need is just like controllers are solutions to many problem in kubernetes.

A

That's how humanities works, weed, Multi cluster controllers, and so what is missing to create multi cluster controllers is a way to watch and reconcile resources in different clusters, say you have a high level API in one cluster and some either communities or other custom api's and another and I want to reconcile them. But if I delete an object in one cluster, I want the dependent resources to be deleted in the other cluster and we don't have garbage collection across clusters.

A

And, finally, if you've tried a few multi cluster solutions out there, it's it's usually pretty difficult to set up. There's a you need to generate queue, configs and store them in secrets, load them, or there is a custom CLI that you have to use and run an inn relative command, and so that it'd be nice if there was a declarative bootstrapping for those things.

A

So the generalized solutions to these problems have are packaged in some of Admiralty's. Open-Source projects like multi cluster controller and multi cluster service account Multi. Cluster controller is in a way sort of a multiplexer version of the controller runtime project, which is an official humanities project that basically drives queue builder or the operator operator SDK and a multi cluster service. Account is a way to as a system of controllers that imports in Odum at odd amounts service accounts from other clusters into pods of a local cluster.

A

But actually we won't see a lot of multi cluster controller. In this talk, I thought it would be more instructive if we built a multi cluster controller from scratch, and so we're gonna start with the sample controller project. Some of you are familiar with it. It's a sort of the like a an example project that you can use. You can fork it vendor it and create your own controller from it. And what does it do so? It's it's it's it's an example right.

A

So there's a high level API in this project, that's called foo and when you create a foo object, then a nginx server deployment is created with some parameters, and so some people like to have higher-level api's for their users so that they don't have to figure out all the details of you know how to deploy this thing yeah. So, uh let's, let's try it so there will be some live coding, live demonstration and I. Don't want to waste your time so I'm getting a lot of I'm gonna. Do a lot of copy pasting.

A

I mean also a shameless slow, typer. So beef before you can create a few object, you have to declare a custom resource and for that there is a custom resource definition, this big enough for people in the back.

A

So so yes, I'll, try one like this, and so this is a very simple custom resource definition that has nothing of the advanced features that you should use, like validation and other things. So but we don't need that for the purpose of this talk so.

A

So we're going to create it in cluster, one of four information: I just tested them before this demo I've traded three clusters in gke that I'm going to use for purpose of this talk and they are associated with cluster 0 cluster 1 and cluster 2 context. In my cube, config and I also extracted individual cube, config files in a specific folder as a convenience, you'll see why so I mean cluster 1 I create the CR D and then so. I checked out the code.

A

I mean just a few tweaks, it's really very, very similar to a sample controller in master in origin, and so yeah I, just added a plug-in for GCP login, and things like that. So this controller is running in another terminal.

A

I can create a fruit object, which is this one? It's just an example. The nginx deployment will be called example, foo and we'll have one replica.

A

Let's see if the deployment was created. Yes, what does it look like it's an engine, X deployment and we can update it with true replicas.

A

And you can see that the deployment has been updated down on to replicas and the usual. You know the usual controller loops things. So when you delete the deployment you want, you want it to be recreated, because the food object still exists. That means, as a user, you want constantly to be a deployment with the same name. So if something that deletes that deployment, they should be recreated recreated, it's nice also to in a control loop to update the status of the owning resource, and so here in the in the few objects.

A

We see that there are indeed two available replicas and it's also nice for the user to broadcast events, and so in the sample control project. Their events are broadcasted when there are errors or the sync loop, it was successful. And finally, if you delete the owning object, which is the fruit object, you want the deployment to disappear to DB delete it. That is using a garbage collection all right, so we want to transform this sample controller project and make it multi cluster.

A

So I'm gonna start with by changing a few things and basically watching foo in one cluster and deployments in another cluster and at the same time, I'm kind of combining a few steps here. um I also in this example, I want all those nginx deployments to be deployed in the same namespace in that different cluster I, don't want the namespaces too much.

A

It makes sense in a single cluster to want to the owner dependent relationship to only exists within a single namespace, but across clusters, not necessarily so that I can deploy that foo object, which could be you know like oh I, need a database in that database is in so I mean clustering, aks and I want a database, but my software needs you DynamoDB and so I declare foo and so in. In the other cluster we can deploy a sorry in cluster 0. Here we apply all those common resources. Central resources from you know like this.

A

In the end of this, at the end of this talk out, there will be a second step where I we can define and declare those who resources from any cluster they're all connected to close to 0 and cluster 0 runs all those dependencies in the same namespace.

A

So, but if we want to do that, we have to understand how a controller works, and so this is a this- is a diagram from the sample controller projects documentation. It was actually originally published in cloud ARC's blog, and so it was nice of the data to contributed to the project and because it's a great way to explain controllers. So how does the controller work in just a few minutes usually create a set of informers in indexers and reflector?

A

So, like whining former index for reflector from a factory, you create one yet there's one created per resource, they can be shared if you have multiple controllers running in the same process for aptitude. So that's the way to optimize API calls, and so the reflector just lists and watches objects from the kubernetes api, which are cuban ATIS verbs and the informer listens to those changes caches. There is the the objects in using the indexer so that all the gets and lists that you need to use in your code are optimized.

A

They don't always call the API, they don't. They use the cache and your controller can subscribe to changes and that's where the controller does and the controller sees some change like a deployment was created. If you object, what's updated anything deleted and used a key in a work queue. That key is what we call the reconstruction and it's important to have the same key for the objects that you need to reconcile, so usually in a single cluster. What we do is the is that we define the key as name space plus name.

A

That is a unique way to identify an object in kubernetes, and so if we want to reconcile a few object with a dependent deployment object so with foo, we just say: okay, let's take your name space on your name. That's your key with the deployment.

A

We need to know what the owners name and name space are, and for that we can rely on owner references and so an owner reference is that is a identification of the owner here at foo object called example, foo and the unique identifier note that there is no namespace in here, because in kubernetes we assume that owner dependent relationships are only exist in namespaces.

A

Yeah and finally, we process the items, and so there's a there. Here's where you put your business logic, it actually often looks the same. It's a it's! The usual control loop, where you, you have a key for that key namespace name, you get if you get the own owning object in this case. That's a very common pattern for higher-level api's, and so you get the foo object.

A

If it's possible, they doesn't exist anymore because in the meantime it was leaden between the time it was changed was noticed and the time the controller processes, the the key in the work queue. First of all, it's disappeared. Well, garbage garbage collection, won't. Take you up the deployment we're good, but if so, if we find the foo object, we need to make sure that it has a deployment. So if it's not, if we don't find the deployment mean, it means needs to be created. So here is the.

A

It could mean that the deployment has been deleted. It could mean that it's never been created, yet we actually don't really care, because this is a level based algorithm. We care about this, the state, the current of the world and what we want it to be, and so, if we find it, it has to be up-to-date. If not, we updated. In any case, it's a good practice to update the status of the owning object and broadcast some events so yeah, it's, uh let's try to to a change sample controller.

A

To do this, like you have deployments created in a different cluster in the fruiting space. All the time.

A

So I'm going to check out a code change I will I will push this branch to a repository after the talk, so you guys can read redo the same things on your own. If you want oh I already built it sorry so build to manage the the manager to the current state. Let's see, let's see what what I did here in the code. I won't go into too many details, but down so many changes actually so in uh in this first change, I was a little naive.

A

I may have broken things we'll see, and so just this controller, this manager has a cube confits that QQ config flag to be run out of cluster and so I just added one, which I use to create a different kubernetes configuration in code now use that configuration to create a client and an informer Factory and in the controller I just replaced every where we needed to create a deployment updated deployment delete. No, we don't delete because we rely on garbage collection and anytime. We need to do things with deployments.

A

I replaced the using the foo namespace with a constant, the foo namespace. Instead and that's it that's all. I did, let's see if it works.

A

I said: I wouldn't waste your time.

A

And what just happened.

A

A

See if the next step works.

A

It's Prime with my cute config, alright, never mind I'm gonna like say my messages first and we'll try to go back to it. A little later.

A

So this had worked, it actually wouldn't work properly, like I'm just here, I'm, just not able to run it, but I should be able to run it and what so in the example, I touch. Hopefully you can do later or we can do later.

A

You create a you, create a few object and Wow the dearies deployment in the few namespace in the other cluster, so you're happy, but a few things don't work. For example, events are not broadcasted the when you delete the deployment in the few name space it is not recreated, and if you delete the foo object, the deployment is not deleted, so garbage collection doesn't work and the the way to figure out what the owner is of a deployment doesn't work either and so to fix that we we need to enhance the owner reference.

A

The owner reference only provides information about the name of an object, but should also, in our case for multi cluster, say what namespace in the other class area did what Nate what namespace the owner is in and also what cluster. In this case, we still have just one cluster that has contains the Foo objects and the next step needed. Another enhancement and garbage collection doesn't work. So why does garbage collection not work? There are so I'd step back a little bit. There are three modes of deletion in kubernetes which are either orphan the dependence.

A

So you just leave them be say I if I delete the owning object. Foo I want to keep the deployment in place, but just remove the owner reference orphan it. What you want cascade, cascading deletion, which is what, if I delete the full object. My intention is to also delete the deployment, and so this can be done either in the background or in the foreground, and that is done by the garbage collection controller.

A

The garbage collection controller is a generic kubernetes controller that maintains a graph of owner dependent relationships and is when it detects that, so it listens to basically all resources and when say a food object is deleted. It will check the dependency graph and see that, oh, that deployment needs to be deleted, I'll delete it, and so that can be done. In the background, so when you delete foo, it disappears right away and the deployment is deleted just a few seconds later, or immediately and Vince called faster cluster runs and or in the foreground.

A

So how does that work? He uses finalized errs so in the foreground that when you want to delete in the foreground the foreground, that means you don't want the owning object. The foo object in our case to disappear until all of its dependence, your just the deployment have been deleted as well, and so the garbage collection controller does that by adding a finalizer to the to an owning resource when it's deleted. This example is actually a deployment.

A

If you deleted a deployment, you would want the replica set to be deleted as well and cascading to the pods, and so when you delete the deployment deletion time stamp is added, but also a foreground deletion and when the garbage collection controller is done deleting all the dependents it removes the finalizer and finally deletes the owning object.

A

Well from in our case is that we have two clusters: each has its own garbage collection controller and they don't agree. They basically each have their own representation of of owner dependent relationships and only in their own clusters. They don't communicate so they're unable they don't know what a multi cluster owner references, so we could and I'm actually working on one create a multi cluster garbage collection, controller, that is generic and we could have a multi cluster owner reference. That actually is something that exists in multi cluster controller.

A

But the solution in this example is actually what I call poor-man's in controller garbage collection using finalized errs, basically in our in our control loop, we're going to add a finalizer right when the object is created and before any of its dependents are created to avoid any race condition.

A

If, if a foo exists, it cannot have dependents, the dependents, unless it has a finalizer that way when we delete the foo object, we'll be sure to delete manually or like not not only automatically in our control loop, but like custom code, deletes the dependents before we remove the finalized the finalizer. So this is a like in in the in our sink handler. In the controller we implement, basically a decision tree based on observations, so is the true object.

A

Terminated does have a deletion timestamp, if it doesn't I mean, is still alive and first things first make sure it has a finalizer and then go back to our control from the very beginning. Is the deployment found no created? Is that today, no updated things like that?

A

I for clarity, I removed, I didn't loop back with update status, broadcast events, but you still need that so if it, but if it's terminating in so kind of typically at the end of the lifecycle, but remember that the lifecycle is not the same thing as your logic in your sink handler because we're edge based, sorry well level pace, note edge based, and so it's so if the deployment is found, we need to delete it if the foo object is terminating and if not what that means we're ready to remove the finalizer.

A

We do that and then we can just rely on our single cluster garbage collection controller to remove the foo object.

A

In this second example, I.

A

Added a cluster to the mix so like I said earlier, and so this example should should work better because I'm running it in cluster. The first example was running out of cluster from my laptop using cube, config files, one of them apparently has been tampered with, so in this example, I will deploy a pod in a cluster 0. This is where we want to run our controller. We want to run it in this example in the central cluster 0, and if we wanted to watch all the clusters 1 2.

A

However many you want and see if, when they create foo object, make sure that we create the deployments that they need.

A

So how does a pod call the kubernetes api of its own cluster first? Well, it has a service account when you create a service account.

A

The service account controller, creates a secret in the same namespace with a token to call the API, and the token is auto, mounted and auto mounted in the pod when it is admitted and germinate his clients like client, go decline. Girl library know how to load the token file that has been mounted mounted as a volume and so always good. Now you can extract actually the token from that secret and create an equivalent queue config file and is what people usually do when they do multi cluster of things. So you take that token.

A

You put it here in the user and then you take the the druvan anus api of that cluster and add some like names to know where to find things and then store that in a secret. So it's a bit of a pain, but it works. So here is a example of an installation procedure for sto multi cluster with VPN connectivity and so run all these bash scripts and extract that extract this, create that it works.

A

But it's it's it's a pain point I believe so what I suggest is a decorative way to do this, to import a service account from another cluster and so Multi cluster service account comes with a Ciardi called service account import and a service account. Import is simply a cluster name, namespace and name of a service account and most cluster service. Account controller takes care of finding that secret in the other cluster importing it and then, if your namespace enables it at admission, pods that are annotated with those service account.

A

Import names will see those tokens as cube. Config files are amounted in a non robot in an own folder, so that then you're Cuban, it is client library like client, go or I. Think the Python library does that, although maybe I shouldn't I, say that because I haven't used it, but most humanities client libraries know how to use cube config files, because that's what you use when in development when you're just dealing with your clusters that you create it, and so you can just use your code as usual, but wait.

A

Some of you may think. How does multi cluster service account import? Those tokens from the other clusters right once the chicken and egg problem you get to start somewhere and so I believe that if you install multi cluster service account, you do that brute bootstrapping process once where the custom CLI. But then all of your other multi placer controllers can use the declarative API of distributes account import. So you don't have to do that ever again, so it probably wasn't a rooster or the urn fertilized eggs.

A

By the way, let's write, let's write a demo of that.

A

So I'm going to check out my code, add a different tag and they see because it just adds an example and I'm going to deploy those stacks and so I have a cluster n stack and a cluster 0 stack. What's in. What's in cluster 1 and cluster 2, we have a custom resource definition for few objects. We do create a foo namespace, even though it will not contain any deployments. It's just a way to host a service account that is allowed to list watch, get update the fool objects and creative X, and it's so.

A

There is our back rules for that in cluster zero, which is our central cluster. In this example, we have a namespace that will host all the nginx deployments and the service account that is allowed to list watch guy create those deployments, some binding for that, and in here our true service account imports to import that other service account from all the dependent federated as a general term clusters and, finally, a very simple deployments for the controller that is annotated. So I can import those service accounts in a way to I.

A

Just call the the with the same arguments that I used in the previous demo, so you configured it from this folder where Admiralty serves the counting port, usually puts tokens.

A

Okay, yeah: it's almost done.

A

It's actually a good thing that first demo didn't work, because we wouldn't have had time for this.

A

So so I applied those tax to cluster one cluster to four cluster n, then I'm. Installing a multi class of service account in cluster 0 and then bootstrapping with a custom CLI cluster 1 cluster 2. These two steps- you should only do them for your first months of cluster setup. If you create more multi cluster controllers, you don't have to do this. Your clusters are already well well known to each other and they can import tokens from each other.

A

So when once it's bootstrap I can finally install the multi cluster controller in cluster 0 I create the example foo in cluster 1 in a different example, called bar in cluster 2 and cluster 0, you can see that our deployments have been created.

A

That's that's it for the demo. So, in short, we create the created this custom multi cluster controller system using cross cluster control, loops cross cluster, garbage collection and decorative bootstrapping, which are the common ingredients at the core of these projects.

A

Now many examples and I be curious to hear about yours, because just just so many things we need to do with multiple clusters, and so here's how you can connect with me if you want in some links to the different projects, I've talked about either from Admiralty multi, just a controller service, accounting scheduler, which I actually barely mentioned. It solves two of the things in that list of examples and then some of the official projects that I mentioned yeah and if we have time for questions I'll, be happy to answer a few.