Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2017 (Berlin), 11 Apr 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Cluster Federation: How to Write a Federated Controller? [A] - Madhu C.S.

Description

Kubernetes Cluster Federation: How to Write a Federated Controller? [A] - Madhu C.S. & Nikhil Jindal, Google

Extensibility is one of the major selling points of Kubernetes. Cluster Federation is built on the same spirit, but the tooling provided in the form of libraries and framework to build federated controllers is quite different than the tooling provided to build Kubernetes controllers. In this talk, we will show you the tools and techniques provided by Cluster Federation to write your own federated controllers. We will also show you the steps involved in building a federated controller by building one during the talk.

About Madhu C.S.
Madhu C.S. (madhusudancs@{github, slack, twitter}) is a Software Engineer on the Kubernetes team at Google where he works on Cluster Federation. Before Kubernetes, he worked on a number of different projects within Google Cloud. He also has a strong background in compilers and has worked on a JIT-compiler project at Google. Before joining Google full-time, Madhu helped the Google Summer of Code team for five years as an open source contributor. He is a long time open source contributor, a Gopher and a Pythonista.

About Nikhil Jindal
Nikhil Jindal (nikhiljindal@github) is a Software Engineer on the kubernetes team at Google. He had started by working on API machinery and is now focussed on Cluster federation. Before getting the kubernetes bug, he was enjoying working on Google Maps.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hello, everyone hi I'm, Nikhil and I'm a friend Madhu here, and we are going to talk about cluster Federation in kubernetes and especially specifically how to write a federated controller. But before we get started, can you get a show of hands of how many people have actually used cluster Federation?

A

Alright, a few so just to make sure everyone understands what cluster Federation is so cluster Federation makes it easy to manage multiple clusters, and why would you need multiple clusters? There can be many reasons like you want clusters closer to where your customers are. You have customers in Europe and you have customers in US, so you would want clusters in both those continents across you don't want cross Atlantic latency or you want high availability that you will have clusters in multiple availability zones. So even if one goes down, the other still continues to work.

A

All right. You might need multiple clusters for scalability or you want to avoid vendor lock-in. Like you have a hybrid make a hybrid setup. You have one on AWS one on GCP or on Prem. So those are some of the reasons why you might need multiple clusters and what Federation specifically helps with is like. In this example, you can see you might have red, blue and green three different clusters, and you want to deploy your app across all the three clusters, so without Federation. How would you do it?

A

Your developer would have to create the same deployment across all those three clusters ensure whenever you are doing a rolling update, you'll have to update all three of them with Federation. You get a single endpoint which ensures that you are all that your app is in sync across all the three clusters. So, instead of sending those commands to all those three individual clusters, you just tell Federation that I want this app deployed, and this is the version I want and Federation ensures that it's deployed to all all the clusters.

A

You do, you can do rolling, update, rollbacks everything works and all the clusters are in sync, and once you have these apps deployed, you also want to create these network resources like service or ingress like if you have an English object. You want to configure that ingress in such a way that it's backends are configured from all the clusters like. If there are pods running in red, blue, green all the tree clusters, all of them should be configured as correct backends for ingress or your global service as well.

A

So if you, if you are creating service or ingress via Federation Federation, does it for you, it will automatically create the load balancer or the DNS DNS rules for this service in such a way that all of those would be valid. Backends and now, since your developer or your admins are talking to this single Federation control plane instead of talking to each of those individual Estelle's, this becomes the single place. You can apply all your policies like should ulam policies or our policies, everything you you apply.

A

Those policies in Federation and Federation gives you a single point where you do that.

A

So if you want to do it, how it really works, you first have to sign up for Federation control plane, and then you can join all the clusters that you want to Federation once you've done that you use the you can use Federation, as in you'll use. Kubernetes Federation is 100% api compatible with kubernetes. If you have your existing replica set definition and you run like you, cal create queue, cut will create replicas set to and you send that request to cube api server.

A

The same request can be sent to federation api server and the defaults are configured in such a way that it will all work out like it by default balance the replicas across all the three clusters and obviously you there are some additional advantages that Federation gives and you can enhance the replica side spec that you already had to take advantage of those Federation specific features as well, but your existing replica set spec that used to work with kubernetes will work with Federation as well.

A

So, like your kubernetes, don't need to know developers don't need to know that, are they talking to a kubernetes api server or a federation api server? They just get an endpoint and the same kubernetes resources work for Federation, so to go deeper into the architecture.

A

This is like a 30,000 foot view of how a cluster Federation setup looks like you see their users on the right and there's Federation control plane, which is talking to you multiple clusters. The users can talk the various ways they can talk to the Federation control plane.

A

They can use the REST API, they can use q, cuttle, command-line interface or the UI and you'll see those are similar to how you would talk to a kubernetes control plane as well, and you send you our request to Federation control plane, which then sends it to the other clusters that you registered with Federation and for those of you who are familiar with kubernetes. This looks very similar to how kubernetes works their users. Who can you talk to kubernetes control plane but then sends those requests to nodes.

A

So your developers no need to talk to the nodes directly. They just talk to the kubernetes control plane and to go deeper into the control plane in kubernetes. You know the control plane consists of like these are the main components in the control plane. There's API server, there's HCD and there's controller.

A

So when users send their request to the control plane, they're actually talking to the API server cube api server, which stores the desired state which it received from the user and just those stores it in its CD and then controllers make those desired state happen. So what controllers do is they'll read the desired state that the user wanted from API server. Then they'll read what is the existing state of the cluster and then they'll take actions to bring the current state of the system to the desired state.

A

So if the user said I want a replica set with five replicas, the replica state controller will observe that this is the users intent and right now in the cluster I see only like there's no replica existing. So then it will go ahead and create those five replicas. So controller is the one that does all that work in Federation again the control plane looks very similar. There's Federation, API server and users are talking to Federation API server.

A

They express the desired state, what they want it's stored in HDD and then there's Federation controllers, which make that the desired state happen. Now to tell us more about how those controllers work I'll invite module thanks.

B

Thank you so before actually looking at how to write a Federation controller or a federated control, let's try to understand why one would want to write a federated controller or like any controller, but before that, let's also do a show of hands again how many of you attended, one of the controller stock, either esterday or today we have talked means ever number of talks about this. Ok, not many of you I guess anyway. So what's the motivation, one of the main, the key powers of kubernetes, actually lies on its extensibility. It's a very extensible system.

B

One way to look at kubernetes is kubernetes is an api server, so it ships a bunch of interfaces and in the official release or the distribution along with those interfaces we ship a bunch of implementations. Those are the controllers and there are other components like your proxy DNS or and all those things which could be considered as controllers, because they they do similar things. But these implementations are our most cases 90 to 95 percent of use cases, but there are always cases which are not covered by these implementations.

B

So, in order to extend this implementation or extend the functionality provided by the default implementations or the default controllers, you might want to either augment or extend the existing controllers. One of the example that I can think of is: let's say you have a replica set in API object and we ship a replica set controller. But what, if you also wanted to create an audit trail or audit log for every replica set created in your system, which is not provided by kubernetes today?

B

There are talks about providing it in the future releases, but it's not there today and you want it now, so you could write your own controller to extend this functionality, create audit logs like who created the log at what time who created a replica sec at what time was it created and you could store in your database. You could do that today if you want to another reason why you might want to write your own controller is to customize the behavior. There are a number of implementations examples for this today.

B

The ingress API object is the best example kubernetes for GCP ships and ingress implementation, but there is ix implementation. People have written their own implementation for fi or any other load balancer that they have on Prem. So these are the key motivations one of the other motivations but lesser. A far lesser motivation is, you might want to implement your own API object, kubernetes code kubernetes as a cluster. Has this concept called TP our third-party resource? We don't have that in Federation right now, but we'll probably have it sometime in the near future, but yeah.

B

That's also another use case where you might want to have your own object and write a controller for it with that. Let's look at the architecture of the Federation controller. Specifically for this a for this talk. We have created a demo I, don't know how much the demo is going to work, but for this demo we have created an API object, called the foo object. It's called type foo, it's exposed by our the Federation API server I'm. For that we're going to write a controller today so like equivalent is controller.

B

If you are familiar with the kubernetes controller architecture, the Federation controller is a binary, it's an executable that can run as a part inside your cluster. So we have this controller called the foo controller and just like the kubernetes controller, it talks to the API server, but the difference in this case is it talks to the Federation API server, which is on the right here. What does it do? What do controllers do in kubernetes? They watch for the desired state and reconcile.

B

So in this case this is the foo controller and because we want to reconcile the stage for the foo object. This watches for the foo API resource in the Federation API server, but in addition to that, this is special to Federation. Every federated controller also watches an API resource called clusters. This API resource lists all the clusters that are part of the Federation. This is the key difference.

B

Kubernetes controllers watch for the desired state in a kubernetes cluster and reconciles the state in the same cluster, whereas federated control Swach for the desired state on the Federation API server, but reconcile this state in the kubernetes underlying kubernetes clusters, which are part of the Federation. So we don't have clusters right now, but after you start watching the clusters, you probably also want to start watching the same API resource foo in all the underlying clusters.

B

So if a new cluster joins Federation, you immediately want to start watching for the foo API type in the underlying cluster. So if another cluster joins, you want to start doing the same thing and the same thing every time in your cluster joins, and you might also want to do some cleanup operations if you want, when a cluster leaves Federation.

B

So with that, let's go a little deeper. This is like lowering the conceptual levels. We looked at the conceptual architecture. Now, let's look at the actual API types and after this let's go little deeper into the code itself. So the Federation API server exposes the clusters API type and it also exposes the food resource type and we want to what watch these API objects.

B

These are the specific specific resource version and kind name, Federation, /v, 1, beta, 1, slash cluster is actually a real API resource type in Federation, and you can watch for this, and for is another type that we added just for this demo. Then you watch for the same food source type, which is also exposed for this demo in all the underlying kubernetes clusters. Once you do that, you basically set up your control loops, which reconcile all you do in.

B

In this particular example, the only thing that we do is copy the object from the Federation API server to the kubernetes api server. It does not really do anything interesting, but it shows it demonstrates the idea of Federation, so you basically watch for the desired state in Federation, API server and update the state in the underlying clusters, and in addition to that, one more thing that a federation or a federated controller handles is cascading deletion.

B

When you warn the when the object is deleted from federated API server, you want the same object to be deleted from all the underlying kubernetes clusters. So with that, let's actually go like right into the code. This is a little too much I think, but let's see how much we can.

B

So we talked about all these concept. Shell things. The first thing that we talked about was watching the full resource. The controller architecture or the controller infrastructure in kubernetes is actually a library that you can use. There's a lot of coding here, but I'll point out the important bits. This is this.

B

Some of this is boilerplate, which is setting up the library or calling the right functions or the packages right the methods, so we have a library called as the informer library for writing controllers informer libraries used everywhere, including a disk controllers, every kubernetes controller talks to API server or sets up a watch using the informer library. So setting up the informer library looks like a lot of code, but it's mostly boilerplate. It's not a lot actually and it can be easily copy pasted and that's how most of us start writing new controllers.

B

We copy/paste from somewhere else and then start changing and then actually implement the logic. So this is how you set up an informer. This Informer is basically setting up a watch for your resource type foo resource type in the Federation API server. So it says foo resource and you need to provide two lambda functions which are implementations which are basically to list and watch watch.

B

Does a watch for the events in the API server and we also need a list implementation, because a watch uses a cache and caches can go out of sync so time to time we do a completely sink like list all the resources from the underlying API server and update the cache. So you need to provide these two implementations list and watch pretty simple. They just call the client functions, the appropriate client functions and then you need to say what object you are watching, then, a bunch of other boilerplate once you do that.

B

The second important thing that you need to do is watch the clusters and the other impart. The third important thing is to watch all the resources in the underlined twisters. These two are taken care by a single library. The other library is called as the federated informer library. Don't worry too much, you can see, I mean seat in the back like you can go and look at one of the controller file files, but the concepts are important. So this this setup looks very much similar to the previous setup.

B

It's just that here you are watching for the objects in all the underlying clusters, every cluster that is part of Federation and in the bottom. Here you also say you are watching the clusters themselves. The clusters object that I talked about. So in the implementations that you provide, you say what you want to do when a cluster joins or leaves a Federation. So here in this specific implementation, we only say we care about cluster available. That is, we want to do something when a cluster becomes available or join Federation.

B

We don't really define what we want to do when a coastal use Federation, but in a real controller, you probably also say cluster unavailable and supply an implementation for it. So these are the key things to set up the watches and once you set up these watches, the other thing that you need to set up is the cascading deletion itself. Again. This is another library function where you call the appropriate function, new deletion help I mean you call the new deletion helper to set up the deletion helper and you provide a bunch of implementations.

B

All these are basically lambda functions or the functions themselves. So you provide these functions as implementations. We use the finalizer api to set up cascading deletion. The details are not that important. It can be like you can look at decode, one of the controller's code and just use the same code. We are also working on simplifying all these libraries so that you don't have to repeat the same things over and over again for controllers, but this is the current API or the libraries available. Once you have all these, you basically hook things up.

B

You say you want to start the informer for the API and then you say you also want to start the Federated Informer and you start all of them, but one key thing that you do here is: when you start the informers. You also register handlers. You want to say you also want to define, or you define how you want the reconcile operations to handle. So you provide implementations for very consulars.

B

There are two reconcile, as one is reconciler that runs whenever the cluster membership information changes, which is called reconcile, force and cluster change, and there is other reconciler which runs every time. An object itself changes. So you provide the implementation for these two types of reconcile errs and the implementation for the reconciles are really simple. This is the reconcile force on trusted change. So whenever a new cluster, we only define what to do when a cluster becomes available, so this handler only takes care of cluster becoming available case.

B

So in this case, if you look at the last few lines of code, which is important first, one is just the error handling part. This is the important part here. Whenever a new cluster becomes available, what to do is you take all the objects you have in Federation and copied to the new cluster? So you basically do a list and copy. That's it. Deliver. Does the copy to the underlined clusters and then the more interesting part. The real logic is implemented in reconcile full function.

B

All the Federated controller logics are implemented if logica is implemented in this function, every single controller, but each controller has its own implementation. So here every time the Federation or API object itself changes. You compare you compare what changed and if anything has changed. You propagate the same changes to the underlying clusters and you usually don't want to do the operations every single time. Something has changed. So there is a facility or library to batch these operations.

B

So you batch these operations and give it to the updater, which does the batch update on all the underlying clusters. So with that, let's try a demo yeah.

A

That was a lot of code and we assumed all of you are software developers and wanted to see code, but you also demo and hope it works.

A

So what I, what I have is I have a liquor, Federation control plane set up. I have two clusters on GCE money on defends owns on GC, one is in your central one A, and one F and I have set up that this expose foo. Both those clusters and Federation he pays server also exposes foo and what I'm going to do is I'll, create foo and I'll created only in Federation control, plane and we'll see how the controller works and propagate said to the underlying clusters as well.

A

Yeah, let's see, if I'm able to connect to my machine.

B

Looks like the.

A

Odds are not in our favor every.

B

Single Federation demo since Tuesday evening, so law of averages has to work. Murphy's Law has to work and we are gone lucky ones yeah, and this is even basic. Ssh.

B

A

B

A

Seen the code yeah.

B

Let's actually move forward and if it actually works.

A

B

Can come back, come back.

A

Yeah all right.

A

So we've talked about a lot of stuff here and all of that is not implemented. We have a lot of stuff working like we have federated ingress, replicas, self.namespace secrets, config maps.

A

We have added alpha support for deployments and we recently launched a new come a new command line tool called cube, fit which makes it easy to set up the Federation control, plane registered clusters and in 1.6 it's going beta as well, but there's still a lot of work left to be done like the ingress, especially works only on GCP right now it doesn't support hybrid scenarios. We want to do that. These federated services. You need a public endpoint for them to work like the DNS rules that we configure.

A

It uses Google Cloud, DNS provider or route 53. If you are on it ablaze, but we want to make it work with private services as well, and we we are improving how the access management and identity work and the work that I talked about, that Federation control plane becomes the single place where you apply all the policies. We are doing a lot of work there as well.

A

So there's still a lot of work left to be done and madelyn I have been just lucky enough to be presenting it here, but we we are not the only ones working on it like everything in kubernetes. This is the community effort. We have a federation mailing list which you guys can join or we meet every other week and we have us. We are active on slack channel as well, and here you can see our email, ids and github IDs, and you can ping us directly.

A

If you are interested in any of the work that we listed, you want to contribute we'd love to have those contributions. Yeah and yes,.

B

A such thing came back, let's see if that.

A

B

A

Can you guys see this.

B

Can anybody, can everyone see this ok.

A

Yes, sir, here I've the Federation control plane, side up and I have two clusters: I sent them up last night and I have food or so side up as well, so I'm getting I I can run cue card will get foods with Federation and I can do that. The same thing with the underlying clusters as well. So you can see here cube cuddle. I am using the same cube, cuddle binary to be able to talk to Federation, API server or the underline clusters as well.

A

The way it works is I, have set up multiple contexts and I have a Federation context, which was talking to Federation API server and similarly I have two other context which are talking to those two underlying cluster clusters that I registered with Federation. So not now. What I'm going to do is I'll create a very simple foo object and I am I've created it in Federation, like you cuddly context Federation, and this is how my food looks like it's.

A

A very simple object, which says: cuke on rocks and so now what's happening in the background is the stuff that madhu talked about. There's this reconciliation loop running in the controller I have a foo controller running which what it does is it observes that the desired state from the user is that this foo object should exist everywhere in my cluster and what the current state of my cluster is that it doesn't exist in any of those underlying clusters.

A

So what the controller will do it will send create requests to both these underlying clusters and now, if I, do the same, Keuka delegate foods in the underlying clusters and see those this cube, cointreau object there as well, so it exists in Federation and it exists in the underlying clusters as well.

A

So foo controller did this panning out of the request to underlying clusters, but it wasn't just at creation time. It's always running, always active. So what? If I go ahead and delete this foo object from one of the underlying clusters, so I'll delete it from money. So since this foo controller is always running, it will again figure out that the desired state is that this fool, this coupon few objects should exist in all the clusters, but it got deleted from one so it doesn't exist.

A

There again send a create request to that cluster and again make sure that coupon true exists in both the clusters. So if I get Fu's again, I see it was recreated 13 seconds ago, and if you really want to delete this coupon few object, you will delete it from Federation API server, because that becomes the place of like the source of growth. So if I go ahead and delete coupon food from Federation again, the foo controller will observe that now.

A

Coupon few is not part of the desired state, but it still exists in the underlying clusters. So then it will go ahead and delete it from the underlying clusters as well. That shows like it's always active, always running, always ensuring what did the desired state matches the current state? So now, if I do get foods in the underlined cluster, it should say no resource found in both those clusters.

A

Yeah, that's all I have glad that bit of a work yeah, that's it.