Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG Multi-Cluster Intro – Christian Bell, Madhu C.S., & Nikhil Jindal, Google (Any Skill Level)

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18)

SIG Multi-Cluster Intro – Christian Bell, Madhu C.S., & Nikhil Jindal, Google (Any Skill Level)

About Madhu
Madhu C.S. (madhusudancs@{github, slack, twitter}) is a Software Engineer on the Kubernetes team at Google where he works on Cluster Federation. Before Kubernetes, he worked on a number of different projects within Google Cloud. He also has a strong background in compilers and has worked on a JIT-compiler project at Google. Before joining Google full-time, Madhu helped the Google Summer of Code team for five years as an open source contributor. He is a long time open source contributor, a Gopher and a Pythonista.

About Nikhil
Nikhil Jindal (nikhiljindal@github) is a Software Engineer on the kubernetes team at Google. He had started on kubernetes by working on API machinery and is now focussed on multi cluster problems. Before getting the kubernetes bug, he was enjoying working on Google Maps.

About Christian
I am co-lead of the Kubernetes Multicluster SIG (previously Federation SIG). I am interested in how users can make use of multiple clusters for high availability, regional proximity and consistent deployments across regions and multiple cloud providers.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

I'm Nikhil from Google and I'm here to give a inter about segmented cluster I'm, doing it on behalf of Christian Bell who's, the sig, lead and I. Think Winton was also here who's the other sig lead, so he can get started to give you a high-level agenda of what we'll be covering in the next 20 25 minutes. First, I'll give a quick overview of what our mission is, what we do at segment a cluster and then how we are going about it. What are the different projects?

A

We have I'll go to some of the top-level ones we have, which is Federation cluster registry and the cube mci tool for multi cluster Inglis and, at the end, we'll have about 10 minutes for Q&A. So that should be good before I get started.

A

I just wanted to get a feel of like how many people know about sig multi cluster and have joined the coil and who are like totally new, who are totally new once can raise their hand, Oh lots of ok, so I yeah I tried to give a high-level overview, and if there's any question you can have, you can feel free to ask them like. While we go through these slides as well, we will keep it interactive.

A

So the mission our mission is to solve challenges about managing multiple clusters and then deploying applications across them. So cluster management and application management over those clusters and the way we are doing it is we have multiple tools. Api is that we design document and implement to be able to achieve these goals and to give you a bit of history.

A

The way we had started was it was called sig Federation before and we were working on Federation very actively and sig Federation was like one of the first three SIG's in kubernetes, so we've been working on it for a while, but now we are also working on a fetish and v2 approach, which I'll talk about here and also trying to come up with smaller, well scope tools for each of those problems, so that we can support those tools better.

A

They can go GA independently and we can support them and you can build and integrate each of those tools together and solve multiple problems together. But if you have like a smaller one like let's say, multi cluster angles, that's the only problem you have, then you just use like um MCI or if you only want cluster less, then you just useless religiously. You don't need to use like the full.

A

So that's so standalone building blocks, that's what we are trying to do and we have our mission and more details which you can look on the community github repo. So this is a high-level list of all our sub projects. We are working on Federation v2, which is like the next version of Federation V 1, with Federation v1.

A

What we were trying was to use kubernetes api to solve multi cluster issues so, like let's say deployments, and you want to deploy your app to multiple clusters, we were still using the kubernetes api, which is for single slicer and trying to use that same and expand to multi cluster, and there are some problems that come in. Like formal declare, sir, you need to provide some additional information about like how many replicas in each cluster, then. Similarly, you want status from each cluster rather than a unified and then like.

A

If you are doing rolling update, how do you do rolling update across clusters? So there were some problems with federation v1. So in federation we do. We are trying to come up with like a different multi cluster api. Then there's cluster registry, a very common use case. We heard four people working with multi color sizes. They just want a list of their clusters. Like I know, people have their cube conflicts and there are multiple people in the company working on multiple classes.

A

Each has their own cube, config and then any time a new cluster is added or removed. Everyone updates their cube config that turns out to be messy and so like list religiously. Is that single API endpoint, where you can list all your clusters? And it gives you a kubernetes api over list of clusters, so you can do like label selections. You can use cube cutters with like get Lester's, create Lester's list Lester's. All of those so like. That was a major use case.

A

So we decided, like cluster registry, is a well scoped project that does just the cluster list and then another very common use case we heard was like multicolor studying, dress people want load, balancing across multiple clusters. So there's scenario is you have your app deployed in multiple clusters in US, Europe Asia? And now you want a single IP address and route users to any of those clusters in any of those GCP regions or anywhere.

A

Your data centers, wherever you have but have a single IP address for that so cube MCI, is that standalone or cell CLI tool, which helps you build multicolor, saying guesses, federación v1, so federation v1 was moved out of the it wasn't cold k, /, k repository and we moved it out and we had a lot of lessons learned from it, like some of which I talked about that like kubernetes.

A

Api makes really sense for the cluster level, so we're trying to come up with like a multi cluster API and there's a lot of work going on that and I'll also talk about Federation v2, so Federation we do where VI right now is. We have a prototype which you can look at, that github repository and it uses less registry for list of clusters, and then it has other controllers built on top and the way we are defining, the API is basically three things template placements and over I.

A

So template is the common template, which should be same across all your clusters, so taking an example of deployments again you have the same pod pic so like that would be template. Then you have place when you want to define like this should go in my US clusters or like cloud clusters or data center cluster. So you can have some placement directives and then they're part cluster overrides as well. So let you want only one replicas in some twister, so you can provide some exact over lights.

A

So that's an example of how we'll do it deployment by trying to extend that same model to other API as well, and the way we are doing it is we implement them as like? Cid is over cluster registry, and so tomorrow we have a deep dive as well. So we are going to show a deeper demo of this. If you are interested, you should join that.

A

The roadmap is, as I said, we want to implement. All of these API says C IDs. We want to implement a third-party plugin, DNS controller. There's another interesting project called cube applier, which pulls changes so the model with Federation was there. Was this global control in running that pushed changes to all the clusters? Cube applier is a open source tool from box which, what it does is you run from your you run it in your an agent in your cluster and it pulls it from a common source.

A

So that's also an interesting model that we are exploring and trying to come up with shared federated informal, Federation controllers. We are going to do all these standard, kubernetes types and we'll do them. We have a roadmap for those and there's some issues. We're still figuring like I, said status. Aggregation like do. People want individual particular status as well, or they want like full aggregated list. So how we surface that in the API and like some of those issues so.

A

Next I wanted to talk about clustered industry, as I said, like one of the common use, cases was just list of clusters, so that is what list registry is it's an API server with a cluster resource and you can run queue. Cuddly create lists, get Lester a list list. Now you can add labels to them. So you can get like a list of all staging clusters or list of all broadcasters, and you can do like developer a gets list of clusters. A and B and developer begets like B and C.

A

You can add our back rules to them, so cluster lists would have like all the clusters, but one developer can only like testing clusters or some an admin can list only all the broad clusters. So you can build on those primitives over this and you can use all existing kubernetes tools. It's a standard, kubernetes api server and you can write your ramen spec.

A

You can be more declarative about it so and it's being used for like we want to use it for kubernetes clusters, but it's actually just any kubernetes api server list and lay castillo is using it for history or service mesh as well for multi cluster service types. If one to keep a list of multiple clusters, the API is alpha right now and we are planning to move it to beta.

A

The only blocking thing we have like now is whether the cluster list should be named space and there's some discussion happening around it in this sake, but we should, we hope, to resolve it soon, and so API should be beta soon enough, so we want to have more users user. It right out, build tools over it, and the game is that it's very simple: to use it's just a cluster list. You can integrate it with your CI CD pipeline or any other tool you want, which wants a list of clusters so like very scoped tool.

A

This is how the cluster spec looks like. So this is the github repository where it is. You can go. Take a look file issues. Try it out- and this is the general cluster kind it has the API endpoint and the C a bundle, and you can add basic cluster information, so this is defined.

A

Then there's been some confusion because it's called Calista there's also a clustered API, that's being worked by a cluster lifecycle, and that is more for, like how do you want to bring up clusters on like GCP AWS on Prem, and they have all the knobs that each of these cloud providers provide like you want to bring up API server, what all flags should it have? So that is a lot more details. Unfortunately, that's also called cluster resource, and this is as well but like. These are different.

A

This is a very simple cluster representation and, like the Amir, is that, ultimately, you should. This could also replace like queue. Conflict could be integrated wherever you use your cube, conflict. Sure.

B

Does it define only runtime configuration of a course like I see the only endpoints, or you also define a spec of how to create a cluster? No.

A

So this is only the runtime configuration that, like the cluster exists, and now you want to do some work over it like deploy some workloads or monitoring logging or whatever you want to do so. There's like this is about how to reach that cluster so similar to the way you use cube, config yeah.

A

Yeah next, what I want you to talk about was multi clustering guys. So, as I said, like a very common use case, we had is I, have my app deployed in multiple sisters, but now I want to configure a multi clustering list. I want a single IP endpoint, which then sends traffic to all my clusters, and you have clusters in different regions, and you want like, if I Europe lists, Europe a customer from Europe its your IP address. They go to the Europe cluster and someone from Asia hits your IP address.

A

They go to Asia and similarly US customer goes to us. You want that for like geo latency, so that, like you, serve users from the closest customers, it's for least latency. You want it for H, a reasons like. Even if your Europe cluster goes down, those customers are still served from Asia or us, so at least like they are not getting full of us they're at least being served. So this the advantages that this makes you resilient two clusters going down, or even on cloud providers, availability zones going down or four regions going down.

A

So you have clusters in like different availability zones. Again we'll have a demo of this tomorrow at a deep dive, so you should come, and Greg and I are also giving a talk on multi class strangers tomorrow, I think around 3 p.m. so. If you want, they will be sharing more details there. The way this qmc I CLI dual works is it takes your ingress, kubernetes, ingress definition and today takes like cue conflict, which is your list of clusters, but again like we want to integrate with cluster industry as well.

A

So if I were cluster is just the endpoint working you could list there, but essentially it needs a list of clusters and your English bike and it configures Google cloud load balancer today to give you that setup that I just described, but you can extend it to support others as well. So upcoming plans are integrated with religiously for lists of clusters extended to other providers and follow Signet works lead to so the ingress pack as it works in current kubernetes api. It only makes sense for a single cluster.

A

We want to extend it to make sense for multi cluster. As well so do more fancy stuff like right now, what QM CI does is. It lets sends traffic based on the location. So it's it's one load balancing word that it supports. There are lot of other load balancing modes you might have like you're carrying a new cluster, so you want like 5% traffic, 90% traffic or so a lot of other fancy load balancing modes.

A

So for that, we need like to define the API as well right, not keep him safe, supports there, zero latency use case.

A

This is like a block diagram of how it works, so you have users from California and Berlin, and you have two clusters, so users get routed to the right cluster. This is how you run the command cube, mci, create you give it an ingress back list of clusters. It configures google cloud load balancer and which gives you a single IP address, and you can give it to your customers.

A

That's the major projects we have and if there any questions is.

C

So just lie I understand more the multi cluster ingress use case. I, see that your implementation has the Google cloud load plan, serene in it. Without that put any DNS feature allow us to achieve similar. So.

A

The advantage with Google Cloud appliance a load balancer is that it has the anycast IP and like with Google's by network presence. You can do that. It's what well do it with the other I DNS service as well, and you can, if it like, supports this- that, like users from a shout out, iteration and users from Europe out to you correctly so then you can extend Cuban CI tool to support those other providers as well. Yeah.

D

So this cloud registry, API that we mentioned before, where does it reside I mean? Where does it leave so.

A

It's on open source, github repos. It will cover native slashed listed registry. The.

D

Api itself so I mean the service itself. So if I have several clusters, so do I select the cluster that I want to deploy it into or it's just replicated across clusters. So how does this work.

A

Yeah great question, so this is a different API server that you have where you install your list and then you run it on an existing kubernetes cluster, one of those that you already have well, you can have a different communities less than over. You run this host cluster, especially for a cluster industry, and you can install like other cid is like I said: can.

E

Extend that question is around having a separate control plane for multi cluster, or that would be kind of federated control plane. You.

A

Mean like in Federation, v2 approach or like generally in that.

E

Case, like do do you think, control planes separate from the from the current kubernetes control, plane, yeah.

A

So one of the use case for like this multiplexer control plane is to be able to run on well. Multiple sources do operations and be resilient to each of those clusters going down. So yes, like the you need higher availability for this rather than like each of those individual sisters. If you run it in any one of those classes, and that becomes your like master critical cluster- that if it goes down, you can't do any multi cluster operations, yeah.

A

And actually other people working on multi-class ii come up as well and as their questions in come up as well again, it's fun so.

F

I think what you questioned was that very exactly is same as registry also and same for FCP control, plane that where exactly does that run so currently, so initially, the way trail ran was that we had to had a dedicated base cluster in tradition, v1, and that was it so, as Nikhil mentioned, that that becomes your point of failure. Also. So now what we are trying to extend or what we are building is that there could be different configurations.

F

It can run with it can run as an aggregated API or it can run as odds say it's same with all the Federation v2 also now edition v1 and closure industry also that we can run it based on configurations. It can be run as aggregated or it can be FCP as a hold mechanism, or it could be a TRD install server. So next work is based on that CRD related.

G

So the federation, we want only supported certain kubernetes object types, but you mentioned that you plan to support every kubernetes type in Federation v2. So I'm really curious. How you imagine that's that's going to work because yeah, of course, some resource types don't make a lot of sense in in a federated context. Yeah.

A

And, like that was a problem we had in Federation, you know, you're always playing catch-up to like kubernetes is moving so fast, so I, let it fun who's working on Federation. We do.

F

So that's actually a very nice question, so in Federation v1 we did, as he mentioned, a catch up with some API is, and there were some really hard. Api is or hard types to integrate into Federation right so currently in v2 our approaches that we have for all api's. As you mentioned, we have some basic sync mechanism which can sort of replicate. For example, if you are creating a secret, it could be just replicated in all clusters, and that sync mechanism is a single sync mechanism.

F

You just have to install API, typer and adapter to it, and you can replicate it to all the types, but it will be a simple replication.

F

Okay, and then we have what we are trying to do is we are trying to build a layered approach to this, so more complicated types might have their specific scheduling kind of stuff or what are the specific stuff that needs to be done, for that particular type can be building a layered approach into a different controllers, so for all api's our target is, they can be simply synced, for example, if you are creating stateful set also, it could just, for example, if you are creating a stateful set in Federation, what this approach will do is will create same stateful sets in all, but not providing.

F

What really stateful sets provide in kubernetes right now, a more complicated approach could be using this mechanism. You can build a stateful set shed euler later or somebody else who wants to actually run. Only that stateful set workload can build their own controller to do what they really want. I.

A

Should probably also clarify the different maturity of all of these projects like Federation, we do. We have a live working prototype, but we are still exploring like a lot of those concepts. Qm CI is like a separate product on Google cloud and LexA studies. This is also we are its I'll find some people are using it and we want to graduate you to beta soon yeah.

H

Can you quickly describe the flow of somebody? Who's got maybe like four or five different clusters are just kind of using switch context today if they were using this multi cluster API? What would like what would the start of the day look like for them? If they're trying to jump around a bunch of clusters and a bunch of different namespaces yeah.

A

So it depends like today, if you're, just using plain, cube, quadrille yeah, you have to keep changing your context and like write, simple shell scripts for for looping around it.

A

So you could varied like you, you could use clustered registry and build fancy Dashwood's around it, which like goes through this list and it just fetches all your diploma means or other workloads that you are running and shows them in a fancy UI or you could build them in your integrated with your existing logging monitoring pipelines that you have that, instead of which you are using for single cluster. Now you integrate it with just religiously and expand it to like multiple clusters and so use your existing tools or you like. We are exploring Federation.

A

You could try that you extend your existing tool so like we want to give all these different options that, like what you want and use these building blocks together and tie them together to come up with your like a multi cluster solution. That's good for you! Your individual needs.

I

Do cluster Federation and kubernetes are loosely coupled how can I run, for example, multiple multi cluster Federation / multi kubernetes clusters independently? Yes,.

I

So the question is, for example, I've got one company and it runs its own kubernetes Federation and another company that runs its own kubernetes Federation over the common set of kubernetes cluster will they'd survive.

F

Yeah, so if I understood the question correctly is what you are saying is that a same cluster can can the same cluster be part of multiple Federation's right so yeah?

F

The concept of Federation is that Federation is actually an user, a user of kubernetes, so the control plane basically acts as currently, for example, I admin user for each cluster, and once you have joined that cluster to a particular Federation, this FCP, the control plane, has the user eyes to do what it can do with those clusters create workspaces or create workloads or that kind of stuff. So multiple Federation's multiple clusters could be part of multiple Federation's used by different users.

A

Yes, this one I did yes, I find her books, son hobby and is working on tradition. We do.

J

Hi last I looked Federation had the ability to send workloads to multiple clusters, but to get or to view the workloads across all the clusters. It wasn't it wasn't there yet. Is that something that's in the works, so I can do coop cuddle, get against the Federated context and see everything everywhere. I.

F

Think Nikol mentioned this earlier that we are trying to, and this gentleman asked what about the plans of integrating all the api's, so we actually had plan on integrating all the API is in the read mode while we were working on v1, but that did not sort of come across as the really needed per se feature because you can have a user access to rest of the clusters and a simple script can give you this view axis in future. Right now. We don't necessarily have a plan to integrate all the api's.

F

What we are sort of doing this. We are trying to build a reference model of what mentioned what Nicol mentioned as building blocks for multi klutzy use cases, so we are taking it in steps and we are doing it in the layered mode.

F

So, as I mentioned in the answer to the earlier question that in Federation v2, we are trying to come up with a sink type of mechanism for all ap ice, where you can, even if it does not exist in the reference model in open source, it should be really easy to just integrate one more API type or a new AP API type, which comes up in Co finatus to integrate to this v2 project. So in that case, what you are asking would be possible, but we are not there yet.

K

D

So are there any plans to integrate? You know some kind of role based access control to the list of clusters, or you know because, as I see from your example, I published the cluster end point and then certificate. So basically, all the old information need to access the cluster. So whoever access API can access all the clusters listed there right so, but would like to probably split it. So some person can only see clusters or some view credentials and so yeah.

A

That's a great question, and so I did allude to that you're having the discussion that whether clusters should be named, spaced or not, and one of the guiding factor is to add I back rules like we want to support a use case where, like I, said that you have a list in the company has like three clusters: a B and C, but you want a developer only to be able to access a and B and, like some other developer, only gets access to B and C.

A

So we do want to support our backs and the way to do it is then we need to add namespaces so right now, cluster object is not namespace code, but yes to support our back. We will need to make it named swisco. Then we do want to support these access control, use cases.

A

Yes, that's exactly so. That is why we are having this discussion that it does get confusing if, like a cluster, is in a namespace foo like what and what does that mean like? Is that foo exists in all clusters and, like is that some special food like if you have a sink controller like? Would it sink and all so like that becomes a special namespace which is like not a federated namespace only in this cluster list, cluster registry.

A

So, like those are some of the questions we are working through like, but we do want to make it our back supported as well, which requires names like that's how the I back modeling covenant is works. If you want access control, it has to be namespace code so like, but we had namespace. Then it adds confusion. So just.

B

To share our experience so I'm, actually, so we currently solving this problem. We introduced another object, which is just space: it's like specific type of namespace, which is supposed to be used only by cluster objects, yeah.

A

Okay, that's another excellent way were discussing like kubernetes. Has this Cube system namespace, which is like a privileged system namespace? So we could have like a registry system namespace? Then you can create cluster list on that namespace. So if you don't want I back support and you don't care about namespaces like you just create them in this, but if you want to go.

L

A

Yes, exactly today, like that's the only concept that the governor's API machinery has names, the only machine, only thing we have available to do this segmentation and to be able to create our back rules so like we are tied to that. It does add some confusion, but that's the only thing you have right now. A great solution would be to introduce another concept that allows us to do this without namespaces yeah yeah. We could define our own custom resource to do that yeah.

A

But to answer like that use case is important. Yes, like that's a very common use case, we have.

M

Right to that, I want to mention that the credentials to access the cluster are not stored in cluster registry. So if the user can see the cluster that doesn't mean that they have access your classroom, necessarily that needs to handle outside class registry yeah.

A

So what we're doing is that cluster history would give you a pointer to a credential store and like if you know that it's a GK cluster, then you can run g-cloud container, get credentials so like it would point you to the right direction to get credentials. It might not have all the credentials itself exercise history, one store all while the information it will make it possible for you to do it.

I

K

Hi, what is the road map of Brynn G we're? Bringing the arrow back of the network policy to the Federation, we're gonna, the state of object, gonna be production already you.

A

Mean adding them to cluster registry or to Federation to Federation Federation.

F

Yep, so what you mentioned is actually one of the difficult to solve problems, so we are not considering adding our back to Federation as a type like other types, which would mean that we have our back support in Federation for clusters right, we sort of are not considering that. As of now, we are first considering the admin level use cases.

F

That means the Federation would actually have admin access to all the clusters and it can control most of the workloads, but in future we would probably be thinking about this hard to solve problems as a next step.

A

Last five minutes: anyone has any questions.

A

Yeah, so you can also catch any of us, there's Greg and fine or me. Are there a lot of other people here who work on signal to cluster feel free to catch any of us any time in the hallways or you can join our call and you're going to have a deep live session tomorrow and Greg and I are also doing a talk on multi cluster angles. So if any of you are interested in that- and Lindsay is here as well for Federation v2l chill do a demo tomorrow, a deep dive, yeah.

A

Yeah, so just to repeat the question, the question is: that was the number in mind for number of clusters. We want to support for multi class two operations, so we've seen like very wide scale. People have like three five clusters as well, and people have like hundreds of classes as well so, and it also depends on what to the scalability depends on what tool you are using.

A

If you're, using like list religiously, it's a community say placer with just less the list that can stay scale, a lot more and depends, and if you're, using some other controllers over it so like what those controllers do can limit your scalability as well. So we don't have a set like any scalable details that this is what we support and like this is what we want to support. We also want to grow with the community like I'd know.

A

If you see a lot of people are like early in the journey, they started with one cluster and now they're moving to like multiple clusters there were few going on. They would have a lot more, so we would scale as the community grows as well. Yeah yeah.

F

And in code there is no limit actually put as of now, so it depends on your use case. You might as well have hundred clusters or thousand clusters. In that case,.

A

Yeah, yes, yep.

N

Yeah I was just going to interject there, so when we, when we conceived this, is I presume in the context of Federation, sorry I'm double-booked with next-door, so I'm trying to multitask we we sort of had in our mind this idea of in the order of a hundred clusters, would be a sort of common use case at the top end and I think the architecture and the design is well suited to those kinds of numbers of clusters.

N

We have come across cases more recently where there are pretty sound arguments, for you know two orders of magnitude more. If someone may have been at certain the CERN the keynote this morning, they have a thousand clusters. We have retailers who have all banks that have you know tens of thousands of outlets and they wanted to federate these things treat Airport as a cluster, treat each retail Walmart as a cluster, and they have more than 100 of those I. Couldn't I can promise you.

N

So you know I think over time we will figure out how to do that. There they're different approaches. You can actually federate Federation's, so you can have hierarchical. Federation's of clusters, these things are all possible and or we can, you know, react attack parts of the internals but I think the api's are sound for much larger Federation's than 1-hundred I. Don't think, there's anything about the api's that that limit us to a hundred as opposed to 10,000 clusters. I.

F

Think we didn't produce him I. He doesn't need an introduction introduction. He is our sigelei Quinton.

A

Okay, going once going twice gone. Okay thanks a lot. Everyone.