Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022 [End User Sessions], 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: SIG-Multicluster Intro and Deep Dive - Jeremy Olmsted-Thompson & Laura Lorenz, Paul Morie

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

SIG-Multicluster Intro and Deep Dive - Jeremy Olmsted-Thompson & Laura Lorenz, Google; Paul Morie, Apple

SIG-Multicluster is focused on solving common challenges related to the management of many Kubernetes clusters, across multiple cloud providers (so-called hybrid cloud), and applications deployed across many clusters. In this session, we'll give attendees an overview of the current status of the multi-cluster problem space in Kubernetes and of the SIG. We’ll discuss current thinking around best practices for multi-cluster deployments and what it means to be part of a ClusterSet. Then we’ll highlight current SIG projects, focused use cases, and ideas for what’s next. Most importantly, we’ll provide information on how you can get involved either as a contributor or as a user who wants to provide feedback about the SIG's current efforts and future direction. Bring your questions, problems, and ideas - help us expand the multi-cluster Kubernetes landscape.

A

Hi everyone welcome to sig multi-cluster, intro and deep dive. uh The three of us here today to give this presentation uh for me: I'm laura lorenz, I'm from google.

B

I'm paul, and my name appears second in the list.

C

And I'm jeremy olmsted thompson, I'm also from google.

B

All right, hello, let's, let's talk about what we're going to talk about so as laura has just introduced. This is an intro and deep dive to sig multi-cluster, so we're going to cover what is the sig about we're going to talk about what is the current state of play? What is the current activity within the sig we're going to touch on cluster set and name space?

B

Saviness we're going to talk about cluster id and cluster set membership, we're going to talk about multi-cluster services, api in multi-cluster, dns, and more than that, as you will see in the coming moments, we're going to do a deepish dive and we're going to have some demos we're going to talk about the about api, also known as cluster id cluster property. We're gonna we're gonna, see uh deepish, diving demo on multi-cluster dns.

C

And we're gonna.

B

Talk about how to contribute to this thing.

C

Okay, so what are we about? Multi-Cluster is still a new space, we're seeing more and more multi-cluster deployments in the community, um and it's clearly where the community is going, but we still don't have those best practices or tools that we have in single cluster. If you want to know how to do something in a single cluster today, you know there's tons of documentation, tons of best practices out there.

C

We just don't have that yet for multi-cluster we're trying to work on as say, multi-cluster we're trying to work on building the tools and primitives and workflows that you'll need to. You know safely deploy your your applications across multiple clusters.

C

We want to work with you to help figure out how to connect services between clusters, how to replicate workloads across clusters and how to manage the rollout of the deployments across your multicluster group. We made some great progress in areas that we're going to talk about here shortly, but we really need your input. Real world use cases help us pick what to focus on next.

C

Many of our projects are still fairly early stage and room to steer a new direction. We're looking to figure out what the next set of projects should be, and we want you to come. You know join us. uh Tell us what you're working on um tell us what problems you have. So we can help you sell them.

A

So uh really important to the sig multi-cluster uh group is our approach, and this has been learned over uh the years of just trying to make sure that we're focusing on problems that we can really solve right now. The use cases that people have today and set ourselves up well for the future. So just a couple points about our approach as a sig is that it's important to us to avoid premature standardization or to solve lots of optional problems.

A

Multi-Cluster is so new, but also there's so many new variables that come into play when you're dealing with multi-cluster deployments.

A

It's really important for us to kind of focus in on what problems people have today and we really try to focus on specific functionality that we want to build, uh some of which we'll be demoing for you later and then, if we can work backwards from these specific problems into something bigger, then that's the direction that we want to take. That gives us something that's really applicable up front, but it can give us some insight and information that allows us to uh work backwards into the larger abstractions.

A

So let's talk a little bit about the current activity and the projects that we are working on today, so the first conceptual um basis and foundation for a lot of the multi-cluster work that we're doing is this term uh called cluster set and cluster set is a word to represent a pattern of use uh that we see all over the place for people with multi-cluster deployments.

A

It's some group of clusters that are governed by a single authority, and importantly, they have a really high degree of trust within that set, and this is something that we can leverage when we're describing different, multi-cluster, tooling or apis that we want to come up with. We can kind of focus on this cluster set basis this foundation as this group of clusters that have high degree of trust and are governed by single authority.

A

So one kind of important side effect of the cluster set or or uh item about cluster sets. That's really relevant to all the work we've been doing.

A

Sig multi-cluster is the idea of namespace sameness, so once we get into multi-cluster deployments, we need to say something about if the same namespace is in two different clusters, you know how what are the properties about both of them now that these clusters are working together and the um link in here is for the position statement from say multi-cluster about what namespace sameness is, but basically the short version is that uh for a given namespace in different clusters, they're considered to be the same, their permissions and characteristics should be consistent, um and even though the namespace is in a cluster set, not every namespace has to exist in every cluster in the cluster set.

A

If they do, they should behave the same across the cluster set, and this is what gives us the ability to operate on a multi-cluster deployment in some similar ways as a single cluster deployment and a new update from for this. um This round of q con is uh about the clusters. Cluster set membership uh being stored in this about case studio, cluster property called clusterset.case.io.

A

So we're going to talk about it on the next slide and deep dive later. But this is where this actual property about a cluster, its cluster set membership will be stored.

B

So when we think about sets of clusters, one of the things that we need to establish that concept effectively is to talk about the coordinates of clusters within that set.

B

The the name that we have given to that concept is cluster id, um and if you want the details, you can look at cap 2149 or you can look at the sigs.kates.io about api link, which will uh send you to the git repo, where the about api is coming together. So about. If you haven't guessed about, is the name of the api group? There's one resource within this api group called cluster property.

B

It's a cluster scoped crd and there are two special cluster properties that are meant to allow discoverability within the cluster of information about the cluster set and cluster um identity of the cluster within the cluster set. So the first one is id.case.io which allows a cluster to identify itself.

B

The next one is clusterset.kates.io, which allows a cluster to identify the cluster set that the cluster is part of the cluster id uniquely identifies clusters within a cluster set for the lifetime of membership within that cluster set, and it provides a reference for multi-cluster tooling to build on within a cluster set.

B

So, for example, it allows uh to disambiguate back ends for headless services between clusters. It's a coordinate to use for scheduling work and it's a possible annotation for metrics and logs associated with that cluster.

C

Now I'm going to talk about the multi-cluster services, api or mcs, which some of you may have heard us talk about before the mcs api was the kind of the the first reset into this new approach that laura mentioned before, where we just focus on specific problems, and here you know, services seem like a logical place to start. You have services deployed in multiple clusters. You want to connect them together um this. This is the basis for a a multi-cluster deployment and it's a specific problem that had real use cases for us to focus on.

C

So the mcs api builds on the concept of namespace statements that laura mentioned and lets. You consume a service just by knowing its name across multiple clusters, and we actually use the service name and the namespace to link services together. So you can even deploy a service with the same name in the same name, space in multiple clusters and consume them as a single service with with just distributed backends.

C

um The api focuses only on the api and common behavior. We don't have a de facto implementation, and this is this is by design. We wanted to allow room for various implementations. You know different platforms, different environments have different needs, some have flat networks. Some have many networks that need to be stitched together.

C

We wanted to leave room for implementers to you know, do what works best in their area, but provide a common api that you can use to describe, how you want your services to be exposed and how you want to consume them.

C

This leaves room for you, know, centralized or decentralized control planes, depending on what you need, but the idea is that a consumer of the service only ever has to rely on local data, so they get a familiar interface and they don't need to understand the topology of all the clusters.

C

The result is that we have the extension of cluster ip and headless services that just basically work as you'd expect across clusters now, with a new name with a cluster set name instead of a cluster local name and laura's, going to dig into dns in a little bit here we have some other initiatives going on as well.

C

We are continuing to work with mcs to figure out how to extend the multi-cluster services api to work with network policy in a in a multi-cluster environment, with policy applied across clusters, we're working on figuring out how to address the multi-network scenario where you are stitching together, clusters on different networks and whether or not that even needs to be something that gets represented in the api, or it can be left up for implementations again we're trying to just focus on the core problems that need solving and not be too descriptive about how um how exactly the problems need to be solved.

C

So we don't shut any doors unnecessarily. um We're also looking into what it would take to build multi-cluster controllers controllers with replicas in multiple clusters, and I think the big thing that's come out here is: we need some form of distributed leader election.

C

This is something that that is very new as an idea, but we're looking for help building it so come join us come help us figure out what to do. Here. We have the work api, which is focused on spreading groups of resources to different clusters.

C

So you can roll out your deployments and, of course we have cube fed cubefed has been around for a while. It has some some users uh who have been interested in a long time, but we've seen it kind of uh go a little bit into maintenance mode and we're considering archival, as we have some new alternatives emerging so come join us help. Us figure out what those alternatives should look like.

A

All right, so I'm gonna take us through uh two deep dives and demos of some of the work that's been going on and uh we're gonna talk about the about api and show you what that looks like and multi-cluster dns and give you a demo of that as well.

A

So, first off with the about api.

A

So as mentioned before the about api um is the source for uh the entire design, is in kep2149, and it's now available at six case, dot io slash about api, which is what I'm going to demo for you. Today and again we mentioned it's a cluster scoped cluster property, crd of just name and value used for the purposes of uniquely identifying clusters in their membership in a cluster set and more concretely over here. I have some examples of what those about api resources may look like so again.

A

They're in this about case, I o um api group they're, a cluster property kind, and they just have names and values, right names and values, and also down here names and values, and um the two special names that we've been talking about is id.kates.io, which these two examples up here are referencing and clusterset.case.io and id.caseio is to represent the name of the cluster, whereas clusterset.kcio is to represent the uh cluster set membership of that cluster.

A

So the exact um specification of these values are actually um quite flexible, so you can see, there's actually two examples up here of what it might look like for an id.k.I.o resources value.

A

We recommend the cube system, namespaces uuid for uniqueness purposes, which are described in some more detail in the cap, but as long as the uh values meet the properties laid out in this cap regarding uniqueness for the length of the membership in the cluster set, um then it can technically just be any value that that meets those properties, even though this one is a likely candidate.

A

So um what I want to also mention here is that this cluster property kind, this just name value where value is very resource specific and only those two id cases that I cluster and clusterset.case.io have any specific properties defined in the kep. um It's important to point out that actually um in the cap, other uses are allowed for this um api.

A

So you could store any arbitrary properties that you want in this cluster property crd um and as long as the names follow the kep guidelines, which is basically not to conflict with any of the well-known properties of which, again, we have those two id and cluster set, um but they must use a suffix and cannot use the reserved case studio or kubernetes.io suffixes.

A

But this means that you could use this to store any arbitrary properties that you want to about your cluster in sort of the centralized crd and make it a little bit easier for you to access them using the about api.

A

um Some ideas that were mentioned over the course of the cap were, for example, if you wanted some sort of fingerprint for some specific company or implementation, the important part is just to name it whatever you want dot your suffix and then you can put whatever you want in the value field or under a discussion right now regarding multi-network um there's an idea of potentially storing network information, so this is just a potential way that that might look in the crd, so any sort of uh structuring use in here is is possible and we're definitely interested to hear what um what people might want to leverage uh cluster property for um and happy to um brainstorm with folks about that, both for single cluster purposes and, of course, for multi-cluster purposes, all right.

A

So I want to give a quick demo of the api right now um and major shout out to ishmael who did a lot of the work on uh this. um This implementation- I am just the messenger, so let me change my screen all right so uh to demonstrate this. I am using a kind cluster uh here that doesn't have any crvs in solar at all and right now, I'm in a local copy of the about api's.

A

I've cloned the about api from case.io here and I'm going to run this make file to install the crd into this cluster. So this crv is built with cube builder.

A

So for folks who have used q builder before this will probably look very uh familiar uh and all the boilerplate that's inside and generated because that's inside this, um uh this github repo, but if you haven't used it before once you get into this cluster property um directory, you have this makefile that you can uh run make install as long as you have controller, gen and customize in the right places. Then the makefile will apply the crd, so we can see now that the crd is here.

A

Cluster properties have about that case.io and again it's just that name and value right, so I actually have the um I actually have the source here from uh the um cr. The custom resource definition um in this uh repo here, uh with uh everything um being just these two, uh these two values right are these two fields.

A

I should say right: okay, so um what I want to do next is to actually switch over to another terminal where I'm in a samples directory, because in here I have this um uh example- cluster property, which I also open up here, which is again from the slides like very simple just this name and value, and in this case I'm using a value that looks something like the cube system, namespace uid.

A

So if I go ahead and apply that in here then now we have this uh cluster property. In the about case that I o um api group, that's called id case. So the whole thing here is that now I can get uh my cluster property called id dot. Kate's gotta spell it right! There, though id.ks.io and I can give myself a I'll look at.

C

A

At it all here- and this is how I can directly access this value right through this cluster property um api now, so I'm doing here from cube cuddle, but you could of course do this from any kubernetes api client, and this is how you can just access, whatever id resource that you or another implementation has put in the crd.

A

So that's the idea behind the about api, simple and sweet. This is a demo with again id.casio that needs to follow some certain properties. But again you can put any cluster property you want in here and use it in your controllers, as as you like, all right. So next I want to talk a little bit about multi-cluster dns, um so I have some slides here that show two of several uh parts of the specification.

A

uh So we have the cluster set ip case that jeremy mentioned earlier for mcs api and just switching to the next slide, really quick, the multi-cluster headless case. So I've just pulled up the a and quad a record definitions, but the specification also um has information about surf records.

A

But the idea here is that for a cluster set ip service, one that you have exported with the mcs api so that, for example over here in cluster, a if you have some pods that we're calling.

C

A

And over here in cluster b, we have some more cod pods that we are calling blue. um We want to be able to treat them like they're the same service from cluster b. We should be able to get backhands blue back ends from cluster a from cluster a we should be able to get blue backhands from cluster b, etc and from a dns perspective, how this works um is for the cluster set.

A

Ip is that we have a dns name that looks very similar to the dns names that you're used to for single cluster blue.test.svc, but you'll see. The change here is that it ends in clusterset.local instead of cluster.local.

A

So the idea here is that this is very similar to what you are already used to um when single cluster deployments, and you can just sort of seamlessly drop in this clusterset.localzone at the end and the mcs api and the mcs controller will route this dns name to the vip that will go to any of the associated back ends um in any cluster. So same idea for yellow.

A

If we have yellow back ends over here and yellow vacuums over here, one dns to rule them all right, we can get to the clustering back backgrounds or the cluster b back ends with just yellow.test.sbc.clusterset.local.

A

And on the multicluster headless side um for the aggregate, dns names same type of story, you can get all of the information from cluster a and cluster b. How headless a and quad a records work. Is you get back all of the individual ips, so this works the same uh for multi-cluster headless. If you uh query blue.test.sbc.clusterset.local for backends that are all headless services, then you'll get back all of the ips, the pod ip for cluster a1.

A

Here then pod ip for cluster b, blue, one blue, two and blue three right, you'll get all four of them um so that you can uh do uh what you want with them um same thing, for yellow dot, test at scc.clusterset.logo and then the example down here is showcasing um how we handle pod dns so for individual uh to access the ip for just an individual pod um in normal single cluster headless services, um the host name uh plus uh the rest of the uh normal aggregate dns, the service um name.

A

Namespace sbc.cluster.local, is how you can disambiguate between the different back ends by dns in a single cluster, but in the case of two clusters or more clusters right multi-cluster. We need one more piece of information to disambiguate.

A

We need to know which cluster uh pod we want to go in, go what clusters pod we want, so um you can kind of see the problem here of cluster a if it has blue one cluster b has blue one, then just saying blue1.blue.test.sbc.clusterset.local is not specific enough, so we need to throw that cluster uh name in there too.

A

So this uh single ip gets uh pulled back from blue1.clustera.blue.test.sbc.cluster.local, whereas this pod ip over here in cluster b, even though it has the same hostname uh we get just its ip from blue1.clusterb.blue.test.svc.clusterset.local.

A

Right so um this really clearly showcases the use of that cluster id uh for the ability for a cluster to know uh what its own name is, so that we can actually um put this cluster name into the dns records for the purposes of multi-cluster headless services.

A

All right now, I would like to do a quick demo of the multi-cluster dns plug-in for core dns and major shout out to jeremier who did all the work on this one all right so over. Here I have a uh version.

A

I have a same kind cluster and I have a version of core dns deployed that has the multi-cluster dns plug-in installed, and I have it configured you can see here um to forward uh the clusterset.local request to the multi-cluster plug-in so with this uh configured and overall, the core gns installed on my cluster here, uh having this plug-in in here now, any request to a dns name that ends in clusters.local is going to be sent along to the multi-cluster dns plugin and um what it will do is then base the ip that it's going to respond with uh on the service imports that are present in this cluster.

A

So I also have a service import um that I have dropped in here and actually, I think up here yep. I have an example of what that service import looks like so my service in the demo name space.

A

It's a type cluster set ip, and it has this. You know fake ip attached to it, but the combination of our multi-cluster dns plugin and being able to see the service import. That has this name and namespace, and knowing that this is the vip for that service import. This is what we're going to get the response back for a dns name of the form myservice.demo.sbc.clusterset.local.

A

So we can go ahead and actually try that out, because I also have already deployed this little dns utils pod in my demo namespace and it has ns lookup installed, and so I'm going to request that very dns name myservice.demo.svc.clusterset.local kind of crossing over here and we'll see that the response we get back is that address. That's in the service import, one, two, three four! So um this is again a property of the multi-cluster dns plug-in that's in the core dns that is deployed on this cluster.

A

That's uh using that plugin to resolve any dns names that end in clusterset.local. So that is what that looks like.

B

All right, let's talk about how you can get involved in sigmulti clustering. um This is the part of the intro and the deep dive where I tell you that one of the most important things that you can do it has high value. It should be fairly low. Effort is to share your use cases, your problems, your ideas, note that use cases and problems super valuable. If you don't have any ideas- and you just have a nice big shopping, cart, full of use cases and problems- we'd still love to hear about it.

B

um So I want to just make a personal request to you. If you're, seeing this and you've thought about, you thought about uh the this functional area like that that we're talking about we'd love to know your use cases, your problems, your ideas check out our home page, it's on the community site, sig multi-cluster, give us a ping in slack. The channel name is, I hope, you're sitting down for this sig multi-cluster, and you can hit us up on the list which is called kubernetes sig multi-cluster we'd love to have you join the meetings.

B

They're bi-week bi-weekly on tuesday um they're at 12, 30, eastern 9, 30 pacific in 1630 utc.

B

If you have something that you'd like to present, feel free to put it on to the agenda and just come and talk to us about it. I'd love to see you there hope to see you soon.

C

Thanks for joining us today,.

A

Thanks everybody.

C

Hope you enjoyed.

A

B

A great kubecon experience get.

A

B