solo.io SoloCon 2022, 5 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitOps and Controllers- It’s Not that Simple for Multi-cluster

Description

SoloCon 2022:
GitOps and Controllers- It’s Not that Simple for Multi-cluster

Speakers:
Alex Ly
Field Engineer, Solo.io

Marino Wijay
Developer Advocacy and Relations, Solo.io

Session Abstract:
GitOps has become a valuable approach to manage configuration for applications and infrastructure. Having a source of truth that can be automated, auditable, and is easy to understand is increasingly important when expanding to many deployments. However, enabling multi-cluster capabilities typically presents new challenges: not every cluster is the same, context is important, and managing every lower- level configuration across multiple environments can get cumbersome (and dangerous) quickly.

This talk will focus on a specific example where multi-cluster GitOps is difficult: application-networking and security with service mesh. The goal is for platform teams to provide the right point of demarcation with abstractions that focus on the intent, while abstracting away the translation and orchestration of lower-level config (mesh-specific API resources in this case). We share our experiences building these abstractions with some of the largest deployments of service mesh in the world.

Track:
Service Mesh and Application Networking

A

Hello, everyone welcome to solocon and welcome to our talk on git ups and controllers, along with using it with service mesh and how it can make your life easier. I'm going to pass it along to alex to go ahead and do some introductions.

B

Hey everyone: my name is alex lee, I'm a field engineer based out of san francisco supporting the west coast here prior to joining solo. I spent some time at red hat and mesosphere focused on container orchestration technology, specifically kubernetes.

A

And my name is marino wijay. I am part of the developer relations and advocacy team here at solo, so I focus on helping the community embrace some more of the service mesh technologies that we have here, along with some of the other great things that we're working on here at solo, I used to work at vmware and I spent some time both in the networking and security space, along with the the devops space with vmware 10 zoo. I'm excited to be here to talk to you about blue mesh and how it works with git ups.

A

So, let's get right into it.

A

So one of the the most interesting things that we see when it comes to istio is the way we go about managing istio for let's say a single cluster or a single kubernetes cluster, and we may want to consider something like taking the declarative approach to simplify how we go about configuring, how microservices transact inside of that kubernetes cluster, and so when we think about it within within a cluster. It might be easy to medium in difficulty to manage artifacts or resources that reside in that cluster.

A

For istio and for kubernetes, but what we also have to consider is the potential for scale and how that would create a lot more complexity and how we go about managing. So, let's think about it. This way what happens if we were asked to go ahead and create a secondary cluster or a tertiary cluster to support our applications or our microservices?

A

Could we potentially leverage something like git and version control to help with that? Absolutely so, in this talk alex, and I will cover off the approach to declarative configuration using git, ops and how that applies to using blue mesh or service mesh technologies and how that can simplify your operations.

A

I will also cover off a brief demo, just so that you can visualize how this all works and see how the configuration is laid out across different clusters and how that interacts with both istio glue mesh, as well as our get ops controller.

A

So when we look at this cluster or in this diagram right here, we have a single kubernetes cluster running istio and this could be backed on any cloud provider. Single cluster istio is very simple. It's an easy way to get started. It's actually. You know the best way to kind of break in and understand how to use istio to leverage things like traffic management, observability and and the built-in security.

A

Now, when we consider having to manage these these resources that istio provides to us like, for example, a virtual service or a destination rule for a few micro services across a few name spaces. This is not um not that difficult, but where it becomes, you know highly challenging is when we have to scale to manage multiple clusters.

A

So, let's think of it in the lens of having to run coop, cuddle and istio cuddle across whatever cluster we're using to stand up the environment make sure it's ready for istio make sure it's ready to support multi-cluster, istio and then go on to proceed, to try and establish, let's say a shared route of trust.

A

Establishing, let's say a common root. Ca or a common root certificate authority can be challenging because you now have to consider how all of the different micro micro services within your clusters do communicate.

A

And so you then become the custodian for having to be responsible for certificate, rotation and management and that in itself becomes very troublesome and and quite painful to manage.

A

The other thing to consider is how you set up your access policies and how you go about allowing different services within your cluster to communicate with other services across other clusters, and the final piece is addressing things like role-based access control, allowing um what services can take action on whether what other services or what accounts can do what what they need to from an authorization standpoint.

A

Now, we've kind of understood the fact that, with a single cluster managing things like virtual services, um destination rules, service entries like to access external endpoints, it can be pretty straightforward for a few micro services. um But when we try to scale to multiple clusters, it becomes very hard, as I previously mentioned. So the the most challenging part is having to create a set of virtual services, destination rules and even authorization policies for three microservices across.

A

Let's say three clusters now you're gonna have to manage this nine times over, and so maybe you might consider using something like a version control system to help manage and facilitate some of the um the approach to making changes or syncing changes across those clusters.

A

But the other thing to consider is even if you had separate isolated clusters and you use the level of git to help automate it. You still have the challenge of your certificate rotation across those different clusters.

A

So when we look at this deeply, if we are assumed to have a single cluster and a few microservices, we know it's easy to manage and when we continue to iterate against that app add more microservices, it's going to grow. So that means our varying istio.

A

Resources will also grow alongside that, um when we also think about the certificate management within a cluster um that that in itself is not difficult but having to set up mtls between, let's say the reviews service in one cluster and then the rating service in another cluster means. We also have to go about establishing the common root of trust and then also establishing a pattern for certificate rotation.

A

So one of the ways we could try to help simplify. This is if we use a git ops approach and a git ops controller, which means we now feed our configurations in through git those those changes that get fed in it are recognized and synced by the git controller or the git ops controller and then in turn reflected inside of the kubernetes cluster. Now this is not doing everything for us all. It's really doing is automating some of the work instead of us having to go, run coop cuddle against every single cluster.

A

Basically, the the get ops controller does that on behalf of us for us, so it's syncing those changes honoring. Those changes which means that your istio resources that you've gone about to configure will change alongside of it. But again as this scales, um you also have to start thinking about how you set up things like access policies or authorization policies across the different clusters for different service to service communication.

A

At the same time, not every cluster is going to have the same level of needs, so you might have certain clusters within your environment that may need to access external services using service entries, and then you also then have to go and define and figure out where destination rules will correspond and where they align to across the different. Let's say virtual services that you've set up.

A

The other thing, too, is you're, now spending more time having to manage that than you are allowing your traffic to flow around within your operating environment. So one thing to consider is: maybe you you want to consider a further level of abstraction, something that can understand these resources and then additionally, translate them for you.

A

You configure that single resource it. It goes on to create the additional resources that istio would normally produce when you need to set up things for routing or when you need to set up security, but it all starts with some abstraction, and so, let's think about the glue mesh abstraction, so glue mesh is a well-designed api. That is intense specific.

A

It really considers things like you know the source and destination of objects to determine how best to align that policy um in, in the case of blue mesh and in our example here we actually have a custom resource definition or a crd defined. That, basically, is our traffic policy that gets translated into a virtual service and this and destination rule in istio, but for multiple instances of it. So what we're doing here is we're defining a route.

A

So if we think of the world of like network routing or ip routing, you know we have to define a route, and then we have to also define um how we get there or what our exit point is or or our exit interface whatever.

A

It might be in our case here we're defining that route and we're telling it where, where we're trying to go so, we may be originating from somewhere in the glue mesh name space, then anything that has a label that says type north south means that we'll will be allowed to route to anything that is containing the label app reviews.

A

So I'm just basically creating a policy that says, use these identifiers to match on this particular criteria or condition to allow you to route to this destination, based off of that criteria or set of conditions, and then, furthermore, within this policy, I'm able to define the weight or distribution by which I send my my traffic requests so perhaps based off of where I'm coming from, I maybe get I may be sent to, let's say, reviews or the review service in one cluster or cluster one I may be sent to maybe the review service in cluster two based off of other um route information.

A

So what this is really helping us with is creating a single crd that that can replace a number of istio crds. This is this is glue mesh, it's a powerful api that uh that provides a variety of functions.

A

What we're seeing here is a traffic splitting kind of function, but there is also the ability to do things like traffic shifting fault injection. We can inject some delay so that we can ensure certain micro services come up before others to prevent any sort of chicken in an egg situation.

A

We can also have we can have blue mesh program, the proxy or the side card, to do some retries in the event of failed requests.

A

We also have the ability to implement like laugh functionality, so we can block certain types of traffic and prevent any sort of harm that comes into the network or into your microservices network, and you can also do things like external authorization and tie into systems like oppa, so that you can provide some enhanced authorization mechanisms to various microservices, so it effectively creates a very simplistic manner in which you can manage your multiple kubernetes clusters.

A

All that run. You know they all run istio. They all run their own version of istio. But what's really neat is you're not managing the individual instances or resources that istio needs to look at, or the istio crds you're not having to manage that you're. Managing these higher level constructs these abstractions called traffic policies or access policies or even virtual destinations, and those are are understood by the glue mesh controller, which then will translate those into istio resources.

A

But then the other part of it, too, is that you're still also leveraging git ops here, because when you feed your changes into git, the get ops controller is going to pick up pick up on that, and then it's going to in turn reflect those changes.

A

It's going to sync those changes into the kubernetes clusters, which then are going to be picked up by the glue mesh controller in order to translate those into steel resources. So this is significant simplification of managing all those different istio resources using glue mesh crds.

A

So what we've effectively done is simplified your management of your kubernetes clusters in istio at scale, using the glue mesh approach and, and with that this, this basically allows us to segue into this opportunity to allow alex to show you how this all works.

A

One thing I will point out is: if you notice this this bar here in one of the previous diagrams, I showed you. This bar was further up top and it was basically pointing out the fact that there was a demarcation point in the in the amount of resources we manage. So if we actually went back a bit it's further up here, which means I am actually having to manage a lot more resources within my istio environments, whereas looking at this diagram over here, the demarcation point means that I only manage these particular resources.

A

The glue mesh controller or blue mesh management controller then becomes responsible for understanding these resources and managing everything below that way. I'm not here having to sit here and manage them individually. This simplifies my overall approach to management. This simplifies my operations. It streamlines how I build my policies both for traffic and access and then how I define how I define my routing so with that. I'm going to pass it over to alex to take over and show us the demo go ahead, alex.

B

Yeah thanks marino, um so let's go ahead and hop into the demo portion of this. um Just on this on the page. Here is the slide, depicting the architecture that we're trying to accomplish here, essentially leveraging git ops, alongside uh integrated with with the glue mesh management controller. In order to provide that line of demarcation.

B

um You can see clearly here that, um under that dotted line, uh we're gonna allow those those lower level resources within cluster one cluster, two and cluster three to be managed and synced through good ops, as well as using control kubernetes controllers, um such as the ddos controller, and glue mesh management controller.

B

In this case, and then you know by doing that and allowing you know software to handle the management and automation of configuration, we're going to allow our devops team and our sre team to focus on the higher level intent, which is essentially what um what source and destination that you want to configure policy on right, so traffic policy. In our case here over here in my environments, I have kind of set up a similar example: cluster one cluster, two and plus c3 using the book info example um in cluster one.

B

uh I have just reviews version one set up in cluster two. um I have reviews version two and cluster three I've reviewed version three.

B

This is just really simply to show um that that here I have um three clusters, all showing you know the same app but a little bit different right in in the sense of um no stars, black stars and red stars, and if I go to my bookstore application, you can see this is my cluster one running on gke with no stars or further reviews in cluster 2, black stars and in cluster 3.

B

I have red stars, so this is just a typical single cluster istio setup registered to glue mesh so that we can actually see you know from a single pane of glass.

B

What our cluster topology looks like our multiple cluster topology, and here you can see that each nco, ingress gateway is only serving the services within their their one cluster right.

B

So when it comes to going into multi-cluster in this scenario, what you would have to do in a upstream istio build would be to configure the access policies across the cluster. You would have to unify the root trust across all these clusters in order to make them aware, and this can be cumbersome, especially if you're kind of talking you know more than a few clusters to tens.

B

If not hundreds of clusters, like some of our customers, have in this example, what we can do is configure what we call a virtual mesh and the virtual mesh here will unify our root truss across our clusters, selected by label selectors or direct reference itself. So here's what a virtual mesh crd looks like pretty simple um and and as you can see here, it is all synced in git, so that git can actually, you know, use that as our source of proof but to dive a little bit deeper into the virtual mesh.

B

Here we can see that I want to unify the root truss between two of my meshes in this case just cluster one and cluster two, but leave cluster three alone and in in this case I want to just uh for simplicity: 6 set the global access policy on my virtual mesh as disabled.

B

For now, let me go ahead and configure that and what you'll see if I go to my um glue mesh ui, is that I'm going from three meshes initially uh three individual istio message meshes um and here you can see cluster one's running uh version. 111.3 cluster two is running version one eleven, four and cluster three is running version, one eleven five. Just to to make a point.

B

You can see by configuring that virtual mesh object that I just showed you. We have now kicked off the blue mesh in order to unify the root trust between cluster one and cluster two, and you can see that is already happening now. I have a virtual mesh that contains cluster one and cluster two. I can see, what's in my virtual mesh, what policy rules that I've created?

B

What workloads are running inside, that virtual mesh that can now be routed together, and I can also actually see through the glue mesh dashboard.

B

What versions of meshes are in my virtual mesh right, so so in this case 111 3 and 111 4.. Now, if I go back to this github repo, I can show you how I can now use a simplified crd to handle that multi-cluster routing using a route table and a um a traffic policy like merino, showed right. So in my route table here, I have defined that in in this particular um north-south traffic, any any um traffic going to the product page on cluster, one or in cluster two get split, 50, 50.

B

uh and and then from from there. As far as the east-west traffic goes within the clusters, uh I am declaring my intent right. So essentially, uh what is the source where this um connection is coming from? So in my case, the north south um ingress gateway selected by a label selector- and this is really important, because you know, as I onboard from one to few to many clusters- labels become pretty important here, so that I can handle the discovery uh without having to hard code. Any um you know specific reference or anything like that.

B

If cluster three or cluster four cluster five comes in to play and they have a a north, south ingress gateway with the label type north, south blue mesh will pick this up and start routing traffic immediately. If I want to actually get pretty specific with that, though, blue mesh is capable of it as well. So in this case um you know in the destination I can select by direct reference, but I could also tie it to um to a label match as well.

B

So in this case, this cube service matcher is matching um all the clusters for specifically cluster one or cluster 2, with the label app reviews as our destination, so there's our our source, our destination, and then the policy or the intent that we want to create and in this case, is a traffic shift from um uh from reviews anything going to the destination which is reviews.

B

I want to shift that traffic to reviews across cluster one and cluster two at 50 50 rate, now that I've already configured that um I can hop back over to my cluster one book info application. That was only showing no reviews earlier and if I refresh a couple times, you can already see that this is now serving traffic across two clusters: um cluster one and cluster two, and you can see that through the nodes, those stars and black stars that you see on the screen here. If I go into the cluster 2 ingress gateway.

B

You can see the same, however, in cluster 3, since I have not added that or selected that as part of my virtual mesh, you can see that that default. You know istio install stays preserved in that case completely outside of the virtual mesh itself and still is showing red stars.

B

If we pop back into the glue mesh ui and go to the graph, we can actually see this right. So um let me select all my clusters.

B

And here you can see that cluster 1 and cluster 2 have been connected together here with the shared root trust, and you can actually see that in the ui with these lock icons across the cluster, so you can see mtls is enabled across across multi-cluster. In this case, and uh and if uh I I so choose with the traffic policy, I can direct traffic freely across the two clusters uh right now.

B

I don't have r back set up, but I will hop to that in in a moment just to show how easy it is again using label selectors to set access policies uh to lock this down over. In my cluster three, you can see that uh the istio ingress gateway in cluster 3 is only communicating with the services in cluster 3 and that's how we saw just red stars showing up there to tie this all back to how git ops works.

B

Everything that you see in this demo so far has been. um You know, synced, with git as our source of truth using argo cd as our git ops controller.

B

So here, switching over to a you know, adding a third um that third mesh into our cluster, essentially onboarding, um onboarding, that third cluster into our virtual mesh is as simple as switching a a get branch right. So let me show you um if I go back into git and I pop over to my three clusters branch- you can see the same crds in the same space, which is just the management cluster being used, and essentially all I'm doing, is adding another reference or label in the mix right.

B

So I'm moving from a virtual mesh that only had cluster one and cluster two now to a virtual match that has cluster one two and three. When it comes to the route itself.

B

um Here I have now kind of added the cluster one cluster two and cluster 3 north-south traffic set with a weighted distribution of 30, 30 and 40 to cluster three and then in my traffic policy. um I have also now added in the service match, to match services in cluster 3 with the label app reviews, and then I have now defined my policy to shift that traffic from 5050 to cluster 1 and cluster 2 to 30, 30 and 40 across all review services uh in in my motor multi cluster setup.

B

Now that I have everything defined and get it's as simply, as you know, pushing a change through your ci that changes that branch in this case, I'm gonna do that manually. So in argo, cd, I'm gonna switch over to the three cluster branch and let argo sync that what you'll see here is that argo cd is now synced to my three cluster branch, meaning that the virtual mesh here now, if I scroll down, will show that it actually has um all three meshes um being selected.

B

If I pop back over into my blue mesh ui, you can actually see that the the process of uniting that root ca across all three clusters has already completed inside my virtual mesh.

B

I have um cluster one with istio 111 3 cluster 2, with issue 1 11 4 and cluster 3 with aco 111 5., and if I pop back into my application now and refresh, what I should see is that I should see a no stars, black stars and red stars being evenly distributed at the weights, that I selected 30, 30 and 40 over across all three clusters here. So cluster two, you can see, has all three reviews now and so does cluster three.

B

So now I have essentially onboarded you know: team three's cluster, which has the new red stars app into my virtual mesh, transparently without my end user, knowing that this service is actually split across three clusters, and I can actually view this in the ui.

B

So let's go kind of double click on what is what our app looks like um in the back end, uh you can see how all these connections get extremely. You know complex to set up uh and think about this with three clusters. uh Think about it. With, with you know, 50 to 100 right. The web of connections is going to drastically scale, meaning that um the amount of configuration that I would have to manage in an upstream istio setup would drastically steal, alongside with that now.

B

Lastly, what I wanted to show is how simple it is now that I have all the connections um across my three clusters working to set up access control policies. So I can lock this down. If I pop over back into git, I can show you how all that kind of gets done with just a uh an access policy uh or a couple access policies um that also define your intent, which is source destination and um and- and you know where I want to basically select and put my access policies.

B

So in this case here is the in book info access policy, um the source selects the cube service account for the ingress gateway and the destination matches to any um any any application within the default namespace with the label service product page.

B

So this again with label matchers allows you to better select or easily scale, because you can select based off labels, but you also, you know in a case where you need to lock down to a specific resource, can select based off of a specific reference itself.

B

You can see that all these access policies follow the same pattern. Choose your source uh choose the destination of where the policy should go.

A

This almost reads like a little bit of a firewall rule set right, alex.

B

Yeah a lot like that, um so so it makes it simple um to to select things, uh especially based off labels.

B

So why don't we go ahead and show again now once I've, I've got that set up, and I can just sync that to get what you'll see is uh access policies being created to glue management plane and from there back to this diagram here the the uh operator or sre team defines the access policy that gets submitted to gu mesh and blue mesh and instead of a human being, will then translate that access policy over to the lower level issue resources across my clusters.

B

Again, this you know becomes a lot more sense at scale when you're talking about 50 to you know or tens. You know her tens of clusters right um having to go into each cluster and create the individual authentication policies. To make this happen to make the connections um you know secure would be pretty untenable.

B

If I pop back over into the blue mesh ui, you can see that my policy has been implemented.

B

You can actually click on the policy rules themselves, so you can see you know the multiple sources of which this policy has selected and the multiple destinations which this policy acts um on right. So in this case cluster one cluster, two plus three ingress as the source and cluster one two and three product page as the destination.

B

If I pop back into my application, you can see with these particular access policies. I am I'm still. Everything is still working and, however, if you were to not have something that was under a access policy, you would be routed to an rbac access denied page.

B

So hopefully, this helps. You understand a little bit more about how to leverage git ops as well with a controller in order to simplify operations across multi-cluster and specifically multi-cluster service mesh.

A

That was great, alex awesome. Well, thank you. Everyone really do appreciate you taking the part of solo con and looking into our presentation, hope you enjoy the other ones out. There take care.

B

Thanks everyone.