solo.io KubeCon North America 2020, 18 Nov 2020

Previous Meeting

⏯

youtube image

►

From YouTube: KubeCon US 2020: Daily Demo featuring Gloo Mesh

Description

Denis Jannot kicks off KubeCon week with a demo of multi-cluster Istio service mesh configuration with Gloo Mesh. Having fun with virtual backgrounds in the process

Learn more about Solo.io at KubeCon
https://lp.solo.io/kubecon-na-2020-virtual

Questions? https://slack.solo.io

A

All right dennis all yours, thanks folks for joining we've got dennis who's going to demo um multi-cluster istio with glue mesh, and if you are having tom, I know you mentioned you're having some issues with the audio. Please feel if you um you're welcome to unmute and folks who can you're welcome to unmute, and um you know, ask questions. Otherwise you can type them into the chat and I'll also bring them up for you.

B

Cool thanks betty, so let me go in presenter. Mods.

A

Broadcasting from the simpsons.

B

So yeah welcome everyone, so we are going to talk about uh glue, mesh and focus on multi cluster service mesh operations.

B

We'll do a live demo, so I'm going through some slides at the beginning, but then it will be mainly a mix of of slides and demo, and so I'm done director of field engineering uh at solo based in uh in emea.

B

So just before we start about uh discussing about the multi-cluster challenges, uh I will briefly show the two current options that people have when they want to use istio cross multiple clusters and they can use what we call a shared control plane, which means that you have one control plane and you have multiple cubans clusters and that works. If you have a flat network, but that's really not um commonly used. uh The most popular approach is what we call a replicated control plane and you will see in istio 1.8.

B

The names have. The names have changed a little bit, but the the idea is still the same and that you can have multiple control, plane or unique control plane and you have. You can have a flat network or not a flat network. So we'll focus on this one, which is the most popular approach, it's more flexible because they can be in different networks. They can even be in different clouds. You can have one on premise, one in a public cloud and so on.

B

So there are many challenges here, uh we'll go through them in in more details, but we'll speak about the federated trust and identity.

B

So how one service in one cluster can trust another service in another cluster so that we can have like uh end-to-end mtls how we can allow communication between the clusters- it's not very easy in in istio, and we will show you how we we can simplify that with blue mesh and um we'll speak also about access control, because as soon as you start to have like uh multiple services on multiple clusters- and you want to allow cross cluster communication.

B

Obviously you also need to think about access control in a global way and we'll go through some other topics. But uh let's, uh let's jump in so quickly glue mesh is composed of several services.

B

The open source version only has the discovery and networking services and the enterprise version that will be available in beta soon, as also two additional components for airbag and ui. But I'm going to to show you that later uh so the two main services, the one you find in the open source version, are called discovery and networking and basically the discovery. One is responsible for discovering all the workloads running on the different clusters and the networking one is responsible for making your life easier.

B

So basically you you define your traffic policies and access policies in a very simple way globally and then this service is. This process is responsible for translating that into multiple istio objects and we'll focus on histo here, but be aware that we are also working on integrating other mesh technology like app mesh and osm. So that's the same principle. It will be translated in this case in app mesh or osm objects instead of these tier ones.

B

um So I'm going to skip that for the demo, I'm going to use a lab. Basically, it's a vm where I have like three kind cluster running inside uh one, where I have a gloomesh deploy that you still see.

B

I I refreshed my presentation everywhere, but I still see that I missed one icon here, because that was the previous name we had, which was like service mesh up, and that's a good way for me to tell you about this change so that if you were already aware of service mesh herb, we now call it glue mesh.

B

But that's the same project so glue mesh can be deployed in a cluster where you already have sto, but it can also be deployed in a kind of management cluster and that's what we are going there in this environment.

B

And- uh and you see, I have two other clusters where I have uh istio running and it's a vm in google cloud, but that could be that could be anywhere.

B

And what I did is that I deployed the booking for application on both clusters. The only difference so you're probably familiar with the this application.

B

If you have already used istio it's composed of multiple micro services and the main one which is called product page, which is the the front the front end, if you want- and this one calls back-end services to be able to display information on the main page- and you see, the only difference is that the the reviews microservice on the cluster in the on the left, which we call cluster one, uh has only version v1 and v2, and uh on the second side, you have v1, v2 and v3 and you'll see the demo uh we'll uh we'll use that to show you that, from the first site, we are able to reach v3, which is on the on the second side.

B

um So, as I said, the first challenge is about like federative, trust and identity, and uh the way uh istio identity works is that um the service has a side cap proxy running with envoy, and this agent that runs there is a creating a certificate request and it asks the um sto demon, the control plane to sign that certificate.

B

And uh basically, when you deploy histo the control plane in one cluster, you use like a ca certificate to sign these different certificates and the the css that you have in the other cluster is different. So the first challenge is that now, if you want um one service in one cluster to trust a service in another cluster, you need to make sure that the certificates are issued from. You know the same root certificate.

B

So you can do all this stuff manually, but it's quite complicated and I will show you how we made that easy in in glue mesh, and the other thing that is quite important is that when you have multiple services in multiple clusters- and you need to have like a unique identity for each um for each service and the way it works in stu, it's that it's using something we call a spf id and if you look at the way the spf id is created is something like spiffy current slash, the trust domain, which is by default cluster.local in any cluster.

B

You deploy, uh slash, ns, uh slash the namespace, slash ss slash the service account used by the pod. So, first of all, you need to make sure that each um pod, or at least its service, is using a different service account, but also you, as you can see, because the child domain is placed on the global everywhere.

B

Then that means that uh if you have uh different two pods in two different clusters that are using the same service icon in the same space, then there is no way to differentiate them. So it's important to deploy your istio clusters with different choice domains so that you'll get like unique identities at the end.

B

So we will focus on on on this one and basically, as I said, you will have like two control planes and uh we have this notion of what we call a virtual mesh in blue mesh and uh you define the number of control plane you want to have in this virtual mesh. You just determine which mesh should be part of it and when you do that, then glue mesh will deploy a certain agent on these clusters and will create a root cell that can be coming from your own certificate authority.

B

If you want and then basically it will create a new c excerpt for each cluster and that will be considered as an intermediate certificate and this intermediate certificate will be used to sign the certificate of all the services. So that's at the end, one service and one cluster can trust a service on another customer. So that's all the magic that happens in blue mesh and uh and if you, if I show you um because I already created it here, um I can show you how I did it.

B

So you just like have like a very simple yammer here oops. I think it's a little bit later.

B

No, it's not there yeah it's here, so you see, I created a virtual mesh just specifying my two control planes. Here I enabled access policies and I will show you what it does after and here I just asked to create or to generate a root cell, but, as I said, it could come from your own certificate authority. So just doing this that stuff or just creating this simple object uh allows you to have the identity federated, I'm not going to discuss about access control.

B

Now, because you will see when we go through multi-close traffic, we will have at some point an issue about the permissions, because by default, when you enable access control, you are not allowed to do anything or no communications are allowed between microservices, even when they are in the same cluster, and I already created the local rules so that this service or the ingress gateway, can talk to this service.

B

This service can talk to this service and so on, but when we will go through the multi-cluster traffic, we'll see that some rules are missing and we will. We will get a nexus denied message and we will fix the access control together at that at that time.

B

So, um let's, uh let's just like, take a look at uh our application now. So you see here, the review is the version one here, because I don't have any stars.

B

If I refresh sometime, I will get black stars, and this is the v1 v2, sorry so v1 no stars v2 black stars and these three red stars, but I will not get any red stars because, as I described before, we don't have the version 3 running here.

B

So let's say now: I want when the product page try to reach the reviews micro service. I want sometimes to go locally and sometime to go on the remote cluster. If I want to do that in uh istio without blue mesh, I would need something like that. I would need to create, like a virtual service, where I describe what I want to do like, for example.

B

Here I want to send like 75 of the request to the service v3 on the remote cluster and 15, on v1 on the local cluster and 10 on v2 and the local cluster, and then I will need to have destination rules created to define the subsets here, so the different versions. Then I need a service entry for the service that resides on the other cluster and indicating the ip of this year, ingress the other cluster, so that it can reach it and and then the second cluster.

B

I would need also like an envoy filter so that I I remove the suffix ear and after that it becomes like a local request and I still need destination rule for the subset and and so on. So it's quite complex just to make that a simple communication happening. I will need to create multiple eco objects instead of doing that with blue mesh. We have this traffic policy abstraction and you see I can really simply describe here what I want to achieve.

B

So when a request is sent to the reviews service on cluster one, I want to have 75 of my request going to cluster 2 on the v3 of this service and, as I described before, like 15 locally on v1 and so on. So let me just apply that.

B

And now, if I refresh, I should quickly see here that it tries to reach the version three it even it's even able to reach version three, but version three is not able to communicate with um the rating service. So if I go back here in my diagram, I see that this communication here from the product page to the reviews service, is happening, but this one is not because I didn't create a policy and access policy for that.

B

So, as I said, we have two kind of abstraction: the traffic policy for the traffic calls and the access policies where we define which service can talk to which service and uh the good thing is that you can create everything with crds like I've. I've just shown you, but.

A

B

Then you can also um use uh the uh the ui so that you see what has been configured.

A

Will sorry hang on dennis will? Do you have a question.

B

No, I think it was just background noise, so you can just perhaps mute him thanks. um So with this blue mesh ui, you are able to see. um As I said I created my virtual mesh. You can see the different clusters. You can see the overall state, my virtual mesh is there. I have my two clusters and I can see the different traffic targets and you know workloads are running and things like that and if I go to policy, I can see the three.

B

The simple policy I created here and you see the the weights I have described in my yammer and what's nice- is that it also shows you based on the rule you created, uh what are the workloads that are impacted and what are like the traffic targets, and you see you see that here. But if I go uh in my other policies, you see, I also have access policies and I already created them to allow the local traffic and if I look at the reviews, one.

B

What I did here is that, as you can see, I upload the reviews version one on cluster one and the reviews version 2 and cluster 1 to talk to the rating on both clusters. But I didn't allow the reviews on cluster 2 to talk to them. So that's what I'm going to do now and that's why we don't see the rating for the moment.

B

So the way you create an access policy is uh quite straightforward. You um just use the the disk. You just describe the service accounts and you see here.

B

I will indicate that I allow the service accounts of the reviews service on cluster one and the one on cluster two to access any any pod with this label service, equal rating. So because I had only that before I didn't see the reviews services of the second cluster, but only the one of the first cluster in the ui.

B

So if I apply this here.

B

And then I can even check that in the ui ear you see now it has discovered so blue mesh is looking okay, based on the rule that I have set. What are the workloads that are impacted, so what it shows you now is that it also impacts these three additional workloads, though the three services on cluster two. So now these services are also allowed to talk to the rating services. So if I go here- and I refresh this page- I should see the red star.

B

So that means that now, I'm I'm going to the second side for this service, and this service is also able to communicate with the rating service.

B

So that's uh an example with um the what we call like uh multi-cluster traffic, but uh we'll go to another example, another use case and before we do that, I will just delete my traffic policy to clean up everything and um what we are going to do here is quite similar, but here what we want is that we just we don't want to spread the traffic between local and remote.

B

What we did before was more like if you want to do bursting, for example, but here we will do like a failover. So that means that okay, I want to use like the local review service. If I can, but if this service becomes unavailable, then I want to use um the the one that is available on the other side.

B

So to do that, we are going to use a traffic policy first, where we will define the outlier detection, so how we determine that a service is not available anymore. So that's also something you can do with the traffic policy. So I'm going to apply that here.

B

A

B

Oops, I don't know why it clicks okay and then I'm going to uh create what we call a failover service, so failover service. You describe that uh this review service, you create a new name like it's like a highly available dns name that um that is backed by uh different services. So here.

A

B

If I send a request to this new name, uh I will send the request to these services and if the first one is not available, then it will go to the second one, and it didn't mean that the first one is not available by this outlier detection that I described just before so here.

A

And dennis we have a question on that from tom. It said he asks if reviews did not exist in cluster one. Would traffic from product page and cluster one automatically route to cluster two, not in a failover setting.

B

uh On in the case of the failover that we described now, right.

A

uh Tom I'm going to need you to text in the message.

B

Because here I mean like, if you have this failover service and this service doesn't exist or doesn't exist anymore, then yes, it will go to the next one.

A

Yeah, so the question is around: is there cross-cluster service discovery.

B

Yeah, yeah, and, and- and here uh what you see is that I I spoke about this new name that is kind of a highly available name. So it's like if it's available locally it goes locally. If it's not, then it will go through the remote site.

B

That's one way, you can do it, so you can say now in my application, the product page, I will update the product page service so that, instead of sending requests to uh reviews, it sends the request to reviews failover and the application is now aware that it's id available, but um some cases you don't want to modify your application.

B

You you want that to be really transparent and you can do that with a traffic policy again where what you just say is that for all the requests uh targeting the reviews service, I want to send them to the reviews failover service, so it becomes transparent for the application. The the application is now having your id available review service without even being aware of it.

B

So again, I'm just going to create that.

B

And um to to check that everything works, I'm going to just like uh make these two services uh unavailable by just adding a slip here so replacing the command by a sleep and very quickly.

B

Here I should see that I'm still able to get my application, even if the local services or just that's, okay, that's done so now. I am able to still see my services and these are coming from the second cluster.

B

Even if you see it's, only the white and red is just the way uh it works with envoy that it will just create two threads and it will use two versions by default, but we can check that it's on cluster 2, because if I go here under logs and just check there.

B

And if I refresh the page here.

B

I still have access to my application like like we see, and then we see the access logs there that proves that uh it's handled by the second site. So again, um let's just like clean up that by restarting your services. We will probably not wait for that, but here I have. We have a description about all the istio objects that are created by this failover service.

B

We have a workshop where we cover all of these things. You can. I really encourage you to go to our website solo.io and you see uh in the events page. We we have like a workshop where we do all these things in in details and you have a lab and a lab environment and you do all the steps by yourself.

B

So yeah, I'm going quite fast on that, but just be aware that if I wait here for two minutes, then it will fall back automatically in the first sight, and you can also see you know the outlier detection that uh we discussed about before we created. uh You can also go in the mesh here and you can see the fadeover that we have configured here and we can see how we configured the failover and so on, so that's kind of it for the traffic and failover multi-cluster traffic and failover.

B

But I wanted to touch on another thing that we are adding in. um We have in our enterprise version, which is about um multi cluster airbag. So before I I show you how it works. I just want you to explain the the idea behind it. So if you are familiar- and you are probably familiar with kubernetes, you can use roles and all bindings to define who can do what in your cluster.

B

But it's not really granular. You can say: okay, you can create this kind of resources or you can delete this kind of resources or stuff like that. But you cannot say how you want people to create these resources. Like you cannot say, I want this person to create. I want to allow him to create a pod, but he cannot do that with this specific parameter.

B

So the problem, if you look at the you, have two problems here: the first one is the one I just described and the second one is now that we have this multi-cluster service mesh. We are able to create access policies to define which service can communicate with which service globally.

B

But what we need now is that we need to define which user can create traffic policies and access policies in our environment. We don't want anyone to be able to create holes for any service in my entire environment.

B

So the way we do, that is that we we've created something that we call a glue mesh airbag and the the the way this uh works. I'll show you that here is that we are giving you like very fine grain control about about who can do what and you see here- um that's an example of uh the namespace admin or so um what what we, uh so you, let me show you something before yeah, so by default.

B

What we did is that if I, if I go here and I take a look at my whole binding with my current user- so let me do mgmt get all binding.

B

B

Bigger, probably it will be better, so you see I have an admin all binding. So if I look at it.

B

A

B

uh I I have like this admin role and if I look at the admin role then I will see that um I can do everything. Obviously that's why it's called an admineral.

B

But you see it's very fine grain. You see. I have stars everywhere, because I am an admin, but you can define uh if if, for example, if you completely remove the access policy scope here, that means that this this hole doesn't doesn't give you any permission to create access policies at all, or you can do the same for traffic policy or failover, and things like that, so you can define that and then you can define very fine grain.

B

You know you can say okay, this person can create like a failover service and traffic policies, but I want them to be doing that only in their own namespace. So that's a very common use case, for example, instead of having like a namespace admin in your local cluster. Now you can extend that idea so that you can have like a global namespace admin. So this person has a namespace on each cluster and is able to create, like all these traffic policies and access policies in his namespaces. But that's it.

B

You cannot create any rule that will impact other namespaces. He cannot allow communication between his namespace and another namespace is really just restricted to his own namespace. So that's one example. We are going to show you now, but obviously I'm going to speak about a few other use cases after so that's my uh traffic policy that I created before you remember I deleted it.

B

I removed uh the binding. Did I remove the binding? No, I didn't so that's the reason. So sorry, let me let me delete that again. That was.

B

Yes, so the first thing I need to do here is that I need to remove this whole binding so that I don't have commission anymore.

B

And now, if I try to create this traffic policy, you see I'm not allowed, because I don't have the permissions for that, but that's a good opportunity for me to give an example with a namespace I mean, let's say I want to now have permission that just corresponds to the default namespace. Where I have my application running now, so I can go there. I can create this role and then, when the whole is created, I can create a new binding and you see I just I will just get the.

B

I will just get this uh default name space admin all, instead of being like a admin of the full cluster.

B

And now I should be able to create my traffic policy again.

B

Yeah, you see so I get like this admin role and and again I did everything with these crds, but the ui is always there to help you. You know making sure that you can understand who can do what globally across all your clusters, you can really easily see that okay, I have these rolls and this hole is only granted to this guy for the moment. So that's also you to make sure that there is no like security hole um somewhere.

B

So uh I mean I think it's uh yeah most of what I wanted to show you um just quickly. As I said, you have multiple use cases for the airbag. You could say this user can only use a specific virtual mesh if you have, let's say 10 cubes cluster and you have sto deployed everywhere, but you want to.

B

I don't know you want to pair them 2x2 that you create like five virtual meshes. Then you can create a role that will give admin all, but only for one of these virtual mesh or you can create like someone who can only create policies on one cluster or can only define access policies or only defined traffic policies um and many many other things like uh I. I gave you some examples, but but it can be really fine grain, so yeah. That's all for this talk.

B

We still have a few minutes. If you have.

A

Questions, folks, you can unmute yourself if you have any questions or type them in the chat and then we can get them answered.

A

And we'll also be doing a demo tomorrow, we'll be doing at least one demo each day in this format. So tomorrow we'll have web assembly um for envoy and then on friday, we're gonna do glue edge, so api gateway, ingress controller.

B

Yeah and that's a good point, and a little bit easier- you see this one here, so that's something we are currently integrating. So basically you will be able so with blue mesh enterprise you'll be able to very easily define okay.

B

I want to apply this wasn't filter in these specific targets and then you can also see in the ui which services have this are using this or this filter and and in the mesh ctl, which is our cli you'll, be able to build your web assembly uh filter from the cli, then you'll be able to push it to uh let's say oci, compliant um registry and and so on. So that's a lot of good stuff coming there, so I don't think we have any questions. So let me thank you for attending.

B

I hope it was useful and uh obviously you can also find us around in our booth, but you can also um you know, chat with us in the cncf uh slack. We will be there.

A

Tom tom is asking, um does glue mesh need access to my to the kubernetes api and does it work for private clusters.

B

Yeah so glue mesh is is not is not an online service. You see it's running there, it's not a software as a service. It's it's really like a software. You deploy anywhere in one of your kubernetes cluster and uh then yes, it needs to access. uh It needs to use the quest api of the other cluster, so it discover uh the different services. You know the discovery piece I have.

B

I have defined before you.

A

Could disable discovery and create.

B

Your targets by yourself, but obviously you get more value out of it. If you you, you keep it enabled, but I really want to highlight the fact that this is a software you deploy in your cluster. You know in the dmz, if you want or like there is nothing in the internet.

B

Thank you guys.

A

Well, thanks dennis um I'm going to leave this open just so that if people stop by our booth and have questions they can pop in and folks yeah thanks for joining and we'll see you later.