Istio Community, 29 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 2020-10-29 Istio Community Meeting

Description

On October 29, 2020 the Istio community hosted a meetup featuring a demo by Denis Jannot, Director, Field Engineering at Solo.io entitled, “Multi-cluster Kubernetes and Service Mesh Patterns."

A

Yeah yeah, that's done it's in progress.

B

Awesome thank you and welcome everybody to the istio community community meetup. I'm glad to see many of you here today uh and in order to get started, uh we are going to have a demo by dennis jannot dennis the florist sears. Would you like to introduce yourself.

A

Yeah sure, thank you very much, so let me share my screen at the same time, so yeah.

C

A

C

A

Denisendo, I'm director of field engineering at emea, in emea, for solo.I, o and uh I'm going to you know, speak about this multi-cluster pattern with uh istio in particular, and I'm going to really like uh focus a lot on demos, so I'm I'm using some slide at the beginning to introduce the concepts. But uh I will really like uh you know, do a lot of different small demos there so that it's more interactive and feel free to interrupt me. If you have any questions, if anything is not clear, I'll, be happy to to clarify that.

A

So let me just like go and present our mods, and now I will jump directly. So basically, what I'm going to do is I'm going to do a lot of demo that we have in a workshop and um we are going to run this workshop again in the near future. So if you are interested, uh you'll you'll be able to join, but the idea, I will jump directly in this part here, where um I'm going to speak about the different. Your way you can deploy istio on you know multi-cluster.

A

So obviously, one way is what we call the the shared cluster or short control plane. Sorry, where you have just one control plane and you have like uh two cubans clusters and you have pods running on on the two clusters. This uh model. We don't see it used quite often because, first of all, you need a flat network and also, if you have an outage on this cluster, then you lose your control plane and the other services will not work uh correctly anymore, and so the most popular ways for multi-cluster is really using.

A

This uh replicated control plane model where you have like different cubans clusters and you deploy um istio control plane on both clusters.

A

It can be two three four or more clusters and uh then uh the advantage is obviously you don't need a flat network uh if you have an outage on one cluster, I just impact this specific cluster, but um on the other side, it's a lot more complex to implement and I will cover I will go through. You know what are the common challenges there and and the way we can, we can handle them.

A

So the most common challenges- and there are a few more- are like about federate. uh You know federated the trust and identity um and I'll go through that uh deeper in in the next slides, and also how you allow the communication between the clusters, uh how you can also like manage access control globally, um how you can manage failover this kind of things. So that's the topic I will cover and I'll try to do.

A

You know demonstrate um some of these capabilities in in the next slides and what I'm going to use is a project that uh we have launched at uh solo.I o. That's called service mesh hub that, as for the goal of service, mesh hub, is really to to make it easier to manage um istio across multiple clusters.

A

We started with istio and we are extending the capability to other technologies like app mesh and osm, but definitely like the the big focus here is uh is istio.

A

So I will not go deep here into how service mesh up works because, as I said, I'll go through the demo, and it should be quite clear in a few minutes but, as I said, I'm going to really um go through the different challenges and I'm going to start with this affiliated identity.

A

So here you are probably all very familiar with the way um identity works in istio. But I'm going to go through that very quickly again.

A

So you have like a ca certificate that is uh used by istio so that it can sign the certificate for the different services that are running in the cluster and uh that's uh allow each service to have its own identity based on the service account that the pod is is um for.

A

uh For I mean each body is using and uh the the beauty of istio is that you can quite easily have encryption between the services using uh mutual tls and uh istio is like uh handling the complexity rotating the certificate, which is something that is really not straightforward to do. If you don't have sto and when two services need to communicate together, they come with their own identity, like I described based on the service account, but it's quite important to understand how it works.

A

So each each pod has like its identity in the form of a spiffy id. So this spiffy is like a standard that mean secure production, identity framework for everyone that is used in uh in istio and uh a spf id is in this kind of format like spf trust, domain and workload identifier and in in istio. A spiffy id is like uh spiffy the trust domain that you use when you deploy your istio cluster ns, the namespace of uh the service account used by the pods sa and the service account name.

A

So that's like the give you a unique spf id and it's important to understand that, because, when you deploy sto by default, the trust domain is clustered.local.

A

So when it comes to multi-cluster sto, then that can become very challenging, because if you have multiple clusters- and let's say you have like pods on these different clusters- that use the same service account name in the same name space, then there is no way to differentiate these two pods. So when you want to tackle this multi-cluster is your problem or the you want to to deploy stu and multi-cluster in the right way.

A

The first thing you need to do is really to have like uh different trust domain on the different deployments, and that means that then, like this pod running on one cluster, has a identity that will be unique across all the clusters as soon as, obviously, you have like a different service account for the different pods.

A

The next challenge of the dust is when these pods want to communicate with this spot here, and you want to have like a mutual tls.

A

Now they have their unique identity, but they are not able to validate that identity because, as I said before, the certificate used by this pod has been signed by the control plane here. While this, the certificate used by this pod has been signed by another control plane and each control plane has its own ca certificate.

A

So what we need is that we need a common uh identity, so we we need like a common root cert and then the that's one of the first thing that uh you get with service mesh hub is that you, you have this concept of a virtual mesh and when you create this virtual mesh, what service mesh hub will do is that it will create like a new intermediate certificate on each cluster.

A

uh It will sign this intermediate certificate with a common root cert, and then this intermediate certificate on each cluster will be replacing the the css that is used by istio in each cluster. So at the end, you will have a certificate chain that looks like um the certificate of the the pods, the intermediate certificate, which is unique on each cluster and a common root cert, and because we have a common root cert now it will allow one service on one cluster to to be able to validate the identity of a pod in another cluster.

A

So that's what you get like uh when you, when you you create this virtual mesh. uh I already created the virtual mesh in the in the environment. We will use for the demo, so I'm not going to focus there and I'm going to go back to access control a little bit later as well. So I, the first thing I'm going to do here- is really to do a quick demo about how I can now have this uh cross cluster communication, and for that I I deployed like two clusters.

A

Basically, in fact, I have three clusters, because I have one cluster where I have still running here on the left, which is called cluster one another one here called cluster two, where I also have this tio and another one. That's called management where I have service mesh up, but this could run also on one of these clusters.

A

I just put that in a separate cluster so that you see it's not mandatory and, as you can see, I I deployed the book info demo application on both sides on both clusters. The only difference is that the review service has only v1 and v2 in this cluster, while it has v1, v2 and v3 on this cluster. So, as you probably know you I'm sure, you're already familiar with the booking for application, so yeah.

A

If I send a request here, I will have just like no stars or black stars and if I send the request here, it will be like no stars, black stars and red stars.

A

So if I, if I jump on uh this demo environment here and I'm trying I'm trying to reach here this product page on the left, so it's just this one here.

A

If I refresh this page, you see I just get like no stars or black stars, because obviously I sent the request locally and I didn't configure any cross cluster communication. Now. If I want to upload this cross, cluster communication service mesh hub makes that easy, with this abstraction called traffic policy. So in the traffic policy.

A

What I will say that when I um send when there is a request sent to the review service on cluster one, uh I want a traffic shift to happen where I send 75 percent of the request to the v3 service on cluster 2 15 of those of the request in cluster 1 on v1 and 10 cluster 1 and v2. So you see it's very simple way to define this cross cluster communication.

A

So if I just copy that and paste this in my terminal.

A

Now, if I go back here and I refresh most of the time, I should get red stars- you see. First, I get red stars here, red star black stars. I can get other stuff, but most of the time I will get the red stars, because obviously the rule is like. I get 75 of the time these red stars.

A

So what's interesting is that it was very easy to set up, but behind the scene, service mesh herb has created a lot of different studio objects that are needed for this communication to happen. So you have obviously a virtual service that looks very similar to the traffic policy. I def, I defined you where you see the different weights, but here you see the destination is like um there is. No notion of this is the cluster, because uh istio is not aware.

A

You know of this other cluster, but you have this host name that finished by cluster2.global and dot global is a special. You know, suffix that you use. When you want to do cross cluster communications, then you need to have like a destination rule that define the different subsets that I use here.

A

Then you need to have a service entry that tells esteo how to reach this service in the other site. And obviously here the address is the address of this. Your ingress gateway on the second side, then, on the second site, it creates an android filter that basically will track all the communication coming with um clustered blue.global and will replace this suffix by cluster.local. So that's the communication seems like a local communication, so everything then can happen normally with the local destination rule and so on.

A

So again, the idea is really just that with a very simple abstraction I can make this cross cluster communication happening and uh and then uh with the service mesh ui.

A

We can, you know, see the policies that we create like this one- and you see here, like um you know, the the percentage of you know the workload that is described in the policy and we can see also like uh the workloads that are impacted and that's an example for a traffic policy, but you can also create like, for example, here to I, I upload the review.

A

Pods to talk to the rating pods and you see, I defined different traffic targets, and I can show you that in fact here, so I already created that before the demo. But you see you create, you have the traffic policy like I described before, but we have also another abstraction, which is called access policy, which um is a kind of cluster aware authorization policy if you want, but um here instead of just you know, using traditional service account sector and stuff, like that, you can also use uh set up some different clusters.

A

So here basically I say I want the service account used by the booking for reviews pods on cluster one and the one on cluster two to be able to reach uh any rating service on any cluster, and that has been translated by the local authorization policies.

A

And then, if I go back to the ui here, it displays as well based on the current workloads that we have on these clusters.

A

These are like uh the current uh source spots and the target pods that they can reach based on on this policy, and you can always also like go to you know this debug mode and see uh you know the corresponding.

A

uh You know authorization, uh policies and service entries and destination awards and so on, but that just makes it easier with this higher abstraction that are cluster aware. So this example was really about showing you how you can do like cross-cluster communication and another use case, for that is if I want to manage failover. So let's say I want if I am not able to reach the local review service, I now want to reach the review service on the other side.

A

So again, I'm going to jump on the demo and and configure that so um the first thing I'm going to do that, I'm going to remove the the traffic policy I just created here.

A

And then I'm going to create a few uh objects, so one is a traffic policy that will uh define uh outlier detection. So I will be able to tell that I want basically to have this outlier detection. If I have like more than one er in 10 seconds, then I want to eject um the the target for like uh two minutes, and so I'm going to apply that here and basically it will be translated when I do that, it's translated uh in uh two different uh in one in two different sorry stu objects.

A

So one is like a destination rule here on cluster one and a destination rule and cluster two for the review service, and you see the outlier detection that is similar to what I just like: defining the traffic policy and then the second step is to create a failover service so that we define you know what are the services that are behind this new host name, that we define so it's kind of like defining a highly available host name.

A

So if I want to reach the review service- and I want to make sure that this I will be able to reach the review service, even if one of the backing service is not available- I I will do that by creating this uh failover service.

A

So let me just try to you, know, copy and paste that and again, when I do that. Basically, what service mesh up does is that it will create um an envoy filter so that it will replace the uh the the dns name is specified here by uh the local under remote cluster, and it will also create a service entry for this new host name so that it's uh known by the mesh.

A

So I could just stop there and I could say: okay now, if I want to access this, then I would go to the product page service and I will modify my micro service here. I will modify my application and instead of sending my request to reviews, I would send the request to reviews failover and basically the application will be aware that it's now targeting like an highly available service, but I can even make it transparent so that uh here I don't. I don't want to modify my product page service. I wanted.

A

I want this id available service to to be in place without changing anything in my app and that's something I can do easily.

A

I just need to configure another traffic policy where, basically, what I say is that, when a request um targets the review service, I send it to the reviews failover so that it's it becomes transparent, and so I'm going to just create that and again, if you think about what it does in terms of istio, it's just like create a virtual service, and the virtual service is just like request going to. Reviews are now sent to the highly available service so to test it out.

A

What I'm I can do is I can just like make this review service on the local cluster. I can make them unavailable by just like adding a sleep command, so if I do that here they are not now unavailable, but if I try to access my app here that should still work, you see it still works. I still get like just the time to make it happen.

A

I can still see I can still access my review service and I have a way to prove it, in fact, is that if I go to, uh if I look at the logs in cluster 2, I should see requests coming so that that's uh just prove that the requests are already going through this second cluster.

A

So here I am on the second cluster and if I continue to refresh here, I I should start to see here. You know I see more requests coming here.

A

So again, it was really easy to set up the failover, and um I am also able like it was the case before I can go here and I can see uh what policies I have in place and I can see the the failover that I just configured here and I can see the backing services for the different failover and a few other things that I didn't describe before. But that are quite nice here when I created the mesh I enforced airbag globally.

A

So it was like an option on the virtual mesh creation that I can show you here. So I created the virtual image before it was just there a little bit higher. I think yeah. So here you see the first time when I created the mesh in the in the in the demo. uh I didn't set the access control here and I just did that to validate.

A

You know the spf id and all these things, and then, when we do the access control part, we just modify the virtual mesh so that we want our back to be enabled globally and when we do that. uh What uh service meshup does is that it creates uh some rules like that, some authorization policy with empty spec. That means that nothing is allowed.

A

No communications are allowed, and currently my application is working, because what I did is that I already created all my access policies here to allow, like the ingress gateway, to talk to the product page then to allow the product page to talk to the details and review services, and then so you see, you know, constructing all these different things that are needed and then, following the reviews to speak with the rating service.

A

But again it was just the reviews on on this site, so that was a load, but still this one was not allowed for the global communication.

A

That's why what I said before is when I did this um traffic uh or multi-cluster traffic here I had to create as well this access policy to specify that I also want the review on cluster 2 to be able to communicate with the rating service.

A

So yeah I mean it was just like an overview of what you can do with uh with service mesh urban, how it can really simplify multi-cluster communications, and I just kept some time for answering questions. So if you have any question, I am happy to to respond. This question now.

B

Thank you so much tennis.

A

B

And uh would you be able to pass me back the host right, so I can continue to.

A

B

You any questions for dennis um matthew. Are you in the call? Would you like a mute? Is that possible, for you.

D

uh Yeah, I was on mute.

D

Great yeah, I had a question: um I'm a novice at all um about istio, um so I was looking into multi-cluster stuff and I was honoring and there's kyali, which offers like visualization of the traffic that's flowing um and also with traffic policies that splits percentages of certain traffic to certain destination. um Is there something like that as well in service mesh hub, because there is a lot of information, apparently in there about which servers that are backing, which rules and what's on the thing, if that is available as well.

A

Yeah, that's a that's, a very good question. So that's the next uh next item in the list of what we are tackling. So we are adding some new stuff in in service mesh hub. um One thing that we're adding is like a global airbag so that you can define you know who can do what who can create these access policies?

A

Who can create these traffic policies and stuff like that? And the other thing we are working on is uh a global. um You know global telemetry or you know global metrics like you just described, um so that's something we we are working on because currently uh you can have kelly on your first cluster, carry on the second cluster and you can use that to to visualize.

A

You know the communication between services on the same cluster, but you cannot do that for multi-cluster, so we are working on having like a global telemetry and that will be part of of service measure.

E

A

F

E

Sorry, sorry, here's only roman from I'm with box and we're actually in an early evaluation and proof of concept stage with istio.

E

um I have actually two questions, but I want to focus on the first one first and then probably hold off on the second to give other people a chance to answer so um istio is very opinionated about how it actually manages identity and certificates, and you actually now, of course, obviously you kind of put that into mesh hub.

E

How can we integrate with mesh hub when we have our own pki infrastructure because we're not using? We don't use citadel? We don't want to. I can't, because it's not technical limitations, more a policy and a governance issue that we're highly regulated industry. So we to give you some some high-level introduction how we manage it.

E

So we actually run the sidecar in every part that retrieves the certificates and then it puts it into kubernetes secrets and then those secrets get mounted and as a file system into the part and then our services access that through this with istio, it's very hard to manage that it's possible. But it's it's pretty pretty much. Hacked so we don't, let istio generate the certificates. How can we do that with uh mesh hub.

A

So so, basically, though, you know the the way it works is that you know like if you, if you look at uh without mesh hub, you would get something like that. You know you would have like, uh and I described that here. Basically, you you would have like a cert uh chain. I think I have an example here.

A

You would have a search chain. I don't have the example before, but uh basically uh you would have like the certificate that um correspond to the pods and then you will have the ca cert that is used by istio on the cluster to sign the certificate, and you can change that see a cert if you want to.

A

But when you create the virtual mesh, then we add another layer with this root certificate and the root certificate is used to sign the intermediate certificate that is different on each cluster and then the intermediate certificate is used to sign the pod certificates and if you create the virtual mesh like I did, then the root cert is generated, but you can also provide your own road certs that comes from your pki. Basically, and then that means that all the certificates will will be um trusted by your your environment.

A

I don't know if it exactly answered the question.

E

Yeah, it answers the question so that works nicely if you're only running in kubernetes, but we also have a ton of legacy services that run outside and we need to do mtls with them as well and that's fine. So that's fine.

A

E

It works, but it's kind of a hack, so yeah I mean your. You answered my question with something that I already figured out myself, but I thought that would be a more elegant easier way, but that's okay.

A

I mean it's a it's, it's it's a act, but it's not really becoming your act. I mean like the support is becoming better and better right and, uh and it works, I mean it's, not uh it's not straightforward.

A

But when do you you put the process in place, it works right and in one age you have a lot of new stuff coming as well like on the dns side, and so on that that will make your life even easier.

E

And we're currently working on integrating with serb manager, which then would make our life much easier because we manage our certificates involved anyway. So all right, I think, I'll yield any time. I heard the second question, but I'll give it up to it.

A

But uh yeah, if you want to continue the discussion, I would be happy to to continue that you can reach me on linkedin or even on the our slack. You know in solo slack channel, you have like a service mesh hub.

A

C

Have a quick question: if there is still time this is the century from zendesk and since you touched on the trust domain, this is an issue that I'm currently investigating. So I have a sandbox cluster here. It was running it's running 173 with the default res domain and as soon as I replace that by whatever name, I want everything all the mtls cluster breaks. If I add the trust, alias cluster.local, then I restore the functionality, but even after I restart stod, I restart all the sidecars. If I leave only the neutrals domain, it breaks everything.

C

So is there any trick or do I need to yeah the trust.

A

C

A

Is a trick that we show in our workshop basically uh the way you deploy that you have to. uh Where is that here? That's the trick.

A

So that's basically you you tell it to you know, to skip.

C

A

C

I'm not seeing.

A

Your screen, oh yeah, you're, right yeah. Sorry, um let me just put that in the chat. I appreciate it. Man or you can just as I say you can also reach to me, but I put in the chat but yeah there is a. There is a trick for that.

A

We took time to figure that out as well, but that's the way you do it. So that's the the way you deploy steel, you just deploy it that way, so that um it's keep that and basically you don't need to validate the trust domain because you have this common identity.

A

So in any case you can really trust the identity of the other components, because you're able to you know, validate the certificates and stuff like.

C

I see I appreciate that yeah I'll try it out.

A

Sure and just reach to me, if you have an issue sure, thanks and uh and just like a quick advertising, we are doing another workshop uh on the service mesh hub like the 23rd of november. um I think it will be like european friendly time, but um if some of you are in europe, that will be easy. Otherwise we will do a other workshop in u.s friendly time as well soon. So I hope it was useful for you guys.

B

And do you have information for about these this? uh What did you say? It was a webinar or a workshop.

A

It's a workshop uh so- and uh I can just like uh put that in the in the chat as well. Yeah I'll put the link so that if anyone is interested, we still have some seats available. I think so and, as I said like it's, it's the next one, but we will have others coming as well. So.

B

And uh any other questions for tennis.

D

uh I had perhaps another one um I got here actually because on github one issue mentioned that in istio 1.8 there was going to be some changes about how they recommend multi-cluster setup.

D

um I was wondering if that is going to break something or if um service mesh hub will already work with the new release, which is, I think, plan to come out mid november.

A

Yeah, I know we are- we are already starting to to to work on that. We always like kind of validate with the the beta before and uh and obviously we are tracking that I I I didn't look by myself. I don't think like there are really uh huge breaking changes. I think the the big change is really on the on the dns side, so obviously we we it will. Basically it will make our life easier more than anything else, even on the service mesh website.

A

But uh one of the advantage as well of service mesh herb is that it will take care of the complexity of like going from one seven to one eight and that's something we will handle on our side for the multi-cluster things um instead of you having to do it in one way in one seven and then in another way, in one eighth for example. So that's also one of the idea of service meshup is to to make it easier for people who want to adopt multi-cluster to then also maintain it in the long long.

A

A

Good. Thank you very much thanks for inviting us uh in this talk, and yes, I'm going to jump now, because it starts to be late here and I have like a dinner coming soon. It was a pleasure.

B

Thank you. Thank you for presenting a demo, um so I we have one more item on the agenda for today's meetup um and it's about a sessions for kyoko north america.

B

I was wondering if anyone has submitted a session, uh if you want to discuss that, unfortunately, unfortunately, we don't have the feature to break people into rooms, but that's ideally what we would be doing at this point, um but I thought that that that topic might be interesting uh to discuss in this group. Has anybody submitted a session about istio.

B

For cubicle north america.

G

um I have for servicemashcon, which is the co-located event.

B

Nice awesome: what is it about lin? Do you want to.

G

B

G

Here yeah, I can dig through a link uh it's about seo simplified beyond single cluster, so it's basically give update on the community work. We've been doing multi-clusters in one eight, and also the simplification of virtual machines.

B

B

B

uh If you can share a link, we can make sure to promote it on the istio twitter handle.

G

uh Yes, uh do you want me to share the link on the chat here? Yes,.

H

Okay, so I've added a link to the chat to the whole schedule for service mesh con. I think we have three talks uh that are going to feature istio so uh and they look like most of them happen around 3 p.m, to 5 p.m.

G

Yeah, thank you mitch. I think you have one along with linkedin and some other service mesh too right.

H

Yeah, we have a really fun talk: that's a collaboration between linker d, console and istio that focuses on how our architectural choices impact our users.

G

Yeah, by the way your link is for the europe, I think the one I send was for north america oops. Thank you. That's okay. It took me a while to find a link too.

B

uh Michy, if you can share the link to your talk, that'll be awesome too yeah. Thank you.

G

Yeah, it's it's that general schedule. You just find out what uh talk uh so there's actually a lot of talk about istio um like, for instance, there is a confident canary deployment. uh So that's uh israel talk uh like sure. I'm uh actually not sure if sharon is speaking now that he left detroit, so there there's a talk about multi-cluster deployment, uh practical guide from tetra. I would be thinking. That's probably it's deal and also um there's a talk about department of defense use istio to do end-to-end encryption from zach and platform, one jeff.

G

So that's the istio talk also.

G

And christian, I think he is going to talk about service mesh a step further with weber sampling. It still might be highlight there too um yeah. So it's probably 50 percent of the talk will be israel related yeah. I think I heard.

B

That at the at the steering committee um as well, I was just curious to see people here in the in in this community. Meetup had some big talks um uh yeah so.

B

That's that's what I was going for, but a mitch when you have the link to your talk. uh I would love to to add it to you to our schedule for social media as well.

B

B

Great, so um I wanted to share with you all that I was. I am creating a topic on a sql discuss a forum discuss istio um to request demos, and I just wanted to extend this invitation to all of you here now, um if you, if you have a an idea, for example, something that you would like to present.

B

um Thank you mitch, um please uh reach out to us uh and we would love to have you present at the next meet up. um I am basing some of the guidelines on what demos we are looking for. Okay, let me give you one second, I need to put this here, okay and, uh as always, I I would love to have your support in spreading the word about this.

B

So if you see, even if you don't have something yourself that you would like to present on, but you'll see that somebody's working on something that is really cool, encourage them to to submit a demo idea on this on this thread that I'm going to share in a minute here.

B

um These demos are going to like this recording what we do afterwards. We upload this to the istio youtube channel and we trim the demo we're gonna make a playlist of demos, so it's always a good resource to have to show how easy works, um and this is the place where you can point people or add your own demos as well.

B

I just share that on the on the chat box, so if you're able to to share that, that would be really cool if you uh within your community or if you're on twitter feel free to share this more broadly so that more people hear about this.

B

um Are there any other topics that you would like to discuss in the community meetup? We shared the agenda in advance, but there were no new topics added. So I'm just curious. If anybody has anything that you would like to talk with, the group now would be the.

B

B

Okay cool, so it sounds like.

B

We are good for today. Thank you, everybody for showing up. I wanted to give you a heads up that the next meetup is going to be aligned with uh cubicle north america.

B

So it's going to take place a week earlier than usual, as you may have noticed, we moved the community meetups to take place on a once a month instead of twice a month, uh so I hope to see many of you there, the next community meetup.

B

We expect to attract more of the cubicle north american participants, so there may be new people around, uh and I hope that you all can still show up and make everybody else feel comfortable as well until welcome to the easter community and I think without any further ado, I will give you a few more minutes back to your day. Thank you for coming.