Cloud Native Computing Foundation GitOpsCon NA 2022, 2 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Secure Multicluster Istio Configuration Using GitOps... - Christian Hernandez & Nicholas Schuetz

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Secure Multicluster Istio Configuration Using GitOps Patterns - Christian Hernandez, Red Hat & Nicholas Schuetz, Solo.io

Deploying, securing, and managing a service mesh multicluster environment can be a complex process. Furthermore, enforcing your organization’s security policies and procedures can be a challenge when you start scaling your environment when things like drift and cluster sprawl come into play. In this presentation we will go over how to manage a production Istio deployment in a secure way with policy enforcement using Argo CD and enable a Developer self service by integrating Argo Rollouts for progressive delivery.

A

All right, all right, so um yeah, so quick introductions right so um Christian, my name Christian Hernandez um again, so I am um a product manager at Red, Hat I'm. Also a maintainer of open get Ops. So um there we go. If you see that QR code of getting involved like. If you want to ask someone directly, you can ask me um um you know any questions about that as well: I'm also a contributor to the Argo Largo project and Nick thanks here.

B

Hi I'm Nick schutz I'm a field engineer at solo.io. My specialty is in application. Networking um previous to this I was at red hat for quite a few years, practicing kubernetes and openshift um uh helping Enterprises all over the place now I do that with application. Networking.

A

Sweet sweet and uh yeah so take it away talking about a service mesh.

B

So istio istio Is An Open Source service mesh. It's also a cncf incubating project. As of recently.

B

So the common service mesh functions are things like enforced mtls application, resilience, observability, uh granular policy, Access Control.

B

Those are just some things that you can allow via service mesh with people's applications without having them change their application at all easily.

B

So one of the challenges behind uh Service istio as a service mesh is traditionally the deployment um and it's fine if you want to do a SEO deployment on a single cluster, perhaps in multiple namespaces and use their low level configurations to do so, but once you start getting into multi-cluster, you start to get some challenges. uh Things like uh communication between those clusters uh root trust between those clusters and when that starts to happen, um you know what's missing and lacking.

B

Is you know that mtls between the Clusters, like I, was saying uh an API that actually understands multi-cluster comprehensively uh and something that people can consume at a higher level? A more comprehensive level.

A

All right there, you.

B

Go and that's where the blue mesh, open source and Enterprise product comes in. It allows for that abstraction at the multi-cluster layer uh uh or for a multi-cluster deployments to a point where it makes it very easy to consume and also everything is done via declarative, configs via crds, and that makes it perfect for things like get Ops.

A

Yeah and um like I said since the all these configurations to manage your service mesh are all just like. You know: crds yaml, that sort of thing right. It's all kubernetes Native. It makes like get Ops and istio uh like I like to call it like peanut butter and chocolate right goes better together right. So, as you can tell I'm a big candy guy it, um it just goes better together right, and so, if you are looking at your um desired state right, I'm, gonna kind of use this term loosely.

A

So if any security guys here are here, don't don't get too mad at me, I'll get into that, but you can look at your desired State as almost like your policy right and when I when I say policy I, don't necessarily mean security. Obviously that's going to be a big part of it.

A

Part of the larger picture is, um is security, but what I'm talking about policy is like your organizational policies right like we're thinking about, for instance, I'm, just going to get really really simple, like like a corporate standard, build of your of your um like Linux VMS right or your um or whatever, servers like that. You're setting up those are actually policies right.

A

You, you have a policy that you want to apply to a system, so um you know bringing that back up to get Ops is that you can view that source of truth right that get that you want to apply that you don't want it to drift as as policy right and so that, in kind of almost like a um policy Governor, almost right, you're thinking more now, you're thinking, like kind of like more as you can tell kind of, like Enterprise architect background, but, like you think about more of it being like a generic um governance model of like how you manage your system right.

A

So it doesn't. Policy, doesn't necessarily mean security right, I'm, not talking about like policy with respect to like authorization and authentication, although that's a bigger part of it. I'm talking about it being more like holistically about your systems right and um service mesh is a powerful tool right and so, as Nick was was talking about it's it's. It's really.

A

um It's really there to help you scale and it's really really powerful, but it gets really really complex when you want to do it, massively at scale is trying to look at it at a single pane of glass right and so that's kind of like where get UPS comes into play, where you kind of want to like distribute that power and make sure you have that power and make sure you have ultimate control across all your systems there, and so, which is why I kind of have that graphic is like get Ops and istio.

A

You know you you, um they work, they work pretty well together right and um this is um and when you're managing a large systems right, um you know both both Nick and I have like you know. We've worked with really really large uh Enterprises to try to like get some of this stuff out and observability becomes Paramount. Very.

A

um It becomes um really especially with the distributed systems right, especially with like really really large uh Enterprises, and you have you know, thousands and thousands of applications across thousands and thousands of clusters. um You know you get you get data with that with service mesh right and you get um you. You have to use that data.

A

Now that you have that data, you know the um you know the someone once once said that what's the difference between um data information right and just information to just process data, now you have now that you have the data. Now you need to have that the information you need in order to do um things like policy enforcement right right and so I try to get a better graphic than this I. Don't know if that makes sense right.

A

You have an infinite Loop going between your configuration and your your clusters, but the idea there is that um your get Ops controller is kind of like your policy enforcer, again I'm kind of using these terms a little Loosely, because if there's any security guy there, it's like well, it's not really enforcing any policy I'll get to that in a second. But um but if you think about it, your GitHub control, what you're using flux or Argo or you know, whatever um you- you know you kind of think of it as like.

A

You know, I'll have this um policy that I want to be able to deploy massively across my cluster right like I, want to be able to, um and then what I mean policy I kind of like I've, been alluding to some of the The istio Primitives um of how to um you know, traffic controlling right, and so, um where get UPS really comes into uh comes into into play with respect to like power. Is that it's it's preventing that policy from drifting right?

A

It's one thing to, and I'm kind of um I'm kind of kind of skip bullet points right here. It's one thing to use things like kyberno and and um and opa, and you know other security aspects of it right because, like you know, when you're using stuff like kyberno, kyverno or opa, that's really more on the like traditional policy side aspect of it where it's like you set something up, and that controller is making sure that that is happening.

A

However, there's nothing controlling. If someone some bad actor, someone with fat fingers a config, someone changes the kyverno or Opa configuration on your cluster and all of a sudden, it doesn't matter if you're running kyverno right like if someone's there changing the policy to something you don't want it to be, then you know how good is that policy right, and so you know, kind of jumping back up is like preventing that that drift right you're going to be using your get Ops controller to um making sure that drift doesn't happen.

A

You want to set your policies and you want to make sure that policy is always what you want it to be right and basically also you want to automate those policy changes right.

A

So if you ever change a policy, whether you um put in a new constraint- or you know you- maybe you widen it a little bit, you want that to automatically propagate massively to to your entire cluster, and so um another aspect of get Ops is what I really like about istio and what I really think about service mesh is that it um it puts in things like Progressive rollouts into um um into it, codifies them right. So, there's a challenge with that.

A

So I find that if um things are dynamic in a service mesh world right I want you know you you, um you want to be able to do something like uh like Canary deployments, right I, always talk about Canary deployments. um Those are really really Dynamic right. You want to. You know, release a new version of your application. You want to release, maybe some like 10 of it. Okay, that's looking good, maybe release another 10 so now you're at 20. Okay, that is looking good.

A

How does that look like in the get Ops World right, um because is that a series of series of commits to a branch right- and you know or a PR? You know a bunch of you know, commits to that Branch or if you're you know, really running, get Ops a bunch of PRS and make every PR. Someone has to look at that PR and merge it right and approving you have to go through that approval process. For every time you want to increase that Canary.

A

That Canary is going to take forever um and another way to do. It is kind of ignore those differences, but then that kind of goes against the spirit of get UPS. You know you really want to have like a policy around that right, and so um so, if your source of Truth is static, how do you do like a dynamic roll out right really like embrace the power of a service mesh right and a lot of controllers?

A

um A lot of projects now have a controller for that. So, um for instance, in the in the Argo world, that's Argo, rollouts um and I know. Flux has Flagger to do some of those things as well, um but you have a another controller Progressive that is get Ops friendly right. That is basically saying hey, given these parameters right I want you to start at 10 and increase progressively to 100 percent, um and until that rollout is successful right and it works with your get Ops controller.

A

What's really cool about uh using something like our rollouts or Flagger? Is that it's also policy based with respect to metrics right, so you can plug into things like Prometheus right. So, like you know, I use this deal a lot so I use. You know Prometheus, you can tap into Prometheus and just run whatever query you you want to in order to do the rollout. So now you kind of have like a policy.

A

um What's the word I'm looking for you kind of have like a low end high-end sort of thing, I'm like okay. Well, given these parameters do the rollout for me and then then it becomes like automatic right and get us friendly, and it has things like roll back and um um Yeah. So, basically like rollback, whereas, like you, um if something goes wrong, Prometheus is catching errors like 500s or whatever you're able to do that as well.

A

So um so the idea behind using um something like istio service, mesh and um and get Ops together is like you really want to scale right. So the idea is scaling. The idea is that, okay, you know we have. um You know you know. Organizations are always talking about like you need to do less with more. We need to scale rapidly. You want to be able to handle anything um you have now.

A

You have like the tool sets to do it and any other tool sets to do it automatically right, and so we have like things like Federated mess, meshes or multi-clusters I forget I, don't know what the right term is so uh so it's called.

B

Federated federator.

A

Message, cluster right so, um um and you know you manage complexity exponentially right. So once once you add like another cluster like an SEO cluster, um the problem grows exponentially. Not just you know incrementally, um so your apps could be really anywhere right and so um get Ops. Basically just helps you manage that. Not only manage that complexity lets, you do um other cool things like a progressive delivery um in a get up friendly way right, and so uh one of the things so I have I.

A

Don't know if you've seen the back of my shirt. If you want to see it later, you can um my back of my shirt says unless it's in git, it's only a rumor another thing, I like to say so, that's kind of like one of my catchphrases well, another thing I like saying is that Autumn automation does not mean automatic either.

A

So um um you know everything basically, that phrase just means that you are ultimately in control of how that rollout happens is that, but what actually does the rollout is automatic, so that's kind of like what I what I mean um from that aspect right and then so um you saw the um the image before right. That kind of how it looks like uh here, where you know you are not only using uh the glue glue mesh to help your your Federated clusters right.

A

But now you have like, like a um like a controller there as well to help you with the the get Ops thing. um I, don't know if you have any yeah.

B

So that this isn't just Theory, we have people lots of companies in production today implementing these patterns um very effectively for hundreds of clusters at a time. So it's a proven methodology that large-scale companies use on a daily basis and it's very effective and cost efficient.

A

Sweet sweet, yeah and um and by the way these slides, I just uploaded them to sketch so I I see you taking pictures, which is fine, but just let you know we uh we uploaded that there so um I guess with that I did do we have time for yeah. We have time for um some questions here. If anyone has any questions.

B

A

Go back a slide. Yes, that is a question. So I'll allow it. Yes, oh there's a question back there.

B

So if you do want to learn more about istio, just raw istio or even a glue mesh, you can go to academy.solo.io and we have some self-paced classes or you can actually take a webinar style class where it's interactive and you can have Hands-On with technology.

B

um So please do come and join us quite frequently, and thank you guys.

B

Oh yeah got a question right here: yeah.

A

C

It all yeah. Oh sorry, so a.

B

Quick question we're also using.

C

Like Easter with Flagger and one interesting problem we had to solve I think is when you have Flagger or argosity you're allowed. You have this internal state of the rollout which managed by Flagger. It is guitars friendly, but you still have um like failure. You still need to handle failure, States and failure conditions and then can a little.

C

So it goes a little bit in a different direction as a GitHub, so your github's repository drip from the source of throw, so you have can have different workload on different versions, deploy it on your on on your like on your system and a different version defined in github's repository. So do you have any like good, cookbook solutions to solve it, or do you need to approach it yourself like we did? We had like cold bags, automatic rollouts and making sure we change the state inside the repository, the reflect it correct, yeah.

A

Yeah, so um so I don't know for so for Flagger, I'm, not I'm, not sure right so I know for rollouts, specifically that um since it Works hand in hand with with Argo CD it'll Mark the deployment as they created, um and so um meaning that, just like just your deployment, is just like just the greatest. So it's no longer like in sync, with with Git, it's no longer like it's basically marking your app as like. It's it's, not not working right and so, um which I actually talked to someone once it's like.

A

Well, that's not technically true, because it rolled it back, so it is healthy but um technically, but it does. It does show it I do think, though, that there's room for improvement, um because to recover for something like that is very, um very imperative um like right now, even like with Argo rollouts, um you have to um do some imperative work right so like to to automate that you may need like another controller, um but.

B

Yeah and also in glue mesh, there's uh validation web hook as well, that won't allow you to apply configs that are bad yeah and you can set that to strict, and you know you can also use open policies to uh you know, check your configs Etc in tandem with that.

A

Yeah, so not sure if we answered your question other than like I agree that like there needs to be a a better way to handle like either like rollbacks or like um remediation of of that.

D

Thank you um just a quick question about the diagram, I guess, and maybe it kind of follows that question about whether things are really get UPS or not yeah, but uh so the top part of the diagram shows glue, mesh being synced with Argo CD, I, guess being synced with Git those glue, mesh, crds, I, guess being synced with Git. But then there's the arrow about glue mesh, creating or translating istia resources and the individual clusters right are those istio resources?

D

Does their configuration ever show up in git, or is it only in the mind of glue mesh? If you understand what I mean um it.

B

Would be, it would be glue, mesh's abstraction of the istio resources? Yes,.

D

A

Well, and also it's very similar with Argo rollouts right, so you see, like our so part of Argo, rollouts and um again I'm I'm, ignorant on Flagger, so I'm not sure if it's the same with Flagger, but with Argo rollouts um you're, not actually even using a deployment either because you're using the art on Argo rollout specification, which has the same spec as a deployment. So it is.

A

It is like layers of abstraction because you have a rollout that controls your um of your replica set versus a deployment so uh very similar to to like to Like Glue, where it's like you're using the glue translation of you know. Glue within you know, create those resources.

B

It's all about making the user experience easy, and you know very readable to in the end. You know to reduce human error to you know, allow replication, easily Etc.

D

So so, if I go so, if I go back to an old version of the glue mesh crds, it should put back the yesterday resources I had before that. So in general, that's the get Ops idea right. Yes,.

B

D

D

Any other questions.

E

Hi, it's pretty much related to the last question. It's about the the sink or or the drift detection on the on the Easter clusters, because in this case, if I understand, we think we are going to synchronize the glue resources through argosity in this case and glue mesh is going to create a daisio classified. But if there is some drift on the istio clusters like at the global failover or the web configuration wherever that riff is going to be picked up by the glue mesh or the Argo CD sink.

A

The Argo CD will detect that the the drift right um I'm not sure, if there's the tool in glue that detects drift um but I think so.

B

In this case, it wouldn't definitely be Argo, looking after the resources that have been deployed, etc for changes. Okay,.

E

Yeah my question was because I I- really don't I, haven't used this combination, so I I come and see the how it looks on on the Argos in.

A

Dui yeah yeah, so in just to kind of like pull the the curtain back a little bit in Argo CD, um so I'll like for for istio I'm, going to use some SEO um nomenclature. You have um uh like your virtual service right, usually you'll have like your virtual service. Like pointed certain percentages, you actually point at the stable 100 at the stable release so like in git. It's um um it's, it's more of like you're, storing, like your end state in git for the virtual service. Now the Argo controller takes care of sorry.

A

The Argo rollout controller takes care of what points to stable right. The stable and I'm using stable in quotes because that's just like they call it the stable release versus the canary release. Right, it's an actual specification. It's actually called that so, um but the um yeah, so the Argo rollout itself is the one that you know looks at what your end State wants to be and then shifts the traffic um accordingly right. So it'll, it'll, Mark, individual replica sets as stable versus Canary and you that's.

A

It kind of you're kind of giving a lot more um control over to the rollout controller right versus um and in an article. You just have your end state to find like how what do I want to see. As my end state.

B

Yeah, it really did Ops is about you know, having a process behind it right having that Gateway into your configuration, so in an ideal world you would have you know, checks and balances before that drift can even happen.

F

um So it's a theoretical question: doesn't this break the principles of care option that gloomesh translates the source of Truth.

A

Yeah, so it's uh this is actually um and funnily enough. There's a conversation I had with a Christian poster um who's uh who's that solo as well. It's um it's so getting into just like the get us aspect of it. The it's points of demarcation as I always say right like because like if you technically think about it, a deployment isn't really you're um right, like isn't really what runs right. It's the pods right so is storing pod definitions, more get Ops than storing a deployment, or are you just storing um what your end State?

A

What your end State wants to be right, and so it's it's really your points of demarcation right like so, and then there's like a balance between that right. You do like if you write like um um a kubernetes controller, that is like three lines of yaml, but like has a massive deployment like that's technically get Ops, because it's like you just have a you know: you're, just selling the controller. What you want to do, or do you store those individual pod, yamls and the individual service animals? All of that can get it's um it's something.

A

We've actually discussed at the get Ops working group right, the this concept of like soft get ups and hard get UPS where and where that point of demarcation is where I come. Where my point of view is like your point of demarcation and where, where the controller lets go of that and where another controller picks it up right and where do you want those to be exactly.

F

Yeah I mean where I'm coming from is more in terms of repeatability and consistency. So today, if gloomish changes in a future release, it might change your end service.

B

But that would be the same with sdo in general. That abstraction is needed, I mean really, it comes down to are people going to use it right? Are they going to be able to consume it and, in the end, we're trying to make that easier?.

A

Cool we have about two more minutes. So if there's any more questions, Commons concerns.

A

E

A

Cool. Thank you. Everyone.