Cloud Native Computing Foundation ArgoCon 2022, 21 Sep 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Creating A Scalable Foundation for Kubernetes Mult... Lukonde Mwila & Kelvin Piroddi

Description

Lightning Talk: Creating A Scalable Foundation for Kubernetes Multi-Tenancy - Lukonde Mwila, SUSE & Kelvin Piroddi, Yieldstreet

One of the biggest challenges when operating Kubernetes at scale is managing unrelated workloads and different teams that share a cluster. In some cases, you might have one big cluster, and in other cases, you might have multiple clusters housing different teams and workloads. Kelvin Piroddi and Lukonde Mwila worked as DevOps leads on a project that used Argo CD to manage multiple EKS clusters, as well as the dedicated Argo cluster itself. As the project grew, more people needed access to the clusters, such as additional operators, solution architects, QA testers, and different development teams that were responsible for microservices. In this talk, Kelvin and Lukonde will elaborate on how a GitOps strategy with Argo CD optimized their continuous delivery, how Argo CD's projects and RBAC enabled secure access control for different teams, and how Argo's SSO feature complemented the organization AuthN model with Azure AD. Furthermore, they'll discuss how Argo CD solves multi-tenancy challenges for GitOps, but can be further complemented by a cluster lifecycle management tool that also supports features like projects and RBAC for other cluster operations outside the scope of GitOps.

A

All right, hello, everyone we're going to be talking about creating a scalable foundation for kubernetes multi-tenancy. My name is luconde wheeler or you can call me Luke and I'm. A senior developer advocate for kubernetes at AWS.

B

My name is Colin perrotti I'm, a senior develop engineer at Hill Street great.

A

So we're going to skip the formality of an agenda, because this is a lightning talk, but also where is the fun and knowing what's coming, and as for that 404 error. That's my dad joke for the day. So I won't take offense. If you don't laugh, but a smirk wouldn't hurt, looks like I got one thanks.

A

All right, I want to start off by talking about why we care about this topic, and so, for starters, we've got first-hand experience in seeing the challenges around kubernetes multi-tenancy and we're aware that this may look different depending on the use case or the context, which means that some some of the solutions or strategies around how you solve the challenges around multi-tenancy may look different, but we're of the mindset that there's some fundamental principles or Universal Concepts that can be applied regardless of the use case, and that will allow you to essentially build a scalable solution around your multi-tenancy challenges and that's what we're going to be elaborating on.

A

The second reason is that we find that multi-tenancy is a very common model. Whether companies are using a single host cluster for several unrelated workloads or in some situations, they'll go with a very strict or isolation approach having multiple kubernetes clusters, but whatever the case, companies and teams are trying to find out the right way to apply security, segregation of their workloads and essentially just structure things in such a way that they can seamlessly build and easily onboard additional workloads or cluster personas. As time goes on.

A

So this image over here, just kind of depicts the idea of trying to find the best way to slice and dice your kubernetes cluster.

A

But let's talk a bit more about some of the common best practices, and these are fairly well known. Things like namespaces to have a general way of organization of organizing your workloads. In addition to that, you got Network policies which gives you a stricter approach to managing the network traffic, specifically Ingress and egress traffic for your workloads.

A

Then you've got Resource Management to try and prevent any resource hogging for your workloads and, of course, there's role-based Access Control to manage the permissions not only for workloads, but also for the different cluster personas and we're both of the mindset that each of these best practices essentially points to the main fundamentals that this whole talk is kind of aiming to drive home, which is organization and isolation.

A

These are the Core Concepts that we believe can help you scale um when it comes to having a optimal multi-tenancy strategy, and something that we were actually discussing in preparation for this talk was the fact that we both believe that your multi-tenancy strategy is something that should actually start on a whiteboard, and you can then have all the different parties of Interest the different projects or teams that we'll be working on. That will essentially have their workloads running on your kubernetes cluster and based on what you strategize in that session.

A

You then translate that to your kubernetes cluster.

A

We believe this is the right way to essentially create a blueprint for the future, and this is something that we got to apply to some degree on a project that we worked on years ago uh together and we were essentially both functioning as devops leads in a banking context on a particular project that was actually going to be the blueprint for several kubernetes projects, for that particular bank, and so this is something that we had to give a lot of thought to, and so there were multiple eks clusters and one cluster was dedicated to running Argo CD, which would essentially manage the continuous delivery to different workloads and when it came to multi-tenancy, Argo CD helped us in a big way, because the concept of projects and applications essentially allowed us to carefully organize the different environments that we had for our different applications and, of course, grouping that into a project, and this allowed us to be able to then easily scale for the future workloads that were going to be onboarded to those respective clusters that we were running.

A

In addition to that, as time went on, it wasn't just a case of onboarding additional microservices. There were also more cluster personas different people with different functions, whether it was solution, Architects or or QA testers, and we needed to find the best way of giving them access to the cluster, and so Argo CD's Concepts around projects applications and our back really complemented that.

A

But, as most of you probably know, if you've followed any kind of multi-tenancy strategy, multi-tenancy doesn't just go as far as your git op strategy. There's also the life cycle management of your kubernetes clusters, and so we want to also talk about how you can complement what Argo already has to offer and going back to what I mentioned earlier about these two main Concepts or two main approaches. Rather that companies typically take with multi-tenancy whether it's a single host cluster or several separate kubernetes clusters.

A

We want to touch on two solutions that could potentially help in this case. One of them is Rancher, as you can see over there, and the reason we're mentioning Rancher is because Rancher also has this concept of projects, which is also meant to be a solution around multi-tenancy. It's a logical in the concept in the context of Rancher, it's logical grouping of your namespaces and you can also apply our back to those projects.

A

So you have the same idea of projects both in Argo and in Rancher, and it shows you how they complement each other and kind of point back to organization and isolation.

A

But we are also aware of the fact that in other scenarios, companies would rather opt to have a single host cluster, and so for that you can make use of something like V clusters, which will still give you the concept of separate kubernetes clusters. But in a virtual context, then you can run all of those on a single host cluster and still make use of Argo CD to essentially deploy to these different Downstream clusters. And that's what Calvin is now going to demonstrate sure.

B

Thanks Luke so yeah when you think of multi-tenancy, there's kind of like two main ideas out there, so multi tendency through namespaces or multi-clusters, the namespace pro multi tendency has the problem of if your developers are using crds and they're building against crds, those are cluster-wide resources, so you can't really use namespaces to divide. Then you get the multi-cluster scenario, which is quite expensive, quite big. To maintain and to bring up a new cluster is quite difficult.

B

So, with the with v cluster and August CD, we can use August CD to bootstrap virtual clusters and the developer gets the experience of he has his own cluster. Obviously you can connect to that cluster and run. So this is the repo that has a little demo of what we can go through, but essentially, as the admin we have multiple applications which are essentially clusters. So we have, if I, just quickly scroll down to projects, we have a project or classes.

B

These are actual virtual V classes, that deployed via August CD and once they deployed they are essentially kubernetes Cutters on the main host cluster and they are configured if you have to go to Argo CD. They are configured as clusters at the class, a DOT V classic namespace. So essentially now August CD can deploy to these clusters, and now we can use this multi-tense Foundation to scale as many classes we want, and then each team has a specific rack rules that allow them only to deploy to their specific clusters.

B

So in this example yeah we have, this is Project a they are back. They can only see app a that's deployed to their cluster a service. If we go to projects they can only see their project project a they go to the Clusters. They can only see their cluster a so they are in the mindset of they have their own virtual clusters so for full isolation and the same would be for Project B. Then I have access to their app in cluster B.

B

The destination is cluster B projects like this, and in order to do this, yes, there's a quick. This is a public repo that um deployed this entire solution. So if you could want to clone it, you can go for it.

B

um A quick thing with the V cluster is every time and we're using git Ops to bring up these virtual clusters and every time we bring up a new cluster. We obviously need to deploy initial manifest onto this cluster to give all the CD access to this cluster.

B

So every time we create a use, a new cluster, we create a new service account and we just aband that service account to August to a service account called Argo CD, and obviously you can use that token to access that cluster and then start deploying it to it as it was any other cluster.

B

In this example, I've used the make file to demonstrate the steps of how to go through each creating the projects, the pro clusters, but in a in a non-production environment, you could use workflows to keep bootstrapping your clusters and scale infinitely and for in this example, yeah to get when we deploy classes. We need to get that token of the um from the v-cluster to tell Olga.

B

He has the token to connect to, and we just have a v-class to command to connect to it, but you can use Argo workflows to do the same logic, and in this way we can achieve a multi-tense environment using August CD, and the nice thing is if we have app sets using the cluster generator.

B

If we have a base, a foundation of applications, we just have an app set there and every time a new cluster gets deployed onto August CD it'll automatically deploy all the necessary applications to this new cluster and that's easy way to just scale using App sets and August CD um yeah. That's it thanks.