Kubernetes WG Multi Tenancy, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubernetes Working Group for Multi-Tenancy Project Overview - Tasha Drew

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Kubernetes Working Group for Multi-Tenancy Project Overview - Tasha Drew, VMware, Adrian Ludwin, Google, Fei Guo, Alibaba & Jim Bugwadia, Nirmata

In this session, the leaders of the Kubernetes Working Group for Multi-Tenancy will quickly go over how you can join the multi-tenancy group, and also do a quick overview of each of the projects we are incubating: the Virtual Cluster Project, the Multi-Tenancy Benchmarks Project, and the Hierarchical Namespace Controller Project. You can visit all of our projects here too: https://github.com/kubernetes-sigs/multi-tenancy/ Got more questions? Join our mailing list via google groups https://groups.google.com/forum/#!forum/kubernetes-wg-multitenancy or talk to us in the #wg-multitenancy channel of the Kubernetes Slack.

https://sched.co/ekHM

A

Hello and welcome to the kubernetes working group for multi-tenancy today we're going to be doing a project overview in which we will cover how you can get involved with our working group, which is part of the upstream kubernetes community, and we will also be going over some of the three main projects that the working group has been working on information about how you can use them, how you can get involved and contributing to them just to quickly kick off for the working group.

A

The chairs are me, I'm tasha drew I am the director of product incubation at vmware. I also work on project I used to work on project pacific. I was responsible for launching the tonsi kubernetes grid service for vsphere.

A

My co-chair is sanji, vrampal he's a principal engineer at cisco, and then we have the project leads uh on this um on this presentation. Today we have adrian ludwin from google who works on the hierarchical namespace controller. We have fake woh from the virtual clusters uh and tenant controller project he's at alibaba, and then we have jin baguadia who works on the multi-tenancy benchmarks, and he is the founder and ceo at nermata, um and each of those uh project leads will be giving a more detailed introduction about themselves and the projects later in this presentation.

A

So you may be wondering how you can get involved in the multi-tenancy working group um so for new contributors or people who are interested in getting started with open source. What I would my number one recommendation is: join our google group. So that's the second link on this slide. You can go to groups.google.com and just search for us for working group, multi-tenancy kubernetes when you join the google group that will give you access to our documents.

A

Our meeting agendas you'll see links to go and watch our meetings that we've posted to youtube, we'll post a link to this presentation um in that same youtube channel as well, um but you will also get a calendar invitation to all of our meetings and so a lot of times. People get really confused about how to join our meetings. We have bi-weekly meetings where we go over project status.

A

You can come and present anything you're working on by adding it to the agenda or suggesting on them on the mailing list that you'd like to talk about something, but that calendar invitation to that meeting and the zoom link will all come to you if you just join our google group. So if you're not going to do anything else, but you want to join that definitely join the google group. It may take a few days for that calendar invitation to show up there's some latency in the system adrian's working on it.

A

Google will solve it any day now, uh but yeah just be patient. uh I promise you. If you join the google group, you will get that invitation, we're also in the kubernetes slack in our multi-tenancy uh channel. You can join that by going to slack.k8s and yeah. We also have a github repo where all of this code is available. You see that link at the bottom of this slide. uh So just kubernetes dash slit, sigs, slash, multi-tenancy, so yep, that's that's how you can find us. That's you can talk with us more.

A

We are part of the open source community super happy to have more people involved, more interest and new ideas and new projects as well. So uh yeah join us and now over to uh adrian to do a deep dive into hnc.

B

Thanks tasha, uh so I'm adrienne, I work at google waterloo up here in sunny and warm canada come visit next year and I'll be talking a little bit about the hierarchical namespace controller, which is our project to add a concept of hierarchy to the kubernetes namespaces that you know and love. uh So why would you want to do this?

B

Well, the reason is that namespaces are the foundations for all policies in kubernetes, so rbac network policies, quota limit ranges uh and also most of the crds that you create will have name space scope as well, unless they're, unless they're a cluster-wide operator, so everything really centers around name spaces and yeah.

B

Sometimes you can subdivide things at a subname space like within a namespace, I should say, uh but by and large most of these um features either work the best or only work at the name, space level, and you can see my talk from qcon eu from earlier this summer. That goes into this in more detail.

B

So namespaces are great, but sometimes you want policies across namespaces such as to represent a tenant who has access to multiple namespaces uh in your cluster, so adding hierarchy. Namespaces is a great way to enforce this idea of ownership and express tenancy and also allows you to create self-service what we call sub-name spaces, which give you permission to create a namespace, even if you don't have cluster level privileges all while adding as little as possible to core kubernetes so agency. It's a great solution.

B

If lots of teams want to share a single cluster uh and if you combine with a git ops solution to uh to spread policies across multiple clusters, that's when it works well in a multi-cluster environment as well. So how does this work well? What it does is it creates. A little agency gives you this little custom resource that you can use to take existing spaces and arrange them into parent-child relationships, uh and we call these full name spaces, because you can. uh You can modify the hierarchy or, as I mentioned, you can create these self-service.

B

What we call sub-name spaces, uh where, if you just have permission to create a sub-name namespace, underneath this particular existing namespace and then that one's locked in place, you can't move it around, and so that's really good for those sort of self-serve uh contexts and then once you have that you can create policies in the ancestor namespaces and they will just get copied into the descendants you can pick which ones get copied in a cluster-wide config things like our back limit ranges: network policies, uh an agency also puts some guaranteed labels onto the namespaces, that mirrors the hierarchy, and you can use those as label selectors and things like network policies to get that kind of hierarchical enforcement.

B

uh As I mentioned, I gave a talk that went into hnc in depth at qcon eu. It's on youtube, it's fantastic! If I may say so myself, so please go look at it, um but we are not sitting still and we've been working pretty hard ever since then. So the big changes since that uh talk have been well.

B

The agency has been increasing instability as we get more users more people evaluating it, trying to break it and poke it in weird ways, which is great: uh we're finding some corner cases and and getting rid of some of the watches that made it harder to use. uh We are adding exceptions and selectors. What this means is that remember how I just said: you put a um an object in an ancestor namespace. It gets copied to all the descendants.

B

Well, sometimes you don't want it to go to all the descendants, because strict hierarchies are too restrictive, so with exceptions, you can basically put a label selector on objects, and that will stop it from being propagated everywhere in the hierarchy uh or even into nowhere at all. If you don't want it for some reason, uh the last thing that we're adding is an improved api. So we did an api review after the um the api had grown in a sort of ad hoc way for about a year.

B

um It survived pretty well, but there were a couple of nice changes that we made uh things to make it more kubernetes compliant um and so we're introducing this in the next version, which is 0.6 not going to be. We don't think there will be many changes before we go to a beta and if you already have agency installed we're going to upgrade all of your objects for you automatically.

B

So if you want to get this, you do not need to get a new version of kubernetes, it's just an add-on. So it's pretty easy. You can add it to any kubernetes 15 cluster. I think at some points and we'll be switching that to 116., uh for if you want to on open source, you can go to that multi-tenancy reboot that tasha mentioned, and just click through to hnc uh or go to the releases page and grab her for more releases.

B

uh If you happen to be using gke or anthos, then uh then you can get it as a product called hierarchy, controller in config, sync or enthouse config management, and it comes with a really nice plugin that you can install using crew, which is cool and it's available for linux and mac os.

B

If you want to learn more about this, as I mentioned, I have uh the video from cubecon.

B

U uh just after that, I wrote a blog post on kubernetes.io, uh so please go check that out and uh read through and learn more about the the thinking behind hnc and the concepts around it, and if you would like to contribute, we are looking for contributors who want to work on features that we like the idea of, but we haven't quite been able to get to yet things like uh setting us per sub-tree configuration if you only want secrets to be uh propagated in one sub-tree, but not another, uh creating sub-name spaces with default policies that are not inherited hierarchical resource code.

B

Lots of people talk about that one and just generally improving productionization support. I was actually working on prometheus today, but more eyes are always welcome, plus more to help with testing more help with documentation any patches that you want to provide.

B

We are always looking for people who want to help out with that kind of stuff, and so uh please stick around, and I hope to hear your questions afterwards uh and with that. I will pass it over to faye to talk about virtual.

C

C

Just quick check: can you guys see the screen? Okay, uh so hello, everyone? My name is faye. I come from ali ababa, I'm in the past one and a half year. I was working closely with the multi-tenancy working group or project called virtual cluster.

C

As anger mentioned in previous plantation, the in kubernetes namespace is the primary uh api abstraction to support multi-tenancy. um But by sharing uh uh for for many tenants, users who share one ips server themselves bring some part problems that cannot easily be uh handled by namespace. For example, they may have performance interference between among each other and uh once when you, when we really use namespace to organize all the dependent of resources and only give tenant users to access to the namespace.

C

It is hard for them to create a cluster of resources such as crd or web hook center, which gives some trouble for management for the for their management stack.

C

I see we do see some cases that uh the namespace based isolation is good enough, but we do see some other cases. People may want very strict or complete isolation, and they won't have a solution which is compatible with the existing cognitive api. So so the so compare people may do this by just creating individual.

C

You know cluster tenant, but in that solution there is a problem that if each people has individual node cluster by managing their individual nodes, it is harder to do the global. Optimization for the node, uh no degradation so ideally we'd like to have a kind of complete control, isolation solution. Well, we can share in the node resource, among all the tenants.

C

As I said before, we reach the solution, um has almost zero integration efforts for the existing system, which means uh we would like to have a solution that uh which, which is fully compatible with the existing kubernetes apis um in this project. So we introduced a new uh kind of architect for the virtual cluster. uh In short, the idea is that we will the media have a tenant operator which we assign on a dedicated control plan for each individual tenant and the the the tenant which we call the tenant master. The teller must.

C

The goal of the tenant master is to do to maintain all the objects that are created by the 10 users, and there is a single controller which synchronizes all the you know. Tenant created objects from the tenant master to a super master, uh which is the kubernetes cluster, which manages the actual node resource.

C

um So, in order for the you know, different tenant to call the global, complete api uh to access or monitor the pod, or do the logging or excc api code to the queue the tenant pass. We introduce a component covered agent which proxy all the you know, cannon's request to the uh copyrighted apis from a high-level perspective. You look at this architecture. Ideally, the tenant master is the object. State maintainer and the supermaster is, can be abstracted as a resource provider owing it only provided a part resource to each tenant users.

C

With this solution, it is clear that, in with this solution, uh each tenant user has, you know complete view isolation. It won't be aware of all the objects that are created by other tenants, since they don't have direct access to the supermass at add-on.

C

With this solution, we also limited the velocity radius, if any tenants hit any security vulnerability, only that tenants get affected and the other tenants won't be get affected. Also, with this approach, since you know, each tenant uh user has a dedicated control plan, it will. It have executed all the uh resources that presented in the tenant master, including all the crd scope, resource, rv, control, etcetera. um Since you know, supermaster only you know treat only works as a part of resource provider. It doesn't involves, in the you know, control plan management step.

C

um So this is what we do in the virtual cluster, uh the the what what what the virtual class really does uh from the high-level architecture perspective um next I'll give a brief status about how this project goes. um At this moment, the the the among all the three major components, uh canada operator sync operator and a v agent, the single controller and the va agent- are pretty much as of now. They are kind of production ready.

C

We are actually working on switching the tenant operator to a new class, the api based design and the implementation.

C

The idea is, we we are trying to have a more provide, a more formal way to do the tenant, tenant master provisioning management by leveraging the all existing cluster api offering that class api c groups has been uh has been done in the past two and a half years. They have very good tooling um around it and they have that we want to follow their. You know specification about how to define a tenant master component.

C

um This project uh was starting as a working multitasking working group, incubator project, and now we we already get a secret uh c class api, uh the support and we are moving. uh We are. We are working on moving the intel code to a new repo called class api provider nested, so the all the future development and the enhancement we have been doing that repo and this is a open source project, and this is a community effort.

C

We are very welcome anybody who are interested in this project to join the project um by by by by failing the issue by giving easy virtual class a try. So I am happy to help people to use it and making this project better. So next, I'm going to hand over to my team for the next translation.

D

Thank you frey, yes, hi everyone, I'm jim beguardia, and I lead the multi-tenancy benchmarking effort within this group, and so both adrian and faye showed us. You know different ways of implementing multi-tenancy and, as everyone probably knows, there's several several constructs and kubernetes that you can use for security for multi-tenancy for segmentation isolation.

D

But one of the common challenges that we heard from folks in the community uh from customers is how do I know that my cluster is properly configured for multi-tenancy?

D

Are there ways to measure this test this and report this right, so the benchmarking effort, our our focus, is to create a set of guidelines for first off, of course, configuring multi-tenancy and then to also create tools to easily be able to validate and test whether multi-tenancy is properly implemented.

D

So in terms of what are some of the definitions we have come up with, and these are, of course things that are evolving in the community, but right now we have two profile levels defined where the first profile level. The idea is that, given any namespace independently of how this namespace was configured, how do you make sure that that namespace has the right constructs as all of the right again the isolation the segmentation required, for you know multi-tenancy of different teams using that same cluster and then with the profile level? Two.

D

The idea is to extend the first profile level and also add self-service for that. So not only do we allow in that profile level uh teams to be able to you know, to isolate and segment their workloads, but also to be able to then do self-service operations, like maybe defining custom network policies for their application custom roles, things like that that would be required across you know: different multi-tenancy constructs also in terms of the categories of the benchmarks and tests.

D

We identified seven different categories, so everything from isolating the control plane from workloads isolating you know: tenant workloads from each other network, isolation, storage, isolation, making sure host resources are not accessed from pods. So these are following, like pod security, best practices, ensuring fairness through configuring resource quotas and then finally self-service which is profile level.

D

Two, as as I was discussing earlier and then in terms of in terms of the tests, what we saw is just checking configuration is not always enough, because there's an increasingly in a number of different ways just take, for example, network segmentation, there's a number of different ways. You could configure a network segmentation or implement it, so, instead of just checking for configuration, what we also do is run some behavioral tests in terms of how multi-tenancy is configured within the cluster.

D

So today you know the tests are mostly focused on namespace-based isolation, but as the virtual cluster project evolves, we'll also be looking at adding behavioral tests uh to be able to check on that.

D

So with that you know, and the you know, I also mentioned that part of the effort of this track was to also come up with a validation tool. So we have a coupe cuddle plug-in which you can run on any cluster or it just requires a namespace and a role and I'll give a quick demo of this and what it will do is it will allow us to check and it will run about 15 or so different tests to ensure that multi-tenancy is properly configured and this list of tests again.

D

All of this is on our repo, so feel free to browse through look at the details, provide feedback and comments there, but let me just show what this looks like in action and then we'll come back and you know talk a little bit about the roadmap, etc.

D

So here I have already a cluster configured and what I'm going to do, just in sake of time, is I'm going to show what this looks like when running. So here I'm using a particular user role. This is a tenant admin alley and the name space here is test. So there's um it's going to go. The mtp plugin is going to run through a number of tests and, as you can see, uh what happened initially was it's.

D

You know a few of the test paths because I have you know quota configured, I have some limits configured, but a lot of the tests are failing at the moment right. So what I can do and all of this by the way is documented on our you know, web web page. So if you browse into the project- and um if you look at you- know the mtb web page, you will see instructions for going through this exact same demo, but I'm going to apply.

D

You know some policies now to for segmentation isolation within my cluster, I'm using kiverno policies. There's also you know, oppa gatekeeper policies, so you can choose which policies you apply and, as you can see, what these are doing is they're, providing host isolation.

D

uh There's different policies for also you know, disallowing things like adding capabilities to your pod, etc. So with that, let's run these tests again and what you know we would expect to happen is we would get much better results this time around as we're running the tool, and since we have the policy engine in place, which is you know, guaranteeing some isolation segmentation.

D

So this time- and you know, as expected, we went through all of these 15 tests we're seeing a pass. So this gives a very easy way for us to now validate that the namespace is properly configured.

D

So a few other things that we're working on- and this is where again we're looking. You know for folks to join in and help and uh extend some of this or even just contribute. By giving feedback is we want to add more behavioral tests for storage and networking? uh There are.

D

You know some pending requests for also converting this tool to run inside of the cluster and produce periodic reports using some, you know a custom resource for reporting and then also you know, there's interest from other working groups uh to be able to reuse these the benchmarking tool for auditing and benchmarking across different other types of security standards like the pod security levels, uh things of that nature.

D

So that's a quick. You know overview of the multi-tenancy benchmarking project and I think upfront tasha gave some links, I'm just showing the repo links and other things um again, but we're here and we're open to take your questions and chat um so feel free to jump in and ask questions there.