Kubernetes WG Multi Tenancy, 6 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: WG Multi-Tenancy Bi-Weekly Meeting for 20201006

Description

WG Multi-Tenancy Bi-Weekly Meeting for 20201006

A

Hi everybody- uh this is adrian ludwig uh hosting the meeting sitting in for tasha who couldn't be here today. uh So this is the bi-weekly meeting of the uh multi-tenancy working group, although we actually ended up skipping all of september due to either lack of agenda or short work weeks. um So we have a light agenda today.

A

The only thing is just an update from all of the projects that are being actively worked on uh by the multi-tenancy working group, um and so those three projects are going to be hnc, which is an update that I will give, and then we will have jim talking about multi-tenancy, benchmarking and faye talking about virtual clusters. After that, I can see a few more people are popping on uh after that.

A

If there are any open, uh uh we can open this up as a round table and failing that we can go back and do more work. So with that said, uh why don't I give a quick update for hnc the hierarchical namespace controller, so we have released uh hmc 0.5.3, which I am hoping will be. The final release on the 0.5 branch, so 0.5 has been basically the first real release of hmc.

A

It's the first one where we felt like we had a decent feature, set a relatively stable api and reasonably good uh performance and behaviors, and so I think 0.5 first came out in I believe it was june um and ever since then about once a month we have been giving it a patch update, uh usually not changing, behavior that much, but we have actually changed some fairly significant pieces of behavior.

A

um Let me just uh bring up uh and just bring up the the list of changes that we made uh over the last little while so, as I said, I'm hoping that this will be the last update, because uh ichi has been working who many of you know. Ichi has been working very hard on the upgraded api, so we started as many projects do with v1 alpha 1..

A

We got a bunch of reviews on that to make it more sort of kubernetes style compliant, and so we've been working hard on implementing that and so we're hoping to release that by the end of this month, and that will be the 0.6 release. 0.6 will not be exactly backwards compatible with 0.5 you'll have to update your client tools. uh However, we are building features that will automatically take all of your existing objects on your cluster from the old api and immediately upgrade them to the new api for you without any intervention.

A

uh So you don't have to to recreate everything.

A

So what we've done in the last release is we've added support for crew, so crew is a standard way to download and install plugins for your cube, cuddle uh command line, and so the the hns hierarchical namespace plugin, which is part of hmc, is now supported by a crew, um and we have made a couple of bug fixes. So, for example, you weren't able to delete something spaces if there were propagated objects in them.

A

It took us a surprisingly long time to notice this, but that was a very quick and easy fix once we discovered it, um the probably the more significant change and the most significant change in this release was that if there were conflicting objects between parents and children, they were propagated inconsistently and what we've done now is we've uh added the rule that if there is a conflicting object in a parent and a child, the parent will overwrite the child.

A

However, we've also added a bunch of web hooks that prevent you from accidentally creating a conflicting object in a parent so that nothing ever gets overwritten. So as long as the web hooks are always working, you can never overwrite your own, um your own data, um but if somehow you managed to bypass the web hooks, let's say you installed a bunch of stuff um and that well hnc was disabled and then turned it back on again.

A

This will actually overwrite the existing objects, because we felt that having guarantees on the consistency of policy application was more important than preventing data loss. In this case, uh and so this was all discussed on the slack channel through design docs. uh Let's see, we also added support for kubernetes 119 and a couple of minor other corner cases.

A

um So yeah, as I said, I am hoping that this will be the last uh the the last release on the 0.5 branch and I'm hoping to see a release candidate of 0.6 out in about two weeks with the final uh version of 0.6 coming out soon. After that, as I said, the only major change we're expecting in 0.6 is the new api.

A

We may have a sort of experimental or possibly even alpha, release of a new feature called exceptions where you can, where, instead of objects being propagated to all descendants as they are today, you can limit which namespaces in the subtree that they go to by using label selectors. So, for example, you can say well. I would like this uh resource quota policy to be propagated to all child name spaces, except that one and then you can exclude that one and add uh your own resource quota. That has a let's say, a higher limit.

A

So this gives administrators more control uh over how they create their policies and how they propagate them throughout their cluster um and so jenny who uh some of you may have uh if you're watching the the pull request. Jenny is, uh has been contributing to h c uh all summer and so she's working on that feature right now um so yeah. uh That is what is going on, plus the the regular kind of stability and testing enhancements that come along with any growing project. So before we go on any questions about hnc right now,.

B

So anyway, a few things so uh first you mentioned cruz crew- is that um yeah? That is a k-r-e-w.

C

P-R-E-W, so you said you, you can use that to install plug-in. I.

A

Haven't tried that yeah, it's very cool crew, is uh it's developed by one of the cigs. uh I imagine like that of the cli sig um and um and as soon as the blog post on agency went out, which got something like 14 000 reads, which was pretty nice. This was in august um as soon as I went out, the the maintainer of crew contacted me and said: hey. You should be able to download this. So it's very nice. It auto detects. If you are on linux or mac and it installs the correct plug-in.

A

We haven't built a windows plug-in yet, um but we will, if there's demand um and so yeah, whether um it will download and install it, for you can uninstall it as well. It only supports one version. At a time, you can't ship multiple versions um on crew at once.

A

So, basically as soon as we release 0.6 with a new api, um we'll only be able to you'll only be able to use crew to get the latest plug-in, but the old way of doing it, which is just downloading the binary and installing on your path, that's still available as well, and we've updated our build process to automatically generate all of the artifacts required.

C

By crew, um why do you store your plug-in binary, actually in github or so sorry in docker.

A

um So so the the the binaries are just stored on github, as they always are they're just artifacts of the release. So, for example, if you, uh if you go to the release, a connection.

A

Can everybody see the 0.5.3 release? Okay, yeah? So if you scroll right down to the bottom, you can see. We've got quite a lot of instructions up here um you get all the the artifacts. So this is the gamma file that you can use to install it on your cluster this right here. This is the tar file that contains uh both mac and linux, as well as the license file, and this is what is used by crew um and so crew itself is um they distribute their um plugin through a different repo?

A

And so basically, I have checked in a yaml file that points to this file in their repo and whenever you say, crew, install or crew update. The first thing is, it does. Is it clones that repo locally uh downloads, this tar file, extract and then extracts the correct files for your platform, which it auto detects?

A

Okay, but if that's not working, you can feel free to download these files, uh rename them to remove the the suffix and as long as they're anywhere in your path. That will work as well. But crew is very nice because it puts them in um like in a hidden directory and it can install them and uninstall them and update them uh as you as you wish. I see I see I see yeah, that's good.

B

Yeah, because um that related to vc we are trying to. uh If you see one of the issues, we are trying to move the vc ctl to a kui, kubi-ctl plug-in. That is more formal. um This is our plan actually.

A

Yeah I can. uh I can certainly share with you um some of the work that we've done there. um It's not actually part of the it's not part of the 0.5 build process, but um um on the trunk actually whoops. That's not what I meant to do.

A

If you look at the trunk, oh.

A

Yaml file expected by crew, so here you can see it's part of our hnc hack directory, and this is the yaml file that actually gets. um It gets modified to have the correct, for example, the correct image, uh the sha-256, the link to the correct uh download repo, and this is what drives all of crew. And then uh our new build process actually fills this all in and we'll include this as one of our artifacts and then your only role is to download that file and then submit that as a pr to the crew index.

A

I see so I can show you how to well. We can talk about that when you are ready to go.

D

Yeah yeah yeah good, I think good to know so I think I can go to the crew and which help us to upgrade. So I was always wondering if we, you know developed a.

B

Plugin, how does the user you know, keep keep track of the newest version for that plug-in for every installation? That's not very good.

A

Exactly yeah and crew is the the kind of accepted way to do that. Okay,.

B

Great, so another thing is um so so yeah at this stage. I think of also the part project at least uh seeing hnc vc is kind of um uh much uh not 100 mature. But what is the progress of moving to the another independent repo for.

A

uh So for us, the the main blocker there is, the api is implementing the new api uh and also we're getting a code review and a security review from um um I forget which thing it is, but um basically mike dinassi and a few other folks are giving us a review. Once we pass the code review once we pass the api review, that is, then we can uh then we'll be in a better position to get our own repo.

A

So really we've been focusing on the api, because that's the long pole, um I've gotten some early code review comments from mike um and I think we're gonna save those to a little until 0.7 at this point, um because they're not hard, but they um you know moving files around and break is always a little bit annoying.

B

Oh so 0.6 won't be overcrossed won't be able to address all of the comments.

A

um I don't think so, just because we have limited bandwidth um and I want to get 0.6 out the door if we waited a bit longer, we could so in 0.6 we're going to finish the api um and then in 0.7. That will probably be the point at which we're ready to uh to to look for our own repo.

C

B

Yeah, I don't have other questions related to hnc.

A

Okay, uh thanks faye um anything else for me before we go on and given that phase already been talking, maybe we can go to vc next, but uh before that, any other questions for hierarchical, namespaces.

A

Hearing none, uh why don't we go on to faye so fade? Do you want to give us an update on vc.

B

Yeah, so in vc side I mean so, we see in upstream version. We have a two major part y is the synchro part. The second is the tenant master life cycle management, which is a vc manager. um I think single party is pretty much enough, so there is one uh which I think is very important. Enhancement for the synchro is now we support. Sharing, not not exactly cody is about 90 percent conflict.

B

Now is uh so so so the idea is, even though um you have each tenant is given a dedicated name, uh aks server, but synchro is the central hub so for the sinker worker queue. You still accept the request from all the tenants masters uh there's potentially that, because what worker queue is a fiber queue, so there's potentially there is in the contention that faculty level. um So now in just in the past a few weeks ago now we implement the fair sharing in the thinker uh worker queue.

B

Now it's a support you know uh round. Robin is actually is a weighted round-robin algorithm, so um by default each tenant has the same weight. So we don't play with the way that and now, but the algorithm support different weight. So now it's kind of uh when we serve the reconcile requests from the tenants masters. Noise can go with the round-robin order, so to avoid any salvation and the priority inversion kind of thing, which I think is a major enhancement for the sinker to support.

B

You know slr uh kind of thing uh which so, if we finish this picture now now is you can see um the tenant master. Has you know isolated environment system? In terms of both you know, control plan and also the performance wise.

B

They have kind of uh guarantee when, even if you have multiple tenants, are actively sending uh real or really sketchy intended master, most likely is the right either you know, creation or update those operations will be populated in a more fair way uh that is um function, functionality, wise, and that is a major upgrade for vc. uh We see um thinker uh the second.

B

What was driving.

A

Those um those requests like that was, or um that feature was that, based on your own internal usage,.

B

Okay, now we don't have real usage to drive this, but in theory you have the power arm for sure, because for the I can make up for the michael benchmark to illustrate this problem, I have one tenant master, uh keep generating. You know, 5k qps, for the for the thinker, that's just keep upgrade and another one. You just generated one. So there is easily you can create a case. You have a kind of starvation that one there's one request need to wait. Another five thousand loop, let's get finished until you get.

E

B

Thinker, so micro benchmark wise, like we can you can, we can make up the case a real use case, uh not really uh until you hit the tender master really hard at this moment. Even our production is not that hard at this moment, but uh in theory we just fix a potential problem which we can get attacked by any you know any people say you still have a weak point in your design. Yes, I have that at the meeting.

B

Yes, we have a weak point that we just want to fix that weak point at this moment,.

A

Okay, cool anything else from you or.

B

Yeah yeah, another part, is the vc manager part which, uh because we are also thinking about the thing- is how to move this. um The receive intel project up to the c repo. um We probably have to go through a different round other than you guys do, which you guys would think of. um We are thinking, maybe we we should do another way which we enhance the tenant master management using the cluster api style.

B

I was thinking.

A

The same thing, yeah.

B

So now now we are working with apple chris, maybe without sick. Today um we are trying to make that happen to get bonds from the c class api. So because now class api has all the the the differential things is nowadays the class api focuses on providing more intel cluster, including node resource. There are quite a bunch of things on um you know, provision machine.

B

uh In our case, we are trying to propose uh a new kind of class api only only create um master component like api server, not not notes, basically it's a major use case and for our multi-tenancy, you know solution.

B

So that's probably a new thing that we are currently trying to completely revise the way that we create tenant master. um So that is our plan, and we hope, after designing that we, we should present this to c cost api to see that feedback. um Eventually, I wish I mean I wish now the the the thing is. um The the project will looks like a little bit a little weird in the sense that okay, it's a class api project, but that component is only useful in multi-tenancy contacts.

B

So I will put the synchro part in that in that repo to leverage the class api thing, that's probably easy to get accepted by class api. That's my thought.

A

That's interesting: it's uh it's really worth having the discussion with them and seeing where that goes yeah. So it is a type of multi-cluster.

B

You type the marketplace yeah. Without knows it looks a little bit weird. We need to, you know, clarify all the concepts of this. So in reality the actual use cases for vc is the best use case for vc I mean other than that. Maybe nobody else wants to create so many canon masters without nodes, so right.

A

So yeah it's it's interesting. um Certainly you could use the kind of kubernetes as a control plane idea. It would be nice.

B

Yeah, I think some of the other issues is that you create you, create many many control plans and you install virtual complete. That's one way that I can think of yeah uh other than that is um vc. Definitely the best fit uh yeah is slightly narrow scope for this. I hope you know other api guys will like this so yeah. I think chris has some initial initial uh conversation with them. um We are just working on some design docs to push forward for that.

A

Sounds good, um I had a question or two if you're finished or.

B

Yeah yeah yeah, that's all at this moment, yeah.

A

Okay um yeah. The question I have was that I've been seeing um some like that. There's gonna be more activity heating up just like in the community about multi-tenancy, and some of them are kind of like in the sort of canon, crd hnc area like um capsule and loft, and um uh and what's the latest one I saw uh angel uh was here to talk about that a couple of weeks ago. uh The name escapes me right now.

A

Caseman, thank you, kate, spin, um and so I've been talking to angel about possibly uh joining forces on some of that stuff, and I've also seen that there's at least one other kind of implementation of the virtual cluster idea. um Faye have you reached out to the like? Have you looked into those other projects?

A

uh Do you know what the differences are? Is the laughter one it may be locked yeah loft was up to credit. It was doing a couple.

B

It is a lot, so largely is kind of the past platform, so the underlying technique. They call it virtual cluster.

A

And is that? Is it a similar idea to your virtual cluster.

B

uh Idea is similar, but it I mean idea is give if the idea can be as simple as give each template a dedicated control master master component. That.

E

Is right right.

B

This is similar, but the implementation is different. My current interpolation about that, based on the documents based on the architecture, design, doc, I think they they they they they they work more like a k3b, if you remember k3b, not so much.

A

B

The rancher guy's greatest case rev, if I I think rajit mentioned last last year, actually it's like so so. Basically, last year we proposed in a virtual crossfit that, um after a few months that the rancher has a case of implementation, so idea-wise the same thing, but the implementation are different. um I can give some key differences here, but since I didn't see their code, so I don't know um the exact difference concept. Otherwise, um first they don't use the centralized sim card.

B

They use. You know each channel as an individual thinker. That's number one number two: is uh they they they don't exactly like. We did things like. We have a one-time matching between the talent and memphis and the supermarket namespace. They group all the tenant objects into one supermass namespace. So the name specifies there is no one mapping between tenant and super, which I think is a problem.

B

If, because so so, basically all the tenant objects when they sink to the super. The name space is cheap. The name space organization is completely changing. There is only one so which means within that hana master. A per name space feature like network policy kind of thing doesn't work.

B

C

I I break quite quite some.

A

Compatibility, so I think um where I was going with this, is that, does it make sense for you to reach out to the law folks and say like hey? Should we should we join forces should instead like because I know that they're selling a lot more than just a virtual cluster, just like kate spin, is selling a lot more than just hierarchical name spaces? Does it make sense for them to adopt our vc and maybe contribute to it when like when they need a feature?

A

That's not there as opposed to going it on their own fully on their own.

B

um I'm not so this part, I I don't. I actually don't know quite quite because this they are a commercial company, they sell it. So I don't know if there are, I I mean I don't have any problem. They use vc, but I just don't know. Maybe I'm not kind of familiar with this process. If there is a commercial company, is selling a project. Is that okay to ask them to use upstream or cut.

A

um Well, I thought I'm just looking on my phone to see if I'm correct, um I thought that at least some of their stuff was.

A

um I thought that at least some of their stuff was open sourced. If that's incorrect, then maybe that doesn't make a difference, but I thought they had a github repo.

F

Yeah, isn't it the same team that also released kiosk, so I think they built loft add-on on top of kiosk just for creating multiple clusters right.

A

Yeah, so I think that's, the thing is like we don't care if they're, commercial or not I mean yeah, most of us are um all we care, I believe, is if the code is apache too licensed right, um and so, given that, if they are, it might be worth re.

A

So, given that they they have something that kind of intersects with both of our products, um maybe say you and I should reach out to them at some point together and say, like hey, you seem to be doing a lot of the same stuff that we're doing um doesn't make sense to join forces because they, what they then get, is that they can claim oh yeah, their standards compliant plus they get the benefit of all of our work.

A

What we get is um their user base and any additional contributions that they are willing to make or that they need to make for their customers. So I think that it's, it could be as long as the the models are sufficiently compatible. um This could work. Well, I mean, obviously I don't know if it will until we we look at it, but does that sound say? Maybe you and I can coordinate on that.

B

Yeah so yeah, so basically that, as I mentioned, you know the biggest difference, what I mentioned is we? Don't they don't do the one one one two one mapping for the name space. I don't know any other theory behind that. There is other reasons they have to do that. So, besides that, I would assume the same thing has no difference.

A

Yeah exactly so, and if they're not wedded to that idea, if that's just what we started with, uh perhaps they're willing to change it, especially if there's a good reason not to do it, which they hadn't thought of.

B

So you know the other guy right. I.

A

Don't think you've spoken to anyone from loft, although, although I'm starting to lose track, I need to go back and and double check. Yeah.

B

They published quite quite some dark. I mean the post in the medium, the indian medium and the new stacks kind of thing. I think we can get this connected person if you you would like so they use agency. I don't they.

A

Don't even, but they have something they have a concept of self-serve name spaces that is um overlapping. What agency does um so? I think it would be. It's really useful to get them to start joining this group and talking about like if there is some reason why the the projects that we are building is not suitable for their customers. That would be useful. I think feedback for us um both okay.

A

So I don't want to take up too much more time on this but say I'll contact you on slack and then maybe we can uh go together. We.

B

Can try to reach them to see if we can, we can talk about this.

F

Yeah it'll be good if they want to come present at one of our sessions right, so maybe they can talk about both of their projects, exactly yeah. That sounds good. Okay,.

A

F

Say I did have one question. You mentioned something about network policies. I didn't quite understand you. You said there was uh um like I guess, as you were talking about, I.

B

Just I just make up, I just make up use case. For example, uh the tenant can create a net memory policy in tenant master so um but decorative policy, narrow policy. You have to specify the namespace okay. So if you keep the 1t1 mapping between the tenant, namespace and the supername space, then we say when someone thinks the talented.

B

The net policy objective is super. You still can guarantee that. Never policy only work on the specific namespace, although the name has been changed, but logically, the same name space, although the name is changing, but logically the path ownership is not changed right only in these parts below this name service, so network policies still work basically because they never was it. But if you get rid of the name certain mapping, you only have one name space in the supermaster.

B

Okay, the network name space doesn't work because everybody everybody's are in the same name space in the supermarket, actually wrong.

F

Sure so that's that's! Just more of a limitation of name spaces based virtualization versus oh.

B

I just make up one using it. I can. I can try to think of other case.

F

Yeah, I think it'd be interesting to come up with that classification for what you can and cannot do with namespaces based virtualization versus you know the virtual cluster concept right.

B

It is, is, um I I don't think I think name space, which is like case beings like more like a name space device mutualization, because they are actually doing exactly what your cluster, but only even because, if you.

E

B

At the api that they provide to the customer, it's a full cluster, it's not a namespace okay, so they are going with the virtual cluster based virtualization. The only problem is that I mean so so the interface wise, our approach and their project the same because they give the people the virtual cluster. The only thing. What I worry is the implementation wise right imagination. They have a single, we have a thinker, so the thinker may do differently. What I see the biggest difference. Is they? Don't they don't directly? Sync the namespace. I don't know.

B

What's the reason behind that, because to me it is more nature to directly synchronize space than not do that.

A

And when you say namespace you mean the namespace, that's on the host like on the supermaster, not on the uh yeah.

B

I think it's much much easier to just simply they choose not to think it. Maybe they have some reason which I don't know so.

E

I mean I'm looking at the architecture right now and I mean to me it seems like they more or less kind of have like a tiller type thing that is just a federated cli that you authenticate with with its own our back and they separate it. Nate it's in it's, it's kind of like a tiller with um uh the uh the open shift, is kind of what the architecture looks like and then a nice ui on top of it.

E

um So that's why they do a sing, a name space per um and their nested name spacing seems still to revolve around kind of a name space per cluster and their virtual cluster is is a name space from their docs. It looks like.

E

F

It was name spaces base too so, but I haven't looked.

A

I thought they had both. I thought they had like within one cluster. You could insult service namespaces, plus you can build virtual clusters.

E

If that is not enough for you, so I mean the self-service namespace provisioning says that I mean at least on their docs. It says that it's from their loft cli, which to me looking at that, and then they have a a loft extension api server. It seems to me, like you, just authenticate with cli and then it'll, basically federate out, to create you, the a namespace if you're authorized, is kind of what it seems like.

A

Yeah, but that main space is not in a virtual cluster, correct.

E

From what I see it doesn't seem like that, but you know I'm just I'm just looking at their their docs and stuff. So I'm not an expert.

A

Okay, let's, let's have them to come and give a presentation, because.

E

A

The same territory that we're covering right.

F

Okay, anything else on bc, or should we move to mtb.

B

F

All right adrian, if you can give me screen, sharing I'll, just pull up the repo and also walk through some of it.

A

uh Let's see if you have a screen more okay, yep.

F

Laptop all right, so just you know kind of quick updates from on the benchmark side. What we did is we moved so much like adrian was describing with hmc. We also have a crew plug-in, so it's scoop cuddle mtb.

F

So you can you know if you go into the multi-tenancy benchmarks for our project and click on the link for running the conformance test uh with coop cuddle mtb, then you go into a sub page, which describes in quite a bit of detail the process of downloading and running the benchmarks right so where we are right now and I'll go back and show the list of benchmarks, but this is fairly straightforward to run on any namespace and the way this has been designed and everything that's working.

F

Is you uh basically can create a role within a namespace run? These benchmarks uh initially you'll get a bunch of fails and then, if you install gatekeeper, oppa or kiverno policies uh which to give you the right level of pod security, so as you see most of these tests are related to pod security.

F

um So then, you will start getting your test to pass right and there's other things. Like quotas, etc, um you know all of the best practices you need for multi-tenancy, so this tool works pretty well and in fact we did a demo to cigar a few weeks ago, and there was interest in perhaps reusing this also for the replacement to psps, uh which is you know, like the standardized spot security profiles,.

A

What is that? What does that mean? Mtb is, uh I thought: mtv is purely client-side.

F

Right! No, but as a way of benchmarking uh for a level or testing or measuring for a.

E

F

Of conformance to uh different pod security profiles, cool.

E

Were you referring to the replacing like the cis benchmarks,.

F

No no, this is um the. I think it's called like the standard profile levels for pod security right, so um this is being proposed within cigar as a possible replacement or as a way of sort of, as I think in 1.22 psps will be deprecated, and at that point it will be up to tools like kiberno or gatekeeper oppa, to measure and report on pod security.

F

But potentially the benchmarking that we're using for mtb could also be an in cluster tool, which anybody can run and say. Are they conformant to a level of pod security or not because a lot of the checks we're doing here- and these are runtime checks, um they're all they're, very much related also to the different security pod security levels.

F

Yeah, so a few other things that we need to complete to get past our so going back on the main page, where the list of profile level one checks are um so here, we've like I mentioned everything, that's complete, is more or less related on the security context. There's also some single name: space checks for quotas limits things like that.

F

The the one area where we need a little bit more discussion and perhaps some feedback on direction, is how do we handle things where we want to do behavioral checks across name spaces right and the problem there is, you know, with each different variation of how multi-tenancy is handled, um there's different ways of creating these namespaces, with tools like hnc or even like, as we're discussing with capsule or loft, or even if you're, using just kiverno and creating namespaces. Each one has a slight variation.

F

So we need some some general mechanism, or perhaps a plug-in mechanism to allow for these variations uh to do tests where we're actually checking if two namespaces can talk to each other or creating namespaces for different tenants, but the plan the so that's a little bit longer term, but the immediate plan. The next thing we're going to wrap up is um tests on the storage side.

F

So there it's more like on pvc checks within a single namespace, and then also um we will you know, there's a few others like the that came up in discussions with other folks like this cap drop. All is just checking again for part configurations of which we will add, but as it is right now, this is fairly usable. It's pretty simple. If you haven't tried it out, definitely just go through. You know.

F

If you click on this running link, you'll see, there's a video as well as their steps, detailed instructions on running this on any namespace. We did test this with hnc. So now, with some of the, I think bug fixes that came in the prior release. This works fine with hnc everything, looks good cool yeah. We haven't tested it yet with virtual clusters, but I don't expect any issues there, because they're creating a namespace is just you know, a regular namespace, but that would be a good good kind of milestone as well.

A

Yeah, I think that the most valuable thing that this can provide is really ensuring isolation, because what agency does in a way is it breaks down barriers between namespaces, because it's like namespaces are the way you isolate the control plane anyway in kubernetes and but they're too. Limiting you don't want to have them completely isolated.

A

So, like agency allows you to sort of selectively, break the barriers and it's kind of hard to imagine a general purpose way where you would measure that breaking down of barriers, because, like all of the the features that you're testing right now, they're all features that are part of upstream kubernetes and you're, just basically testing how they've been set.

A

Whereas things like hnz agency versus capsule, there is no agreed-upon standard between them right and- and perhaps it's premature, especially if we start talking to the loft folks and the kate spin, and eventually we start to say well, yeah, let's standardize on agency, which is at the lowest level like it's at a lower level than either of those features, um and then they and then we can start saying like okay, so that is the primitive.

A

Does. Capsule and kate spin uh use them in the correct way.

F

Right, yes, if there's a common resource which represents a namespace or just using a namespace itself with some other mechanism around it, then you're right. We could start.

F

You know leveraging that as a standard, the other option, what we're thinking of with mtb is just to maybe allow a plug-in model where, based on the tool that you're using you know, the user can provide how a namespace gets created exactly as long as there's some coupe cuddle or some scripting available to create that namespace, um then you know, mtb can call out to that and perform checks across multiple namespaces.

A

And that will work as long as the concepts are similar enough right. um Yes, the matter if the command is different, that's pretty easy. If there's some kind of conceptual difference, that's a little bit harder, not maybe not impossible, but harder.

F

Yeah- and here all we're concerned about, is how do we, how do we get the user to give us two spaces which belong to two different tenants, and you.

E

F

Automatically create those namespaces as needed for various checks. um So if you can say tenant a here's, a namespace and b here's a namespace. Now we can validate that. Oh.

A

F

I see you can't talk, uh you can't have two applications talking to each other or referencing each other directly, right.

A

Yeah, you could say: okay, um here's this or you could just have a setup script. The subscript would be like the name. Spaces are called a and b and in this scenario, they're part of the same tenant, and in this scenario they're not show me what is that accessible, because that will depend on all kinds of stuff. For example, if you set up your your network policies, maybe you want to talk, and maybe you don't.

F

That's exactly yeah, so you know I'm checking that we can run pods in each namespace and they shouldn't be able to talk to each other, or you know I guess an advanced case would be even do like dns lookups and things like that. But those are the sort of checks we want to do across namespaces.

A

Okay, yeah, that that I can see, is being very valuable. Give me a pair of name spaces and tell me how they can interact.

F

B

Well, hey jimmy, yes can, can you talk a little more about the required capture poll? So what exactly that which this one because cap drop 4 is pretty powerful. If you do right, it's kind of the container, can you can almost do no communication but compute.

F

Well so the best practice that's recommended- um and I can I'll have to pull up some references to it- is to to drop everything and then add back the capabilities that are required. So.

A

F

Container does require a few privilege capabilities. Then you can add those in explicitly. The problem is, if you don't drop everything, each container runs with a very um fairly large set of privileges by default right, which are yeah.

B

People there are about 20 some right right, yeah. I know, because uh because libra brought up these questions um recently, if you, if you are aware there is a cbe which is right, uh the the the network raw capability has a security attack. So I I know we are. We are thinking about the solution. How do you work around that? I know one solution is drop on and add one by one, but.

F

B

The the the counter argument is that this is too too much work.

F

Well, but it's probably well worth it and I think the challenge is yeah, detecting which pods and you know, classifying pods. So this is one, so I don't know if folks are familiar with the scoopsack.io, but it's pretty good reference for most common pod security level.

F

You know concerns like this, so this is one of the places where they recommend and it's fairly, even in the docker community. This is recommended when you're building secure containers- uh I don't recall often if it's a cis benchmark necessarily, but certainly for pod security. It's recommended.

B

Yeah so yeah uh I understand now my question is um I see the reason so actually I don't quite see the reason to put these mkp because we are empty starting to address security issue. If you.

E

B

That, then, a lot of them need to be done.

F

Right so, and that's a good question right and I think one of the things we need to decide for mtb and is for multi-tenancy.

F

Obviously you want pods to have a certain level of security like you, don't want them to access host resources, things like that, but then there are other security concerns which may not directly be related to multi-tenancy, but which are best practices right. So now so I mean you, could you could sort of argue that if you don't secure your pods, some a malicious user or somebody who accesses the pods could potentially get access to other cluster resources and that's sufficient reason to secure your pods for multi-tenancy as well, but you're right? It's not directly.

F

You know promoting multi-tenancy, it's more just a security best practice. So one option there is to just reference the standardized pod security profiles right and, if those become you know, become widely accepted and used, which I think they will uh as psps are deprecated.

F

uh At that point you could, we could just say for multi-tenancy you're required to have, um let's say the restricted profile of which is the highest level of spot security. In that, in their definition,.

B

B

uh Let me see so.

B

So so is, I are you saying that is the requirements? This is optional.

F

So yes, a require would be that we would check for it now whether we you know how we score, whether we say that that's you know is it again. We have to decide and uh get some feedback also from folks who are working on some of these pod security profiles for what they recommend. But I think this is uh one of the discussions I had. I think it was with rory who works on some of the security standards he had recommended, uh adding this in.

B

So so so what I try to understand so if I didn't drop all the capabilities so so I will my has a fail.

F

So this would yeah so if you're running the benchmarks, the mtb ben, you know what would happen is this would show up as a um as an error or, as a you know, failure right within your for that particular test. Now the other you know kind of nuance on this. Is we have to decide of each one of these tests just like with cis? There are some benchmarks. They recommend but they're not scored right. So they're saying that yeah we highly recommend you do that, but they're not going to count.

F

You know for multi-tenancy, so we could say it's an unscored test or.

A

You have different levels.

F

A

F

A

Would be the highest level.

F

Exactly so, there are.

B

Nuances like that.

F

B

You're right, I think the those things could be optional. We shouldn't you know 100 percent of you people to do this um because, in my opinion, the other ones the otherwise is about isolation for sure you need to block it. So if you didn't block it, you fail it all right, but for others um which which not directly correlated to isolation like security one. This should be just recommendation kind of right.

F

Right so we could yeah, so we can still check for it and you know kind of recommend that as a best practice, but not score it for multi-tenancy yeah. I see your point and I agree with that right because it's not really.

F

I mean there is an indirect. uh Like I mentioned again it could. You know lead to.

F

I guess somebody breaking into different tenants if they're running pods in this manner, but um it doesn't in one way or another, directly impact multi-tenancy.

B

Yeah, okay, there's one more thing because um coming to my mind, so it's random things. uh So it's about the block use of the host networking. So um for this kind of thing you have are you checking uh is the part using the same ip as a host network, or are you checking? Can the pod pinhole.

F

Yeah, so, um for so, are you this is the one you're referring to right, host networking and host sports? Yes, yes, yes in this particular check and you can, when you click on any of these, the actual checks and everything's in here.

F

So if I recall correctly for this, we're checking for configuration options on the pod spec right, so we're making sure that this is not configured with uh host network as true um and then the host ports are also set to nil right, so that yeah yeah and the way this works is we actually do the so we run the check and of course, if we are able to run a plot with that configuration that the test will fail.

F

So then you know we apply policies with engines like kiverno, which will actually block that configuration right, so that will then prove that. Okay, now, if you're running you know with those right setup policies, um you are compliant to these checks.

B

Okay, yeah, because I I thought you are checking you probably will do most strictly check. Like you know, you do exactly opinion test you see. uh Is your eye is complete correctly, so configuring the way that your product can never access the host.

B

You see what I'm saying so so you are now so you you are saying that you are checking nowadays, you just you probs, back checking for this network, but in reality I can do something like I configure my theme eye. No, I just write.

B

We, we virtual internet, you just plug directly plugging into the host, so I still can have access.

F

Sure so, if you have, if you have the admin permission to configure a cni, then of course yeah you can do whatever you wish right. So, but here we're checking more at the user level uh and most of these checks. Yes, so some will do things at the in a configuration level, but others will be behavioral right, so we will try runtime access, depending like we were discussing across name spaces. Things like that.

B

Yeah so yeah, so that that's more about the wrong access thing right right! You you, you better just start apart and try to do some.

F

Yes, something.

B

Yeah, if I can just accidentally access somebody's else, so right, yeah, okay, so yeah! If you.

F

If you have some ideas on how to you know what exactly that test would look like, uh let's discuss, because those would be interesting, so we do want to add those tests across namespaces like like we were just previously talking about, but even if there's some way to check for well-known, like host resources, things like that uh we can.

B

Add yeah yeah, because um that that's the main thing I want to talk about since the moment. If I glance all the tasks it seems like configuration based most mostly so what I'm trying to print out is the random base is still very important.

B

Now we have a lot of ways to go through this. I mean all this configuration like I said just if the c9 plugin is not complete correctly, you still get you can do whatever you do.

F

Right right, so some of those yeah if your cluster is not configured or yeah. If there's some other issues there, then yes yeah. So we do. You know adding some runtime checks and I think there were a couple here which are more just runtime, which so this will actually, you know, try to access. It will bring up a pod, try to access different things in the cluster, uh but there's most of these are more configuration and checking at the pod level for various uh security, best practices.

B

Yeah, okay, yeah. I wish you know we have one more long-term thing so now, uh because again, if this becomes standard, you know I can say, we see positive tests, so we have right so, which is much more.

B

I mean important than the config check, because uh in the production level, if you want to talk about with hyper level guys they don't look at your detailed implementation, they just say show me the bank, you have the runtime chat.

F

Sure yeah and that's the frameworks there to start adding those and that's you know a lot of the effort was in building the the test, suite and the way of reporting, etc.

B

F

Can add more runtime checks, so, let's, let's discuss that.

B

Yeah indeed, okay, good.

F

Cool all right, any other questions or comments, feedback.

A

um Okay, well thanks everybody for that update. That was a little bit more involved and interesting than I thought it was gonna, be um so yeah. If a you- and I will get we'll we'll talk over the next couple of weeks.

A

Okay, thanks everybody, um so this meeting will be posted onto youtube as always and uh and feel free to reach out on the mailing list or on slack. If there's anything we didn't cover here have a good week. Everyone all right, yeah, bye,.