Kubernetes WG Multi Tenancy, 28 Jul 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: 20200728 Kubernetes Multi-Tenancy Working Group

Description

- MTB demo (Divya, Anuj, Jim)
- HNC update (0.5.1, v1alpha2, etc)
- Live from sig-multicluster: Naming Survey Results with 112 responses

A

Hey everybody and welcome to our regularly scheduled multi-tenancy working group session uh today, jim divia and anush will be going over the multi-tenancy benchmark uh project and doing a demo and then adrian's gonna give an update on the hierarchical namespace controller project.

B

All right, thank you, tasha, so hi everyone. This is jim. I am going to give a quick update or just a review on the multi-tenancy benchmark. Work that we've been doing and um anuj and divya are gonna. Do some live demos on what they've built so far? So the goal here is to kind of checkpoint and get some feedback see. You know what thoughts are on next steps.

B

So just a quick recap for those of you who might not be familiar with the benchmarks. um You know effort. So what we're trying to do here is establish a set of guidelines for different levels of multi-tenancy and then, along with that, have some automated tests that we can run on. The idea is to be able to run these on one or more namespaces and report if those namespaces are properly configured for multi-tenancy.

B

So there is a if you on on the github. You know. If you go into the multi-tenancy folder, you can read about the categories definitions, the benchmarks we have so far, but today what we're going to focus on is a subfolder uh cuddle mtb, which is what a new gen divia have been working on, and you know we'll see that live so just to kind of quickly set the context for the demo and then I'll hand off to your news.

B

So we're going to see two demos one is you know with the 10 controller, we'll see how namespaces are set up through using the tenant controller crs and then how the mtb tool can run a set of benchmarks, report results um and then to expand to rectify, or you know, fix particular problems, we'll see how we can use gatekeeper, oppa as well as kiverno to install policies then rerun the benchmarks so that we get a pass on like all of the profile level.

B

One checks that we have defined so far, so anuj is going to show this first with you know the tenant controller, then divia will show also a similar demo, but with just a free form, namespace and a role that she sets up.

B

So the goal here was to be very flexible and allow in a broad usage of this tool, so anyone can run it on any namespace, independently of how that namespace has been set up or configured.

B

So with that, let me stop sharing and I think um a new divia with whoever one of you wants to go. First, you can share and go through the demo.

C

Okay, this is announced hi everyone, uh I'm gonna, show you how like running the benchmarks on our tenant controller. So if you can see, let me show you first, that I have our tenant set setup already using the tenant controller. If we get a tenant, so we have our tenant zero cr and if you get the tenant admin name space, we have a tenant, zero admin and if you look into the tenant admin namespace, we have one tenant, namespace already created.

C

C

That okay, so we have our tenant name, space, zero, cr, already created by our tenant controller and uh it's tenant. T0 ns0 is a reflection of the namespace 0 cr. It is actual. So this is a crd object. You can see if we describe.

C

C

Okay, so this is our tenant. Namespace cr object, so it it is actually t0 and a0. That is its own namespace. So we are going to run our benchmarks on p0 and a0. So if you wanna listen to the benchmarks, so the there is a simple one: cubical mtv get benchmarks.

C

So at present we have around 14 benchmarks, uh so most of them are profile level benchmarks and we have a single sec uh second profile level benchmark, and we can filter this benchmarks according to the profile level, if you can get apply the flag at benchmarks dash p1, so it will show you all the level one benchmarks.

C

So we don't have uh right now more than we have only uh benchmarks, all of one and single of two. So there are the total benchmarks. So we are now gonna run our benchmarks over the.

C

C

So you can pass the user uh user that you want to impersonate while running the benchmarks with the as flag. So if, when you run the benchmark, you will see a default console log that so that the show the description and you get a scorecard of all over what was the uh what happened while running the benchmark. So you can see, we have classified uh the benchmark result into the four categories. Past failed skipped and the error pass takes place when it, you know it passes the benchmark.

C

Failure is like it didn't pass the benchmarks escape. I didn't pass skipped any so it is zero and the error like we have set the prerequisite for the benchmarks that uh need to be passed to actually run the benchmark like we have any benchmark that is for pod. But if the user don't have any code privileges to create or something that so we will uh put that benchmark into the error category. That user can't create codes.

C

So let's assign the user uh the role to create the um uh ports deployments and other things that we need our in our benchmark. So we, I have already already created thing.

C

C

C

C

C

uh Can you see my vs code, hello, uh so I will. I will create a now a role that is uh for the our user, that it has some privileges like board service deployments and a role binding to the my, the my user and resource code for a benchmark.

C

I will describe later so I will apply this roll and roll binding.

C

So now we I have create created. So let's run again, the bank.

C

C

Okay, so it actually passed all on getting so yeah.

C

uh Give me a second.

C

C

So yeah uh it when I applied all the thing like uh user can uh create port so and when I run that run the benchmarks it passed according to the configuration you have applied to the tenant, namespace or the tenant user, and there is one more thing like you want to create policy report. That is a new repo that gym has been working on, so you can take the output as a policy report object for that.

C

You first need to install the policy report, crd that is here you have to before and create the crd in your cluster, and then it will uh run run so at the end we have got like it is. We have got a policy report object in in itself the namespace for to get that.

C

Yeah, so we have uh the status that 13 pass. One fail uh warning: zero error, zero skip zero. Now we have some more flags. I wanna show you like. If you wanna skip some benchmarks like you, can pass it in the skip skip flag.

C

And if you wanna go for multiple skipping benchmarks, just copy the ids.

C

So that I have skipped 2, so it will run 12.

C

C

so yeah we have got a new policy report that is created in again in p0 and s0, so you can get the policy reports here, so yeah yeah. This is, can be used for the multiple purpose that you can put it on a job. Jim can explain this better than me, for the policy report uh aim to create that and yeah and we have another flag, like you, wanna run only the profile level, one benchmarks to profile level, two as well. You can just pass dash p one.

C

It will just run uh only one, uh only first level benchmark, so it will now run 11 benchmarks out of 14, because two were skipped in through via skip flag and one was skipped in the profile level player yeah.

C

That's it from my side.

D

I'll take over now.

C

E

C

E

A question before we go on uh two questions. um One question was we we started by setting up um a tenant. I see that's the tenant's credit right. um The tests that I saw it looked like you were: writing them, mainly in a namespace. uh Did the tests take advantage of any um of any particular feature in the tendency, or do you or or was the 10 series just the way that these these policies were enforced?.

C

uh Actually, the main aim is of the mtv benchmark is just to validate your name. The namespace you have provided is validating the bad lines or not, so it was just okay. You can say just an example: we used for our demo. We use the tenant controller in case of device. She will use just a normal namespace. She will create her own, like cubecode, create namespace demo, or something like that.

C

For me, I just use the tenant controller to just uh show you the uh application of that. We can use different multi-tenancy aspects.

E

Cool and the only other question I had is um what was the motivation to make the policy report a custom resource, as opposed to, for example, like outputting it as a file? um What's the benefit of keeping it on the cluster.

B

Yeah so that yeah yeah, I can provide some more context to that, so that the policy report is something that's being worked on in the policy working group and there you know there has been quite a lot of discussion on different ways to capture outputs from different policy engines uh like oppa, gatekeeper, caverno, falco, uh even.

E

So this is a standard exactly oh, this is not. Oh, I see it's a standard, cr, okay, cool, exactly.

B

So the benefit here is for cluster admins. It gives one common way at a high level. You can get output from different policy engines and scan, for you know, reports within namespaces or at the cluster level, um and then you would link to details. You know I guess, for different policy engines as required.

E

For like audit and stuff like that,.

B

Yeah, okay, yeah. So one question I know so I know you're like when you installed your manifest. Did it also install the kiverno policies or how did those test cases pass? Were you.

C

Actually, I'm also uh I was, uh I had already set up all the things, uh but uh actually I had to apply the given policies I have uh you can see like uh I work all in our in this hair, but when I saw maybe there is a something cash problem or I don't know it so it's applied uh already. You can say I didn't have to apply so it passed everything.

C

Okay, uh I can just.

D

In a single file actually so.

B

Okay, yeah, because typically I think, if you do this on a empty name space, you would expect the test to fail and then you would need either gatekeeper or giver no to yeah get the test to pass right.

C

B

C

F

Hi, this is faye uh quick question, yeah a few minutes late. So I I see your test kind of pause. um I see some of the tests say block using abc, so um I feel uh that thing can be done by um kind of validation, webhook. So do you need a web hook or how? How does it test around to make sure you cannot add ipc in your parts back or that is not.

C

B

Yeah, so I I can explain it. Maybe uh nudity can also show like one sample for a test case, so the tests are actually um some of the behavioral checks are actually trying to perform that um application like the operation.

B

So, for example, if you're trying to see can you know can a user, for example, uh like um you know, let's say change resource quotas right, so it's actually performing that application or that operation, and then you know, recording the result and then, of course, for things like uh where we want the user to not access like path or other checks.

B

Those would have to be enforced by a policy engine right. So we focused on gatekeeper, oppa and caverno. We did not use pod security policies. I guess as they're still beta there, and I guess the plans are to- I think it's a few years out, but the plans are to deprecate them. So, instead of that we're using policy engines to enforce the actual uh behavior to get the test to pass.

F

Oh, so, okay, so okay, and so you already installed a policy engine. So it's not a. This is not empty. uh No.

B

Yeah, so I think maybe when deviate shows we can try it on an empty namespace just to see so first, you will get a number of failures, but then you need either. You know a policy engine or you could potentially configure psps for some of these to pass, and then you could rerun with with that.

D

So I'll be sharing sharing my stream okay, so I don't have permission to share the screen. Actually.

A

Okay, let's you should be a co-host.

A

D

So currently I'll be demonstrating the tool using the local user and name space that I created manually without using the tenant controller. So I created this namespace using petrol, create ns, and this is the local user that I have in the system so I'll be running the benchmark on this.

D

So, as you can see, the most of the test failed uh this passed because uh the current year doesn't have the privilege that is create, update and use privilege. So that's why this test passed, but other tests failed. So what are you doing either picking one of the tests? That is a block use of notepad services and I will apply oppa gatekeeper here. So with the upper gatekeeper, uh we need a constraint template. So actually I've already applied the constraint template because it takes some time to activate so I'll show it here.

D

So you can see, I have a constraint template here that is kts block node code. So what consent template is actually uh it defines the schema and the rego logic of the oppa policy, and then, after that, we need to apply a constraint which basically defines the scope of object to which a specific constraint template applies to so I'll, be applying a constraint template here. I constrained you, so you did and I'll run the test again.

D

Like, as you can see, uh it passed so yeah, so this this test was done on a simple name: space and a user data created locally on the system without using a tenant.

D

Controller, that's all from my side.

B

Sophie, I don't know if that helped answer your question a little bit or.

F

F

I understand I'm also part but, uh for example, the node part of service. um I think yeah, I'm a little bit confused, because this seems has nothing to do with namespace right. It's the node setup or.

B

Right so I think one of the and these tests, you know the the ones we decided. I think we'll have to go and re-review the list and see what uh how we want to position most of these checks. But the idea here was, of course, if you're, using a noteport service in a multi-tenant environment, that's a shared resource right, so I I know it's pretty common to use noteport services on for several things. So the question is: is this in a multi-tenant environment? What is the perspective? uh Is it recommended to use or not right?

B

So, oh I I okay. I see so this one.

F

Is just particularly check service object so make sure you're right now it doesn't mean that in the node you're blocking certain parts, no, no.

B

So what it's trying the test will actually try to create a noteport service and see that that's denied right. um So I think each one of these is written.

B

You know what anuj and divi have done it they've created a framework and what we wanted to do actually in the next demo is we'll focus on what it takes to write a new test case uh and add a new benchmark.

B

So there's a small framework to run these to test this, and it's all done through using the client code. So through you know, programmatically you can check and validate and then provide output for this benchmark test.

A

G

Hi this is johnny uh and for this node node path service. In addition to try to create and see, if it's rejected, do you try to list any existing node part services.

B

No, so here we're just checking on that namespace, as the user provided to see if that user is able to do the operations. I.

H

B

We're not auditing on the existing configuration in this case.

G

uh For this listed rules are there all the rules are just a sample sample list.

B

Yes, so if you actually, let me let me share my screen quickly, um so this is there's. It's still somewhat work in progress, but I'll show you what we have so far and you know then we're actually adding to this right. So if you go back to our wiki.

G

B

Under multi-tenancy and then benchmarks so there's a table of what we're calling profile level, one checks and then.

H

There's some proposals for profile level.

B

Two checks right, so these are all what deviantart showed up these checks. We have automated and now we are able to test on any namespace right and for each one of these, like, for example, block privilege containers. If you click in there's details, um you know, and what the rationale is why we think this is. You know important for multi-tenancy and then how to remediate right.

B

I see thank you so this structure we in the next couple of weeks, the plan is, we will so with what the work anu gender we have done.

B

This is also it's very straightforward now to add new benchmarks and it will generate a lot of these files and definitions from some yamls, so we'll show that in the next you know in the next session, like two weeks out on how to do that, and then we have to update this wiki with the latest tests and we need to re-review what we do for profile level. One two and three.

G

Thank you, sorry. I will I'm not familiar with the detail of this, so it's just a simple question: can this uh benchmark tool be running against any kubernetes cluster or it requires some? Oh okay,.

B

Yeah, so it's a coupe cuddle plug-in. You can download that we will publish it with crew at some point, uh so you can download it run it on any namespace. So the idea is, you would run it in a namespace as a user, so that those were the two two parameters you just give it a name space and what role you want to test with and and it'll do the checks.

G

Yeah, because I'm thinking we probably can also use it to run it against the actual cluster, if some rule probably not apply, but most rules still apply. We can get some insights from that.

E

Okay, yeah, that would be.

B

E

Have you considered running it inside the cluster, like maybe as a an hourly process,.

B

Good good question, so we were so one option. We could follow uh something similar to what coupe bench does is right. You know they just containerize the command line and run it inside the cluster, so we could do the same, and since since mtb can also generate policy reports, uh it could also be run as a cron job where periodically it would just generate policy reports right.

E

So I know that um I don't know if gatekeeper has audit mode, but I know they were thinking of adding it so yeah some kind of audit mode, basically right right.

B

Yeah not sure if gatekeeper does kiverno does so, it scans does background scans. I think gatekeeper does mostly admission controls at this point yeah, but the but mtb can run as a separate cron job and report results.

E

Even better, if it could detect the kinds of changes that might cause something to go from pass to fail, or vice versa and run.

B

Lock that yeah.

E

I'll not necessarily block that that is more than the job I would imagine, is something like gatekeeper, but like at least detect that something relevant might have changed. We should probably run the thing again and right. Some mirror cool.

B

Yeah, so maybe every time a new namespace is created, for example right uh it, it's possible to then run the benchmarks and report for that namespace.

E

Yeah thanks. That's really cool.

B

B

All right, yeah, great job onwards, tivia, and I think you know the next focus will be in making it easy to add new benchmarks and then we'll have to we'll go back and re-review the list of benchmarks and tests that we have.

C

Okay, so there was a some error like given a policy: do you want to show again.

B

No, that's all right, we'll, I think in for sake of time, we'll you know move on with the rest of the agenda. We can demo in a couple of weeks again.

E

I should probably have a chat with you, jim about um or the other members of your team about uh seeing how hierarchical namespaces interact with all of this as well. I know that you asked a question or two on uh on slack, but um I haven't really thought about it systematically, but it was I enjoyed watching the presentation, so I get a better idea of what you're up to.

B

Yeah, definitely, I think we should, let's you know both for virtual clusters as well as agency. I think we should see how we can kind of uh tie the threads together on on. All of these sounds good.

A

Thanks cool, so uh I guess next up is adrian's update on hnc.

E

Sure so I just wanted to give a quick verbal update of where we are um so, uh as some of you may have seen if you're watching the, if you're watching the repo we have a new contributor called named jenny.

E

I don't know if she's on the call today, but she is, uh she has been working on taking all of our terrible end-to-end bash scripts in the hack directory and turning them all into a suite of automated tests that we can run easily and I've been using it to qualify the latest uh agency release, which is 0.5.1 so a lot of hard work on her part, which has saved a ton of work for me because uh for the last release it took me many hours to run all of the tests and verify the results manually.

E

uh So that's one really nice improvement, which is already paying dividends other than that uh ichi is working hard on the results of our api review. So basically we did a an api review with jordan. Leggett, who is this is one of like passing. The api review is one of the requirements that we have of graduating from the incubator directory into our own repo, um and I believe that the the virtual cluster uh project is also looking at a going through a similar process.

E

The security review is the other thing that will be required after that, so she is working hard on that and our goal is that we're going to hold the next minor release of hnc, which is 0.6 until we have that until we have that new api in place, so that will be v1 alpha 2.. uh We will have a one-way upgrade process so that anybody who's using agency now.

H

E

Has a bunch of crds of our custom resources in the cluster we'll be able to install the new one? It will automatically update all of your existing crs to the new format. You will not be able to downgrade, but it's a fairly easy process to backup your cluster. If you want to ahead of time um and so she's working hard on that um other than that, as I said, I'm about to release zero five one, it's got a couple of um minor bug fixes on top of 0.5.

E

um Basically, uh it makes it harder to delete an object by accident that you shouldn't be allowed to delete uh and then the only other news is that we've actually integrated hnc at google we've integrated it into config management and config sync as a beta product.

E

um So if you, if you, uh if you're using either of those on gke, you can go in and turn on what we're calling hierarchy, controller and it'll install agency in a couple of integrations, so the uh config sync and acm have actually adopted the hnc tree label model so that you can use your uh you can use a common way referring to the hierarchy. That was in your git repo, which wasn't instantiated on the cluster as any um namespaces that you do create on the cluster that weren't available again.

E

So that's the update for hmmc any questions from anyone about that.

F

Can you can you uh briefly talk about the api review, so it sounds new to me. So what is the process? Oh, um I.

E

Apologize, I think it went out on the slack channel, but we may not have sent it out on the mailing list. If you you go up, you should be able to see the results of the reviews so uh yeah. Basically we went to. I don't know how you become an api reviewer. To be honest, um being charging is probably a good qualification, uh but he and mike dennessy went over the v1 alpha one api uh and looked for all the kinds of any kind of stylistic inconsistency.

E

So, for example, we had enums that were starting with lowercase letters, whereas the the standards that is the facebook uppercase, um we weren't using a standard condition format which apparently um we're all supposed to be moving to uh standard wave reporting conditions.

E

um And then there were a variety of of problems mainly on the reporting side, like our specs are pretty much unchanged in alpha 2 as they weren't all for one, but the stat a bunch of the ways we report the status is changing um so yeah we basically just went to sigoth. We asked him for a review. He gave it.

E

um It was the the attendees were myself uh ryan, attended, um ichi and and jordan and um and mike, I think, were the main attendees and you wrote up a report about what's changing and I believe that got sent to the slack channel um ishii. If you're on the line did you send that out to the mailing list as well or just slack or someone else,.

A

uh I think it's in includes the it's linked in the dock.

E

Yeah, I can see it on slack and it was on the mailing list as well. um Why don't I I'm gonna copy the link, address and paste it into the chat.

F

That sounds good to me. I'm afraid you know they asked to change too many specs api that would be trouble to me or because I I was afraid you know they asked to change quite often fields in the spec and that would be problem.

E

Yeah they didn't ask this to change, certainly very little structural, um nothing that a web hook can't handle uh and especially because we're still just at the alpha level, we actually got the advice that we shouldn't even be bothering with my books. We should just make people upgrade um by hand, didn't really like the the sound of that myself.

E

So instead we decided to sort of split the difference and go with a one-way upgrade conversion web book, which is really not what you're supposed to do, but is, but we felt was better than having no automated upgrade tool at all.

E

Okay, okay and we've discovered all kinds of entertaining um dependencies between um uh between cluster ca, bundles for validation, web books versus crd web books. uh Yeah. If you want to know about web hooks, talk to ishii because she's discovered a lot of fascinating stuff lately.

F

Okay, that's good thanks.

E

Thanks anything else, uh any other questions for cnc, not, I believe, josh is next on the agenda.

H

Hey there, so if you look at the agenda document, I've pasted in the results from our what should we call a group of kubernetes clusters survey?

H

um Unlike our internal survey, this one actually has a pretty clear favorite um cluster set won out for the aggregate scoring because it was loved and liked by more people and hated by fewer people than any other name.

H

um It was also 50 of the respondees of the hundred. There were 112 respondees 50 of them picked cluster set as their favorite.

H

um I did look through the additional name suggestions we put a blank in there of what terrific name did we miss. um I didn't see any winners in there. There were a lot of suggestions for names that we already disqualified, particularly cluster group. um The only new ones were cluster pool cluster fleet and cluster array.

H

um From my reading, none of those are compelling enough to disregard the sort of clear mandate of the survey. um There were a couple of blockers mentioned, but those blockers do not apply to uh cluster set.

H

So it kind of seems to me that cluster set is our clear winner here. Discussion.

B

So josh, what? What is the context of this? Is it.

H

For what do we? What do we call the object? That is a group of kubernetes clusters that share the same name space, since that is a thing that we can do now, at least in alpha.

E

I think this was discussed a lot in the sig multi-cluster uh group. I don't think it's been discussed a lot in the multi-tenancy working group. uh Oh.

H

I'm in the wrong meeting.

E

I mean it is relevant because the the.

E

Attribute of tenancy because all of our tenancy models use name spaces, except for perhaps virtual clusters, which are a bit of a special case. So it is maybe interesting to have this idea that, across a it looks like cluster set, is the winner yeah? You could have a common set of tens that are using multiple clusters, either across regions or availability zones, or other things like that.

E

So, even though, um even though I think this has been discussed here is probably a good reminder for people on this group to to keep an eye out on sick multi-cluster, because there's some interesting stuff going on there that could yeah be involved into us.

H

Yeah I mean to put a background right multi-cluster. The actual implementation of multi-cluster is a number of different concepts that are actually being implemented separately right.

H

There's network sharing, which there are several different tools for um there's job sharing across clusters, which is the old federation approach, um and it's still a thing, and now one of the new things we've gotten is namespace sharing right, as in um uh you can um say that this particular namespace is going to be and its objects will be shared across all of these clusters and there's a reconciliation loop to make sure that the objects in the name space exist in all of these various clusters. But we needed a name for that.

H

The so what happened? Oh, I see what happened so paul canceled, the meeting for a sig multi-cluster, and this shows up in my calendar as multi-tenancy and but because the calendar so small. I only see multi.

E

Yeah, I sometimes call it multi-x when I used to attack.

H

Yeah so anyway, so you know that's what that's going on that hasn't been announced yet, but when we have our next monster meeting, the survey is over so we'll be picking that as a name um the um and incidentally, if you all need to run surveys, I've become the sort of community person for um manipulating the cncf uh surveymonkey instance so feel free to ping me on slack. If you need a survey.

A

A

Cool uh well thanks everybody good meeting. uh We enjoyed the naming update, josh thanks for thinking of us and yeah I'll post this to youtube, and if anyone has anything they'd like to talk about at our next meeting, please add it to the agenda.