Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive: Kubernetes Policy WG - Zhipeng Huang, Huawei

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Deep Dive: Kubernetes Policy WG - Zhipeng Huang, Huawei

The deep dive session of the Kubernetes Policy WG will introduce the newly formed CNCF level WG and Container Policy Interface we will be kickstarting. The main agenda is to discuss - external system policy enforcement - common policy description

To learn more: https://sched.co/HaH4

A

See working group deep sigh deep time session, we are a small, persistent, sometimes passive-aggressively, schedule working group, sometimes no meeting at all, but sometimes working a lot of stuff, so Erica and I, and also with the help of East Way, will walk you through. Like overview of the working group and some of the interesting cases we are looking for in the next year. Ok, so a quick overview, so you was really thrilled for us to see.

A

Brian G mentioned policy as part of the big picture for next year for Cuba nineties, so well from his world in region kubernetes as the policy framework. Of course, there are many choices and problems that we need to address, which Erika will talk in depth later well from ourselves. Of course, we'll find policy like are needed and also designed like scattered around the humanities ecosystem policy.

A

Descriptions are domain-specific in nature, so it's not only in the sense that Brian mentioned you know in the humanities domain, but also in a larger context like, for example, the utilization of the policy, whether it is used for audit security network or like vertical industry adoptions. If it is you use in finance, you can need many financial jargons or using telco. You know pharmacy work or whatever and also languages right.

A

Usually we treated like out of scope for work, our tasks, so it would be great to see maybe someday there emerged a unifying policy. Language might be great, but not core work here. What we are focusing on is like policy semantics or the control mechanism. We believe that could be universal in general generally described.

A

So our mission is like try to avoid you know to rerun many of the views or available and try to provide a oral architecture that could encompass both the past and the current and the future policy related stops in communities. So just give you a sense of the scope as like common in defining cuber natives working groups are something that works across the six.

A

So basically we are touching upon policies that beyond you know so many six, for example, sykov scheduling will be like storage. You know you can do policies for like snapshotting. You know, given certain time interval to do that also network, not only in the sense of network security. You know, but also like Sdn controllers, you can define like which way you want the packet to flow right and also domain-specific architectures.

A

We have so many of the things now for AI, which could be also policy driven and also multi-tenancy, and our output is something like those right design document we have. Maybe you have new policy proposals Erica. We talked about the unified format for next year, so this is what we do basically a little bit history, so the discussion started almost a year ago and we start the momentum this year. I think so can believe is already almost a year. We have weekly meeting.

A

We try to have weekly meetings, so we try to have a science of a sense of development cycles. So basically our cycle is like half a year, so we call it's like summit target, so we corresponds to like two cycles of communities.

A

Okay, many other things we do is to have a running list of our interested items. So this lease is for the fall cycle. Basically, one point 12 and 13 for communities, so, as you can see so we have like issue is work on the security profile and many the sub KPS discussed and in issue like alpha implementation and for the 600 I think, and we have something expect to design conclusion in 13 services.

A

Technology achieve that scheduling, policy I think also, but need to migrate to KP here, I yeah, quick, which is great name space, initializer part of the security profiles to in discussion and also long term. We just came to Iowa like metadata and Q batch since you have to treat the batch ones and number ones so there's a co scheduling policy in place.

A

Okay, new items, so I'm gonna invite you. So you have to talk about the new multi-tenancy use case.

B

Thank you. My name is su a software engineer working for Google, so this particular use case is brought up by send you from Cisco regarding the network, the policy to be using in some sort of monitoring environment, so Angie with parentally know so I will describe the use case brief phrase.

B

So the idea is, in certain martin say use case when the namespace administrator has full control of all the policy objects inside namespace, and they will set up a baseline of the network policy to block certain traffic between the namespace and also like outbound and inbound traffic. While the element want to grant the developers the permission and allow them to define the firewall rules by themselves and particularly for their own ports with, but they want to, they want their rules will be restricted by some ground rules. They cannot a break based on today's network policy.

B

It's impossible to do that, because either you don't allow anybody to write a new network policy or you you allow them to write a network policy. Basically, they have capability to overwrite anything in the existing system, their policy. The reason is when there are multiple nella policy rules appears they are aggregated to use and finally use the most relaxed rules and that's the basically problem and in practice there's a few ways.

B

We can do that, for example, and old-school way, is the developer write an email to the network, Adam iing and say a bunch of justifications say: can you open this port for this port, but that's less? That's just automated and the other way is kind of like if the canary cluster is Mary by a configured code.

B

That means when you want to know something all the manifest files are managed in a gated repo and use send out a coded review say all this is the change of network policy can review that and then a prevent a commit. The still relies on human review process, not something like quite automated, so this actually bring up a problem in the policy space.

B

How can we define policy that can be monitored and Martin user-friendly, and this is still quite open. Question I see us and you became so. Would you like to describe your use case in more detail? Why, in a specific use case, is in your case.

B

C

Yeah, so as some of the multi-tenancy efforts are still at an early stage, we're looking at what are the different areas that need support from art, a tenancy and when it comes to network level, isolation starting off with namespace separation. Today, the way our back is set up for network policies.

C

If I use network policies to isolate namespaces, then I cannot give right permission to write permission for network policy objects to developers because they may inadvertently start using another namespace, but we do want them to have full Network policy capability because they want to do micro segmentation within namespace. So this is this is the thing yeah.

B

So just leave this open questioning, particularly for this policy type. This type of policy objects. So maybe we can have further discussion off plan if you have any idea or sorts. Thank you. Okay,.

A

So next our Erica is going to talk about the new proposal.

D

I think about policy a lot so stepping back of what do we mean by policy? It can be kind of a catch-all it. That seems like there's some kind of administrative thing here. It sounds like policy. The definition I'm kind of going with is deliberate system of principles to guide decisions. The important part about it is that it is a guide to make decisions. So it's kind of a policy has a meta-level to it.

D

Policy is not your configuration policy can be configured.

D

Configuration can be how you define your policies, but they're not the same thing: it's not authentication or authorization, while those can play in to it as well. It's like in the space, but it's not it's not just like a program. If that makes sense, we have a little bit more that we're trying to usually do with it, and it's not just security. It's another thing and if I would add another bullet point, nor is it just about compliance and government stuff.

D

So, what's painful about policy today, in terms of the format, cuz I got a lot of other things that are wrong with policy, but they're all scoped differently.

D

Defaults are specified differently, there's no common terminology, no common way that they're composed. So what we could do better some standardization, consistent composability and giving people the tools into that to do that. So the base concepts of what we need for policy is we need or the identity of who or what we're applying policy to. This is kind of the input to a policy. So an example is how we the container image.

D

If we have policies around that that applied, you know we can identify by tag or better by digest most of the time the identity we're dealing with and kubernetes is the resources, including, like the network, via the authorization steps we needed to know what the set of possible outcomes is will be.

D

If we don't have a clear idea, we can't guarantee anything correctly with policy, so we need to know if you know this is a routing if it's a network policy or something what it is that can like actually apply to the decision of what to do. The other thing part is the state. This is a hard one, because kubernetes doesn't deal very very well with this.

D

What what is the other information? We need to uniquely determine an outcome, sometimes so, for instance, if we break down for a pull policy, all of the things that go into should the cubelet. We pull an image this state about where it, what node the pot is in matters it spec matters, whether the that image is on the node, whether that always pull images, admission, controllers, enabled etc.

D

And, of course, we need the rules, so we need to make our mapping from with it's like a functional with the input identity, what our kind of state space is going to be and what the outcomes are.

D

So these are the parts that allow is kind to identifying consistent policy policy composition.

D

Conflicts arise when you have more than one policy that would give different decisions and no authority. One doesn't have authority over the other, necessarily so ways you can deal with it or make sure that there really can't there are formal ways to compose role bindings and work very well, because there allow only there's no like deny kind of other way that could make things conflict. It's ever things additive.

D

Did you topology for nodes? Do this as well? We can do what's called the lattice model, which is priority ordering.

D

This is probably one you're familiar with, and it's because it's really in the way, we think usually isn't this kind of thing that this has a higher rank than something else the other way. Sometimes, though this is done like very arbitrarily or who knows how they're sorting something and then applied, or sometimes there's kind of sort by how restrictive they are.

D

Only allow one policy per thing: that's one way you can get rid of any problems or you can kind of use a combination of these things to make something that kind of works. So the example I was thinking. It was priority classes, you have the user, the admin creates the classes and the the pods can kind of pick the the class, and it's that separation of concerns by having a named kind of class boundary.

D

So with all that in mind, some of the baby steps we would like to take to can have a consistent format, use priority, ordering explicitly define a priority on every object or policy object, mostly keep to admission controlling because it gets very complicated. If you attempt to try to do kind of are like runtime decisions, which means mostly, we keep it clear with allowed tonight very clear outcome, set mostly cluster scoped and then used namespace selectors. This we're basing this on one. What customers or the users seem to need.

D

They need the admin usually to be the one defining the top-level policies and they don't want. They want it to apply to even namespaces that don't exist yet so using a selectors further, you can further kind of narrow down what applies the other thing to that? These are most. These are static policies as much as possible as kind of the example or the scheduling policy. We've been using it's kind of a way to talk about and think about these decisions. So we can see the priority.

D

The allowed deny the rules for namespace pod, where you are selecting on just key value pairs for like in in thei spec. It's very.

E

D

Sometimes we have you've policies on top of basically policies when you have other selectors and defining other selectors that gets more complicated to compose I. Think a little bit still of an open question of how best to handle that.

F

You'll probably want to make it a naming more consistence, the namespace selector, that is a dash below it, I think it's already right. You probably want to using blue robe and and all other things I think, and you want to make it a naming. I did a bit consistent and I. Think it's very hard to for those fall. Rules is kind of difficult to correlate.

D

Make sure I understood some of the names consistency.

F

D

ha Well, that is kind of a weird one, because you can think of it either way right, yeah, oh, wait! Tim did you ever the.

F

Center design is that you're talking to people who have no idea what it is and and see what their first reaction is. Yes,.

D

F

You are probably the expert in this area, but you want to make this work for readers.

D

G

Just on that specific one pot, selectors and namespace selectors there's a standard metadata label, selector object and it has within it a match, expressions, field and so I think this was just a sketch and and I think the actual API would yeah.

D

Yes, I should say this is nothing that there's no like alpha API that you have to worry about. You can speak and still fix things. Yeah, it's kind of weird selectors are set based matching operations. That's it my long-term vision of what would actually get us where we need to go huge majority of our policies are just about a kind of validation which could be defined and done much better with strongly-typed system, if we're also kind of with the latest in programming languages.

D

Having algebraic types allow us to more easily define complex resources as well as have things that can act on those and compose in a certain way. Yes,.

H

So how do you do that in the face of custom resources so.

D

Yeah, that's! It can go mostly on top of a of existing resources right. Think of I, don't.

H

Understand how you would have a strong type system when you don't know what the types are so.

D

No, their type right now is maybe not is somewhat weekly fine yeah one of the problems, this kind of like typescript they'll, be type any probably for a while, but we in general I think we you people would could we could start with like Union types or something like that with the base objects of like a pod or something and where, at the end, your output can still be the same. Pods.

F

The new version of OPA using this coming expression, language, the common expression languages- can handle both JSON and the schema. So.

E

F

Is that if you amp and JSON inside a schema and the expression can go deep into the the JSON right I'm, so the other one is that you can still.

D

The schema for custom resources and don't know how widely it's used so.

F

That means like, if that the trees, don't laser, has a schema. The policy doesn't know they need to change. You just say: a dot, B, dot, C and either basically chase down the past. What.

D

I'd like to kind of wouldn't move towards is having certain guarantees for different parts of the system.

D

If something needs to only you know, have a stricter number of allowable fields have a way to define that if we won't have different kind of admission chains where we can have them explicitly define the ins and outs in a stronger way, that's the kind of thing I'm thinking of I think it will fit in very well with OPA, because once you have the just kind of strong guarantees on what these types are, you can really use any language like a OPA to compile your policies in a reasonable way up we're going.

D

The other thing is where the policies get evaluated and put, there are a couple places we have like the the I call it lifecycle. You know the admission deletion kind of cycle there. We have a lot of like admission, hooks deletion hooks, not really it's kind of weird there's some kind of open question we'll discuss later on that one.

D

Of course, the traffic is one place, things come in and out there's we have like a BPF placed kind of insert certain kind of in a similar vein when pods start up and down like actually are executed and, of course, API calls web hooks often don't work well, the latency, the problem that if that connection is down you lose your like. Your system is like gated from you or completely open, so I think we should think about what would it look like to have a different type of plug-in mechanism within the API I?

D

Don't have a very good idea for what that could be, but use of capabilities. This is not the keep like capability like I, can access to an object tied to its to the resource itself, no kind. Okay. So like an example, just thinking pods kind of leases that make sense your we used to pass around access as to objects so that in a delegated way, constantly you've kind of changed and reevaluated.

D

That's when I need to flush out more I think we need to be explicit. Yo yeah.

E

On the previous.

G

E

Are you thinking that the access control is represented as some token that gets passed around sort.

D

It is sort of right now, yes and making that even more explicit in terms of how you can pass that around and scope the access and who has rights to it. You.

A

Know T's just passed down the capability.

D

Kind of yes, this, the the my pod leases thing was one of the things people ask me: okay. Well, if something changes like we just got new CVE we were able to. You know, stop a pod from being created, but now that we have new information, we want to not have this vulnerable pod running. When is the policy gonna be re-evaluated? This is always sort of a problem of when do you re reconcile with the current state? What how often that is, is a risk calculation really it's how what level of error is?

D

Okay, but a common part point of this is maybe no pod should ever live forever, but should constantly kind of have to be renewed. Their lease renewed.

I

I think that's one of the problem with admission since its admission time, once you pass the admission gay you'd have to reconcile about to reapply the passes, so yeah I think that's something we there is worth to explore because policies I my opinion are not something static. They move forward. Yeah.

H

Though I mean to a certain degree that like if you look at like our back as a policy right like if you can do something at some point- and you also happen to be able to like create role bindings at that point, you can give it out at that point, have your access for both, but the just stays around, because it's not it is. The positive decision is being made at a point in time and right. The fact that you can delegate it. It is also being made it at that time.

D

All right, then, the other thing is kubernetes is now I.

D

There's a kind of oh. Is this conflict between having a richer set of information that you can pull on and the way kerbin is is designed at CDs, not Postgres. It's not gonna work very well for like pulling out relational data, like policies can be if we have information kind of tied externally or to, for instance, signatures on images or or the vulnerability scanning results, there's quite a lot of external information that can be pulled in and that doesn't work very well.

D

So if we think very explicitly about how the flatten that this is sort of the compiled state, not the one that can show you exactly the human readable thing that it came from which we through kind of the typing I, think this can make it a lot easier. We can preserve a lot of the ways that people want to, like. You know, share components and such and be able to have share like share different cluster plugins without having to stuff a bunch of history into an annotation.

D

So what are like some actuals like next steps that you could we could actually do rather than some like. Well, that sounds maybe whatever okay number one I think it's everything in name space. It doesn't have to be a namespace so, for instance, the like an organization for groupings of users or yes, or if you have like now the zones note pools. The important part is every resource is in one, and only one of these the thing that makes like label selector so difficult and kind of hard to reason about is there.

D

It can be anything can be labeled anything with as many kind of labels is it if you want, which means you don't have a lot of ways to really kind of narrow down to a specific thing. Having a really this is it's the beginning of a type? In some sense, it makes it a lot easier to reason about these things, especially also when we start mapping between our different topology layers. There's the work load.

D

We have work loads, we have services in network connections, we have the data plane of most volumes and the data contained in those there's, the external access users and the API I might be missing another one were layer. All of those then can have policies and things that interact with each other. Very obvious example is scheduling. The scheduler sort of is a big policy machine that makes decisions about what you note.

D

Something goes on now your so we have this relationship and we trying to define certain policies around from the that was an infrastructure, the actual read machines, that's the other layer. We need your mapping between, like your machines and applaud or really the cube leading the pod.

D

This is if it can get very confusing, very fast with like taints and toleration and affinities, but I think most people would find it much easier if they knew like if the this is a master node- and this is a worker node- who was much more explicit and you knew it was going to be labeled one of those and you wouldn't be able to create it if it wasn't.

D

That can be very useful to make things also bad some way to keep some context kind of in stamps of approval or something so Chet, so maybe just immutable labels or annotations.

D

Rather there's been suggestion about like having access control on the annotation level having metadata objects. That's still a conversation to have I could see where you actually don't want anyone to be able to read the annotation but you're starting starters. If you just make sure that if something puts on the annotation, probably one it won't, it shouldn't collide and that the user isn't someone else, isn't gonna be able to stomp on that yeah.

F

Immutable labels or annotations, and can we just pick a one they.

D

Kind of would have a little bit different ways: you'd be using them. I guess yeah.

F

I know that yeah.

D

I would say annotations because.

F

I know, like every feature is used by someone and now the problem is that if some policy applied to label some apply to annotation and some will apply to both, then it's very different code. You know as an amine and look in the cluster I just very difficult to reason about it. This.

D

Is true, I believe the intention is more annotations containing more not like machine, readable categories categories but larger information. So, for instance, you could have a like verified label and a signature in like in the annotation right there, overlapping, though for sure the last one improved ownership, this is very I think would make things a lot easier for a lot of people if they could just make sure things are cleaned up and tied together properly, we can make kind of tries to see how to get ownership, cross, namespace, sure, API machinery.

D

People are going to tell me exact how hard that is and why it's never gonna happen, but we can push for object, pairs or something that kind of can put things together very easily. You know, I think it's kind of you shouldn't have to create a custom controller, just to kind of have something. That's one layer on top of a pot. It seems like we could help a lot of people build things by just you know. If you have. This thing is just tied to this one very explicitly.

D

All right that is all linked to I have a doc with some of this and I'll link to it in the slides. If for more discussion,.

A

So another set of issues that might be we might take a look next year, like you, will reverse sapote, delete for the external systems and also the answer guys has communities policy controller, which is pretty nice work with OPA.

A

We are also looking forward to work with them next year, so just feel free to join meeting, and if you have something want to discuss, just comment on mini talk and we have been try some of MIT talks. I think it runs pretty well like OPA Syrian. You know if you have a certain like subject that you want to discuss, just feel free to reach out to us. Okay, since I have wide, we have a steel information. Security sick depending on the two seas and I got many questions.

A

So there's a slight, although most of these same people but the same safe side. We are mostly a top-down approach because everything is so nascent. It's still possible to do top-down, but in communities we are mostly bottom-up I. Try not to you know, disturbed many of the things are working now.

A

Okay, thank you very much and any questions now being they'll be serving the Mike and Erica. Yes, sir,.

C

So this is a just newbie question, because I haven't attended this group before it seems like you're dealing with objects and features that are being spread in a number of other groups: art security policies, network policies, all kinds of other things right. So what is the mechanism this group chooses to adopt to either directly make proposals or influence those groups?

C

How what is the in.

D

General, it's not too hard to do that, just based on the kind of more democratic model of could realize itself. We, you know, keep an eye on the PRS and we're able to comment on those and really engage in discussion right then so I don't think. We've had too much of an issue with people will want help standardizing so I think we get people pushed towards us quite often. So it's pretty that's how I think we just do it keep an eye on the proposals.

D

Comment if to see if they'll kind of change a word do you'd be a little bit more in line with like the other policies, so.

C

All these proposals that you were running through are those some things that you would need to work with different teams that are dealing with that support security policies or whatever.

D

Yes, yeah. That's, for instance, we're doing scheduling policy. So we kind of hear like that's done in scheduling and there's input from both sides from the policy and from the scheduling side. And it's going through that sig of.

J

A general question about the effort for the unified policy types, so there seems to be a trade-off here that, yes, the more general-purpose your configuration is and more flexible. It is you lose usability right so or other way to say the more specific there's.

D

There is a trade-off, but I don't think it's a linear trade-off or curve at all, because if you don't make it general enough, people have a lot of stuff built around it. That can be more complex in a black obtuse. Then, if you had a better more generalized way, sure.

J

But like if you like, look at current are back yes, the types are very hard time, seeing how it can be simpler using a more general mechanism, but my question is more general than this is like. Do we have an opinion on where that balance lies as a whole?

J

D

Was what I was trying to kind of think about the flat thing.

D

That was where I was. My thoughts were kind of. If you have something very, they being able to express, it is complicated. Oh, but probably is a better choice, which means we need something that we don't have to mash. All the information together, like I, would prefer a little bit more for our back a little bit more squashed, no I, don't if there's a way to not have dead objects. That would be good.

D

I

To follow on the questions.

I

We could do something like having you know some resources at the kubernetes level and, at the back end having some policy engine that compute the policies. So we we can. We can have both and also one having something that is more generic, like, for example, one of the slide where we showed.

E

I

Policy, this allows us to be flexible because open users have use cases where they wanna, for example, a low by default, but some other users wants wants to deny by default. So having some mechanism generic mechanism to say you could override the default. Behavior is often helpful. I think we.

D

Didn't touch on defaults in that right now.

D

My normal answer for things like this is have good interfaces right and places where you kind of plug into that.

I

One question about.

D

I

Metadata policy I think it was merged back in 2016 or something like that, which.

D

I

Policy there's a design proposal that is yes, there's one that predates I, think which was merged back in 2016 I wanted to know if it was inspired- or it was a rework of this of this policy, or is this something that we want.

I

So there was if it's okay, if it's this okay, if it's this one, then it refers to the design proposal that was proposed by a by Brian back in the day, so I think.

D

See I think there's been a couple different ones right, it's hard to tell images or issues can stay open for quite a while. It's always fun when you're like wait. What's the answer? Oh no, there's a disk to hundred comment discussion thread and they didn't come up with an answer.

D

A

Okay, thank you so much and it's already like ten minutes past lunch.

D

A