Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2017 (Austin), 15 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Securing Cluster Networking with Network Policies - Ahmet Balkan, Google

Description

Securing Cluster Networking with Network Policies - Ahmet Balkan, Google

In a secure microservices cluster, you should only have the pods that need to communicate with each other to be able to establish network connections, and block all others. But how? Until recently, Kubernetes users could not enforce policies for container networking.

First introduced in Kubernetes 1.3, Network Policies are now a stable feature in Kubernetes 1.7. In this talk, we will discuss use cases for network policies, the Network Policy API, how to configure network policies, and how the configured policies are enforced. We will also present some network policies that address some common use cases and are relevant to securing your Kubernetes clusters.

Also, we will discuss the roadmap for Network Policies feature, other methods you can use to secure applications at network and application layers, and how Network Policies relate to service mesh projects such as Istio that offer similar functionality.

About Ahmet Alp Balkan
Ahmet is a software engineer at Google Kubernetes Engine, working on optimizing the developer experiences. He creates developer tools and tells stories about complicated features.

Previously, he has worked on Microsoft Azure on projects like porting Docker to Windows and Azure Container Registry.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Alright, let's get started, my name is Ahmet Balkan I work at Google cloud as a software engineer and I'm here to talk about Korres, Network policies. So one of the things that keeps me interested about this feature is its simple, so I hope many of the beginners here, if there are any, can also understand this without knowing too much about currents, but I hope you know some amount of coronaries. After a talk, you can come pick up some Google cloud stickers. We are actually have a pretty cute logo.

A

I want to show it much better than they won our there and we have some GK stickers if you're interested in that. So as I said, I work on I work at Google and the Google query. Engine team and we're gonna be talking about they're, never policies today and me and my agenda is like what are the network policies? How do you write them? How do they work?

A

And hopefully the port item is on you: go home and actually start using them, so network policies control traffic coming out of the pods and coming to the pods. So this is a construct that manages traffic at the pod layer. Like you basically say that I want to. You know, control what comes to this port and what goes out of this cloud network policies. Let you do that yeah, so who's using Eric policies already really curious. So for the record there are like less than 10 people out of a full room. That's amazing!

A

Alright I hope you were gonna, learn something today. So without the crew network policies, your clusters are like. Actually they don't know they're on fire, but you know I try to do the this is fine dog meme, but it didn't work because copyrighted anyway, your clusters are actually not fine. Just so you know, and so the network policy API is started around the work around and our policy has started around late 2015 from mainly the folks interested in the networking space.

A

I'm bringing you know, closer network integration through kubernetes- and you know, as you can see, it took quite a long time for the API to actually evolve. We have some couple seats up here. If you want so, the API went through some stuff period and that period wasn't necessary, because if you actually look at the initial proposal of the API, it's completely different than what we have today actually want to say.

A

Thanks to the people who who contributed the feature a lot, namely KC Devonport from Nigeria I'm, like Christopher Luciano from IBM and danwon from Red Hat, these people made the future happen by the way I did not work on the feature, I actually liked the feature. So much. That's. Why I'm here to tell you about it? But yet these people actually did all the hard work. So it goes. You know, I, guess the things go to them, so the feature has been stable since 1.7.

A

It means that this API is in your careers clusters, if you're using 1.7 all about let's talk about in their writing. Network policies like how do I write one right. So first I need to talk to you about labels. If you don't know what labels you know, every part and communities better have labels, it's not mandatory, but this is how we do things in currents, and this is how pretty much everything in coronaries works.

A

So labels basically associate some metadata with the pods, and then we have label selectors which can select a set of pods using these attributes- and you know in this case you know in on the right. You can see: I have a label selector that says mesh labels at equals shopping and, in this case, I selected these two parts. So in this case I said you know, choose the database tier and I selected those two pods on the right, pretty simple right.

A

So the next thing is what like the empty selector is that there is a thing and I want to. Just like ask this question: what do you think the aunt the label selector is going to select who says it's gonna select, none of the pods to two people. Okay, who select it say, likes everything perfect. This is amazing.

A

Alright! So let's talk about you know. What's in the network policy like how do I write a network policy, so first you got to specify which pause. Does this thing apply to you like? What is it control right and we do that by using label selectors or pod selectors, which can select using labels? So the second thing is: where does this policy apply? To is this for which direction right? It's like controlling the incoming traffic, or is this controlling the outgoing traffic?

A

So in this case we call the outgoing traffic egress and the other way is ingress and what are the rules like for? Let's say, an ingress room allowing traffic to the pal who can connect to the spot, so we got to specify some sources that are white listed to connect to this pot. So this is roughly how the object looks. Like you know, it's in the networking case that I obi-wan in API version is called Network policy and I scan all the metadata.

A

As usual, it's got the spec and in the spec we definitely need to have a pod selector, and we definitely need to specify a direction you can either specify ingress or egress or both and those can have multiple rules. So the first rule of network policy is the traffic is allowed by default unless there is a policy in place controlling that traffic.

A

So if you are selecting the pod from a Nanette from a network policy that is actually restricted in traffic, so if, but, if you don't do anything, all the traffic is allowed right now, if you use two rays and if you don't use network policies which is pretty much all of you, network policies are not restricting anything so the rule rule 2 is traffic is denied if there are pods selecting that traffic.

A

Those pods sorry if there are narrow policies, selecting that traffic, but none of those policies are actually allowing that particular you know source or destination. So I'll come to that. I'll come back to this in a moment, but I think this is important to understand from rule 1 and 2. I can say that you can only write rules that allow traffic like you cannot. If you watch the keynote today, they were showing Abe was at the keynote I'm trying ranbir I.

A

Guess it was the keynote they show they never policy which was specific to Tigers implementation. That's not the Koreans core network policy, but in those places in those in their api's they had a deny rule. You cannot have deny rules like you can say, I want to deny this explicitly in Korean every policy. So what you do is you only allow things so as an example. So this is denying everything. This is a policy object. It will deny all the incoming traffic to all your pods and the way it does is.

A

First, you see a poll selector right. It selects all the posits empty selector and then it specifies ingress means that this policy enforcing the incoming traffic and then it's not specifying any rules. It has an empty array in the ammo, and this means nothing is whitelisted. So once you put this policy in place, all the incoming traffic to all your pods from you know outside the cluster. From within the cluster, everything is blocked, so this is how it works, so the room 3 is traffic is allowed, if there's at least one policy allowing it.

A

So, let's say you deploy bunch of policies. You know you deny bunch of traffic here and there, but if you have like one policy that allows that traffic it'll go through.

A

So let's give more concrete example. In this case, in the middle I have a database application on the Left I have a back-end, which is you know. You can't talk to the database just fine, but the front-end probably should not be directly talking to the database. It should produce something else. So how do I write a traffic Network policy that enforces these traffic rules? The way to do that is I have a pod selector that matches to the database. First right, it's enforcing the rule on the database pods and then in the ingre section.

A

I have a single rule like if you just look below the ingress you're going to see that there's on a dash indicating this is a rule and in that rule I see a from statement. It means a lot of traffic from these pods, which is in this case it's selecting the backend pods and it's allowing the traffic only from those pods. So this is, if you deploy this you're gonna, get basically a system where they can just talk to them. Talk to the database, but not too big.

A

The front-end cannot do that, and in this case no other pods cannot talk to the database either. So we were basically blacklisting everything except the backend.

A

Let's take this example further and let's say the database is exposing two ports: one is port 80 and what the other one is like my circle, port 3306, so you don't want the port 80 to be accessible. So you just take the same rule and say: I want to exit I, want to add these ports explicitly, if you say poor story or a PCP, the back end will be only able to use that port. If you don't spice, don't specify any ports, all ports are all from open by default.

A

So this is like further adding that you know a little bit more security in terms of you know, whatever minimal access is needed in this case, the back end only needs to access the port 3 308. Then you just give access only to that port a little bit small, not about the port numbers. You can't currently use the service port, so you have to specify the actual plot numbers. I, don't know. What is it why this is the case?

A

Currently, all the network policy implementations are just going this way, except other, like the spec is not saying that, but it's, this big is not blocking that either. So we'll come back to how the network policies enforced, so I'll have a chance to talk about it later. So we talked about it. We said that you know all the policies are. If there's like one policy allowing the traffic, the traffic will definitely go through right. So let's talk about like how the policies are evaluated.

A

So if you, if you have a bunch of let's say, ingress rules in your network policy, basically, if any of those match the traffic, then the traffic will be allowed. So it's not like you know it should match this room 1 and then it should match this rule 2. It's not an end statement, it's an or statement so to give a I guess a little bit more concrete example. In this case, I have a single ingress rule which has two pod selectors.

A

So if any of the pods are matching volunteers condition one of these pod selectors, then they will be able to get in. So these are combined with an or statement, not an end statement. Similarly, I have two rules here right. They I have two from roles, and this is this has the same effect as the previous one, by the way, just because everything is- or in this case the rule, one is ordered with the rule two so you're at the end, you're getting the same effect.

A

So let's talk a little bit about empty selectors, because you can do cool things with them. So I want to say, allow all the monitoring pods to connect port 5000 of all the pods. So I come up with a very simple network network policy. Spec like this, so this spec says you know, select all the pods and then you know allow port five, a lot of traffic to port 5,000 of the role equals monitoring label right. So does anyone see any problem with this at all.

A

A

He has anyone Canada container with them role equals monitoring. That's true, but that's not that's, not gonna, it's not what I was trying to get it so the this is this works only if the monitoring pods and the spa's you're selecting are in the current namespace, if you're monitoring part is somewhere else. This is not gonna work, so the reason is that the pot selector is selecting only pots from the current namespace and the never policy object itself is an it. Namespace scope object, so you have to.

A

If you have a network policy, you have to deploy that to every single name space. You have to enforce the same kind of thing. You know to illustrate this better, like let's say: I have the same setup in both namespaces. You know about and below the the one below does not have any network policy enforcement, because you did not deploy your policy there, so you have to deploy your policies to every single namespace you have so then, in this case, how do we allow traffic from the other name from other namespaces?

A

Because you know the pod selector is matching only there. You know the local namespace it is deployed to so we need a new kind of selector in this case, which is the namespace selector. It's like the pod selector. You know it still uses labels it, but it selects namespaces by the way who is actually labeling their namespaces I'm really curious.

A

Some people, like very very few.

A

Okay, so the gentleman back there said we only only label our namespaces, because never a policy needs them. That's that's cool, so yeah that they came up with this thing. So if you want to use up with namespaces, you told the label your namespaces, so I guess, let's give an example.

A

In this case, I have two namespaces, let's say: foo, Pradhan, food storage and I want to. You know, give access to my apps in prod storage, so I basically specify that I want this namespace selector, I specify label purpose equals prod, and then I basically allow traffic from all the pods in that namespace. So there's a problem with that.

A

If you, if you use namespace selector it does not, let you specify traffic selectively from another name space. You either allow all traffic from that particular namespace or those namespaces or you don't allow any traffic from those namespaces. So you can't do what what I just illustrated in the picture you can't print. You know we can enforce traffic to selectively different parts because the rules are applied with or once you allow traffic to all of the namespace. You can't reduce it back so so far we learn about.

A

You know three ways: we learn about two ways to specify where the traffic can come from there's a third way which is called IP block. So if you don't know what I see I see the IRS, it's a way to specify IP ranges. So I put a little diagram, I put a little table on the left. It shows you, you know if you do 10.000 slash eight. That basically means all turned up something IP addresses.

A

So that's what CDI CDRs in the example on the right, you can see that I have a from rule and I'm in this case, I specify a CIDR range and I. Basically, traffic allowed traffic only from there and in the example below I, say: ok, exactly a lot of traffic from that IP block, but except this particular smaller block.

A

So that's a feature: let's talk about the egress rules, so egress rules are pretty much the same thing as the ingress rules that allow traffic to your pods and egress rules and forcing traffic to whatever you know is going from the pod to outside. So you can send traffic to some places, but not everywhere. This is actually pretty helpful and it's as helpful as the ingress rules.

A

These are very simply added in kubernetes 1.7, if I'm not mistaken, so here's the same example for the one we have with ingress: how do I all the egress traffic so in this case we're looking at selecting all the pods again different than the ingress one. I specify your policy types and one of the elements in there is called egress. You have to explicitly specify this.

A

If, unless I'm mistaken, it's just the thing, if you do with the egress, you can also expose to specify the ingress but I think it's inferred because of the api compatibility reasons. So in the egress section, I specify again an empty set of rules. This means I'm allowing any traffic to go out right. So that's what it is. So this is a more concrete example.

A

You don't want the front-end instance to call anywhere except the back end, so I mesh to the front end and then in the egress section I say you know, go to the only tier equals back. End pause, so this is enforcing that policy, pretty simple!

A

So if I actually go into the front end pod, you know if I do like QC, DL, exec and I do like pure HTTP back end. It says, cannot resolve host back-end, didn't I, just allow access to back-end ways of giving you that error. Does anyone have any ideas?

A

Dns? Yes, so this thing turns out actually blocks cube, DNS or DNS resolution in general, so you have to either allow all the DNS traffic or you have to figure out the IP address of your queue DNS service and give access whitelist that particular IP address in this simple example: I do basically just added ports here, it's a new rule. It's like the rule. One is that the one at that accused access to the back end. The rule 2 is giving access to ports in this case, I don't specify a destination.

A

So all access on port 53 can go out, can go to internal IPs can go to external IPs, but in this case I'm giving access to both UDP and TCP or port 53, because this is the DNS traffic that we want.

A

So you know more fun with egress. You can actually block the external traffic and limit your cluster traffic to internal stuff. Like let's say you have an app, you definitely don't want that app to establish connections to outside your cluster. Like you, don't want it to go download packages, you don't want it to go. I, don't know, send your passwords to somewhere else or something. So this is how you do it and in this case, I'm again much matching to all pods, but in the two section, I'm selecting namespaces but I have an empty selector.

A

So this is selecting all the namespaces and all the pods in them, and it's not selecting anything else. So this is only restrict. This is restricting traffic. Only two pods other pods in your cluster.

A

So that's useful: let's talk about how the network policies are actually enforced, so I have a surprise. If you don't have a networking policy installed in your cluster, you can go home, deploy these rules and nothing is gonna happen there silently ignored like that was my little surprise. I thought it was just kind of magically Yorick, so it actually does not work out of the box. You have to install something called something like calico, Venus or Romana I, don't personally and there's anything but I'll come to that later.

A

You need a plug-in to enforce your network policies, so in Google play's engine we use calico and it's pretty simple to create a cluster with network policy enforcement. We basically just say gee called beta containment clusters create enable Network policy. So this is installing the Calico networking plug-in to your cluster. This is something that calico and Google manages together like if there's any problems with your network policy pods. You know we know about it. The uncle comes and fixes it. So the Google series are on this.

A

This is like you, don't have to install and manage this yourself. We can still manage that for you and if you got a GU club back google.com slash Carini's engine, you can find more documentation about how this works so more on. Never policy enforcement, so you gotta, know the connections or duplex like meaning they're two-way. Once a can connect to be obviously be, can con, we can send data back to a on the same connection. That's how HTTP and everything works right so, but does not.

A

That does not mean B can't connect to a so don't Phunk, don't think of it. This, like I, don't think of it like once you have a policy restricting the outbound traffic. You cannot send any packages ever. That's not the case at all. This is pretty much how you would expect I think it works. So the second thing is: never policy is a connection level filter. It's not a it's not apply to packages. It's a.

A

It applies to connections that are established so after you start I guess after you deploy a network policy that would block some connections. If you already have some connections in flight that are established already, network policy is not gonna kill. Those this is not in the specification some providers out there might be killing that, but the spec does not say anything about it, so don't expect to really kill, but any new connections established after the nerd policy takes effect. Those are going to be blocked and they look like they're timing out most of the time.

A

I guess that also depends on the implementation.

A

So I want to talk about a little bit about you know what are the performance implications of using network policies because I like when I was using that I'm like this definitely changes the performance because I just think you know they definitely create like an overlay network and like they a bunch of rules in there and they really ate the rules anytime.

A

There's a new pod comes up of orders, and even every policy comes up, so I thought it probably you know, adds a lot of latency, so good things that the Romagna folks actually publish the blog post about it, and the blog post basically says that you know we deployed so many thousands of rules, and it only affects like only a little and that's like during the first time to bite or something. So it's not a big deal and I guess this also depends on the implementation. Some providers use IP tables.

A

Some just establish an overlay Network on your network, so also depends on how they actually implement the algorithms like. How do they do how they do caching and how they react to you know how the pods coming up and so on so because, basically anytime, a new pod comes up. You have to update your entire cluster with the new set of IP tables rules, so there's a little bit of work there. So, let's talk a little bit about best practices. I showed the default, deny rules for ingress and egress I think those are really important.

A

The reason they're important is, they I think establish a baseline because you're blocking everything and then you're whitelisting one by one, based on the need right, if you're developing a new micro service- and you know where it's definitely going to talk to you're- establishing that network policy, that's allowing that traffic one by one, so you you're, basically starting from zero and building up, as opposed to you know, having everything able to access each other. That's not great, so I think this is one of the best practices.

A

The second thing is, you have to understand how the rules are actually evaluated.

A

It's always good to be explicit about what you mean. One of the problem is that like I right now, I am up here. You know I'm, giving this talk, but I cannot confidently say the difference between those four things. I can tell some of them, but, like some of them are blocking all traffic, some of them are allowing all traffic and depends on. Where do you know the empty object is specified. It can allow or deny all traffic. So you should be. You should be careful about what you're doing, which brings me my next topic.

A

You should test your policies and once you deploy a policy, make sure it's actually doing what you expect it to do and test it with like different conditions like test the external networking tested, I, don't know kind of excess DNS. Can it like you know, go outside the cluster? Can it access another namespace? Can other namespaces access this? So if you try all these things, you'll have a better understanding of how the feature works as well.

A

Also, the I think, if I'm not mistaken, kubernetes 1.9 is going to ship with a better qct I'll describe printer, which is gonna, explain. The network policy is much better than today, so that should actually help the so I've been working on some narrow policy tutorial repository, it says tutorial, but it's not really a tutorial. It's like your recipes repository. So if I figure out like a you, know, cool thing to do with narrative policies, I just put it up there and you can contribute to it as well.

A

I have some you know drawing some diagrams explaining how everything works and I have the policies you can directly download from there. So there's that, but you know never policy is not the end game. You know you. There are already several Network policy implementations in different parts of the stack right. It still claims to be doing some part of it.

A

Some never policy providers are claim to doing what SEO does, and you know everything everybody is like attacking one layer above and below so I think that is fine, but in this case you know, your applications have larger concerns such as. How do you the authorization? How do you do part identity? You know, how do you do? How do you make sure you're enforcing the right set of policies like I think these are these? Are these can be used if additively?

A

This is more defense in death, like you can use steal, but you can also add the network policy on top of it. It would just work fine. So this is not the end game. You know this is not going to give you the this is silver bullet. This is pretty much my talk if you're actually interested in the narrow pulse recipes. Repo, definitely check out that repository. You can find my slides also on the talk page. So if you wanna give a try to the coronaries engine, definitely go. Try it out.

A

I think we have a pretty good free trial and if you want some stickers they're here, if you have some questions, I think we got actually some time well, I'm impressed yeah, please go to the microphone. Thank you. I.

A

Have to repeat it, if you don't say it to the microphone.

B

So I have a question about egress policies. Actually so cuz.

A

Repeat that I couldn't hear about.

B

Egress policies, so if a pod accesses a service IP address, basically the question is what is supposed to happen, especially if the egress policies allow a subset of the pods that could be accessed by that service. If.

A

You're trying to access another service I would just copy the labels of that service and edit the egress route. So.

B

I'm from a just from a spec perspective yeah, you know if you're doing an implementation, hypothetically you've never yeah. What would what what what is actually supposed to happen? That's.

A

A good question, I'm, not sure, actually I mean. Would you whitelist the pod IPS? Would you whitelist the server side piece I.

B

Mean it's: it's actually pretty awkward anyway, because at the level where you're applying the network policy, you are you're, seeing just the service IP. So it's not really clear if you're supposed to be pulling information about this, the service endpoints into the network policy and applying it at that lab level or it's it's problematic in a way. The ingress policy is not.

C

B

Definitely won't work, but the question is like what is it supposed to do? For example, you can imagine I intersect the labels on the service with the labels of the network policy and apply only to the subset of network policies, and then it would have to be integrated with the load balancer. So the load balancer would only load balance to the right set of pots. I'm.

A

Not sure why it wouldn't work with the cluster IP directly.

B

Because the the cluster IP is not an IP address of a pod that is selected by one of the network policies, because the network policy applies only to pods, it does not apply to services. That's.

A

True yeah I, honestly I, don't know how its enforced on the services yeah, I would say, sticking to labels will probably solve it solve it. Oh that.

B

Is the labels are what Scott the problem is because the latest like I, guess that I'll have to maybe bring it up in and Signet workers, probably.

A

Yeah sinking, that work is definitely a good place as these questions. Let's go ahead so.

C

I think if I heard it properly when installed given it is, it doesn't have a network policy implementer quite default right. It.

A

Does not have a default implementation now so.

C

What's the reason for that, that's.

A

A good question, I think the reason is because everybody has a different way of doing networking and every pretty much everything every platform you're using is using a different networking model and there is no single implementation to rule them all. There are ways like the way you live and with bare metal versus clawed. It is wildly different, so I'm guessing that's why also? The current implementations are offered by commercial companies, so I think that's one of the reasons why there is no default. Yeah.

C

But that doesn't quite resonate if it's a part of the spec and then you write it in your EML files. I.

A

Find it interesting too I agree like there is it's a core object and there should be probably a reference implementation that works on, say, mini cube, right, yeah, I, agree with you, yeah.

C

A

Any other questions, yes, I will repeat your question.

A

You would probably yeah so the question was what, if, like I, have like a matrix system like Prometheus and deployed in I, restrict the namespace and I have like a multi-tenant cluster, where everybody's got their own namespace and the customer wants to access that their own metrics I would deploy I, don't know my own custom solution or multiple Prometheus instances. I, don't know.

A

Problem every wall policy will help you yep. Yes, that question.

B

A

Is that as so, the question was, did you have any question? Did we have any discussions about cluster level network policy? What does that mean? Like applies to every pod or something.

A

Yeah, so that is not part of the core API I think the private providers like calico they're, enforcing that in their separate implementations and API, so I think that's possible with their api's, but I, don't think, there's anything going on with the Koreans core API right now, yep.

A

Okay, so the slides are actually on the event page. You can go to the event. Page click, the event and download all the slides, I'll be also tweeting about them. Yes, question there I use iPad just with a pencil you all right. Thank you. Thank you all for coming appreciate.