Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Network Isolation and Security Policies for Kubernetes Bare-meta... Girish Moodalbail & Liel Shoshan

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Network Isolation and Security Policies for Kubernetes Bare-metal Nodes - Girish Moodalbail, NVIDIA & Liel Shoshan, Mellanox

Running Kubernetes at scale in a multi-tenant Cloud requires strong network isolation and flexible stateful security policy enforcement for the bare-metal nodes used for both the tenant K8s clusters as well as in the Cloud control plane. Such isolation and security needs to be implemented in a way that consumes as little host resources as possible, while being immune to potentially malicious host root user. Additionally, the preferred implementation needs to be compatible with a high-performance (offloaded) K8s CNI. This presentation provides an overview of such an implementation for Software Defined (SDN) K8s node networking, based on Open Virtual Network (OVN) and Open vSwitch (OVS)) and offloaded to “bump-in-the-wire” Smart NICs.

https://sched.co/Zetj

A

Hello: everyone, my name, is girish moral bell. My role at nvidia is to design develop and manage sdn networking for gp accelerated compute. The computer itself can be virtual machines, bare metal servers, kubernetes pods and each of them having nvidia gpu networks and gpu cards in them. I'm also upstream contributor and maintainer for oven, kubernetes cni project. I deal with the ovs in uh in my day-to-day work. Yeah. You want to go. Introduce yourself.

B

Hi, my name is leo: I'm a software architect at melanox, now nvidia working on cloud, accelerations, kubernetes integration and switching offloads.

A

Okay, great so we're going to talk about how to secure bare metal nodes um that form the kubernetes cluster using oven, logical constructs and bumping the wire smart nick and in our case it's going to be melanox bluefield network adapter. So this is going to be the flow of our talk today.

A

um We're going to kind of explain as to why deployers are running kubernetes cluster on vms, why they should be running the cluster on bare metal nodes, we'll have a quick overview of oven and we'll see how one control plane runs on the smart nic, how the data path for one gets offloaded to the hardware and we'll also kind of look at the other smart nic advantages.

A

So deployers have been um building kubernetes, customizing vms as nodes, and in this model they actually take an application containerize it they run the containerized application inside a part. The part itself is running inside a vm and the vm itself is running on the bare metal box. um So there is several layer of uh fencing if you will and- and the whole reason here is uh they want to protect the data center from uh from application, which could which can be compromised.

A

So the reason they do this fencing is because they're not confident about the container security, and the hope is that if an attacker breaks out of the container they're still trapped inside the vm, and then they don't act, they don't have access to the bare metal node and therefore they cannot cause any harm to the data centers itself.

A

However, um this comes with a huge performance cost for the application. A lot of the hardware resources like the neck, the gpu has to be virtualized, and this virtualized thing has to be passed down to the vm and the vms. The next pass down to the container there's also an hypervisor uh hold it for the application performance.

A

So the thing is: uh there's a trade-off: the trade-off between performance and security and violence and isolation, so the deployers kind of tend to err on the side of security, and they don't kind of focus more on the performance, because security is uh of paramount importance.

A

So that need not, however, be the case um with a help from bumping the wire spartnic and distributed sdn control plane. It's possible that we can get both performance as well as security and isolation, and this kind of moves. The zone, the trust zone from the hypervisor onto the spartanic hardware, and we're going to explain how this bump in the wire sparknet and the distributed sdn control plane enables uh achieving both performance at the at uh and not at the same time losing security in isolation for the bare metal nodes.

A

So um so, where we are going with this, is that um so with the smart, nick and bom and the distributed control plane, it is possible to push all of the axles and the firewall rules. If you will uh to the smart mic itself, um the parts running on the bare metal they're, totally unaware of this bump in the wire all the kubernetes network policies uh for the pod are implemented on the spartnic and then the accols for the host itself is implemented on the spartanic.

A

So if, uh if an attacker kind of jumps out of the pod and is now on the bare metal host, they cannot flush ip table rules or the nat rules or any host based firewall rules, because all of the firewall rules are on the smart neck, which is acts as a bump in the wire thing.

A

So in that way, uh you you can secure the bare metal uh nodes so oven the distributed, sdn control, plane and the smart kind of join hands here together to kind of provide the bare metal uh network, security and isolation.

A

So one is open source uh project. um It's a distributed, sdn control, plane uh that provides network virtualization solution. It's developed by the open, vswitch community, the github url here is where you can find the repo for the project.

A

um The thing with avon is that the whole data path for r1 can be offloaded with the smellinox nick. So obviously, with the offloads you get the reduce utilization and the cpu can be used for applications and and in addition, we also get increased throughput. um One can be better explained um with the exam. You know with an example and a diagram, and that's what we're going to do next.

A

So in this diagram you have um the two things right. You have a logical topology to your left and then you have- and we have realized that logical topology on the two bare metal nodes. um So on the logical topology, it's a very simple, logical, topology, it's a logical router with two logical switches connected to it and each logical switch has a logical port attached to it.

A

So logical router is a l3 entity. It provides routing, ecmp, nat, load, balancing policy based routing logical switcher, on the other hand, provides uh support for icos support for thcp. It also provides support for load, balancing and dns the logical port captures the mac address and ip address uh it captures it it's where we apply axles and then also it provides anti-spoofing support.

A

um So the thing is the logical port maps to a kubernetes pod network interface. So the logical port is an api to control the parts interface attributes like so you kind of apply uh policies on the logical port. It kind of uh reflects on the the on the physical on the pod interface on the on the pod.

A

So um um so, on the right side, what we have a two bare metal notes um on these notes: you have a obs integration bridge called brnd. This integration bridge implements the whole logical pipeline, the whole oven, logical, topology, using your obs pipeline using open flow tables and flows in those tables.

A

So then, on this particular uh in this particular diagram here on node one, a pod zero got scheduled on node two part: one got scheduled port zero is connected to the integration bridge through a ovs internal port, p0 and part 1 is connected to the obvious integration bridge to the internal port p1.

A

So what we do here is that we associate this physical port with a logical port and that process is called port binding, and this is one of the important aspects of oven. What this port binding does. Is it tells you, the physical location, of a port, for a given logical port, so with that physical location known now, the the oven control plane knows where to send the packet to for a given part.

A

In this example, when a when pod zero wants to talk to part one when the packet arrives on the br end, and this br implements the oven, logical, topology through open flow tables, it already knows that to send the packet to part one, it has to just send it to the node two's uh vtep ip address and then using open flow flows. It just forwards the packet to that next hop.

A

So the whole logical topology is realized uh on this integration bridge um and then the the physical or the port binding thing enables that end to end connectivity between the between the ports. So aman has a collection of daemons that work together to push this logical, topology information and the physical location of this ports to each node and and then each node's integration bridge.

A

So in this slide we basically this slide. Basically talks about the oven communities project. One kubernetes is an open source project, contributed or worked on by the oven community, the github url for the same uh he's here, as shown in the slide here, uh it implements container networking spec um it. Basically, it's a layered architecture at the higher layer. You have one kubernetes cni and which in turn depends on the r1 and i1 in turn depends on obs, so the cna itself is implemented as a client, server architecture.

A

It's basically acquitted is watcher and it has such several set of parts. um It kind of takes the kubernetes resources and then it maps into open resources. For example, kubernetes network policy is mapped into our accounts communities. Cluster services is mapped into one load. Balancer kubernetes parts is mapped into uh logical switch ports and kubernetes node is mapped into logical switches, so it watches on these cubities api resources as and when these api objects appear.

A

It kind of translates us into oven resources, so the oven, control, plane um kind of uh acts on those oven, resources that create that that was created by the one cni and then converts that into our logical flows, and these logical force are, in turn, translated into ovs open flow rules on each of the uh node that forms the qubits cluster. So on the diagram to the right, you see that you kind of explain all the components running at various layers or on the components of various layers running on various nodes in the kubernetes cluster.

A

So the whole point here is that to show um that on some of the one control plane components they run on the nodes and when we move to the smart nic model, these control plane components will end up running on the smart nic itself.

A

So in here you can see that on the host we had ovs bridge and the amount controller which kind of was it adding open flow flows into the obvious bridge uh and the one cni come components itself running on the host. But when we move to the smartening model, uh the the entire sdn control plane, the distributed ascent control plane starts running on the sr on on the smart nick itself. The smart nik comes with the arm core.

A

It has its own os running on the encore and then all of these control plane components move to that arm core. Obviously, this kind of provides increased performance and isolation for the bare metal nodes.

A

So this is uh this diagram kind of dives, deep into how things look like on the smart mic. So on the smart nic itself, we have one control, plane, components running which is our controller, and we have our obvious components running, which is obvious. We switch t and obstb server and on the one cni side we have an agent running which acts as a kubernetes watcher, uh trying to watch on certain kubernetes resources and then acting on those resources uh lifecycle.

A

The amount controller takes the logical flows from the oven and then translates into the open flow tables and open flow flows um and then kind of creates the data path for the pawn on the bare metal host itself. We have multus to orchestrate multiple interfaces into the pod and we have a very basic cni that takes the next available vf from the pf and then provides it into the pod itself for every vf in the host we have a corresponding representer.

A

um We have a corresponding control entity on the blue field itself. We call it as a representer, it represents the vf on the host and we add these representers to the integration bridge and this representer maps to the logical switch port in the oven topology. So this is the port binding aspect of the oven functionality.

A

So when a representative04 gets added to the pr end, we we kind of annotate represented 0 4 to tell that it belongs to certain logical port. So now anything anytime. We change the logical ports, attributes the representers attributes change through open flow flows and therefore um the vf2 that is attached to representer04.

A

It will see that the changes all of these things are being done on the next hop. You know next, stop it acts like a next stop switch, it's like a bump in the wire. So all the attributes of the logical port gets reflected on this representer and then it gets reflected on the vf itself.

A

um So that's pretty much what this slide talks to.

A

So the this this slide basically captures how we can use our apples to kind of add axles at a different priorities and therefore override the axles that the tenants themselves configure. So in here, um avon eccles priority range from zero. All the way up to thirty two thousand seven sixty seven, so higher the priority of the accol higher will be its precedence.

A

So in the design here we have all the tenant accols being represented uh with priorities less than ten thousand. However, all the cloud provider cycles are at the priority more than ten thousand. So therefore their uh precedence will be higher as compared to the tenant tackles.

A

So whenever uh the kubernetes network policies are translated to urban apples, they get uh priorities in this range, the cloud providers they can come in on the side and then add their axles at a higher level, so that you know we they can kind of make sure that the pink tenant here does not uh somehow talk into the blue tenant, because the cloud provider have added ankles to make sure that such a communication is not possible.

A

In fact, the cloud projects can add accols to their own infrastructure services, to make sure that the the tenants host or the tenants part do not talk to infrastructure services that they're not supposed to talk to. So in this way, using our priority accounts, we kind of make sure we provide that isolation of between the bare metal host and the data center services and the data center itself, and all of this echoes the flows is all offloaded onto the integration bridge and leah is going to talk to it uh in the next set of slides.

A

So then, so in here, what we have done is um we have two different bare metal servers um uh and the bare metal itself is uh part of ammon, logical, topology and then the kubernetes cluster also is part of one logical topology. So uh the paths are kind of connected together within our logical topology, orchestrated by the urban cni. On the other hand, the bare metal servers themselves are part of our logical topology, and then the interconnection between them is provided by our.

A

So in here, uh the parts if they want to talk east-west traffic between the parts is represented through the green color. It's a overlay traffic um and the network policies for this overlay traffic is defined uh by the communities network policies.

A

On the other hand, the e-space traffic between the bare metal server itself is achieved through the oven, logical topology and uh um using an overlay and, and then we'll have one apples define which bare metal hose can talk to each other and which bare metal hose, cannot talk to each other. This is defined by the one actuals see both the the green overlay and the orange overlay.

A

They are transported over the blue underlay. This forms the underlay where the blue fields are all in blue fields in the data center they're all interconnected together, and they form the underlay over which the geneva tunnel traffic is transported between the pods and between the bare metal server itself. um So the red underlay network here is for access to the internet.

A

If these spots want to access to the internet, they kind of exit out of the tunnel or they exit out of the oven, logical topology through a oven gateway router, where they get source knighted into the blue, fit ip and then now they're on the red, underlay and heading towards the internet.

A

The same thing is true for the bare metal server host trying to access internet, so they exit out of their logical topology um through a gateway, router, uh oven, gateway, router, where the source netting happens, the logical topology ip is so snatched to the bluefield underlay ip and then the packet head towards the internet.

A

um So that's this diagram, so here what we're capturing is how one can provide a multi-tenancy model uh in a dc using oven, distributed control, plane and a smart nic in it. So in here we have two tenants: um a blue tenant and a ping tenant. The blue tenant um uh has a bunch of bare metal servers which are all interconnected together using our logical topology.

A

On top of this bare metal servers, they're running kubernetes cluster, uh which is orchestrated by the urban cni and the the cluster the qubit is cluster itself is, it is in its own logical, topology.

A

um Similarly, here for the ping tenant, they have a bunch of bare metal servers in the dc that was assigned to them, and the bare metal servers themselves are connected together, using a logical switch and a logical router um and then on those bare metal servers. They have their own kubernetes cluster running, which is again orchestrated throughout cni.

A

As you can see, they all use this overlapping ips and- um and this is all made possible because of the oven, virtual topology and they all exit out to the underlay and then finally, they reach out to the internet. um The cloud providers can apply their axles on the bare metal um on the smart neck, to kind of make sure that the blue tenant and the ping tenant are isolated from each other and they're also isolated from the cloud provider services itself and the next slide kind of talks to that.

A

So in here we show a dc networking with uh thor's and l2 within the tors and everything across the tarsus l3. So here you see, a blue tenon has a bunch of bare metal nodes assigned to it.

A

There's a four bare metal node assigned to it, and each of the brain metal node has a blue field adapter uh assigned to it, um and then these four bare metals they form a vtep network amongst themselves, there's a concept of trusted zones, so you create uh this island of bare metal servers with bluefield on them, and these uh the the geneva traffic um or the oven traffic, only kind of uh happens or occurs between these set of bare metal nodes.

A

um On top of this bare metal, node is where we build the kubernetes cluster and then they kind of live within this island. Similarly, the pink tenant also has its own island with a set of bare metal servers with each of them having a blue field associated with it, and they have their own logical, topology, and the tunnel is formed only between these four pink bare metals.

A

So in that way there is no interconnect at all between the blue island and the pink island and then in the same way, the cloud provider services, the cloud provider can provide their services either on the physical network directly or they can provide it on the oven logic network itself.

A

um So in that way, a blue tenant can access some of the services provided by the provider provided by the cloud provider um through the logical, topology or directly they exit out to the underlay and on the underlay they kind of connect to the services running on the physical network itself.

A

uh Next slide uh leal. Can you proceed.

B

Sure, thanks greg, so uh in addition to the security, enhancement and isolation, this solution also enjoys uh datapass hardware offload. This is done using the asub square framework by melanox accelerated, switching and packet processing.

B

It's a software and hardware integrated framework which utilizes melanox nix to accelerate and offload the data plan while maintaining the control plane in software, so it minimizes the needed changes in kubernetes, cni or other sdns.

B

The solution supports different configuration, srv and virteo, and it's available both upstream and inbox.

B

It consists of three capabilities: classification, offload action of load and data bus offload, which are required for uh switching for switch data, pass offload.

B

So, as girish explained, we worked with oven control plan, which is based on ovs switch. Ovr switch is the most popular virtual switch is a flow based one with many capabilities uh among which are geneve connection tracking, mirroring and more, and uh there are many multiple control planes that are built on top of it. uh Ovn is an example for that it has a ova: it has a user space module and a credit space module and in the regular configuration each pod is assigned a vth, a pervert.

B

B

So in the traditional model, the first packet of a flow will go up to the user space there. It will be processed and sent and then suitable rule will be inserted to the kernel. So the next packets of the same flow will go directly through the kernel without going through the user space with connectex hardware offload the suit this rule would not only be inserted to the kernel, but also to the hardware, to the connected embedded switch in this case.

B

The next packets of a flow will not go to the cpu at all and will just be processed in the. How hardware.

B

So this way we keep the first packet miss architecture by using an additional hardware layer, which is the embedded switch which is located on the neck.

B

Each pod is assigned an srv virtual function and while the controller or the user inserts the the policies by using obvious open flow rules, these rules are offloaded to the hardware in a transparent way using linux dc rules.

B

Packets are being processed by the nic and therefore there is no host cpu consumption and we increase throughput and decrease latency.

B

B

The ace of square has many benefits. It achieves uncompromised performance, it saves cpu using the smart nic. You can achieve also full isolation, and it facilitates the solution for both bare metal and virtualized clouds.

B

So if for virtualized cloud that both network virtualization and storage virtualization were traditionally running in the hypervisor.

B

Using uh connectex, we can offload the network data plane by by leveraging the asub square framework when we use bluefield the smart technique as a bump in a wire also, the control plane of the switch can run on the smartening on the arm cores, as well as the storage, virtualization and other security services.

B

This is also true for bare metal service servers.

B

So I'll summarize bare metal clouds raises the demand for network isolation, since the host is no longer trusted.

B

Network resolution can be achieved by using a dedicated hardware, which is which, which sits in front of the in part of the host. We use bluefield smartnic as a bumper in the wire to run ovn control plane in a secured way, we're running both the virtual switch and the utilities configuring it on the neck.

B

We also leveraged the s of square framework in order to achieve data pass hardware offload, which resulted in bus, drastic reduction in host cpu consumption and enhanced performance.

A