Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep-Dive: Packet-level Debugging of Bridged and Non-bridged CNI Plugins - Jay Vyas & Sedef Savas

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Deep-Dive: Packet-level Debugging of Bridged and Non-bridged CNI Plugins - Jay Vyas & Sedef Savas, VMware

Finding the fault for k8s connectivity issues can be a daunting task and require several teams (DevOps, networking) to do their own due diligence. The best way to track down a connectivity error from the perspective of a DevOps team is to deep-dive into packets’ journey through the Linux Network Stack. In this session, we will walk through the following challenges for various open-source CNI providers, in real-time with the audience: If packets from your pods aren’t reaching their destination - how would you use tcpdump to determine where in the packet’s trajectory? Do you know what destination NAT’ing is, and at which step and how it happens (via iptables or IPVS) for bridged and non-bridged CNI plugins? What about existing network policies and routes on your hardware - are there iptables rules conflicting with the rules added by kube-proxy?

https://sched.co/Zeob

A

Okay, so hi. This is jay and sadef here from vmware we're going to talk about uh how these cni plugins work and how you can look at what's happening packet to packet, um to understand, if you're having issues with cni's or network policies or other networking related stuff in your pod and service networks and kubernetes.

A

So so that's gonna start us off and then we'll get to a demo in a little bit.

B

Hi, hello, everyone, so we will start with introducing some kubernetes networking concepts and then go into detail about how cni's are providing connectivity, and we will conclude with a pack of devil walkthrough.

B

So in kubernetes, networking is a combination of many concepts and it is important to know them to troubleshoot any connectivity problem, so potterpod networking is provided by third cni plugins and since pods are transient. Kubernetes has an abstraction called services which allow us to reach a group of paths using virtual ips and qproxy is enabling connectivity to services by using load balancers like ip tables and how you find services is also a part of networking, there's kubernetes dns service. That provides that once you get basic uh port, what about what service connectivity?

B

How you secure your applications by isolating them is another networking aspect and cni plugins provide this as well. The last aspect is how kubernetes handles external traffic. There are several ways to do it: for example, layer, 4, layer, 7, not balancers or using notepad kubernetes construct.

B

So out of these 5 concepts in the talk we are gonna uh mostly talking about cnn plugins.

B

uh Okay, let's start with cni plugins, so kubernetes adopts a cni model to interact with third party cni networking plugins. These plugins are responsible for ips assignment connectivity and pod security.

B

Every time a pod is initialized, cni is being called. It creates a pseudo interface, assigns it an ip and attaches it to the underlay network and when a pos gets deleted, cnn is again being called to clean up the resources.

A

So the cni api is pretty pretty simple in terms of what cni supports, because all you, you know it says, add, delete check. You know it's it's it's a relatively simple api in terms of what needs to be called in order to launch a container, but.

B

It is a separate.

A

Api, that's managed by separate group yeah, that's cool.

B

And let's go over some traditional linux constructs that are used by most of the cni plugins to provide connectivity. First, one is linked network namespaces, so they provide a separate instance of kernel. Networking stack like a different set of interfaces, routes firewalls to isolate processes. All parts and holes are sitting on different network namespaces. In this diagram we have a single node. We have two parts and you see all of them are sitting on different network namespaces and communicate to communicate between pod network namespace and uh not host network namespace.

B

We use uh a linux network construct called wheat pairs here you see eti0 and wheat. A is one with pair and the other one. Is this one and they are nothing but page cables. What goes in one end comes out the other.

B

We also want uh pots to talk to each other and use a very familiar construct called linux bridges. So it is a it is a networking switch.

B

It is nothing but but switch and does layer to uh forwarding also when we want, when our pods wants to communicate with the horse and hospitals communicate with the paws rod. Tables comes into picture, so here you see, uh everything that is destined to pot ciders will be sent to bridge and bridge knows how to forward them. As I mentioned, hosts and mods are sitting on different network name space and they have different route table rules, as you can see here,.

B

So we looked at how posts can communicate with each other in a single node when there are multiple nodes. There are different networking approaches c and I offer we can categorize them as non-overlay and overlay networking approaches.

B

So the first option in non-overlay networking is layer to networking when the mods are in the same subnet or vpc virtual private cloud traffic can be routed using l2 networking.

B

Another option is providing a pure layer, 3 networking, so this option requires some configuration in the underlying network, so it is only suitable if underlying network can be controlled. So this is mostly possible in maybe private data centers, and there are several ways to achieve this.

B

Kelco uses bgp or private data centers may implement their own cni to provide layer, 3 networking and then when, when arnolds are not sitting on the same subnet or we don't have any control on the underlying network, there is overlay networking option where packets gets encapsulated so that underlying network doesn't have to know anything about pod siders and there are several encapsulation methods that can be used for encapsulation and the last one is hybrid mode whenever so, since there is encapsulation and the capsulation process in the overlay network, there is some overhead.

B

So to avoid this hybrid mod comes into picture. We encapsulate a package only when it is necessary and entry and calculate supports uh this hybrid mode as well.

A

Yeah I'll just quickly say: eks is um we're going to go over eks calico and in a little bit and you'll see the differences. But um you know in eks you don't need an overlay if you're using their cni, because the type control that they have over ip addresses and being able to use enis and stuff other cni's. You know use those overlays yeah keep going.

B

Yeah so we mentioned, uh we need encapsulation for all overlay uh networking. So let's have a look at how encapsulation works.

B

Cmi has support for several encapsulation methods, but uh let's look at two of them here: ipip and vxlan tunneling.

B

So basically, what happens is source node encapsulates network packets, packets in another header, so that underlying network can route them without knowing pod ips so, for example, in ip and ip tunneling, you see this is the ip header which has pod ips and it it gets encapsulated with another header with not eyepiece, so that underlying will only have to route node eyepiece.

B

So when you capture the packets using wireshark and when you, when you check those packers, you will see two ip headers in the packets and for the extent on the link, ethernet frames gets encapsulated into udp packets and again, when you see the shark packed captured, you will see the ethernet packard and on top of it there will be a udp packet.

B

B

Okay, now we are going to briefly mention how some cni's are providing networking at a high level, so it is important to know underlying constructs. Your cni is using to better debug connectivity issues. Let's start with calico calco supports multiple data planes like standard linux, networking or linux ebpf and windows. Networking also it has three networking flavors. It supports pure layer, 3 networking where traffic is not encapsulated for this, it uses bgp a route sharing and underlying network needs to support bgp as well and set up underlying growth accordingly. Otherwise, this is not an option.

B

It also supports overlay network option supports both ip and ip and vxlan encapsulation. Also it has hybrid mode and uses layer 2 networking, for if the nodes are on the same subnet and uses overlay for different softness.

B

We are not going to go into detail of these cni architectures. It is out of scope of this talk.

B

Let's talk about another cni called entria, so calico did not have any bridge um in its um while doing networking, uh but here we see a virtual bridge, that's being used uh pretty similar to linux bridges, so it routes traffic using multiple file tables and supports what windows and linux. If you like, look at the diagram.

A

When we talk about flow tables, so open v switch is the back end for, like s mentioned, um andrea and so open v switch is based on these flow tables and that's the whole. That's the whole way that you architect a networking open v-switch compared with you know what calico is doing in the last slide, with um just using bgp for the routing. I just wanted to repeat.

B

Right, yeah, and uh if, if you see um the diagram here, are we have two parts, they are connected to obvious spirit using read pairs. This part is the same with a regular linux architecture, but on the obs spirit side, we are seeing a gateway, zero port, which is the gateway of the node subnet and zero port, which is the tunnel part that is used for creating overlay tunnels between nodes and andrea is also supporting overlay mode. It supports many many different uh encapsulation techniques.

B

uh It also has support for hybrid mods, where it uses encapsulation only for the inter traffic in in intern network traffic. So we are going to see.

A

We'll actually see when we look at the entrance stuff, we'll see that all the traffic actually flows into that into that gateway, um specifically even local traffic, which is interesting from a control and security perspective.

B

And one one important difference here is that network policies are installed by obvious flaws, instead of, like click using iptables, ebpf.

B

So here we are going to show some examples of manually and overlay routing, so here we are seeing uh non-overlay which which, where we do not encapsulate, packers and use a regular layer to networking like arp and route entry. Here we have two parts that are connected um via a switch that that is to demonstrate they are sitting on the same l2 network, and these are the subnets of the pos inside nodes.

B

So when a pad one wants to talk with pad three, it sends traffic through the wheat player hits the bridge be rich, knows that it is not destined to some pod in this node has its sense traffic to the route table in the route table, so.

B

Every node's port cider is known in the route table, so here you see 172 16.1.0, 24, air traffic that comes to the pod. Sider of this node has to be sent via its eth0 device, and next up for those packets will be this one, so will be the not to id when.

A

You just literally run route n in your linux terminal right. Every linux operating system has one of these routing tables, so this is some a command anybody can run at home if they want.

B

Yes, and after uh it is throughout tables, the packet will be sent through eti0 and routed, using a standard layer to networking and and then arrives to the second node and when it arrives, it is gonna again check the rod table and every packet that is destined to the pot.

B

Sider will be sent to bridge br 0 interface in in in both nodes, so the packet will arrive, br, 0, bridge and bridge knows where to forward it through which wheat player it should forward the packet, and hence this is how it works in in the same subnet, non-overlay networking.

A

And in calico you wouldn't have that bridge right. So that's you would just have just straight from the routing the routing you'll hit.

B

The linux kernel and the linux kernel says: okay, I know exactly how to get.

A

To that ip address and it will have a attach.

B

Exactly so, this actually is a good point. This example uh applies to both calico and andrea, except that calico, instead of having a bridge, has more rod, table entries.

A

B

See that yeah and so yeah, if, for example, a non-overlay network is not an option for you, because you are sitting on a different uh subnets or you don't don't have any control in the underlying network and cannot do layer 3, networking overlay is the way to go and encapsulation uh can be done in many ways. So you see the only difference between this picture and this picture is this. One has ton ton0 interfaces that is used to encapsulate packets and you see there's a change here too.

B

We added a router here to demonstrate that this is. This is not a l2 network, but it is. It is a layer, 3 network. This can be the internet or private or public clouds.

B

So here again uh when the pod one must communicate with part two, it will hit the bridge. Then it will hit the route table, but in route table there's a difference. So every packet that is destined to the other node spot cider is a gun. Is, is going to be forward to two ton: zero.

A

First yeah so um on the left. I've said this before, but just to repeat again um on the left is that's your that's your cloud stuff right! That's your eks! That's your gke! That's your! If you're, using something like a an nsx or something like that, you can have this fancy non-overlay stuff set up for you using hypervisor native network tooling on the right is typically when you're spinning up your own clusters, that's when you're using a cni provider that will kind of abstract that all the way for you and using an overlay.

A

um You know for for people that don't have a whole lot of experience with this stuff.

B

Right and uh yeah the packets will be sent to town zero and there's a process attached to this sound zero interface that encapsulates the traffic. So it adds another header with node ips on top of the pod ips and sends it over the underlying network for underlying network. It is simply a packet that is destined from node one to not to so. It knows how to route it packet reaches to not to, and then it gets encapsulated sent to the bridge and reaches to its final destination.

B

So now we are going to see a packet walkthrough for whatever we explained.

A

Okay, so here we go so um the first thing that we're going to do is um I'm going to hit play so that, if I don't know if it's playing or not yeah um yeah, so first thing we're going to do is uh run art which so def mentioned a little while ago. So we can see there that uh you might want to pause. It um see there that um oh, oh yeah go ahead.

B

A

Just keep it running the whole time. Don't worry about it: okay, yeah! um So once it's our once we run arp. We can see all the stuff that's locally available. I think like the uh um what's. It called uh the um the the local pods and the local nodes. So once the def hits play now, we're gonna see it again: okay, cool yeah, so and and that this isn't a calico cluster, I'm going to swap back and forth. The blue terminal is android, the black is calico and then there's another black terminal.

A

That's uh eks as well! I'm just going to jump around and show you the commands here so um yeah. So we do that and then we can see that there's these 100 ips and there's these 192 ips, the 192's. Those are the nodes. Now, if we run ipnet ns, um we ran it really quickly, but you can see it again here right. We um we don't see it in eks, which is interesting I'll leave that as a homework assignment to the viewers here. um Why? My eks cluster didn't show me any network name spaces, but.

B

A

Provides the cni net network namespaces, and here you can see that the four pods, the hundred pods all have a cni network namespace. Now turning tcp uh dump on now we can jump in and we can see that. There's some traffic going from that 64.0.1 to our 100.98.

A

We jump to the bottom and then we can see that um the ip address that 198 ip.

B

A

That um so, by the way that tcp dump is tracing one of those calico interfaces, so you do tcp dump dash. I on one of the outputs of like, for example, the ipa command, and then you can trace the exact traffic going into that um interface in into that pod and nothing else so, okay, so we can see that that 198 is actually core. Dns makes perfect sense. Coordinates is constantly chatting with the api server trying to figure um with the kubernetes api server, trying to figure out, endpoints and stuff like that.

A

um So um there we go now we now we can do iptable save and then we can go, look and see how that routing happens right. So um we can see here, there's a routing rule, that's going to send um anything to 164.0.1 um to this, to the kubernetes service role dash mpx right. So the easiest way to do this if you're not good at ip tables.

A

Like me, is you just say: iv iptable save into a text file and then you just jump around and you can use star when you're, on top of a word, to jump from one chain to another right from one rule to another. So literally, so you go ahead and hit star and then it takes you right down to coupe service. And then you can see the comment there. That's the kubernetes api server service. Now we're.

B

A

On a time zone mission control uh cluster here, which means our api servers, are cluster api powered they're running on health support, our api servers on host port, and so um we've got the api server running on a node local network, 192 168 5.156, and we can see that that coupe service endpoint role, that the serv that the service pointed to actually finally forwards you to 192 168 5.156, and if we go and we get our node.

B

A

We'll see that that corresponds to the control plane node. So this is um how fort just one example of how routing is going to how you can trace through a packet flow from a pod to the out to the host network.

A

There's a million other things use cases you may wanna trace, for example, pod to pod or whatever, but it's essentially the same logical series of steps that you can go through um and it would be the same in andrea as well, if or or or any other cni provider, because the coup proxy is going to um basically have to do roughly the same thing um do keep in mind. There are different ways to do: coop, proxy and there's some systems may not use ip tables, there's ipvs and other things.

A

Other technologies we've gone the simple route, though taking those for granted. So, okay, now next, looking at the andrea thing, um so we can see here the route. The output was a little different and I jumped over to eks, but you can see now in eks. All the gateways are zero: zero, zero, zero! Now back to andrea, you can see the output. Is this? The entry gateways are zero: zero, zero intro gw zero. Let's just keep going. I think we're going to show it again later. So f, it's not a big deal so.

B

A

Anyways, so here we are now and what we're going to try to do is we're going to try to make in we're back on our calico cluster, where I'm going to make a damon set. So that means I'm going to have an engine x, pawn in every node right and um I'm going to save the iptables rules and we're going to show you what like vanilla routing you just get for free that you have to kind of start you that.

B

There's a few iptables.

A

Rules that we use as a baseline, that get created um after you uh create a pod okay. These are just these are just generic they're, just being created for us by calico. The thing to note here is that they're all named um they're all being added to the dash a kali dash tw dash calek418e, that's the interface name, ignore the comment. The name inside the comment is a calico specific thing to do comparison between um between which version of the iptables rules was being written. So it's a cni specific thing they need.

A

But if you want to look at the rules it's making for a given pod, you can look at the dash a what it's being appended to right. So that's the change being appended to now.

A

We can see the pods are up now, we've saved that file and then the the simplest stupidest thing you can do to figure out whether your network policies are working correctly um in in calico and um is going to be you just uh you well first, we we need to make our policy so uh yeah, okay, so we coup ctl, create this is a network policy and that network policy is going to um deny traffic to those engine x pods, and we know that we're going to see that network policy for sure on this node, because this node uh it's a daemon set, which means one of these pods, is running on every single node in our cluster.

A

So we're definitely going to see it. And now, let's see what the difference is between our last policies and our new policies, so we just get diff and now you can see. We've got a few a few different lines here and we can see that there are some network policy, stanzas um return of policy, accepted, drop of no policies, past packet, etc, etc. Right so those are going to get applied after the pre-routing stage um and I'll show.

A

You briefly later that this doesn't change your service level, routing your coupe proxy routing to your pods and all that stuff still works just the same. The only difference is the packets are being dropped once they once they get routed to the pods.

B

A

Okay, um so let's see here um yeah here it is so this is so that just so, if you've never seen these before you see these decimals, this is how load balancing is done from the coupe proxy level. That's how it decides exactly which pod is going to get traffic from a service right, because obviously you may have 10 different pods behind the service and these clusters.

A

I've scaled up the amount of core dns replicas, so that we have plenty of examples of that um again now we jump over the andrea here in blue and we've got okay, namespaces, okay, so um yeah. So we show the namespaces and then um uh we delete everything and then after we deleted this stuff, um we just go ahead and recreate it. um I'm not going to do the whole diff thing.

A

I assume it doesn't matter as much this time anyways, because we're not mangling as many iptables roles, we're using open v switch for security in andrea, so the android plugin um once we create that daemon set and the policy well, we haven't created the policy. Yet we can see these three rules.

A

If we look at uh table number 90., so you executing to the andrea um containers directly, if you want to see this stuff and you can run ovs, ofctl flows, uh br int and that's essentially, the equivalent of like looking through ip tables and open v switch. It's like the same. It's the equivalent kind of operation right. You can look at those individual flows, and so now um we've created the daemon set and we've created the uh network policy.

A

And once you look at the uh network once once you look at the network policy, uh you can see. We briefly showed you there's a couple, there's an extra row in there and that row shows the reg one, the packet's getting marked for getting dropped and whatnot out of that flow table. So that's how you can see those policies being applied in andrea and I think that's it. We made it thank.

B

You for watching nine minutes.

A

And 38 seconds, okay, great so yeah, that's pretty much uh the whole! That's pretty much our story. So please do reach out to us. um I'm junit 100 on twitter and saddaf is, I don't know if sadaf has a twitter account yeah, but.

B

But we're easy to.

A