Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Understand and Troubleshoot the “Magic” of Kubernetes Networking - Minhan Xia & Rohit Ramkumar

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Blackholes and Wormholes: Understand and Troubleshoot the “Magic” of Kubernetes Networking - Minhan Xia & Rohit Ramkumar, Google (Intermediate Skill Level)

Networking is hard. Kubernetes networking can be even harder. On one hand, kubernetes provides a nice abstraction of infrastructure underneath. On the other hand, k8s contributors and cluster operators bear the burden to seal the gap. Especially for kubernetes networking, it has to work seamlessly with both k8s internals and underlying infrastructure. This brings challenges for understanding and troubleshooting the system. In this talk, we will share real-world experience in running Google Kubernetes Engine and explore the pain points in the current kubernetes networking design, troubleshooting best practices and future improvements.

About Minhan
Minhan Xia has been a member of Kubernetes networking team at Google since 2015. He has contributed to various aspects of K8s networking, including pod networking, K8s service and K8s ingress.

About Rohit
Rohit joined Google in August 2017 as a SWE working on GKE Networking and has been working on better monitoring and testing of core GKE systems. Previously, Rohit interned with Google working on Cloud Pub/Sub.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

So good afternoon everybody so my name is min Han I'm, a software engineer on the GK networking team and.

B

I'm Rohit also software engineer and GK networking so.

A

Today, we're here to share our experience running and supporting kubernetes at Google, so networking is hard and kubernetes networking can get even harder. So at Google we have levels of support for customer issues and we are the last line of defense.

A

So when the customer issues get to us the better be tricky and this talk we don't want to make into a lecture or training sessions, although some of the slides coming from our internal training sessions, so we first pick a few fun cases and then we're gonna talk about what are the lessons learned and what are the best practices? We think we should share with the community so first case it's a black hole.

A

So let me explain the set up a little bit, so it's a service with three backends and the deployments has like had three replicates, and this and this service is actually a UDP service with the port 53. It looks very suspicious, similar to a cube DNS.

A

So a climb pod comes in so the plank climb. Pod in this case is a green one. It sends a request to the service cluster IP and we're gonna use. Matthew McConaughey from interstellar to represent a network packet.

A

So the packet gets to the service cluster IP and the service supposed to low balance, to request the service, picks one back-end and then delivers the packet to the backend, and this, in this case, matthew is happy and, let's say, the service back end changes for whatever reason, rescaling or no disruptions or upgrades whatnot. So one of the pods is gone and the cross represents that deployment is really relevant here, so it's removed.

A

So the green pod again is sends another network packet and it goes to the service and this time a black hole shows up and it swallows all the requests and no back end gets it, and matthew is sad in this case, so this actually happens like a while back. But- and this is consistently like the Green pod requests- will get swallowed consistently. So to explain what happened we need to have.

A

We need to talk about some of the background knowledge so first its contract, so contract stands for Linux kernel, connection tracking, so it basically provides the functionality to remembers the network address and the port add report translations happen inside the kernel, so it's based on the five tuples like protocol source, IP, source port destination, IP and destination port and the and like the packet will get reversed. The network address translation will get reversed on the way back so that the client will get not will not get confused.

A

So on the example on the right, so the order on top you see, the original request is sending from part one to service one and it got translated to part the destination. Ip got translated to part 99. This will this happens when the service, but when request gets to a cluster IP in the classroom and cost IP low bounce it to one of the back end pod.

A

The next background knowledge is net filter. The filter is a very important Linux component. It stands for Linux packet filtering framework. It gives like hooks to allow users to intercept and manipulate network packets, and it's capable of that. We're filtering, which is used by network policy and also network address translation or NAT in port translation, which use for a cue proxy to to to do the service load, balancing and IP tables, EB tables, contract tables and etc. Are all part of the net filter framework.

A

So on the right we have a snippet of the net future processing flow chart. So it's a very complex, but we're only going to talk about what's relevant here. So in this case the network packet comes from the network interface and then it enters the pre routing hook. It's provided by the net filter and then within the pre routing hook. He first checks. If there's an existing contract entry matches the network packet, if it it is, then it will proceed. If not, then it will first look at it.

A

Will it will pass the packet to the NAT table to see if it matches any of the Matt natin adding rules and then take it from there? So so let me explain what happened in the black hole case. So first, the packet comes in and the service picks a back-end and a contract entries created in this case the source, the original destination. Ip is the cluster IP and then got translated to the yellow parts.

A

Ip, ok, so the second time when the request comes in there's an existing counter entry that matches the packet and then the black hole shows up because contra entry decided to send it for the two stale endpoint, which is the yellow pod, which doesn't exist anymore, causing the black hole. So what we learn from this, so it's when we have contract contract can only be turned on for all the network traffic. You can turn it on or off for all, but you cannot turn it on for partial, like traffic and then you have.

A

We have NAT in which we use extensively with queue proxy to increment this community service and then the nature of UDP, which is doesn't guarantee the reliable delivery and there's no UDP like there's no connections there so and then the infirm or nature of pods, which has come and go all the time and then that's a recipe for black hole.

A

So the fix for this is actually happen in queue proxy. So what we did is that we flushed the stale contract entry. When the backend changes, we will find a corresponding back-end at the corresponding entry, and then we flush it so that the next packet comes in. It will go through the NAT table and then go deliver to the right back-end. But this is still a problem because there's a student there's slight gap between like flushing, the endpoint flushing, the backend and the new back-end, and then the backend changes right, there's always a gap between.

A

So you will get a UDP packet loss, basically and also for UDP in general. It's very tricky because the timeouts, how the kernel decide whether this UDP is timeout or not, is just by looking at the source. What we're looking at the whether the packet mat mat, matches the existing contract entry and there's a kernel parameter to configure it. But let's say you have a very long time out. Then you are risking the black hole. But if you have a very short timeout, then probably the request and response cannot make the round trip.

A

Then it's always tricky for for UDP. Well,.

C

A

Have another case for a black hole, so this is a typical part. Networking set up on a VM, so you have a big giant, grey rectangle to represent a VM and then inside this VM we have multiple Network namespaces. So you have a root namespace, which contains the east zero that connects the VM to the network fabric and then the CBR 0, which is the container bridge, is also in this root. Namespace and each part gets. This gets a separate network namespace.

A

That's the default mechanism to for the pol isolation so to make to connect the network. Namespace part one uses a network P and each part gets a virtual network interface pair and then it's basically like a network cable. It's like once. One n is in the pots network namespace. The other end is hook up on to the network bridge and within the root namespace. That's a very typical network set up on the kubernetes node and in this case, network traffic can flow between part 1 and part 2, and basically between the paws.

A

On the same note, but no traffic can come in or go out from the VM. Basically, nothing can reach the parts on the VM and the parts cannot reach out. It was asked if there's a black hole in between that swallows all the traffic, so this actually happens and then and then multiple customer issues get to into our pipeline and then finally, there's an internal customer which is a Google, another Google team, and that usually we don't have access to the customers node, and this time we had a internal customer.

A

Then we asked them what hey gave us the permissions to log on to a VM and take a look at what happened, and then we debug the hell out of it and then what we found is quite tricky, so memory pressure happened on the VM and then and then the kernel star, oome killing processes. So then, in this case, the network D, which is a very important Linux component, that it is basically in charge of the networking, stack that one killed, because it got a very bad um score and then there's a particular network.

A

Debug in that kernel, which resets Colonel parameter on restart the kernel parameter decides whether east zero on the VM is allowed to forward packet or not, in this case reset to zero. That means the easier are not allowed to fold packet. So that's why all nobody can get in or nobody can get no nope traffic can get in or no traffic can get out. So so what we learn in this case, so you have to think deeper.

A

So so, sometimes before we get to this the root cause, we always suggest that customers say: oh, hey, don't try to overload your VM and then you might get into this problem and then, when you get into this problem, restart because restart we're going to refresh the refresh the does kernel convicts but like in this case it's actually a kernel bug and then fortunately at Google we have a whole team to support the host OS, which we call the container optimized operating system, which runs on ones 4gk nodes.

A

And then the team gets the bug and they fix the bug. So this will not happen once again and then. Secondly, is the kernel come fix? Really, matters, kubernetes depends. A lot of communities features depends on the kernel itself.

B

Ok, so we talked a little bit about black holes now we'll get into a wormhole, so the setup for this one on the right and the yellow we have a daemon set called a and a daemon set exposes a host port 443 and in the blue we have a service type load bouncer, which is also exposing port 443 and that's kind of fronting a deployment of two replicas, and we can call that B.

B

So how this kind of looks, we have our load bouncer and we have three nodes, and the demon set obviously exists on all three nodes and the deployment exists on the left and right ones, and we can represent our packet again with our astronaut and he hits the load balancer VIP at port 443 to talk to B and he gets routed to this node. This is a particularly unlucky, node you'll, see in a second because it contains both a and B.

B

So what happened was the request was intended for B, but it got sent to a, and that makes him very sad so before we can get into why this happened. Let's do a little refresher on IP tables, so IP tables is how we implement service routing and quote unquote. Load balancing in kubernetes IP tables rules are configured by both Q proxy and cubelet. Cubelet takes care of one specific case, which is the host port case.

B

Q proxy takes care of everything else and, like Manhattan mentioned before, IP tables works on top of netfilter and so on. The right is a really really simplified view of how it kind of works we'll go into a much more detailed view of how it actually works in kubernetes in a little bit, but as you can see, we register at Hook's could be like a pre routing or output, and we install a bunch of chains, I'm Andrea.

B

In reality, we have a couple layers of nesting of chains and eventually we get to the actual rules, which you know it can be a bunch of things, but in the simplest case it can be a if your destination IP. Is this or your port? Is this then do something right, and so now we can look at what happened. So in this case a was created. This is the daemon set and, as I mentioned before, cubelet programs, the IP tables rules for the host port, and so what happened?

B

Was it programmed one rule in a chain? And it basically said if the destination ports 443, then Dean at two A's IP now B is created. This is the deployment and therefore cube proxy programs, diabete Abel's rules and because it's separate X Bossier type as in service, is different from the host port. We actually have to create a new chain for that. It's not on the same chain.

B

Therefore, we have another chain here and the rule in a really simplistic view says: if the source IP is the load balancer of it then dnat to be is IP, and so what happens? Is this created a wormhole right and so any packet coming in that had a destination of port 443, which in this case was both? It always got Dean added two A's IP, regardless of what the intended destination was, and so we kind of like to coin this as a wormhole, and so some things we learned here.

B

Iptables is really tricky and there's two reasons why. The first is, you need to make sure that rules are as explicit as possible, as you saw that first rule, which basically swallowed up all traffic on port 443 was not explicit. It was really wide and we should aim for having rules that are really narrow and cover a very, very small set of use cases, and the second thing is that rules should be precedence, agnostic, it shouldn't really matter where in the order your rule is executed. It should be this.

B

It shouldn't matter right, but in this case it did. If we had moved that first rule to be the second one we haven't met, we would have never seen this issue okay and so now we're gonna get into some best practices, stuff that we've learned, debugging, customer issues and I know debugging networking at least for me.

B

It can be really hard to just get started can be really daunting, but what we like to do is kind of identify sources of truth and these sources of truth kind of give you a definitive view of how the system is going to act and what the state of the system is is in at any point in time, and so one of the first sources of truth is IP tables.

B

This is kind of how the IP tables dump would look if you went on a node and just called IP tables save this is for me when I first saw it really daunting, but if you kind of break it down step by step, it makes sense. So let's do that.

B

So this first chain here is called cube services, and we can see that any packet, that's destined for 10 0 at 16 at 10, which is just a cluster VIP cluster IP VIP, sorry goes to jumps to this chain called cube service in the green, and this is kind of representative of a service right. So we have our cluster IP. There.

B

Okay, and so now we have two more chains here: the cube SCP, which is just stands for endpoint service endpoint, and this is just kind of a really. This is what we say quote: unquote: load balancing, because we basically use probability to distribute between the endpoints, and so you can see. There are two endpoints here and that's how we have represented it here at the bottom, two endpoints and then for each of those endpoints.

B

We can see that a D net occurs to a specific IP and that's a pot IP and that's represented by those right there.

B

Okay, next source of truth contract contract, is a really good way to see the state of the system in a dynamic way, whereas iptables is kind of static, it doesn't show you what's going on at any given time contract does, and so we have kind of two examples here. The first is a really simple, just kind of pot to pot communication case. As you can see, the stuff in the blue tells you that it's an entry for traffic from one pot to the other and then they read, is the way coming back and then.

B

The second case is a little more interesting. Actually, you can tell by looking at the source IP that would be a load, balancer IP and the destination port, and the blue is a very common node port. So this kind of gives you a clue that this is an entry for a node port type service.

C

A

Right so generally, when we look at a customer issue, we first determine whether this is a control, plane problem or data plane problem, so that you can tell by like, like the symptoms and what are the problems you show up so when it's a control, plane problem, generally speaking, go to the logs first logs events and we'll see what happened and then take it from there for data plane is a little bit tricky, because the stack is a little bit deeper.

A

So first from the user space you have the application coming from the users place and then it enters the kernel space and then finally get out to the network fabric. So there's many places to look at so. In this example, we're gonna show you like a sample like troubleshooting mythology or process that we do. In this case, the customer is like complaining. Dns queries are timing out. So first things. First, we need to figure out how how's the traffic flow. What's the data plane looks like so. First, the traffic comes out from the client pod.

A

The grandpa censor request goes to the CBR, zero and traverse the IP tables and out of the east, zero interface to the network fabric and then magically the network fabric delivers the packet to the destination node and then the destination node route, the packet to the DNS server pod. In this case, the request doesn't ends here, because cube DNS only serves cluster internal DNS queries when he finds the queries it cannot serve.

A

It will forward to upstream DNS servers in this case the server a DNS server pod process it and decide: ok, I'm not in I'm gonna afford it, and then the traffic will come out and to the CBR 0 bridge out to the narrow fabric and go to the upstream DNS. So DNS query is timing out so like what? What do? We look at so first thing? First, that it actually sent now the request right.

A

This can be tailed by the application logs to tell it certain to to tell it definitely you need to dirty use TCP dump to see if actually the request comes out from the interface, so just keep in mind that TCP dump works at the interface level, so you have to attach to an interface the it doesn't. It depends on how the traffic flows and whether the traffic gets knotted or not.

A

You will see the packet as it arrives on the interface, so in this case you can dump it on the east zero inside the pot namespace or outside in who, if you do TCP dump in the root namespace, then you need to find their correct, VF pair of the pod and then dump on that interface.

A

And then you can tell whether there's DNS requests actually coming out from the pod, and then you can also dump on the CB r0 bridge and see if the CBR 0 bridge is actually routing is actually except taking the traffic and routing it correctly. And finally, you can dump on any interfaces and see if there's if the IP tables are actually processing. The request correctly, whether it's for the forwarding, the traffic to correct back in and when the traffic when the packet goes on to the network abric.

A

Then it's the typical stuff on to look at on your network fabric, where you need to look at routes if you are using overlay, whether the overlay is functioning correctly, your firewalls and all the other stuff so and then on the destination.

A

Node check on the east, zero to see if the destination node actually accepts the traffic and then finally in the in the DNS part and see if the requests are actually coming from the client part and the server logs can tell you how whether the server is actually receiving it and how is getting processed. If you want more, you can use s trace to attach to the server parts the process ID and then see.

A

What's this cause, it says it is making and then similarly to check if the, how going request the for the requests are actually heading to, in this case, ad ID and then Excel. So eventually, you can see ok, what breaks and the whole data plane.

A

So that basically concludes our talk, so networking is hard but like with some patience and more background knowledge, kubernetes networking is not magic anymore. All right.

A

B

I can talk about that.

B

Yeah, so you said what about ipbs yeah so good that you mentioned that actually I PBS for anyone who doesn't know it's just another form of load balancing in the kernel, it's actually more adept to load balancing than iptables. Is it's actually beta and kubernetes right now the plan is to go GA with it in the next release, which is 1.11, but to your question I PVS is still built on top of netfilter, just the way it kind of uses. Those hooks is a little different.

B

Like iptables, we do dnat and pre routing where, as an IP vs, they do d net and input, and so stuff like that varies. But for the most part, the underlying like stuff is the same. So.

B

um Yeah I mean it's: it's a different technology and it's one that all of us on the team have to get on boarded to, and so we definitely expect problems. But it's good that the underlying technologies are the same and we're familiar with that. So.

C

B

A

B

Can come down yeah.

C

A

It works now, okay, so so the question is like any other tools that you that we recommend I see so.

A

So there are more monitoring and logging like support on gke, but, generally speaking, we can't touch the user's VM. So that gives us a lot of a barrier, so we have to. Sometimes we have to ask the users to give us to execute something for us and then just give us more clue what's happening, and we are also having like we're, also thinking of like kind of like a debug tool, kit or something that can be deployed as a privilege, pod on any node and then have a better debugging. So, but that depends on.

A

There has a lot of dependency. So first the know has to be functioning and things like that. So, but.

B

For now, we use that, whatever you saw is what we use.

A

So so the question is that wormholes are common right. That's what you experience. It depends on the setup. I think it depends on your q, proxy implementation and all your network network provider. So if those are compatible, usually these happens when they are not compatible. They say: okay is too like two components are programming iptables and don't they don't know each other? They collide that happens right. So it really depends on your network setup and whoever manages the setup needs to make sure that these are compatible right.

A

That's I think that's the right answer where it needs to be fully tested out in all the corner. Cases like hairpins, hairpin, traffic's or other like mattad traffic or no port. So, basically there's a there's. A variety of a networking test suite exist on the kubernetes repo, where it has to pass right and then and then yeah.

C

A

Basically, my answers: yeah, like not end-user I, would say whoever manages the cluster right. Yeah.

C

C

C

A

We had a lot of ideas, but the thing is that troubleshooting, the every issue is very different and then hopefully, because cubanelle is still very young compared to a lot of other projects. It's still like three years old, believe it or not. It's still having so much adoption, so there are like if, if Google, if you are very mature project, engineer, should not get any like customer issues at all right so right now we are still getting the fact that it was still get.

A

It means that it's not perfect, so we are also learning and exploring. Like you said, it's a very common problem where you.

C

A

Many pages of notes and and things like that, yeah that that can get very daunting. But.

A

When you ask about security, the answer is always defense-in-depth right, so it provides you a certain level of security. You say: never policy can you rely on it, yeah, maybe but a certain level, and then and then you are talking about container level security. Then you probably want like, like gee visor right, where you have real a sandbox around your container thing.

C

A

Ask the Calico guys they were our targets, mostly about like what doesn't work well. So.

A

Any other questions, or we can always chat offline right, oh yeah. Thank you.