Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How to Break your Kubernetes Cluster with Networking - Thomas Graf, Isovalent

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

How to Break your Kubernetes Cluster with Networking - Thomas Graf, Isovalent

One of the best ways to learn about something is to understand how you can break it. In this talk, we will look at all the different ways you can break your Kubernetes cluster with networking. Learn from this before you learn in production. Misconfigured DNS, DNS, unreliable network protocols and DNS, DNS rate-limiting, network policy side effects, NodePort surprises, MTU and Kubernetes, effects of Kubernetes networking on distributed databases, network rate-limiting effects, bootstrapping race conditions, unexpected service behavior, sudden breakages at scale, and more. The list of potential ways to break your cluster is long. Half a decade of CNI development experience while working with many, many Kubernetes users as a maintainer of one of the leading open-source CNI projects have gone into this talk. It will be fun, you will laugh, and hopefully, in the end, you will avoid some of what others have already experienced.

A

Hello, welcome to my kubecon session, breaking your kubernetes cluster with networking.

A

My name is thomas groff, I'm cto and co-founder of I surveillance, I'm also one of the co-maintainers of psyllium, a cni plug-in for kubernetes, and in this role as being one of the maintainers, I had the chance to gather with the rest of our team to work with many of you on networking related topics and, as part of part of this, we've learned a couple of things around networking on what could go wrong so in today's session, we'll kind of dive into them and give you an opportunity to learn from what we have experience experienced over the last couple of years.

A

It's quite obvious that networking is everywhere. Like we take networking for granted every day in our life, we have wi-fi connectivity all the time we kind of get this immediate itch if internet connectivity is out so networking is here. Networking connects everything we're taking for granted. It has to work. If the network is not available, it usually leads to very serious consequences, and at least I saw a couple of mentions in network for networking, even in the kubecon title slide. So the previous one. Let's have a quick look.

A

So when I saw this slide, I saw a couple of things I saw like an app team scheduling, 5000 services. In there I saw like my pulling on cue proxy. I saw a q proxy trying to pull up ip tables, trying to kind of make it scale trying to kind of implement intent from the app team uh hey. I need I need. The app team wants me to schedule, uh 5 000 services. I need those packets, low balance.

A

Okay, um we have like the liveness probe up there without network awareness, just celebrating, obviously not seeing the network being down and she's saying. Oh, my my services are happy, I'm even even seeing like a person looking away kind of resembling the platform team, ignoring crashing codiness parts which are affecting the app team and even the moto 4 watch together is kind of networking related.

A

It made me think of, like all the different components involved in kubernetes networking, cni's united chaining cue proxy ingress coordinates mesh and cloud networking all to get all coming together and trying to forward our packets together.

A

So, let's dive in kubernetes networking the dark side, um so kubernetes networking is simple and I think it has a lot of great advantages and in general it works really really well. There are a couple of dark sides to it and that's what we will kind of enlighten today.

A

Obviously we cannot have a networking session session without a special appearance from dns, very obvious.

A

If you like dns stories, you will definitely get enough of them today, but before we dive into concrete example, a bit of context, where are the stories coming from? First of all, as I mentioned, I'm a civilian maintainer, so these are stories from our users.

A

If you have never heard of psyllium syllium is a cni plunging among, among other things, it is ebpf based and it provides networking security and observability and don't worry. This talk is not going to be about cilium. I will leave it at this. If you want to learn more about psyllium check out selim.io, it has pointers to all the source code repositories, docs and so on. You will. You will learn everything that you want to learn on that website, all right.

A

So, let's dive in before we go into stories themselves, we'll do a very, very quick 101 on kubernetes networking, because if you boil it down, it can really be simple. Obviously this is slightly oversimplified and and in some cases it can be a bit more complex than this, but in general these others are the assumptions that kubernetes networking is making.

A

First of all, all parts have an ip and second, all pods can talk to each other. This is typically called a flat layer, 3 network and then in general this is no longer always true, but in general part sizers are allocated to nodes and then nodes allocate out of that cider to the parts. So in this example, we have 10 0, 1, 0, 24 and the local parts, and that node will get ips out of that range.

A

Then, if you want any sort of load balancing, you will use kubernetes services and you can use dns for service discovery and network policy for segmentation.

A

So, in general it is as simple as this, which I think on its own, has a lot of merits and a lot of benefits, and most of the stories we'll hear today are actually not about failures of this model.

A

It's about details and about like small little pieces that you can't get wrong so, first of all, of course, dns kubernetes dns, as we learned it's used for service discovery, it's usually implemented with core with coordinates, although you can of course implement it in another way, it's often run as a multi-replicate deployment, but you can also run it as a daemon set. So you have a dns responder on every node and it typically needs no app changes. So cubelet will inject a resolve.conf into the part correctly and it looks simple it's or it in general.

A

It is a simple model if you believe everything I've just said you probably have never seen this. It's not dns. There is no way it's dns. It was dns if you are new to devops. uh This is the first thing you should learn.

A

It is always dns. In fact, if you remember this graphic, if you have this graphic handy, you probably get a free level of seniority in devops right away.

A

So let's look at um at first. I have a couple here. First, I think this is a very common one. The ndot's default, so cubelet injects a bunch of options into etc. Result.Conf, obviously like there is a search parameter.

A

This search parameter is a list of names, and these names will be appended at the end of each name. If, then, the name that you look up is not fully qualified. This is what allows you to just look up the service name without the namespace and the service suffix and so on.

A

This is what makes this work and the second important one here is n dots number of dots.

A

This defaults to five in kubernetes n dots basically says that if you have less than five um dots in your name, it will always append the search list to it, and if you have no clue what I was just talking about, maybe a quick demo on how this looks like will help you.

A

So, let's switch over to our kubernetes cluster. Here I have a couple of parts running and I will run hubble. Hubble is psyllium's observability layer based on ebpf and we'll run this, and this will in a second, this will show all the dns lookups that we're making and in the upper window I'm basically just cube control executing into the client and then I'm running, curl google.com.

A

And this has succeeded, so google has returned 301 and you can see here. In fact this was google.com responding and then in lower windows. So you see a bunch of output, we can actually scroll back up like lots and lots of lookups were made.

A

Let's scroll all the way here to headed store. So here was the first lookup, so it actually did a dns query for google.com.default.svt.cluster.local and it was actually for 4a. So this was ipv6 and then the second one was done same name, but for ip4, uh then the dns response server is responding, uh not found or not existent and so on. I can look at all of these names, so it basically goes through the search list and tries all of them until let's go all the way down until the very end.

A

It actually does the lookup that we want so google.com and again it does this twice for ipv4 and ipv6, and you see google's dns actually returning here and returning ipv69 before address of google great.

A

So what we just learned is that for any non-fqdn, lookup you're not really doing one or two lookups for ipv4 rp6, but you're. Actually doing five in general five, if you have five search engines or four stars, four, so plus the fully qualified one per address family. So, basically ten, so you we just did curl.google.com, but it actually did 10 dns lookups.

A

This is not ideal. It will lead to a massive number of dns lookups. So you please educate yourself about n dots. You want to understand this and change this default and it actually plays into the next problem that we have dns rate. Limiting.

A

Most cloud providers actually do rate limit dns, for example, aws limits dns to 1000 packets per second per enr. This sounds like a lot but think of last slide. You'll have lots and lots of lookups, it's actually very hard to notice, so you've actually very likely been limited, but you never knew uh it leads to. It leads to random connectivity errors because it will happen and it will fix itself and it's also often hidden because p99 latency measurements often don't cover dns, so that it's not shown.

A

Often, if you have latency metrics in your dashboards, it will only show tcp handshake metrics, like how long does it take for a tcp connection to be established, or how long does my http request responds uh latency or what's the latency between the request response and so on? It often hides this dns latency so hard to hardly notice the way we found this is with the hubble observation, observability layer that which you saw in the demo.

A

Don't worry, we'll have more dns, but for now we'll switch over to network policy network policy. If you have never heard about it very easy declares who can talk to whom? So, let's say you have a front-end pod and you have a back component with network policy. You can basically inject or start a firewall in kubernetes and then define roles. For example, front-end can talk to back-end and everything else should be denied.

A

So one of the most common failures that we are seeing here is something like this: a user wants to allow front-end talking to back-end, uh writes a simple network policy network policy. Spec pod selector match labels app front end egress two parts: electric match labels at back end cool right, easy, straightforward: what could what could go wrong? This is allowing front end to back end right. Easy well, did you think about dns often dns is overlooked.

A

So, if you add in an egress network policy, you will automatically put egress of that part into default, deny which will block dns. So you have to allow dns it's surprising how many users are being bitten by this. The problem is, and yes with some cni's, including some you can define global policies to define, is cluster-wide, but we stand in our policy. You have to allow this for every namespace separately. So as soon as you go from default allow into default. Now you have to think of dns.

A

So what you think is what you need is something closer to this, where you're not only allowing front end to back end, but you also now allow to uh core dns. You can now either edit your yaml or I will demonstrate this real quick. We have actually made our network policy editor available for everybody, so this is a free offering that you can use you can access it by editor.cilan.io and it's basically a visualizer and editor for network policies.

A

So I've loaded the policy that we have uh looked at so you can see there is like a pod, selector app front end and it allows in the name space to app equals backend great. So you see the green arrow, the green yeah, the green arrow here, and you can also see that kubernetes dns is actually still red, so we can go in and actually allow dns great, so it actually added the yml. We could now download this and apply it.

A

Kubernetes services and load balancing um a couple of things here that will bite you, the first one and probably the one that you're you've heard before is simply scale. So, if you're running q proxy in default configuration, you will be using ip tables, it uses linear lists of rules. So, as you grow, your number of services and endpoints latency will go up.

A

I've taken a screenshot here from a q con talk of 2019, a team member of hours presented like a bpf implementation and also show numbers comparing bpf ipvs and ip tables, and you can see that ppf and ipvs both have basically fixed cost, no matter how many services you're running, whereas the ipad tables based implementation will simply uh show increased latency, as you scale up your number of services, and it's not actually observable only like 5 000 services or something like this.

A

I think there was a claim earlier on um by another kubecon talk that you need to run a lot of services. A thousand services is not that much and you already start to see uh the difference service loopback. um This is essentially a part talking to itself via a service. So very common, if you have like a front-end and you have a front-end service and the front-end is talking to the front-end service, this will lead to packets being routed back into the same part with many cni's.

A

This will just fail silently and the the reason for this is because linux or linux, networking the actual kernel only accepts network packets where the source and the destination ip are the same. If the. If that forwarding is happening on the loopback, the lo device for kubernetes networking, that is not the case, packets will go out, they will go to the cluster ip of the front end and they will be translated and will go back in this will lead to random connections. Breaking linux will simply silently drop them.

A

It's worse, because often you're running multiple replicas of front end, which means that the majority of connections will succeed and then, depending on how many replicas you're running, only some of them will fail. Some cni's, including psyllium, I'm not quite sure, which, with other ones as well, do support this specifically, so they will translate the source ip to a different ip to to fix or to work around. This limitation of the linux kernel all right. The last one here on the service load.

A

Balancing side is a manual attack around external ip, which basically allows you to redirect any traffic with the help of an external ip service. And before I go into lots of explanations, I will just simply demo this: let's go back into our cluster and to make sure we have a bit of visibility, I'm actually going to run hubble again and I'm going to limit the visibility to a particular client part and I'm going to curl this ip and actually, let's curl this ip from my laptop first here.

A

So this ip is basically an ip of google.com right, so I've curled the ip here and google is responding. So let's run hubble again and let's curl this ip from my client, I'm actually expecting the same right. So the same output should happen. So, let's see what we see here, huh this doesn't look like google at all like what is this, so this is like some output, like some emojis, even here like you're, successful running a json server. Definitely something else!

A

So, let's see, let's look at hubble and hubble is actually telling me that we are seeing packets from like the client going to echo a which is kind of is that is that another part yeah. So this is another pod running here, so I can look at this service here that I have injected. So I have this echo a service which is of type load balancer.

A

uh It has an external ip and what I've did I've simply added the external ip off or one of the google ips, and this will then load balance to echo a like the according to the the part selector, so I've successfully managed to to steal net or packets intended for a google.com ip and redirect that to one of my local parts, this is, of course fixed since I'm deliberately running an unfixed version here that to to still demonstrate this all right, crds at scale.

A

Crd is custom resource definitions; they allow you to create custom objects in kubernetes stored in a city of the api, server can be created, watch deleted are often misused for anything configuration, state, storage and so on sounds good so far.

A

How does that affect the network? Crds are often watched. This means that you are watching for changes on them to illustrate what that means and how that can bite you a simple example here: let's, let's assume you have a c or d and it's on average 50 kilobytes in size. That's already quite big. Obviously you can change the math here and you will still be have a similar effect, so let's assume 50 kilobytes per crd on average and 5 000 nodes, and let's also assume this crd is updated by each node. Every 10 minutes.

A

10 minutes is not that frequent, that, like it, could be like a heartbeat or something or like just kind of a safety update or something that already means that we have one update per 10 minutes per 5000 notes. That's eight updates per second.

A

This gets worse, eight updates per second times 50 kilobytes times, 55 thousand watchers, five thousand notes, that's 16 gigabits per second or two gigabytes per second for the non-networking nerds.

A

This is the amount of traffic worst case that, if you're running a single api server that your api server has to push out on the networking side, your api server, the vm running this- probably doesn't have that type of network connectivity.

A

You can make this worse assume a perfect storm demon set updates to crd on startup, something that's very common and somebody for some reason deletes all the parts of that demon set and let's say that, actually takes 10 seconds. While all the parts are being deleted, all of these parts will come up at the same time. Around the same time they will each update the crd.

A

This will lead to 900 plus gigabit per second of network traffic.

A

Obviously, there's no network that will that will do this, like you, won't be able to find a box doing this, so this will simply mean, like your ap ips will be busy just pushing out these updates for a very very long time and it will stall everything.

A

This is a major, a major cause for scale bottlenecks, often in like application code that uses crds in this way.

A

All right last topic here: cni configuration wars and this is getting a bit into the weeds, but I think it's it's interesting to know. um First of all, it's dni basics, so cubelet reads: cni configuration from cnn c, c9 at t or similar can be different depending on your kubernetes distribution.

A

um Often cnn plugins are displayed as daemon sets and they drop in their configuration file. For example, zero five dash cylinder conf first file in alphabetical order in that directory wins so cubelet will take that file. Read that configuration and then invoke that cni plugin so for every part scheduled it will do a cni add for every part before it disappears. It will do the cni delete and very important. The node is not ready until a cni configuration is found and valid. This prevents pods from getting scheduled before a null can actually provide.

A

Networking sounds simple, sounds cool, sounds easy. Let's look where this can go wrong. So, first of all the uninstall leftover surprise. So, as we learned, cni plug-ins typically drop the cni configuration as they get deployed.

A

It can be done with a post start hook or any container, and so on, but c9 plugins cannot remove the cni configuration on pre-start or in some other way, because if they would then every time you would restart the cni part, another configuration file in the cni directory could win. So if you would schedule a new pod while that, while your primary cni is being restored, that part will would get scheduled with a different cni.

A

This is you don't want this most likely that pod will not have network connectivity, um so this means that cni plug-ins will simply leave the configuration behind when you remove them. So when you just delete the parts the configuration will leave behind this means in a lot of cases. If you uninstall a cni, you basically leave a non-functioning networking in your cluster behind.

A

Then you need to fix it up manually, the bootstrap race, so users deploy a cni via daemon, set, typically with system node critical, so they get scheduled first, but another cni plugin is already pre-installed. This is basically the standard on any managed kubernetes offering because of that pre-installed cni. The node is immediately marked ready, so parts can can scheduled right away.

A

There's the cni of the the or the demon set of the cnn that you want, then the races to be scheduled. First on that new node, it will probably win because of system node critical, but if that race is lost, the intended plug-in or like your primary signal plug-in will never see the cni yet so again the random new parts will have no connectivity a bit of a bonus. Complication here scheduled, doesn't mean running.

A

So even if the cni part is scheduled first and starts running first, it's not guaranteed to run to the point where the cni configuration gets, dropped or written to disk before the first part is scheduled. So you might still have a race in there in the end, the most reliable way to make sure this never happens is to make sure that you have a cni configuration for yours for your primary cni written as you want it when the node comes up.

A

Last but not least, the asymmetric cleanup um cni configuration x is present like for some plug-in. Some parts get scheduled, they start running new cni configuration file is written. You then delete the pulse to restart them. You know what happens. The old cni will never be invoked on delete. Only the new one will be invoked, but that new cni plugin actually has no clue about these existing parts.

A

This means that routes, interfaces and other resources are simply leaked. Often this is not an immediate problem right away, but it often bites you a couple of weeks later on, because you will have leftover ips leftover routes like laying around in in, in your notes, very very common.

A

All right, so this was a quick warp through lots of failure stories. As you have learned, we have obviously have many more, but only so much time. I want to take a bit of a kind of a summary at the end and and cover three lessons learned um overall, like and and kind of give you best practices.

A

um Obviously, you can avoid the specific mistakes or like learn from them, but in general I think we have learned three. Three things: um shiny objects are cool and kubernetes is one of these shiny cool objects. You can build fantastic stuff with kubernetes and it's it's awesome to build it and can be highly effective, but keep it simple like don't over, complicate it um in moments of complications. Any moments of disaster. Simplicity really helps. I think that's tip number one. It's obviously very generic, but I think it's in particular true for networking.

A

Second, one may be even more important visibility matters, so connectivity itself is not is not enough like. When we talk about networking. We often talk about benchmarks and connectivity and maybe latency yes, that's important as well. But what really matters on day, two and and beyond, is to have visibility into what's going on in the network, we've quickly looked at hubble, that's only the tip of the iceberg. Obviously it's one so losers. Many other solutions out there as well in general.

A

Don't look, don't just look at the connectivity angle. Don't just look at maybe some shiny benchmarks, even though that matters as well look at the visibility. It will really help you when, when issues hit when issues hit the ground last year, I think this is a little bit biased, get yourself some superpowers selim is not the only project using ebpf, there's more and more projects in the cloud native space leveraging evpf and it's leading to innovation.

A

That is truly awesome for you, as a user, it leads to to to kubernetes native solutions where, instead of mapping, this awesome new, shiny, kubernetes objects and concepts into all technologies to really map them into something that it that is intelligent and that can benefit from these concepts the most.

A

If you want to learn more about ebpf, I highly recommend you dig into one layer: deeper, learn a bit about the concepts check out the many awesome, ebpf projects, tracing security, networking visibility, observability and so on. Ebpfio is a great starting point to get started with that. I would like to thank you very much for listening to me and, and I really hope that you have learned a thing or two. If you want to contact me, feel free to do so on twitter.

A

If you want to learn more about psyllium feel free to go to similar, I o or check out the cilium github project. Thank you very much.