Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2019 (San Diego), 20 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Is Your Kubernetes Cluster's DNS Working? - Jonathan Perry, Flowmill

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Lightning Talk: Is Your Kubernetes Cluster's DNS Working? - Jonathan Perry, Flowmill

Your Kubernetes cluster is gaining traction and more and more developers are bringing up new services. That’s great news. But you’ve been getting reports of intermittent service failures that you haven’t been able to track down. It doesn’t seem to be anything else...could it be DNS? Maybe we’re not running enough DNS pods or they are restarting too frequently? This talk will explain how to actually measure DNS health for your Kubernetes cluster and properly plan its capacity. We will share some specific mechanisms to gather DNS traffic information per service both with some standard Linux tools and systematically with eBPF.

https://sched.co/UafQ

A

A

Dns is used extensively in kubernetes service discovery, so when it fails, pods cannot reach. Other pods cannot reach external api's or managed services, and the system fails. An application makes a get adder info call, which the see library then picks up and generates. Dns requests destined to the cube DNS service that services cluster IP then mapped into an individual replicas, and that replicas then either answers the request with kubernetes using kubernetes metadata or forwards it into an upstream server.

A

And today I'm going to concentrate on two. You can go three misbehaviors in the Linux kernel and Lib C, so the Linux kernel the dog in one of their talks. They revealed that the Linux kernel has this miss behavior, where, if accordionist pod is removed from the pool of available endpoints, the Linux kernel doesn't clean up all of its internal state, so under extreme load.

A

Some of the DNS queries will get mapped to the removed pod, essentially dropping them on the floor.

A

The kernel also has this race, where parallel DNS queries, which actually happen in practice on community clusters, get has this race where some packets are dropped so, for example, on the bottom here you can see if a single socket makes two DNS requests. Those race through the kernel dnat, might assign them to different replicas here, for example, core DNS replicas, a and B, and the first packet to win the race gets confirmed and that mapping becomes the authoritative mapping for the connection and the other packets just get dropped.

A

So this is misbehavior in the Linux kernel. Lib C also can have these slightness behaviors and, for example, in alpine, the muscle library, so first that makes all the its queries in parallel. So it's very hard to mitigate the criminal race that I talked about just now, and also it has this slight: stricter enforcement of the DNS standard. So all if you're running on a cloud provider where the DNS is not strictly compliant with DNS outlined, can break your pods resolution.

A

So you really want to monitor how DNS behaves and core DNS has very good metrics out of the kind of out of the box. So here on the top, you can see monitors for metrics, for the volume of requests and on the bottom left to right. You can see errors, duration and response sizes, so this is great. You can monitor core DNS. It gives you pretty good visibility with the following two caveats: one.

A

You need to make sure that this monitoring is up even when DNS is down so, for example, if you want to send an alert to your phone, a page, you need to make sure that works even with das, is down I'm not going to talk about that anymore. In this talk and the other one is, it doesn't have visibility into the application. The three failures I just mentioned: don't even hit core DNS. So how do you get application visibility into DNS? So luckily you can get this from the operating system.

A

If you monitor UDP sockets and look at all the DNS requests and responses, you can analyze the requests and responses and get the same type of metrics. For example, here the volume metrics the timeouts, latency and incise that we just saw by monitoring, also processes and containers. It becomes possible to use the kubernetes api and see what service has been making these DNS calls and to get a per service view of dns. Behavior here is an example dashboard with timeouts, latency and volume, and all of the data in this dashboard is per service.

A

So, for example, you can see I hope you can see a misconfigured service where that experiences. A lot of gayness, timeouts and core DNS is all so a container running on kubernetes. So you can get visibility to core DNS as well. So this technique enables to get both core dns visibility and visibility into how applications perceive DNS. How is it done using BPF e BPF?

A

Sorry, the BPF system call has been around for several years and it allows user space programs to instrument the kernel using tiny snippet of code in a way that is safe and high performance, and this is in fact so useful that I want to tell you about other use cases that use practically the same technique. You can use it to map your application architecture and see if there's a dependency that you don't want you can. When you deploy new versions of your service, you can see how that deployment affects upstream and downstream dependencies.

A

You can detect Network issues and identify misbehaving services that might be causing trouble. So to summarize, you want to monitor DNS. You should monitor DNS. If you can use EBP eff to do it, you can get application level laundering and BPF is useful for many other things. So please come reach out and I'd love to see. You come meet me in the team at the booth. Se 51 thanks. Everybody.