Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 15 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Monitoring Kubernetes with BPF and Prometheus - Jonathan Perry, Flowtune

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Lightning Talk: Monitoring Kubernetes with BPF and Prometheus - Jonathan Perry, Flowtune

Kubernetes has made it incredibly easy to build distributed applications out of large numbers of microservices. But monitoring, or even accurately tracking, the interaction between each of these services can be a challenge. In this talk, we will explain how BPF (Berkeley Packet Filter) can be used to monitor your Kubernetes cluster and its services without any per-service code changes or instrumentation. Specifically, we’ll show you how network flows between pods gathered with BPF and captured with Prometheus can be used to visualize service dependencies and identify issues in your distributed applications.

To learn more: https://sched.co/GrV7

A

Yeah, so in this talk, lightning talk will cover how you can get a hundred percent visibility into your services communication. The flow of data between your services on kubernetes I'll first go through. Why you would want to monitor the flow between your services, then go through an example of how you can get Mis data cover technology that would allow you to get this efficiently run through an example cluster of how this data looks like and briefly touch on how to productionize it. What are the challenges?

A

So why would you even want to get flow monitoring data well with flow data? You can get a complete picture of your application architecture. Do I, have a service to service dependency that I don't know about, or that is undesirable. Is there a dependency between availability zones that could hurt the high availability strategy? Is there dependency between environments, for example, production and staging you'd, be surprised, second, monitoring health. It's easy to introduce bugs configuration errors and incompatibilities for each service that you're running for the upstream services, which interaction is unhealthy.

A

How can you find that which versions are affected which services are affected in which availability zones and finally, in the public cloud communication between availability zones and regions is expensive? How do you monitor which services are using this expensive resources? You can optimize, let's go through an example of how you could get this flow data. So let's say you have a pod. A wants to communicate to a service. X and X is backed by a pod B in the default implementation. Using iptables cube proxy configures iptables, with this mapping of X to B.

A

So a opens a socket to X, then iptables Maps translates X's address to b's address so that B's sees the connection is if it arrives directly from a to get this flow data. You use three data sources from the kubernetes api. You can get the names of the pods and their IP addresses from the socket subsystem in the kernel here using the SS tool. You can get socket information here, a 2x and also metadata on the socket. So throughput, RX and TX packet drops round-trip times from IP tables. You can get this mapping here.

A

A 2x is mapped to a to B. So using these three data sources, kubernetes api, socket subsystem and IP tables. You can get this flow data. How do you get that efficiently? Well, BPF 4e b PF has been around in the linux kernel and in the form that I'm talking about since 3:18. It allows you to space programs to run code snippets on different kernel events, and so, for example, you can get instead of going through all the sockets in the kernel. You can get just streaming updates of what changed.

A

So it's much more efficient and you also have access to much more data from the kernel. There are a lot of safety mechanisms for BPF. For example, the BBF has an internal verifier that ensures that programs cannot access invalid memory and, in the context of this talk all of the programs we run in read-only context, so they cannot garble in kernel. Data structures.

A

Programs are also just-in-time, compiled to very efficient machine code. So this is super fast, all in all, using the EPF it's possible to get a hundred percent coverage of all the flows in your cluster directly from the operating system, without application changes and with extremely low overhead. So that's a big win. Let me give you a tour of our one of our staging environments. Here you can see the kubernetes nodes and some of the health checks that cubelet performs to containers in the cluster.

A

Here you can see out of cluster traffic, hitting our API s and our front-end, and here you can see the flow analysis, server and Prometheus scraping it. I mentioned collecting throughput packet drops and round-trip time by analyzing the time series data it's possible to establish baselines of what the behavior should look like. So the system could alert you on changes on anomalies.

A

There are several challenges in getting such as into production. I don't have much time so I'll cover those briefly. First, you get is extremely efficient, but you could shoot yourself in the butt with it. So you need to be careful using it. Well, Network, overhead: if you ship a lot of data out, you make sure that you do this efficiently, so compress. There are also security implication.

A

You want the system to be real time, so you can alert on it, but I'm you and Prometheus has cardinality issues you need to tackle and more and more and more my contact details are in the bottom. So if you find this interesting, please reach out.