Cloud Native Computing Foundation Open Observability Day NA 2022, 28 Oct 2022

Previous Meeting

⏯

youtube image

►

From YouTube: “Are These Things Talking to Each Other?” - Observing Kubernetes Networking... - Anna Kapuścińska

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

“Are These Things Talking to Each Other?” - Observing Kubernetes Networking with Cilium, Hubble and Grafana - Anna Kapuścińska, Isovalent

“Are they even talking to each other”? - asked almost every engineer on the planet when debugging their system. Although the question is being asked forever, the answer isn’t always easy. As the complexity of our systems grows, networking becomes more and more a magic glue: it's hard to understand and even harder to debug. Good observability helps in both understanding and debugging what happens. Nowadays more and more people are looking into eBPF as a solution for networking and observability. That’s what the Cilium project provides: eBPF-based networking, with a visibility layer - Hubble. At the same time, observability in the cloud native world is commonly achieved with Prometheus and Grafana dashboards. We will connect these technologies. Using Hubble metrics, we will visualize how applications talk to each other in a Kubernetes environment, famous for its complex networking. In a real-world scenario, we will discover a mysterious communication problem. It won’t scare us though. Equipped with open source tools, we will quickly debug the system and uncover the underlying issue.

A

Hello, everyone I work as a software engineer at isovalent, and today, I will give a talk about networking and also observability that, let's start with networking, because as a violence in general, what the company is mostly known from is a project uh called psyllium that uses EPF for networking in kubernetes, and not only so, let's start with quick networking one-on-one in free slides. It will be very basic just to make sure that we are all on the same page and are using same terminology.

A

When people talk about computer networks, uh they usually refer to the oci model, and in this model there are seven layers from physical up to the application layer, each layer Builds on top of a player below. So the idea is like the layer. 4 will wrap data from layer 3 in some way and Define protocols on top of layer 3.. Now this is a model, a theoretical model and it's not exactly aligned with reality.

A

Like nobody really remembers what's layers, five and six are but many people remember what are layers three four and seven, because these are very commonly used terms. So, let's take a closer look at three of them: layer: 3 is the network layer where we operate on IP addresses, ipv4 or IPv6 layer for bit. On top of it and it's the transport layer.

A

Two main protocols on this layer are TCP and UDP, and the layer 7 is the application layer and there are many layer, 7 protocols, the most known, some of the best known, are HTTP Kafka or DNS, um and layer 7 Builds on top of layer, 4.. So HTTP and Kafka use TCP under the hood DNS to use the CDP now layer. 7 has many protocols because they are serving completely different, um end user application level needs.

A

Some people might be wondering why there are two layer, 4 protocols and uh well. This is my favorite explanation of of wine so and of difference between two main two main layers or protocols. Tcp and UDP um anticipate how the communication looks like how Services talk to each other is that they go like that. Do they wanted to hear a TCP joke? Yes, I do. Are you ready to catch up? Yes, I am and so on and so forth, it's their boss. It's super verbose.

A

That's thanks to that verbosity uh we can have reliability, so TCP is uh resilient to, um for example, packets, coming in incorrect order or duplicated or missing ones. Udp is not like that. Udp is simply. This is a leafy dog and I. Don't care if you get it or not. That's it! That's the whole terrific communication.

A

Of course it's faster, but less reliable, okay, so that was a networking 101 in free, slides and and these days we have kubernetes and networking in kubernetes is well um it's widely perceived as very complicated following the kubernetes nautical theme.

A

I think that networking should be representing something like that, like this Tangled rope dumped somewhere and sometimes really feels like that, there are a few networking related kubernetes resources like service in Grass Network policy, and there are also a few components running inside every kubernetes cluster providing networking um there is CNA plugging providing basic connectivity between containers. Cni stands for container network interface, so this is for basic connectivity.

A

uh There is Cube proxy, providing a layer for load, balancing and implementing kubernetes services and endpoints and there's English controller optionally, providing layer, 7, load, balancing and implementing Ingress resource layers of it. Ingress there is corieners providing DNS. There are a lot of things happening and I think. One of the reasons why this is perceived kubernetes networking is perceived as so complicated is that um there is no one way to do that.

A

You basically choose all of these components: uh you're just cni, plugin Cube proxy has like default standard, canonical implementation, but some CNA plugins provide replacement for it. You're just Ingress controller. If you need it, accordions is also kind of standard, but I often need to configure it. So there are many many different ways how it can be configured and the different solution will provide different capabilities, including different observability capabilities, so different visibility into what's actually happening inside the cluster.

A

When it comes to networking so today, I will show what can we achieved with psyllium as a cni plugin for kubernetes? Okay?

A

So, let's introduce our new characters, but that's new really, but evpf stadium and taboo EPF is- and it probably will hear that word many times this week and it's a very hot technology these days, which um I like to think about it as plugin system for Linux kernel, so ebtf allows you to write code that is sort of injected into Linux kernel in a safe way and the main use cases, for it is observability networking and security uh psyllium is it's a CNA plugin, but a very rich and comprehensive CNF?

A

That means there are many features uh there is. There are layer, 4 and layer, 7, Network policies, Network policies are acts, sort of like firewall or pots inside kubernetes cluster um there's two proxy replacement, so.

A

um Implementation of services and low layer for load balancing- and there is also English and service service mesh powered by Envoy proxy built into serum agents. So this provides layer, 7 implementation and load balancing and see your Muses evpf as the underlying technology for networking. So I like different plugins base, usually on iptables um does networking with ebps, uh and there are seven and in a moment we will see how the in action, um but Hubble is basically observability layer for Cinema there.

A

There is CLI and Cy, and it also exposes Prometheus metrics uh and one point before we get into Hubble details and evpf is widely used for observability and they're kind of canonical use case, economical way. How to do. That is that um there is evpf program that hooks into some events in Linux kernel. It can be a Cisco, but can be many other things.

A

It records what's happening in three parts, that back to the user, and there are multiple false. These states that do some variation of that. One of them is, for example, tetragon tetragon is focusing on security events. So it's like security observability tool, um and this is not Hubble. Hubble sort of uses evpf for observability, but very indirectly, Hubble Is visibility, layer for psyllium, so it basically pigbacks on psyllium Stadium uses Epps for networking and have all just watches what stadium actually does okay.

A

So what happened? Does it have collects flows.

A

So a flow is a network event. A flow indicates some Network transmission. It's a very wide event. It can contain information from only all networking players from layer 2 to layer 7. It also contains kubernetes metadata if it's run on kubernetes, because it actually doesn't have to be run on kubernetes, but if it is.

A

We can also have things like Trace ID in the flow. If the applications are instrumented with Trace context propagation, then Hubble will automatically extract Trace IDs into individual flows and Hubble also exposes Prometheus metrics. So here you can see example, hand configuration how metrics can be enabled in home. They are enabled different groups can be enabled independently, so different groups correspond to different protocols and, like some there are a few other groups.

A

There are also. Labels are also enabled separately, so you can control cardinality in that way, and this is it and no additional component is needed when installing the cni plugin that you need for connectivity, you can automatically get metrics about network activity and configure them to your needs.

A

There are also a few other options you can enable for convenience like service, monitor or automatically created config maps with grafana dashboards, and there are exemplars. So if application is instrumental stress, context then have a will extract, Trace, ID into flows and the metrics will actually contain example on an exam first to these traces.

A

This is enabled with enable open, metrics option.

A

Okay, so let's look into how Hubble actually Works how this actually looks like we have a serum agent, this blue box, there saving agent runs on Evernote and it injects evpf programs onto the node, and this green box are Epps programs that are injected into kernel and provide network connectivity. We call it data buff, evpf Stars, its state in BPS, Maps PPS maps are basically like hash Maps or there are different kinds of stuff, but basically a way to store states of Mbps programs and celium reads this BPF map throw that perf ring buffer.

A

There are also a few other components. Inside stadium agent, there is Envoy proxy and the DNS proxy for handling um layer, 7 traffic, clay, server, routing and providing visibility of optional. You can enable that if you need there's a routing and visibility or not, that is used for identity for identifying endpoints that are communicating with each other in kubernetes that will be kubernetes metadata and there's Harbor and Hubble Is.

A

This purple box over there uh runs inside stadium agents and collects all of that collects data from uh proxies for layer, 7, visibility from Identity cash for kubernetes metadata from psyllium monitor that collects them, collects events from BPF maps for the basic Network visibility, Hubble parses, packets, that it collects and stores them in the ring buffer. It's a ring buffer implemented in go, and then um it exposes Network events in two ways: grpc endpoint with raw flaws and metrics.

A

And so grpc endpoint can be queried by either a UI or CLI through components. We call habarella, which is like centralizing, um centralized component collecting data from different from all the nodes and metrics can be it's a radical or open metrics metrics, so they can be collected by any prompt user grafana instance all right. So uh now we have an overview of that. That's the back!

A

uh Let's talk about some networking problem, uh so I will walk through a problem with communication between services and I want to do it live because I, don't trust computers that much especially when it comes to networking, but I will work through like debugging scenario, so we have a new demo app deployed the cluster or maybe not anyone, maybe all up redeployed new version of an update, the cluster. It's a very, very simple app. There is front end pot with front-end service and two worker pots with worker service user is talking to front-end service.

A

Front-End pod is talking to worker service on um it's connected, but by its name by worker on part 8080., and if we try to call a front-end Service as a user. Let's say it's part. Four part: two localhost, we are getting an error. We are getting an error that fronted can talk to work to the worker.

A

um If we open a grafana dashboards that we have, we can see something like that and what we can see that we can see a spike in that in graph with a DNS errors.

A

So uh this is something that we can get alerted on uh if it's suitable in this environment, um but we can also query Hubble CLI for more details and we can call it hybrid CLI with filters filtering by various kubernetes method like namespace, and in this case we are identifying apps by labels and that could be like World Cup name. But yes, we can make inquiry by labels and we can we query DNS responses to this app.

A

These are only responses coming from core DNS in Cube proxy in in Cube system namespace, uh and we can see some interesting things there. First of all, we can see how actually DNS happens in kubernetes. We see that there are both a and quadruple a queries, so these are itv4 and itv6 DNS queries and we can also say that there are queries for different domains.

A

uh There is domain worker demo, apps uh SBC cluster local and then um that it's shortening- and we can say that we are getting this.

A

This non-existent domain errors and if we look closer and we can spot that yeah, actually we made a typo and the worker is is spelled by with double error, so yeah we are actually carrying incorrect domain domain.

A

um Let's fix it, let's fix it. uh This is actually configured in the front-end deployment, so in front-end deployment has environment variable with an incorrect address. We can easily fix it, but when we fix it, we and we call front-end service. Again we get exactly the same error and what now we can take a look at another grafana graph showing drops and what uh what are drops.

A

So when there is some network connection happening in Linux kernel, it can be either forwarded by EVPs implementation in this case or dropped, and here we see summon spike in dropped connections with reason. Policy denied. So policy tonight tells us that this connection was denied by Network policies. Network policies are generally are acting like firewalls for pots, and it's generally considered a good practice to configure Network policies for connectivity in the cluster when we have spike in drops because of policy denied.

A

That means that may mean that there was an attack in a cluster, but very often that actually means that we have misconfigured resources and, for example, some policy is missing or there is something incorrect in the resources labels, for example. So, um let's, um if we try to investigate.

A

Sorry here, if you try to investigate with Hubble, um we can see, we can query about CLI again with same filters and verdict drops and we can see again some interesting things. We can say that there are retries, it's not only one request movement, only one request, but there are retries. So there are a few different connections made and that's because network is generally considered unreliable and in general, when you make specific connections- and it has- there are retries always built in so there are a fewer tries.

A

We can see that they are happening with exponential back off, so that first will try is after a second or so and second one after two seconds, and then there are and some other ones four and eight seconds later, and we see that we actually have like it's colored in red outputs, that there were flowers that were dropped and policy denied.

A

Let's take a look at the network policy for uh for the front end, and we can say that oh well, we made actually the same typo in the network policy for front heads because well we were just copy testing it right. So uh the network policy for front end doesn't allow to connect to workers to proper worker service and, let's fix it, and when we fix it, we again are getting the same, exactly same error. uh What happens this time.

A

We have another graph in grafana that shows us missing uh sin Acts, so um in the TCP protocol, as we saw earlier, we have. The communication starts.

A

Is that first there is a something sent with sin Slack, and there is it's what we are expecting after that it's a response with sin: Arc Slack, TCP protocol has this concept of flux, where the whole communication and whole CCP life cycle is controlled and is managed with using this plug, so that we know when there is something unexpecting happening and how we accept communication to start is that we have this so-called freeway handshake, so that there is a requests with sin response with synac and Dennis rather request with Arc and then communication continuous.

A

Here we have a requests with a which sins without without snacks, and if we investigate that we have a CLI again, we can see that um we can filter a in Hubble CLI by protocol filter of TCP traffic. So this is entire TCP traffic that involves front-end app and what we can see that we can. We can see indeed, that it sends a few requests with reflection.

A

um Again, we see retries, it's not just one request a request: there are retries and there's nothing obvious telling us what's wrong. There is no word error, nothing Reds as it was in previous examples, but it tells us exactly what is happening if this has exactly that. Some of the one of front-end spots is trying to reach one of the worker pots on porch, eight thousand Noel.

A

um If you remember we configured our front-end deployment to hit work and or port 8080 and worker is exposing port 8080 so where this port 8000 came from, we actually they didn't want to call it well, it came from the service it turns out uh missing since uh can mean a missing synapse. Sorry, since without snacks can mean a few things, but very common thing is that the port is just incorrect.

A

The app is not listening on the part we are coming, and here you can see the mistake that this worker service is configured so actually listen on port 8080, but on Port bounding to Port 8000 on the pots. This is a mistake and if we fix that mistake, we finally can fix our problem and finally fronted and work reports can communicate so yeah.

A

That was quick, walkthrough um what's what's possible and we used hubblecla here a lot and we we love sea life, but we thought, um wouldn't it be convenient to query flows directly from grafana if we already have metrics in Prometheus that we can graph in grafana dashboards and we are doing that and if you are familiar with grafana UI until pet attention to previous slides, you might have noticed something that would suggest that we can actually do that when I saw the table with flowers, this table was actually table in grafana.

A

So grafana has this concept of plugins has data source plugins and tunnel plugins, and there are many built-in data source plugins. The most common ones are like for parameters for tempo for Lucky, but you can write your own data source plugin with your own data source and me and my team thought like hey. We have this type of thingy.

A

Let's do that. Let's write gathana plugin for Hubble. That would allow to query flows directly with very granular filters, so yeah um we. This is a sneak peek into what's coming up, because the stacking is actually not released. Yet uh that will be.

A

We are working on grafana plugging for for Hubble data source and Governor provides great tutorials and libraries for developing plugging.

A

Actually following the tutorial after following tutorial, we get um as scaffolding were even without much front-end knowledge, it's kind of clear what we need to do and how it looks like and how a backhand data source plugins look like is that the stronger component, that's written with typescripts and react and back-end components, everything in go using grafana, plugin SDK that allows us to, for example, a core grpc endpoints that Hubble exposes, and there can be multiple underlying data sources in data source tagging.

A

But what I mean by that is that we can actually connect, for example, Hubble grp standpoint and Prometheus and Tempo in science. One data source plugin, which is super convenient for correlating things like, for example, in our case, correlating um correlating metrics traces and raw flaws that have Trace ID and um all visualizations in grafana can be. Are basically available and can be can be handled by uh by our own plugin, so this will be quick preview. uh This is how the plugin.

A

This is how it looks like we can query flowers with very granular filters same as we would do that in CLI. You can see this mysterious process, industry service map tabs and let's keep the mysteries for now. If you are curious about website, then feel free to hit me up during keep con um available time and to learn more to get started with Hubble. You I would recommend, following tutorials in film documentation, observability section and uh ISO valence and announce partnership today.

A

So if you want to see more use cases for using Hubble and grafana for debugging various issues, including layer, 7 or hardware issues, then there is a blog post available and also Hands-On demo available on GitHub. So I encourage you to check out that. Thank you very much.