Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Hubble - eBPF Based Observability for Kubernetes - Sebastian Wicki, Isovalent

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Hubble - eBPF Based Observability for Kubernetes - Sebastian Wicki, Isovalent

Troubleshooting network issues in Kubernetes often requires deep insight into different layers of your stack. Hubble is a new open-source observability platform that aims to assist you in understanding what is going on in all layers of your Kubernetes network. Based on the Cilium CNI and the Linux kernel eBPF technology, it is able to obtain fine-grained visibility into network traffic and applications behavior, with low overhead and without having to modify applications. In this talk, you will get a introduction into Hubble, and the technologies that power it, the Cilium CNI and eBPF. You will be presented with practical examples of how Hubble can be used to interactively troubleshoot complex network issues. The talk will show how to write custom Hubble metrics which allow you to benefit from eBPF's superpowers without having to write or understand any kernel code.

https://sched.co/Zeoz

A

All right, thanks for tuning in I'm sebastian, I'm a software engineer at isovalent. We are a security and networking company who is building hubble, which I'm going to talk about today and cilium, and so the talk today is about hubble an ebpf-based observability tool for kubernetes.

A

So what does observability in kubernetes mean? What are the kind of questions that hubble tries to answer? Kubernetes is kind of a complex piece, so from different perspectives, there can be different questions that we might want to ask. So if I'm an application developer, I would, for example, like to know what kind of services my application is depending on. I would like to know which http endpoints or which grpc or kafka calls are being made from my application and, of course, I also want to know the performance characteristics of of these calls.

A

If I'm operations persons, I might have a different set of questions. I want to ask the my tool, so I I, if I'm an operations person, I might be more interested in the network communication. I want to know if the network communication is failing and if it is failing, I want to know the reason: why is it a problem with the application? Is it a problem with the network if it is a problem with the network?

A

I would like to know why and on which layer this problems occurred, and if I'm a security person, I might have a different set again that I would like to have entered so, for example, if I have a kubernetes cluster which has network network policies enabled, I would like to see if these network policies are being enforced and what connections are being dropped.

A

If there's services that are being accessed by outside entities or that are connecting to outside entities, I would also like to know uh what services are being accessed and what outside entities are being accessed. So these are the kind of questions that hubble tries to answer and the tool it it has chosen to to implement uh to implement its functionality is ebpf. So I briefly want to talk about ebpf what it is and how we use it.

A

So ebpf stands for extended, perk, pack-up filter, it's a linux kernel feature that allows you to attach small programs to the linux kernel in a secure and efficient manner. I will again briefly explain what this means in more detail in a second, so ebpf allows us to attach programs to the kernel on various attachment points. So on the slide here I have an example program.

A

That is invoked whenever there's a outgoing packet on a certain interface, so I can attach this this program, for example on the socket layer and whenever there is a new packet going out on that interface, this eppf program will be invoked. So it's an event-driven mechanism and this sample program here is written in c, but it will be compiled to bytecode.

A

This example program will extract the ethernet and ip header and then display us the source and destination ip address of this packet, as well as the protocol of the payload that is being transported and while ppf originates in the in the networking space, uh it has been extended to support uh quite a few other attachment points, so we can, for example, also attach ppf programs to system calls.

A

This is a common use mechanism, for example, to implement sandboxing, where we can inspect system calls and maybe block or even rewrite them if needed, and the kernel also has some additional capabilities as well to inspect user space processes, but there's also layers deeper down in the network stack where we can access bpf probes, and these days you can even push down bpf programs to the network card, and the network card will invoke your bpf program whenever it sees a new packet.

A

So why do we want to use this technology in hubble? Why do we want to use this technology for observability, and the biggest reason is that it is completely transparent to the application? The application doesn't know that ebpf is being used at all. So this means we don't have to modify the application, which is important, especially when we are troubleshooting issues that we have never seen before, where we could not modify the application ahead of time to make sure we cover that use case.

A

Ebpf is also because it's dynamic, it has a minimal overhead, so we can. We can attach probes only when we need them and we can enable and disable them whenever we want to get deeper visibility.

A

Eppf is already widely available, so there's a few large-scale users. I have a few names there on the slides that are already using it in in production in quite large clusters, and it's also widely available on most cloud providers. So hubble, for example, only requires linux 4.9 to run.

A

If you are interested in more interest more in ebpf and its history and how I came along, I highly recommend you check out uh the kubecon talk by my cover cro daniel he's also one of the epf subsystem maintainers in the linux kernel as well.

A

So how? How would you use an eppf program? I have a small example here on this slide, so ebpf programs are typically loaded by a user space agent.

A

This is a user space program that is taking the c program, so we have the z program there in the middle that we want to attach, in this case, to the socket layer of the linux kernel. So what the agent does is. It will take this c program and compile it to bytecode and then load the bytecode, so ebpf is always bytecode similar to, for example, how javascript works.

A

So this bytecode is then loaded into the kernel, and the first thing the kernel does is verify that this byte code does not do anything dangerous. By that I mean that it terminates so it kind of blocked the kernel and that it doesn't crash the kernel or access any data structures that it is not supposed to access once. We know that this program is safe and kind of crash the kernel.

A

We will then optimize it using a just in time compiler, which will take the byte code and emit machine code, which is basically as optimized as a kernel module, and so once we have the machine code, we can attach this machine code to our attachment point. In this example, the socket layer of the kernel, and once we have attached the ppf program, it will get invoked for each, in this case outgoing packet, that, for example, a process will emit by using the send system call.

A

So whenever there is a there's, a sound system call the packet will be processed by the linux kernel and we will be informed about each outgoing packet and this andy can then either decide to, for example, log it for severability purposes, but we can also implement things like firewalls or other policies.

A

A nice feature about ppf is that these programs that we attach to the kernel they can have state and the state is managed by what we call bpf maps. There's different kind of maps available.

A

uh The kind of most common one is a basic hash table, but there's other one like ring buffers that you can use to stream events, and so this is used to communicate with our user space agent. That loaded us into the into the kernel and the bpf map can be used for transferring of state. It can be used for configuration.

A

So, for example, we can add filter lists to the program using ebpf maps, but it's also used to send down events from the bpf program to the agent and ancient can see what what the ppf program is currently seeing and doing so. Two technologies that use ebpf quite heavily to implement networking on kubernetes are psyllium and hubble, and the two are the two come together so psyllium is, uh is in its core a cni, a container networking interface, which means that it implements the part-to-part network connectivity on your kubernetes cluster.

A

It is the piece of software that is, that is responsible for forwarding your pow-to-pod traffic, but psyllium has a few additional um features on top of that, such as, for example, service based load balancing. So it comes with a cube proxy replacement, which completely replaces the cube proxy.

A

No port and load balancing implementation, but tilum also has additional features like uh it supports kubernetes network policies, which means that we can enforce network traffic rules. We can you can um block certain parts from communicating with each other and psyllium also provides note to node encryption hubble. So, while zillion kind of manages the network connectivity in our cloud, cluster hubble allows us to observe it and more so it's the it's. The network, observer observability part of psyllium, and it can produce things like service dependency maps.

A

It has a troubleshooting api that we can use to dive deep into what's currently going on, and it also provides us with mechanisms for to obtain metrics and to do monitoring so hubble and psyllium are quite intertwined.

A

We released a preview of hubble late 2019, where we released the basic functionality of metrics and the command line tool, as well as a preview of our graphical user interface. This was built for celium one six, so in cilium one seven uh we open source, the ui and psyllium 17 is the first release of psyllium to also contain additional features specifically designed with hubble in mind. So it supports visibility, annotation and it also gives hubble more metadata about the trace events. Event study limits.

A

In march, we had our first kind of proper release where we also provide some apa stability and psyllium. 0.5 is also the last release that we support for syllabus 17, which is kind of where hubble and psyllium are kind of separate entities, because in psyllium 1.8, which was rela, which was released in june uh 2020, we integrated hubble and psyllium even closer together. So the hubble api is now directly integrated into the cilium agent, which allows us much much better performance and the release itself also had a bit more stability.

A

Improvements in terms of the ui and stability improvements in terms of the ui and hubble 0.6 also comes with initial support for hubble relay, which allows you to do cluster wide troubleshooting as well.

A

So how does hubble use evpf, as I've mentioned before in sydium, 1.8, hubble and xilium are both part of the same user space process. The xilium agent, which is responsible for submitting ebpf programs to the kernel to implement all the features that I've mentioned.

A

So psyllium attaches ppf programs to all kubernetes pots that are in the system or on the local node, and it subscribes to the kubernetes api to be informed about kubernetes resources such as existing pods, existing services, nodes that have been added to the clusters and, of course, network policies that cilium needs to enforce.

A

So it takes this data obtained from the kubernetes api and writes them into ppf maps, which are then accessed by the epf ebpf data path.

A

So if a part wants to connect to either a part on the same node or to entities on a remote node, it will have to go through the ebpf data path, and so the ebpf datapath will take the incoming connection we'll inspect the packages see if it has to do some load balancing. So it will access the service map to find service, backends and frontends and maybe rewrite the packet. Even it will annotate identities.

A

So it will check the source and destination ip and see if these are known identities, which is important for policy enforcement, so policy enforcement in the policy map. We will see whether this first identity is allowed to access this destination identity, and while the data path is working, it will emit, trace and drop and policy events at certain points in the code, and these events are also pushed into a ppf map. The event ppf map, which is a ring buffer, which is read by a hubble instance running inside the psyllium agent.

A

So this hubble instance will collect these networking events, um store them in a historic buffer and expose it via grpc and metrics, and these can then be accessed by other hubble components such as the clients that we have like the graphical user interface and also prometheus.

A

So, to see a bit more details of this, uh we have the hubble ui, which is the graphical user interface, which will give you service dependency maps. I will have a demo later with the ui to see how it actually looks like in in practice. The ui also allows you to display flows, filter flows, so you can. You can only see a subset of flows if you're interested, and it also is able to display your cellular network policies.

A

The common line interface is a bit more for a bit more advanced troubleshooting, so it gives you all the details, the details that are available in the flow you can filter the output quite extensively. We have quite a few filters that are implemented and the command line utility also has a json output that you can use to um that. You can use for scripting, and so these are the kind of two, the two troubleshooting tools that we have, but we also provide metrics. So hubble has built-in metrics for both operations and application monitoring purposes.

A

I will later show a bit more how they work and and what they provide, but basically they can be used in your typical grafana and prometheus stack to to monitor things going on in your cluster.

A

So the hubble api: how does it how what does it provide? The hubble api provides access to recent flows that occurred in the cluster. um Typically, it's uh 4k events that are stored in each node, but this is configurable. You can also stream currently ongoing flows as well. You can submit filters and only see the flows which match a filter or you can exclude flows based on the filter and in xilium 1.8.

A

We also added a new feature for a cluster-wide visibility, so the hubble api is re-implemented by this component called hubble relay, which you can talk to, and it will push down the filters to all the different nodes to make sure that you can. You can find all the flows occurring in your cluster and, as I've mentioned, the api is accessed by both the cli and the ui.

A

So what what does the? What does the api provide in details?

A

It has a few information that are obtained directly from the ebpf data path, so flow metadata, such as ethernet, ip headers, tcp flags, but also things like http and tns traffic, as I will show that in a second it annotates kubernetes information such as pod names, labels, service names, uh node names, and if, if available, it will also match um the domain names and will show you the domain names of of either the source or destination of a certain flow event, and we also attach a few psyllium based information such as security identities, um drop reasons and policy verdict decisions.

A

So in the in the screenshot there, you see how the cli looks like so we we have a small four part example. This is stolen from the psyllium getting started guide and we see that we can use hubble, observe the cli to filter uh by label.

A

So, in this case, I'm doing a class equals x-wing label filter to only see events from parts emitted matching this filter and we invoke it in follow mode, which means we'll see all the live flows, and in this example, we see that the x-wing part is accessing coordinates to do a to do a dns lookup. You see this on port 53, which is udp traffic, and this is then followed by some https request to an outside entity. In this case it's uh apparently accessing disney.com.

A

So we see the full tcp connection, including the shutdown as well and at the bottom there we see how it looks like if the part, if a part is accessing something which it is not supposed to so. In this case, the x-wing part managed by the rebels is trying to access the death star, which is managed by the empire, and the empire has installed a policy that doesn't allow this flow to go through. So huble informs us about this drop as well.

A

All the information that I've showed you in the previous slide is information obtained directly from the ebpf data path, but hubble can go deeper if we opt into l7 visibility. We can also see more details. So what this will do is that psyllium, if we, if we annotate a part, so in this example, we annotate the tie fighter part with a visibility annotation, which says that we are interested in outgoing traffic, uh outgoing http traffic, on port 80..

A

So as soon as we do, this psyllium will transparently insert a proxy between the part and all outgoing endpoints, and this then allows hubble to observe the metadata emitted by the by the android proxy. In this case, so we have support for http, dns, tls kafka, but the android proxy can also be extended with uh different protocols, and we can then access this data from from hubble.

A

As we can see in the screenshot there, we are using the cli to filter based on uh the so we're filtering from the tie fighter, part which we just annotated, and we tell it to only display us http request, http responses which have status code 200 and we output this as json, so using jq.

A

We can actually then parse this json and in this case I'm just extracting the l7 information and we see all the metadata that hubble has access to, which is things like the latency, but also details like which url was accessed and the http headers. So in this case we see that the request was apparently made by curl.

A

So this is what the api can do, but, as I mentioned, hubble also provides metrics. So these are based on the openmetrics format. Hubble exposes a matrix endpoint on each node, which then can be scraped by prometheus.

A

There is a ton of built-in metrics that I'm not be not able to show all of them today, but you have metrics for http details, so the l7 visibility details can also be integrated to metrics. We have metrics for dns requests and responses, we have metrics for tcp flags, we have metrics for icmp and many many more.

A

We also provide uh dashboards for graphana, so you can just use our example dashboards to to try this out and the metrics can be enabled and disabled on a metric biometric basis, and it's also quite easy to write new metrics as well. If you want to so, this is kind of the overall overview of hubble. So, let's, let's see how it looks like in action.

A

So what I have here is a kubernetes cluster running uh psyllium and hubble, so we have a 3-0 cluster running psyllium and as well as hubble and hubble relay and ui, and if we check the status output of psyllium, we will see that we can access or the table is enabled and that we can also access metrics as well.

A

So in this cluster I will now access hubble ui. I've done a port for roading already ahead of time. So we now we see the hubble ui. This is how it looks like it displaces a service map for this. This is an example application and it generates the service map only by the flow events emitted by hubble.

A

So in this case we see an application which is kind of a job portal where we have recruiters, which are posting applicants, and we have a job posting part which is creating new jobs and listing them and people applying to jobs both via an api which, on the backend, then access elasticsearch. We have a crawler which is crawling twitter to find cvs and kind of information and loads them into a kafka topic which kafka, as we know, also needs zookeepers.

A

So we kind of see the service dependencies here, quiet quite easily and we also see which endpoints are being accessed thanks to alzheimer's visibility. We have the policy view, so we can see that there's two policies, one on the elasticsearch and one of the crawler part, and so for the crawler. We seem to have a policy for egress or for outgoing traffic, and we see that the crawler pod is only allowed to access twitter and the loader pod that it that it needs to access and the dns for cube dns as well.

A

So this make sure that the crawler pod only accesses the network connections that it is supposed to, and we see indeed in the service map that we only have flows going to twitter and the loader. So now, let's try to see what happens if you actually try to access an an entity which is not part of the of the policy which would be backed by the policy.

A

So in this example, I'm now cube cuddling into the trial report and I'm accessing a current request to the psyllium.io domain, and we will see that after three seconds, this will timeout, because the policy doesn't allow it, and indeed we can double check this by using the hubble cli. So I will ask for drug flows for for the crawler part, and we will see that we have policy denied events coming in.

A

We can also see this in the ui, so in the ui we have the flow view as well, and if you refresh refresh the page here, you will see that the drop flow will also occur. If you filter by crawler pod, we can see the drop flow there as well, so this is kind of the ui and the service map and the information it provides.

A

Another thing that hubble provides are metrics, so on the second tab there I have grafana already open. These are the metrics that hubble provides. So we can, for example, see the general metrics that hubble produces. We see the amount of flows per second per node. We see what kind of flow types that we have.

A

We see also things like drop reasons and so forth. So these are all generic metrics that you can enable and disable based on your preferences, so we have yeah. We have drop forward flows.

A

We can also see there where trace events are being omitted, and here, for example, you have the uh the drop reason and if you look really closely, we can see that there has been a slight increase in policies, policy denied drop reason, which is exactly the flow that we just tried to emit on the pod.

A

We also see other information like tcp, icmp, udp information, so there's a ton of metrics that you can collect. I'm not going into details here right now, but uh there should be something for both application operations and security persons here.

A

So let's take a look at an l7 metric, so these are http metrics being collected in the whole cluster. So this is all http metrics for all the http traffic in the cluster that we have visibility into. So we see things like methods. We see that most requests seem to be successful, so almost no 500s and we also have latency information. So we see the medium latency, which seems to be below 10 milliseconds, and you have to tell it and say which in this case starts to look a bit weird right.

A

So in the tail latency we seem to have these bikes and if we, if we look closely, we see that there's uh so this is bucket based. So we see that the um up to 200, millisecond bucket seems to have quite a few spikes that are occurring, maybe every 30 seconds or so so there seems to be a performance problem with our application. Something is a bit fishy in in this application or something especially in our cluster, and we want to now use hubble to figure out what exactly we want to do.

A

So, let's start to investigate the the way I will investigate this is using the hubble cli using the json output. So this is a jquery which will sort the all http and requests by latency and we only showing the the maximum latency for each url, and so we see here that indeed that there's two uh endpoints job endpoints and core api, which seems to have quite a high latency while the rest here seems fine, there's only two endpoints that have this really weird highly high maximum latency.

A

So since this is on the core api, let's start to investigate this. So we can filter by recent traffic say for the last three minutes. You want to see all traffic for the core api part, I'm using a label filter here and I'm just using less to kind of be able to grab the output. So this is the hubble output. Again we see all the all the events emitted by the core api pod and I grab for the end point.

A

So here's the request that seems to have only two milliseconds late, see that's fine and if I go to the next one, I now see there's uh yeah, there's now a request which has 106 milliseconds latency. So this is one of the degraded requests, so we have the request coming in up there and then 100 milliseconds later we see the response going out at the bottom right there.

A

So, let's see what happens in between so we seem to have a bit of uh a bit of uh unrelated traffic for fro from the front of the proxy. um This is fine. The tcp flags look fine, nothing, nothing too conspicuous here.

A

uh So this is just just a request coming in from the proxy and being being sent to the core api.

A

But then we see uh we see on this line here we see the core api trying to access memcache d, so it seems to seems to be using caching and we see that the tcp flags indeed for the response are reset. So this connection is being dropped. So it seems like the core api is unable to access the cache for some reason, and we see then the elasticsearch um requests going out to the back end to fetch the information from from the backend instead of the cache.

A

So let's see kind of what connections we have to this memcache d um instances. So I've copy-pasted the the domain for the memcached service. So I can again write with the click and write a filter where I say show me all the traffic to this domain, and this will show me all the traffic to the domain.

A

And we see that there's a few different connections, but all from the core api. Now I'm doing a second request as well, which will get rid of the of the name resolution and just show us the raw ips, and so um here we don't have the annotations anymore.

A

This is just the raw data, but if you look at this, we see we see the requests at the resets being emitted there, and so, if you look more closely, we can actually see that the reset only seems to occur on ip23, the other, the other, um the other destinations like point 25 and point 24 seem to be fine.

A

So um let's quickly double check what kind of um ips are associated with this domain, so we can filter by dns request, and indeed we see that we have three ips associated with the memcache and one of these ip is the top 23, which seems to be behaving really really. Strangely.

A

So, let's, um let's let's see if we can reproduce this, let's see if, if there's really a problem, so now I'm keep cuddling into the core api part and I'm just accessing the ip directly on the memcached port. And if we act, if we execute this command, we should see whether or not we can access memcached and indeed it seems to fail for this particular instance. So let's try a different one, the other one served.

A

So we have established the root cause now um that indeed point 23 seems to be unaccessible, so we have let's check again. We have three back ends for the memcache d, and so we've we've established that there seems to be a problem with one of these one of these nodes, and we can. We can check that- and in this case, of course I I know what the reason is, because this is a demo that I've prepared, so uh let's ssh into the ins into this instance and see what's going on.

A

So I will ssh into this faulty memcached part and show me the list of containers that are currently running or have been stopped. So if I use docker ps on that instance, I see that there's indeed a memcached that has been turned off an hour ago almost so that would explain why we see dropped connections, so I can now start this start this container again by just starting the docker container and once the container is up again. We can now verify whether the connection works.

A

So again, I'm using cubecode directly access the part, and now the instance is up so a few minutes later. We can now check again how things are so we still have access to the memcache. That seems to work fine, and if we now check the metrics and refresh, we will see that now that we have fixed the memcached backend traffic is healthy and performant as expected.

A

So this has been a showcase of hubble. uh I hope you have understood uh what hubble can provide for you and what you can do uh if you want to try it out yourself. Please do so on the url on the slides. Here we have a getting started guide which will guide you through a few examples as well and hubble, of course, is open source. So please feel free to check out the source to open, uh prs and issues so yeah um thanks a lot for your attention and goodbye.

A

All right, so thanks a lot for watching, uh we do have a few minutes for a more additional q. A I already tried to answer some during the talk in the chat, but I will now also answer kind of the the most common ones here in the uh in the in the q a session. If you have more questions afterwards, uh please feel free to drop me a message either on the kubecon slack or also make sure to join the psyllium slack.

A

This is psyllium delay slack for especially for more details, technical questions um so yeah an answer that that came up or a question that came up quite quite often is the question better.

A

It is mandatory to have psyllium as a cni in order to use hubble and what this means for things like running hubble on cloud providers like gke or aws, and so the the the answer is is is that, yes, you will need to deploy psyllium as a cni to use hubble that the two are really tightly coupled, however, cilium as a zni is able to chain uh with other cni's as well, so you can actually use psyllium.

A

On top of the cheek on top of gke or on top of the aws cni without problems and psyllium, will integrate quite nicely with the existing cni. So psyllium will do things like the observability for hubble, but the actual part. The pod traffic is then handled by the aws cni, for example. So, yes, it is tied to psyllium. But, yes, you can also run psyllium with other cni's as well.

A

Another question that came up is uh whether we can also use deploy hubble for vms and parameter workloads, and the question here is kind of or not yet so. Psyllium has recently gained in the latest tiller release. We have gained the ability to uh use uh host firewall rules, so you can now protect hosts using serium as well, and this also means that psyllium will get traffic can observe the traffic on a vm. So this is something we're actively working on intellium and hubble will gain the we'll gain the uh the workloads as well.

A

um Our hubble will will be able to to see. This is we'll be able to see the the uh data produced by vms and parameter workloads as well, but this is something that's currently um being worked on in in uh enthelium, so yeah. I think the other questions I've already answered in texts.

A

um There has been a question about security, so I didn't really talk that much about security. uh It's, of course you can use hubble and and uh stomp the dump. The uh events in your seam or or another kind of um security focus tool as well. If you have more questions in this regard, please reach out to us uh on the slack.

A

um I think this is kind of a more detailed question where uh that the use case might might be a bit more um where we have to discuss the use case first, whether it's something but that is in scope for hubble or not, but generally speaking, we absolutely also use hubble for security um observability, uh especially like the network security policy. Monitoring that we have in psyllium is this. One is one use case that I've demonstrated in the.

A

Talk all right.

A

I think this answers almost all questions that we have. I will follow up with the rest uh here in the in the platform and again, please feel free to reach out to me on slack. This is either the cncf slack or also the psyllium slack. I will be available in both uh the the uh the channel on the cncf slack. Is the kubecon observability channel and yeah thanks a lot for attending and enjoy the rest of qcon. Thank you.