Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 24 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Network Observability with IPFIX, Prometheus and Elastic Stack - Rastislav Szabo

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Network Observability with IPFIX, Prometheus and Elastic Stack - Rastislav Szabo, Pantheon Technologies

In traditional networks, protocols such as Netflow or IPFIX are used to export packet flow records from networking devices. In k8s, each node effectively becomes a switch/router of the traffic passing between the PODs. Some k8s CNI plugins allow using these protocols to export traffic flow information as well. Together with good analysis tools, this can provide better visibility into the network in the cluster (both in real time and in history), which is crucial for debugging networking issues, identifying the bottlenecks, or investigation of security incidents. In this talk, I will showcase how a powerful IPFIX collector and analyzer for k8s can be build using existing open-source tools: GoFlow and Logstash for collecting and post-processing of the flow records, Elasticsearch as the storage and search engine for the flows, and Prometheus + Kibana for easy observability of the flows.

https://sched.co/MPbp

A

Hello welcome everyone, I'm Brian, Boram I'm. The track host for this session. I have done a little bit of networking myself but you're all here to listen to the the speaker here. So let me introduce a rostislav Zabul.

B

Hello, everybody so I know it is possible, and today I'll be speaking about network observability with a pickax for me to use and elastic stack before I begin something about me. I'm, a staff engineer at Pantone, Tech, architecting software solutions, mostly for networking industry previously I, was mostly focused on network manageability, using net convene yank protocols, currently working on cloud native networking projects and I'm contributing to to several open source projects such as repo, which was not convey unrelated FDI, all mostly in the GUI sub sub project, and we got Owen County VP.

B

They are cloud native, regulated projects and I'll be briefly talking about them. During my talk as well. So, let's start with some motivation. Why do we need Network observability in kubernetes?

B

Well, I would say that many kubernetes cluster operators do not care too much about the networking, while everything worse works as it should, but at the very minimum you would at least need to be alerted whenever there is some some networking related problem in your cluster, either some photo issue or just unexpected congestion or a packet drop, and that might be just between the poles on on a on a node or even in the fabric between between the nodes or in the underlying network.

B

Apart from apart from network tailors, another motivation would be identification of the bottlenecks in case you, you want to distribute traffic in the cluster so that so that everything work works correctly, usually want you to see whether your network would would properly handle your water distribution and, of course, in case you are doing large-scale deployments. You want to know. Where are the limits of the scale?

B

Another motivation would be detection and investigation of malicious activities, for daddy, of course, may need to for the detection. You that's quite easily doable for the investigation. You may need a little bit more data, that very usual and since I'm I'm, an engineer in the field of Clyde Navy network functions, which is network functions built as a set of micro services, for instance a firewall or or carrier-grade NAT running in a kubernetes cluster as a set of pots for those use cases.

B

All everything with what I have just said becomes even more important so now, knowing the importance. What are the current options for network observability in kubernetes? I would say it's matrix everywhere, right, pretty much.

B

Every piece of everything that you deploy in your cluster exports a matrix and the same is true for the networking you have seen ice the export matrix actually only few of them export the metrics that are useful for for for the observability into the network itself, some of them just export some, some internal counters of errors of programmed interfaces etc, but, for instance, not package ride on individual interface over port.

B

Another option would be service meshes. Of course, if you are using service mesh, you get a lot of visibility, but mostly on the application layer not into the lower lower layers of the network, but, for instance, still can support even even TCP telemetry data, and you can. You can collect that with chromatids and display.

B

The third option would be doing the metrics yourself. You can do something like your own monitoring tool, which would monitor each one network namespace of each one pot that is running in your cluster and and export the statistics on the interfaces.

B

That's a bit of work, of course, and bring some challenges as well, but the most important point that I want to what I want to mention in here is that matrix may be node enough was imagined that okay, you have met this actual example of a oversee and I called County VP P, which provides from at Prometheus metrics for each one put interface.

B

You can you can display packet, counters, byte, counters packet drops and that kind of data you can get all of that into primitives. You can display that in Granite, for instance, as shown here, but the issue is that in case of what's imagine that you have a security incident in the network, you can easily spot that something is happening in the network right right. You have a spike in the graph, but you cannot really tell what is causing the issue. You don't know from where the attacker comes.

B

What is the actual traffic which would use this for further attack etc? So for that, you need a little bit more data than just matrix and if I said that metrics are not enough. So what would be a solution of the issue in the term? In traditional networks we used protocols such as methyl ethyl and IP fix, we'll just ask how many of you have used net flow or IP fix in the past.

B

Okay, like one third of the people here, that's nice.

B

So, for the rest, what net flow and IP C IP fix essentially does in traditional networks you can enable that feature on a router or on a switch in the network, and the router would would collect some most important, important information for each flow in the network, such as source and destination, IP, address and port type of service interface, or to reach the flows for flow passes, of course, packets and bite counters and times some information.

B

The flow exporter, in this case, a router or switch or specialized Hardware probe in the network, would just send this information for each flow into some flow collector, which would be a dedicated component in your in your network. The collector is responsible for receive all of these informations there, pre-processing and storage, and the third component in such a deployment would be some analysis tool which pretty much accesses the data stored and flow stored by the full collector and shows either some user interface or or gives ability to search for some particular data.

B

I also need to say that this flow collector and analysis tools are often commercially sold product, but there are also open source options. Okay, so how we can use the same principle in kubernetes well in kubernetes. What what is actually a router or switch in the deployment is a see. An iPod is a CNI plugin it pretty much either switches or routes the packets between the poles on the same node or or between the nodes.

B

So so the CNI plugin can can provide this data.

B

Cni plugin is actually the only correct place from from where we can get such such data, because the traffic between the nodes is often encapsulated into some tunnel or or something similar. So you cannot really export those information using the switches and routers that you would have in your local network between the between the nodes and as a as a collector analyzer.

B

We can use something that would be twelve native, such as a lock stack, which is which is a shortcut for for elasticsearch, workstation, Habana, three open source components which go together and, and they create a row ecosystem from this third component tree components. A walk stash can be used as a flow collector. It supports it, supports IP, ticks net flow and s flow processing.

B

Elasticsearch can be used as the storage plus the search engine for the false store in the in the elastic search and cabana as the as the analysis tool or the UI which you can use to access. The distort phones.

B

I'll show you a demo in a while. Where will enable IP fix in 1c and I, and the scene is called from TVP. The cni is primarily focused on cloud native network function deployments since it uses FDI of EPP as the data plane.

B

Essentially it is a it is a forwarding, something like a router and switch functionality in a user space process that runs as a religion in node in the cluster. It is a very high performance. It uses something called vector path, a packet processing which means that it can follow art wall set of packets instead of a single packet in in each operation.

B

It provides key proxy functionality all on VPP, so q proxy is actually bypassed there and it also provides a special type of interfaces called mem ifs, which allow a package to traverse from a pot into the v switch through shared memory, and the best news is that it already support our IP fix. So it pretty much needs to be just enabled for the demo allows. The oh I have enabled it only through the debug CLI of the VPP, but for proper implementation.

B

The the process would would go as shown here and I think that eventually I will be able to my upstream that into the CNI. So, first we need to write a plug-in into the county of CNI, which is quite easy. It is easily extendable the CNI programs, DPP management component called legato IO VPP agent, where we need to read IP fix support, that's actually work that which in progress- and this we gotta be BP agent- is able to enable the architects on each of the boat interfaces.

B

Okay and as the flow collector and analyzer I mentioned that I'll use the Aleks tack.

B

Luckily enough, I found a project called elastic flow, whose outer is actually sitting in the audience which provides all the rest glue that needs to be used in order to build this solution pretty much. It is said of codecs and filters in the works and the dashboards in Cabana. So all that I had to do was actually package this into docker containers, deploy into kubernetes and configure the cni to send the flows into the locks pod, as shown in the picture.

B

What you can also see is a list of ports, so, as I said, elastic search box and cabana are just tripods running in the system. This is a result of such an deployment. I will show you this on an actual running system in a second. So this shows the flows between pots in the cluster. The cluster was just very simple: if you cube DNS pots and the route itself, then you can see the different conversations between the pots, so the clients and the server's- and these are the individual flows.

B

But let me go actually into my VM, where I have this running.

B

Okay, so let me first go into cube cattle, as you can see, the elastic search, elastic flow and Cabana running a sports part from that I have one job which is elastic for Kiba now setup. It just pushes some some configuration into Cabana when when it starts, apart from the elastic flow, I.

B

Have the county of CNI running on the note it is just one node cluster, so only aeration. What you see here is actually running on on this note.

B

B

As I have told you already, the County VP PC and I around the VP PV switch on each note. I can show you that.

B

So this is the VPP instance, which acts as the V switch between the bots.

B

These are the individual, inter part, interfaces that go towards the bots. You can see a lot of packets coming to them and.

B

These are the flows that are currently seen by the VPP and export it into the log stash. So let's go back into Cubana, which, as I said, accesses the Falls which are stored in the elasticsearch and they were stored in into the elasticsearch by the works dash pot, which pre-processed the full packet and send it to elasticsearch.

B

So this is a dashboard view where you can see some some top servers and clients. We can go into some top and we, where we all see top clients and top servers running in the cluster. Of course, you can order that by packet count by accounts, flow record, etc. You have the abilities of filtering, for instance, we can look only at the DNS traffic.

B

We can go into four-wheel where we see on this nice graph, exactly which pot talks to which one in the cluster again you can. You can filter out by server by client or by service.

B

There is geoip functionality as well in my cluster. It does not show anything in here, because all the communication is not local but in case that you would have some some clients accessing the cluster from the internet. You would see from where the clients came.

B

We have traffic details in here where we can look into individual client and server flows in more detail.

B

We can browse bit bid rate packet rate between clients and the servers, and the last one would be the actual flow records that are stored in the elastic search, which is the the base data from which we are generating. Those wheels was open some, so this was communication between.

B

This client IP address and sorry it was yeah it. It came actually from the hose itself and it went into DNS spot. So this is pretty much the route flow data. You can use that to build your own dashboards in Cabana in case that you are interested in something else that I've just demonstrated the GZ. Since all the data is in the elastic search, you can use all the features of the elastic search to build your own queries into the database and we use using Cabana one more thing.

B

I had this on one of my first slide, this is actually the graph fauna, so these are the metrics exported by the County VPP CNI. You can see now that I played a lot with with cabana and elasticsearch. There was spike in the traffic from cabana to do realistic, search.

B

Okay and now let me back go back to my presentation.

B

So what I just demonstrated is just a proof of concept pretty much. It was a 1 node cluster which, with 1:1 works instance, or one elastic instance, and one cabana instance in larger clusters. You would actually want it to have the elastic clustered so that it can handle all the all the flows.

B

The number of Falls that are coming from VPP works can be quite huge in clusters, where, with with heavy workloads, so in that case, you actually want to have a log stash pot running on each node so that the flow records that do not have to travel between the nodes.

B

Of course, you don't have to cluster qivana. That's just that's just user interface. One instances you know another possible enhancement would be usage of the memory interfaces between the VPP and and toll collector, as I already mentioned. Memory interface is allow package to traverse between the VPP and the pot shared memory. We should save a lot of processing in the kernel for that.

B

We would need to have a flow collector that would be modified to use mif and one candidate for that maybe agro flow, which is a full collector and go since for the memo we do have a go library which is able to talk to VPP that can be easily integrated together and also what you could use. If you have all the data are stored in elasticsearch, you can export some more metrics, some specific metrics.

B

That would be interesting for your particular deployment into Prometheus in order to to have alerts that that would be interesting for you, of course, because parameters supports elastic search as as a source for the data.

B

Okay, that's all from my side. If you have any questions.

B

Don't free to ask.

B

B

If you, if you need multi node deployment with VPP using the County VP CNI yeah, you usually want to use the PDK interface so that the packet can be grabbed from the Nick into the VPP process directly bypassing the kernel, if your, if your note or VM or or or server, does not support DP decay in able to interface.

B

There are options for for for doing that. To the kernel you can, you can use a packet to grab the packets from from the interface that that would be normally connected to the car to the kernel.

A

Any more questions.

A

Surely somebody right at the back has a question you can watch me run the front. Thank you. I.

C

Have a question regarding manage kubernetes solution: how does it work with managed like UK, you need to setup the VM there gently.

B

So the CNI yeah one thing of the solution is the CNI I used County, VP, PC and I I'm, not saying that it's, the only one CNI that supports IP fix I could imagine enabling in enabling it in others, for instance, by using some some UNIX to some demon that is able to export the false. But for the country with CNI, you can deploy it in some some clusters. But you need to do a little bit hard work yourself and.

C

It requires a physical CPU. The PDK requires no.

B

No, it can be virtualized. The the most important thing that you need is the support for the neck, even its need higher performance, but, as I said, that can be that can be bypassed as well, but you would get modular performance, okay,.

C

B

Okay, thank you. Yeah.

A