Cloud Native Computing Foundation Cloud Native eBPF Day EU 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: eBPF-Powered Observability for Telco CNFs - Junichi Kawasaki, KDDI

Description

Lightning Talk: eBPF-Powered Observability for Telco CNFs - Junichi Kawasaki, KDDI

One of the critical KPIs for telecom operators is 99.999% availability. Of course, it should be assured even after introducing Cloud-native Network Functions (CNFs). How can we achieve this carrier-grade quality in the coming networks? How to maintain observability for CNFs? eBPF is expected to help telcos manage CNFs and keep the environment resilient with its high-performance, flexibility, and scalability. This talk will share how you can apply eBPF tools such as bcc and bpftrace to collecting fine-grained information (e.g. TCP retransmit) for network operation as well as how such data can be leveraged to developing ML models. This talk will also walk you through a test result in a 5G core network deployed by Kubernetes to show you what network data derived by eBPF are essential for failure prediction.

A

Hello: everyone welcome to this session eppx powered observability for teleco cns. My name is george kawasaki, I'm a network engineer at kdei telecom operator in japan. Currently, I'm working on monitoring and orchestration for autonomous network.

A

Let me start my talk with background.

A

We telecom operators are responsible for keep providing reliable network to customers in 5z for the next, beyond 5z era, network will be provided to more various use cases, including mission critical ones, so we have to maintain carrier-grade quality, 5-9 availability or more than that in future.

A

Of course, we have to keep this carrier grade not only in pnf pnf, but also in cnf cloud native network function. For this purpose, we need to transform our operation style. That's productive operation.

A

Productive operation here means to act before impacting customers. One of the key enablers for this is predictive failure, detection using aiml.

A

However, only ml technologies is not enough. As you know, machine learning models are able to provide outputs based on training datasets. In other words, future failure. Prediction is impossible without good information or observability.

A

So ebpf is expected to take another key role for proactive operation because it can provide fine grain information without modifying cnf codes.

A

So this is our proposal. It performs predictive failure, detection by the combination of machine learning and ebpf.

A

It consists of data collection and analysis part.

A

Our target environment is privacy, core cnns deployed on the container infrastructure.

A

In the data collection, there are two ways of collection: one is ebpf based collecting fine grain information from infrastructure.

A

Another is exporter based one collecting basic metrics by she advisor as well as 5g specific data from content layer and in the analysis, step ml models are developed with the stored datasets and they do time series analysis by pharah prediction.

A

Next, I will explain how to empower the 5g cnf observability by ebpf.

A

First, let me share which ebpf tools we implemented.

A

We take these two feature scenario: cases on our experimental privacy, core testbed number one is increasing packet loss rate in seaplane number two is increasing computing resource usage on one of the ups.

A

Considering these scenarios we selected tools in pcc and ppf trace libraries to derive useful insights from the over 100 tools. We chose these ones listed on the right table.

A

It includes many tcp related tools, as well as some processing related.

A

We selected the promising tools, but here raw data itself, derived from ebpm, are not provided by cnf content, each tool, outputs all the data placed on the server without considering which data are related to amf or smf.

A

From the ai point of view, the data should be differentiated for cnf type 4 analysis, so we coordinate data by key information.

A

We use two types of mapping: pid-based and ip-based in pid-maze-based mapping, pid process id or pld namespace. I used as a key after creating cf container, execute doc ps command like this to get pid and type ls command with the argument, including this specified pid to get the corresponding pi namespace, as this is shown here by referring to pid namespace. In this case, the collector is able to know from which cnf data are coming.

A

The other type is ip based mapping, especially for tcp related tools, it outputs the traces with the source and destination id, for example. This 0.14.45 is one of our mf ips, so the connector is able to take the data whose source id is total file, as data related to amf for additional information in this mapping. It also calculates the maximum minimum and average values from this histogram and add a supporting information.

A

Here is a data set that is obtained after mapping.

A

Our system collects this json file every 10 seconds.

A

As you can see, it includes collection, time enabled cnf 5g, related information, energy advisor information, including basic cpu and memory matrix and the ebp of data related to amf is surrounded in range, presenting run, qrad, tcp drop and tcp matrix, where only partial information is shown here due to the space limitation.

A

This json file includes all data related to 5g, core or cnf type.

A

Regarding analysis part, we use lstm modem. That is one of the major deep learning models for time series analysis.

A

Finally, I will show you the experimental result to share how much ebpf can contribute to lstm based future prediction.

A

We used this 5g core cnf network that was created on the linux server by kubernetes, two types of data ebpf and she advisor are collected from the network for comparison. We developed two lstm models. One is a model trained with all data, including ebpf.

A

Another is trained without evpf in the evaluation, the trained models inputs, multivariate test data and then outputs the predicted number of registration failures.

A

We evaluated the prediction performance by determining if the output future prediction can detect problem or not with a threshold.

A

Here is a result of prediction performance at 150 seconds after the federal event starts.

A

This is a result for case number. One.

A

The first model using ebpf shows high performance. You can also see the prediction accuracy on the above graph. The solid line here is true, and the dashed line is predicted values at 150 seconds. This model predicted the future increase of the failed registration.

A

On the other hand, the other model without ebpf did not get a good score and also there's a big difference between the predicted and the true values. This result shows ebbs highly contributed to developing the lstm prediction models.

A

In conclusion, from our practice, experiment, ebbf tools with several mapping processes enabled 5g cnf, observability and lst models trained with the dataset obtained by this observability achieved high performance weather prediction, which will hopefully lead to productive operation in future networks.

A

Thank you for watching. I hope you have enjoyed and run something useful for your activity.