Cloud Native Computing Foundation Cloud Native Telco Day EU 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Network Anomaly Event Prediction and Optimal Resource Control... Takuya Miyasaka & Takaya Miyazawa

Description

Network Anomaly Event Prediction and Optimal Resource Control in Cloud Native Network Functions - Takuya Miyasaka, KDDI Corporation & Takaya Miyazawa, National Institute of Information and Communications Technology

The use of Cloud-native Network Functions (CNFs) in the telco industry has attracted attention in recent years, and examples of CNFs adoption such as 5G and telco cloud have started to emerge. While CNFs enable quick service deployment, can typical requirements of telco service such as high availability and optimal allocation of the computational resource be realized in CNFs? In this session, we will share and discuss our experience in leveraging CNFs as an infrastructure for telco services while satisfying these typical requirements. Takuya Miyasaka will share lessons learned on eBPF observability in a 5G core network deployed by Kubernetes. In particular, he will show that the detailed metrics of each 5G container measured by eBPF enable the detection of future anomaly network events in the 5G with Deep Learning. From a different perspective, Takaya Miyazawa will share his ongoing activities on an autonomous computational resource control system for CNFs being compliant with the ETSI Open Source MANO (OSM) and ITU-T Y.3177 standards. In addition, he will share his work on deploying the control system in a Japanese public network testbed called JGN.

A

Good morning, good afternoon or good evening, my name is takuya miyazaka. I am a network engineer in kddi, which is a japanese network operator company. Today we will talk about the network, anomaly event, protection and optimal resource control in cloud native network functions with my co-presenter takaya miyazawa.

A

Actually, we would like to attend this conference on site, but due to the copied 19 restrictions, it became very difficult for us to go to spain, so we appreciate the event management team for preparing this remote presentation.

A

This is an introduction and our motivation of this talk. The use of cloud native network functions in the teleco industry, such as 5g and mec, have started to emerge from the network operator perspective. The benefit of cnf includes quick self-deployment flexibility of operation and management and diverse business model accommodation.

A

While these benefits are attractive to us, but as a network operator, we need to fulfill the typical teleco services requirement, such as high availability, resiliency and optimal resource utilization.

A

To realize this requirement, there are some challenges we need to tackle, with, for example, to realize the extremely high availability we need to protect network anomaly event and resolve the anomaly event before our customers have bad experience.

A

This slide summarizes our lessons shared in today's talk. In the first section I will talk about the network and nobody even prediction in the 5g network using the ebpf and deep learning technology.

A

After my talk, takayama zawa will share his experience on the optimal resource control in cnf.

A

So let me begin the introduction of the network. Anomaly event projection part.

A

The network anomaly events are inevitable for network operators and occur almost every day on our commercial networks, typical example of the network. Normally events increase network latency and increase the packet loss rate and increase the cpu utilization rate.

A

So this network diagram shows a 5g core network on the kubernetes environment and, let's consider an example of a packet loss event in the control plane of the quad g core network.

A

So for some reason the packet was late in the control plane of a 5g core network built on the kubernetes starts to increase at some moment.

A

If we don't take any action on this issue, the pocket knows late will gradually increase and at some point it will have a significant impact on the communication between network functions in the control plane. And, finally, it will cause a major problem in customers 5g services.

A

To avoid such a situation, the monitoring system needs to detect this network anomaly event as soon as possible.

A

This is a motivation. Our motivation is to detect these network anomaly event rapidly and precisely, but is it possible?

A

Our hypothesis for this project was basic metrics such as cpu utilization rate and not enough to realize the rapid and precise detection?

A

This is because the amount of information of such a basic metric is very limited and we consider it insufficient for the future protection by deep learning.

A

So our idea was ebpf observability.

A

So, before jumping into the details, let me show you an overview of our approach.

A

Our approach consists of two parts and our target application is 5g core containers which is built on the kubernetes on top of a linux kernel.

A

Firstly, as I said in the previous slide, we use the ebpf functionality in order to correct the per container detail metrics.

A

So this diagram shows an example of ebp metric and shows the number of tcp transmission of 5g core network functions container like amf, and this you know the ebb metric measured under an anomaly event that the packet lost late increases gradually.

A

So, as you can see from this diagram, as the packet lost rate increases, the number of tcp transmission also increased due to the tcp packet drop.

A

On the other hand, this observation also implies that it may be possible to predict network varia from these ebpf metrics. Secondly, ai and ml.

A

We used an aiml model from the ebpf metrics to predict the future key indicator of 5g qoe for customers.

A

This example diagram shows the prediction of the number of ue registration failure in 5g network. If we can predict the key indicator precisely, we can detect the anomaly event that violates a threshold or sla.

A

So these two things are overview of our approach and, from the next slide, I'd like to talk about each part in detail.

A

So ebpf, so this table shows some of the ebpf metallics collected in our work, so we collected some cpu related metrics like a long queue latency.

A

Also we corrected several tcp related metrics such as you know, the tcpd transmissions.

A

So next these two graphs show the ebpf metric behavior when an anomaly event occurs.

A

The top graph shows the behavior of ranking latency in the case of the cpu utilization rate increase event, and the bottom graph shows the behavior of tcpd transmission in the case of a packet loss event, and, as you can see from these examples, so several ebpf metrics are positively correlated to anomaly event. Therefore, we expect these detailed metrics can be applied to deep learning for the future projection.

A

Next is the detail of the aiml part. We use the lstm model, which is a recurrent neural network, based deep learning model for the future prediction of time. 3 data such as the number of ue registration failure as a data input of lstn.

A

We apply the basic metric of each container collected by c advisor and evpn metrics, and using using these past action metrics as input nstn can be used to predict future metrics.

A

This slide shows a variation result of lstm in the packet loss event in the control plane of the 5g network. In this variation we simulated many anomaly. Events in our experimental 5g network and collected metrics of 5gc network function container for the lstm learning.

A

Our first evaluation results shows the ebpf enables more accurate reduction in the case of ebpf and c advisors. Basic methodic, accurate prediction was achieved at 150 seconds after the start of a normal event. On the other hand, without evpf at the same 150 seconds time point, the accurate prediction were not possible.

A

This upper line shows actual value and the lower dashed line shows the predicted value by lstm. So, as you can see, there is a large estimation error without eppl.

A

So although these results were obtained in our test environment and not evaluated in actual commercial networks so far, but it is expected that the ebpm will enable rapid and accurate detection of anomaly events in 5g network.

A

So this is a summary of my part. Thanks to ebpf, detailed per container metrics can be collected even in cns 5g network, and our first evaluation result shows ebpf and aiml will enable faster detection of future network anomaly event in cnn's 5g network.

A

With this I finished my part of the network anomaly event projection in 5g network and from the next slide. Miyazawa will talk about optimal resource control,.

B

Next, I'm going to introduce our identity on optimal resource control in cloud native network functions, my name is programmer server from nic to japan.

B

This is the summary of the stroke. An icd operates, a public network testbed called in japan. There are several access points, but this time we have created several vital machines and constructed a small scale of virtual network in oblique region, middle and north parts of japan.

B

The pattern network consists of several nodes, each of which has several vital machines.

B

Honorific network functions are directly deployed on a virtual machine, as vnf cloud native network functions often called the micro services are deployed on containers created for virtual machine or host os and as cnf we have installed hc open source manual, osm on the jzn hot click region, and it operates these network functions.

B

We have also deployed our ai machine learning engine for analytics of data such as ncp utilization and uncomputational resource control, especially for cnfs and connected to the osm system, as a northbound component.

B

Important thing in this talk is that we have implemented an additional interface, especially for auto scaling. Data interface is the main topic in the store.

B

This is a framework of air assisted and computational resource control and management for network functions. This framework is in compliance with itu-t wide 3177 standards. Ai machine learning should be installed into both data analytics and resource control decision to provide high ability to find a solution of dynamic resource adjustment.

B

The data analytics system analyzes and predicts the resource usage based on measurement results obtained from the only network and or cloud infrastructure structures that is usually completed in seconds or in minutes by utilizing machine learning.

B

Based on the analysis result, the resource control system decides a solution to execute source arbitration among services and or network function, migration, which is compressed in seconds or in minutes by means of machine learning.

B

Our goal is to simultaneously achieve maintenance and enhancement of quality of services, effective utilization of computational resources and agile, processing of data analytics and resource control in the time currency of seconds or minutes.

B

This figure illustrates our verification pattern, network infrastructure and consisting of four network nodes, seven rings and two end holes that we have and constructed a part of it on an sct public network, testbed called jgn, ahoglic region and constructed other parts in cochrane city in doctors and interconnected both networks on around 400 kilometers.

B

We have uh installed on uh hd open source version 10 into our network function, operation system by quantities, installation as northbound and components. uh There are ai engine, osm, client and virtualization system.

B

The ai engine obtains data from the underlying infrastructure, such as cpu utilization of each network, analyzes function data and decides the solution for adaptive, resource reallocation to each network function.

B

So what is lagging or insufficient in the osm version 10? Firstly, it cannot obtain some information that is necessary for adaptive resource allocation to cnfs, with higher accuracy and higher scalability.

B

Concretely, the osm cannot guess the current real-time situation of fluctuation in cpu utilization in every second, although it has a function of autoscaling.

B

Secondly, it has no function of designated scaling, for example, designating another virtual machine to migrate a containerized network function from the current battery machine.

B

Besides, it is impossible to designate the amount of cpu resource to be increased or decreased to each cnf.

B

Thirdly, the managed ability is relatively low, especially in terms of support for visualization.

B

Firstly, the operability is relatively low, especially in terms of arranging the phone by manual operations, for example, encoding a web api and receiving json reply.

B

Finally, no aiml function to analyze, brs data to be reflected in network function, operation.

B

So to solve the issues that I showed you in the previous page, we have added two styles.

B

One is an additional interface for a function, enhancement called direct interface, the other one is our ai engine to automate cpu data analytics and computational resource control for cnfs by utilizing ai and machine learning techniques.

B

I'm not going to explain the ai engine in this talk, but, briefly speaking, we adopt several machine learning algorithms, such as support vector regression, lasso and encoder decoder, early current neural network for time series, cpu data analytics and automate automatic resource control decision.

B

This talk focuses on this direct interface for a functional enhancement.

B

This is some overview of a direct interface which is added to the osm, the noiseman on components such as ai engine and the visualization system, and a mineral component such as bim quantities and nfp infrastructure, are connected with each other by the direct interface via kafka bus, without going through the osm internal components such as lifecycle manager, a resource orchestrator and so on.

B

The main objectives of this additional interface are to obtain necessary resource information and resource arbitration among cnfs and nf migration. From a server to another server, in other words, the additional interface may only be effective for these purposes, and the original osm interfaces and internal components may be effective for all other objectives.

B

So it would be better to equip both interfaces in the system just by adding the direct interface to the current osm.

B

So what are the benefits of a direct interface? The first is to obtain a rename cnf cpu utilization in every one. Second, we utilize c group to realize it by tuning for each vrm, which means each and quantities cluster. We contain higher accuracy and scalability in obtaining a real-time situation of cpu utilization of each network function.

B

The second one is this signature scaling in horizontal scaling. We can specify a partial machine to increase support for each in cff. The current osm communities cannot do this without our direct interface in particular scaling. We can specify the amount of cpu allocated to network functions again. The current osm commanders cannot do this result or a direct interface.

B

The third one is higher manageability. It's supposed to visualize a lot of information obtained by multiple osm nodes file interfaces.

B

The first one is higher availability. We have obtained a convenient command dctl, which enables an easier measurement of processing time due to a function to function, to designate, asynchronous or synchronous to each request.

B

So finally, general conclusion for two things:.

B

The use of cns in the telco industry has been receiving much attention in recent years, while cnfs enable quick service deployment. We have still been investigating a solution to realize the typical requirements of telegraph services, such as high availability and optimal allocation of the confidential resources for cnfs.

B

In this session, we have introduced our experience in utilizing cnfs as an infrastructure for technical services, while satisfying these typical requirements in the form about. We have introduced lessons learned on ebpf observability in a five sequel network deployed by kubernetes and showed us. The detailed metrics of each 5c and container measured by ebpf enabled the detection of a future anomaly network events in the 5c with deep learning, and then we have introduced our ongoing activities on autonomous computational resource control system for cns being compliant with the ac osm standard and deploying the network function.

B

Operation system within- and I say the network testbed called jgn. So thank you very much.