Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2020 - Virtual, 4 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Observability Brings Clarity in 5G World - Yamini Sridaran & Praveen George, Affirmed Networks

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2021 Virtual from May 4–7, 2021. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Observability Brings Clarity in 5G World - Yamini Sridaran & Praveen George, Affirmed Networks

Kubernetes has become the de-facto standard for running 5G core virtual network functions (VNFs) - a set of standards defined applications, each implemented as dozens of microservices and often run across multiple K8s clusters. Centralized monitoring, early detection of problems, and deep troubleshooting tools are essential to ensure telco-grade availability of this complex set of inter-connected microservices. In this talk we present our experience building and using our telco observability platform using CNCF projects. This platform provides: End to End 5G call tracing across multiple VNFs using Jaeger 5G network performance monitoring using 1000s of base and derived metrics using Prometheus Fault detection and troubleshooting using AlertManager, Fluentd and Kibana Attend this talk to learn how observability brings clarity to complex, high-reliability applications such as 5G VNFs.

https://sched.co/ekCx

A

Hello, everyone welcome to our talk on. Observability brings clarity in 5g world. My name is praveen george, I'm a senior principal engineer, working in 5g system, test team in a firm networks.

A

Along with me, is my colleague, yamini sridharan, who is a senior principal engineer, leading the platform as a service team in a firm home networks is a telco company. As of april 2020. We are a fully owned subsidiary of microsoft and from november 1st 2020 we'll be completely integrated into microsoft. Azure team.

A

The agenda for today's talk is to first provide an overview of 5g networks and then we'll see what are the key observability requirements in 5g and how they are satisfied using the cncf observability stack, especially for metric collection, logging and tracing, and also we will see the demos as we move along telecom network architecture.

A

The various network elements in the network, it supported protocols, the interfaces that each network elements, support, etcetera, are defined by a standardization body called 3gpp in 5g. The network elements is called as network functions and they are deployed in.

B

A

Kubernetes clusters in different geographic locations, each network function will contain at least 10 to 15 different types of microservices within it. As I mentioned before, in a 5g core network, there are various virtual network functions deployed in multiple kubernetes clusters across different geographic locations.

A

This is a very simplified network, diagram of 5g core and whenever user equipment tries to registers with a network or attaches to the network, there are a series of 3gpp defined messages that are exchanged between various network functions over the 3gpp defined interfaces before the user equipment can either browse the data or make a phone call.

A

This brings our first requirement that we should be able to continuously monitor performance of multiple network functions.

A

Also, as the messages are exchanged between different network functions over different network interfaces, it's highly likely that there can be a fault anywhere in the network at any point of time.

A

This brings our next requirement that we should be able to detect the problem at the very early stages by some sort of automated mechanism or alerting mechanism in a typical legacy network.

A

The people will be staring at different guis for any anomalies or any alarms.

A

We need to have a way to automate that process here, and the other requirement is that, once a problem is detected, we should be able to troubleshoot the problem at the earliest and then resolve the issue at the earlier, so that there is minimum impact to customers like us right. So for that all the relevant debug information, whether it's stats, whether it's logs, whether it is traces everything- should be readily available in a centralized location. I like a dashboard kind of thing, so that it's it's.

B

A

The developers or the support people to debug the issue. Let's now see one of the important key features in 5g, which is called as network slicing, with the advent of 5g the applications which access the 5g network has different requirements. So there are some applications like gaming applications which would need high speed internet connectivity.

A

Then there are iot related applications which would need a huge number of devices to be connected to the network, but they may not be using the complete bandwidth or they may not need a high speed bandwidth at any point of time, and then there are more mission: critical applications like an example as an autonomous car driving mechanism which would need high speed, internet connectivity as well as very high reliability in the network. There should be zero latency for those applications so, depending on the applications requirements, it's not possible to provide separate, separate 5g networks.

A

So the solution is something called as network slicing, where the underlying 5g network will be logically sliced and depending on the applications uh subscription and other parameters, they will be allowed to use the network functions that are part of a particular slice.

A

This brings our next requirement that, though we can collect data from different network functions, we should be able to segregate the data collected from different network functions based on the slides from which it is coming.

A

Let's now see how these key 5g observability requirements are satisfied using cncf observability stack. Yamini will walk us through that.

B

Thank you, praveen hi guys. I am yamnish sridharan. I work as senior principal software engineer at fm network, a microsoft company.

B

In today's session we will see how cncf observability stack brings clarity in 5g world, as praveen mentioned earlier, the 5g world consists of different nfs deployed in different geological locations.

B

The challenges we had when we considered bringing clarity in 5g world are monitoring the performance of nf fall. Deduction. End-To-End call, tracing fear of our cncf observability component helps us to resolve our challenges and bring more clarity into 5g world.

B

Monitoring the performance of nfs in 5g, world primitives and graphana has helped us bring clarity in monitoring the performance of nfs. In this slide, we have few sample nfs like smf and amf.

B

We would like to monitor the performance of smf as per the slide. Primitius library helps us in creating the gorgeous counters and histogram and the url for the primitive server to pull the metrics. This.

A

B

Are helpful in monitoring the performance of ml, for example, we would like to know number of request reached smf every time a request reaches smf. We will have a counter and the counter keeps incrementing by the premature library and the value of the counter lets us know that number of request has been reached smf.

B

So the metric value helps us in giving a clarity on how the performance of the nf nfs are taken care.

B

The primitives pulls the metrics and saves it in the time series database.

B

The other tools have expression, they use the metric value with the arithmetic operation and the expression evaluates in a particular time interval and files, the alerts to the alert manager, the alert.

A

B

Forwards: the alerts to alert receiver example, slack or email the graphama visualization tool, it's a popular visualization tool, which has a data source and few visualization plugins.

B

So we have used few of the plugins example graph speedometer table and it gives the end user a visualization f x, of the metric value, which brings more clarity of the performance of the nfs.

B

Let's see a demo on how we use the primitives and graphana to bring more clarity in monitoring the performance of nfs in 5g world.

B

Now we will look into the primitive server. Primitive brings a clarity in monitoring the performance of animals in 5g world. Let's look how it brings a clarity.

B

This particular metric is from amf cc microservice, which is a 5g related microservice, and the metric has the mfcc, which is the microservice name, as well as the message transfer request.

B

So this particular metric will give us a clarity that how many number of times the message transfer request has been achieved in this slice, as praveen mentioned in earlier in one of his slides about the network slicing in 5g word.

B

So this slice, which is a label over here, relates to the network slice, so we have. As per this matrix, we have achieved 136 message: request, transfer request in this particular slice, like that. We have many metrics, which has been pulled from each container of 5g and saved in the time series database by primitives for aggregating the matrix and the alert rules and any other arithmetic expressions.

B

The rule is the record. Rule is a rule which gives you an expression for any arithmetic expression which primitive supports. For example, we have mentioned here a sum of a particular metric name without considering the label pod in primitius. Once we click do a record rule, this particular record name becomes a metric itself and it's stored in the time series database once we click on here. This particular metric, which was a record rule, becomes a metric name with a corresponding labels per slice with the values.

B

How many atoms has happened for this particular slice in this manner? We get clarity of the performance of nf in the 5g world. We have the other truths. The other truth has the alert name and the expression.

B

This expression includes the metric name or the record rule name with a customized expression as per the business logic, and this expression is evaluated every minute, as per the for duration here with the label matching, and this description and summary will be displayed in the receiver, which was configured in alert manager.

B

This expression, if it is the evaluated and the evaluation result, is true.

B

The premathius starts firing alerts to the alert manager and the alert manager will further start forwarding the alerts to the receiver configured in the alert manager, and this here the metric name is used with the full expression will give the proper value and that value will be considered with the evaluated expression value.

B

This is an example of an alert rule, and this gives on. We will come to know because of this alerting system, that if any performance of the nf's is going down or it's not, is it unexpected of the performance we have to. We can have a clarity in the prematures or more clarity by seeing the grafana.

B

Let's see the graphana, which corresponds to this primitives.

B

This is the grafana. We have configured the grafana with multiple dashboards for each container and microservices and we have configured the dashboards and data source. The data source from this rafana rafana by default supports many data source, but we use prematures primitives.

B

This data source is configured in a auto provisioning method using the grafana api and the dashboards are each and every microservices has its own dashboards and it is provisioned in grafana using the grafana apis for grafana dashboards and data suits. We are doing automatic provisioning, api, graphene master dashboard, for example, by viewing the dashboard.

B

We will come to know that what are the metric values and even in a visualizable way how successful it is- and we can achieve this by writing- queries this query once we select the data source, this query is the prom ql query, which is going to query the primitives and the metric name mentioned here is in primitives and it gives the value according to what is available in primitives.

B

And for more clarity, we can have a database query as well.

B

So this particular dashboard has graph and tables, as well as more queries, which gives a more clarity of what. What time and what phase the particular variable values are. Metric values are available. For example, here we see a metric value as per.

A

B

Wherever we keep the cursor, we will come to know the exact metric value which gives.

A

B

Clarity and a way of visualizing, the matrix.

B

Now we go into the next challenge of fall. Deduction using centralized logging, so smf and amf are the 5g network nfs, which has which can go down or which can create any fault in the animals.

B

So we have structured, actionable logs which are being created by the nfs and which are being created as a standard out and standard error, and we have the fluency, which is the forwarder which collects the standard out and standard error outputs and it forwards the log to the elastic, search and stores the log in the elastic search.

B

The elast alert is an alerting component, which has some alerting rules configured with the receivers. The receivers can be slack or email or web book. So it depends upon the last alert rules that this rule has to be evaluated with the structured actionable logs and the particular events. The expression is satisfied or evaluated.

B

Fine. The particular alerts are being forwarded to the receivers after forwarding it to the receivers. If we want to see more clarity in the logs, the kibana is a user-friendly ui, which gives a more clarity in the logs. Let's see.

A

B

Quick demo on the chibana.

B

Now, let's see the kibana, which gives more clarity in visualizing the logs, the fall deduction happens in nf, the structured actionable logs are saved in elasticsearch, and the last alert, which has the rules, will query elasticsearch and if it sees a rule which is having the similar log, it will start forwarding the alerts to the last alert receiver.

B

So the kibana gives more clarity which log has given the actionable log. These are few logs which we have in our anna's.

B

So the clarity which the nf gives is is in adjacent format, and once you expand it, we will come to know each and every label value and what is the level of severity and kimara gives us more flexibility by creating indexes, we can create indexes index patterns and we can use our own filters.

B

This filters are very useful when we will search for the particular value or a particular label in a huge number of logs, and these are the already configured available fields. Thank you.

B

B

Now, let's look into the end to end call tracing the end-to-end call tracing is a challenge we had to bring clarity into 5g. For example, we establish a call in the 5g world and a call drops off and we need to debug the reason for the call drop up to rectify the issue. So the.

A

B

And amf are the 5g nfs. Well, we need to incorporate the open tracing in those containers and the open tracing creates the traces and spans, and this traces and spans are collected by the eager agent and forwarded to the elastic database.

B

The elastic database stores these traces and spans and the eager ui displaces the traces and spans for more clarity.

B

A demo on eagle will be provided by my colleague, praveen.

B

A

Before I start with the end-to-end racing demo, let me ask you one thing: how many of you have faced issues with your mobile network connection? It could be different. Issues like we are not able to browse internet or our friends or relatives would have complained to us saying that they were trying to reach us, but the calls are going to voicemail.

A

So several those kind of issues we would have faced and it's common that we would have called the customer support center to raise a ticket with them to look into what is going wrong with our connection in.

B

A legacy network.

A

If you raise a ticket with customer support center, what they do is they'll collect logs from different nodes, look into the logs and then see that node one is healthy, then look into the node next set of logs for node two they'll see node two is healthy, then go to the next set of low logs or debug information from node 3, and then they will identify the problem, which means that the debugging process is difficult.

A

It takes time which ultimately backs the customers.

B

A

Okay, so we are trying to see how we can use agar tracing to make the debugging process easier.

A

Whenever a customer calls a customer support sender and raises a ticket with them, each user equipment will be having a unique identifier. So using that unique identifier, we can start again tracing in the network say here. The access network gets terminated on on the amf, so amf using command line interface. We can enable a gut tracing. The requirement is that we should be able to trace messages exchange across all the network functions in the network. We should be able to um capture the traces towards nssf, nrf, aevs or udm smf across the network.

A

We should be able to trace the network. Also, each network function has at least 10 to 15 different micro service type. We should be able to trace the messages exchange between the different micro service type as well.

A

Now, let's uh see how we can utilize niger tracing to debug and a problem that is reported to the customer support center in the interest of saving time. I already ran a call at the back end and collected the trace. There is one trace for a 3gpp defined procedure called video establishment procedure and it has 37 spans.

A

Let's look into the details of this phrase I mentioned before that the traces will be started at amf and on the amf it. The traces will be started at amf into microservice when the message is exchanged from amf n2 to other different micro services within an nf nfr across a nf, all those phrases will appear as child spams.

A

The first phrase is the first child span is a message towards a mf call, control and from amf call control. The message will be sent to nssf, which is a separate nf. Then the message will be sent to nrf, which is a separator naf for smf discovery and then, once the discovery happens, the message will be sent to smf nf itself, so on smf we can on on any trace. We can capture the various internal events in the log section, so these are more like similar to the logs that we see in kibana.

A

This gives more insight into what is happening whenever a message is received at the particular micro service.

A

It also captures the duration or the delay uh or the latency associated with each event happening within an nf and, as I said, I'm trying to simulate the problem where a user has released a ticket, in this case the video establishment procedure, the last trace or the child span. As for discovering a pc of data here, as we could see, there is an error for discovering a pcf from nrf and there is no pcf configured. So this is why the calls are failing for the reported customer it could.

A

The failure could be due to different reason. It could be some configuration missing or it could be that the pcf is not dynamically registered with nrf, but at least we got a starting point as to what is actually going wrong with the network. Now we can browse the data in kibana and see more details, more specific details into it and then take corrective actions.

A

Also. Another important aspect with eager tracing is that, as I mentioned before, it helps to understand the latency or the bottlenecks in the network, especially how long each function on each network function is taking to process a particular message and then 5g is all about low latency right, so we can see where the bottleneck is and then address that problem as well.

A

That concludes our demo and our talk. We hope you all enjoyed our talk thanks for your time,.