Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Network Service Mesh to Address Cloud Native 5G Telco Networking... Roshini Ratnam & Anders Franzen

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Network Service Mesh to Address Cloud Native 5G Telco Networking Challenges - Roshini Ratnam & Anders Franzen, Ericsson

Some of the key networking challenges of cloud native telco applications are VPN/traffic separation, no NAT protocols, path diversity, accelerated user plane traffic, legacy protocol etc. Kubernetes networking cannot address them. Currently they are addressed using approaches like Multus, Danm, CNI-Genie. All this still only works on a node level but will require a substantial amount of complex case specific, on-site DevOps to bottom-up build a functional multi-network cluster, with all challenges that come with that, like LCM. This seminar will cover how Network Service Mesh (NSM) can address networking challenges of cloud native telco applications in 5G with the details of proof of concept and how NSM decouple infrastructure from applications. The speakers will also give details of the ongoing work with CNF test bed network separation use case.

https://sched.co/Zelt

A

A warm welcome everyone to the presentation of topic network service mesh to address cloud native, 5g, telco, networking challenges.

A

So myself, roshni ratnam working as networking and connectivity architect at ericsson and.

B

My name is anderson. I'm working as a technology specialist at ericsson.

A

So these are the agenda items that we will walk you through today, uh first is yg, and why cloud native 5g system architecture and its complexity, major challenging telco requirements and different telco traffic classes and also scenario of when to use secondary network and how nsm can help and which will be followed by a demo using network service mesh.

A

Why cloud native for 5g, the main reasons include speed by adapting market changes quickly? Scale micro service architecture allow scalability with telco grade reliability, efficiency by simplifying complex telco network functions and also increase capacity flexibly when service demands grows.

A

Why kubernetes and uh what do we get from kubernetes primary networking model? So kubernetes is a cncf uh recommended cloud platform. It provides life cycle management, scaling and auto healing service discovery, external service exposure load, balancing.

A

And this is the 5g system architecture based on 3gpp reference, the rectangular boxes represent different network function and that basically does network functions with most of them being software based, and you also see interfaces between different network functions.

A

So some of the network functions. Examples are network slice, selection, function, nssf, network exposure, function, nef, network repository function, packet policy, control, function, unified data management, application function, etc. You also see below user equipment, which is a mobile, laptop, etc. Radio, access network user, plane function and data networks, so the highlighted area. uh The interfaces in this highlighted area are service-based interfaces, so basically they can be implemented using kubernetes primary networking because they are http, 2 or api based, so they're based interfaces, so they can be implemented using kubernetes primary networking.

A

The interfaces highlighted in the below area are non-sbi non-service based interfaces. They are telco interfaces over l2 l3. The telco interfaces are with high requirements such as a high performance, a low jitter high bandwidth and not suitable for kubernetes. Networking example is user data, plane traffic between a user equipment and data network, which is mostly encapsulated in gtp protocol, so for non stand, service based interfaces, we need to use secondary networks as humanities network cannot handle them and need uh different networking secondary networking solutions such as motors or network service mesh.

A

So, based on uh all these details, why we are showing this picture in this slide is because to show how complex uh the architecture is, and also to highlight the challenging requirements that telco has.

A

So, based on that, we are outlining the major challenging telco requirements. One of them is operator requirement on end-to-end vpn separation due to security or legacy reasons. Operators expect end-to-end vpn separation from external network till application boards with kubernetes single network model. Only external traffic separation is possible and no end to end vpn separation and the second a challenging requirement is that support for no nat protocol, so protocols that are not sensitive like session initiation protocol, ship or media and the third main challenging requirement is eager sweep for outgoing traffic originating from application ports.

A

So outgoing traffic from uh port leaves kubernetes cluster with ip address of worker nodes. The outgoing traffic need to come out with a virtual ip address without nat.

A

One reason why this is required is the ip address limit because of too many ips of uh for outgoing traffic and customer expects common ip for incoming and outgoing and another major reason uh requirement is uh support for non-ip protocols and, uh last but not least, we need to support high performance traffic, that is, traffic with low latency low jitter high number of packets per second and high bandwidth.

A

So, based on these requirements, there is a need to support different traffic types. So here we outline different telco traffic classes based on different traffic types characteristics and packets per seconds.

A

So first classification is a low performance with high functionality, that is, it doesn't require uh too much latency, so no latency sensitive. uh It includes protocols that supports a protocol level encryption, but it the traffic requires services like uh load, balancer, service, discovery, purchase management, etc, and the number of packets per second that needs to handle is like 0 to 1 million packets per second, and example, is operational and maintenance protocols like snmp, netconf, etc, and the other classification is high performance, low function, which is only a delay, data link or l2 level.

A

They need to handle high performance, user plane or data plane, implemented normally using dpdk and sriov. The traffic needs to handle 5 million packets per second and, more example, is user, plane, function, radio access network, etc. There is also a middle category that is a medium performance which lies between these two categories. They include high performance control, plane or low performance data plane and the traffic that needs to handle one to five million packets per second example: is a diameter, sip, etc.

A

So, with these details, anders francien will show you visualization of when to use secondary network in the next slide, so anders.

B

Yep hello, so we put together a small like decision graph on how to like uh how to decide when to go for a secondary network. Of course, the strive it should be to always use the primary network to be as cloud native as possible.

B

So we have, we start with a question saying: can we use a primary network of kubernetes and then we say yes and then, of course we use the primary network. So this is the easy solution.

B

If, for instance, that would say no, we cannot use the primary network of kubernetes. So we think that why can we not do that? There must be a reason and the reason might be because of protocols I mean kubernetes is like supporting http and web protocols and stuff like that, and it's it's not really good at tcps, but we have a lot of different protocols.

B

So then we can, we can think about. Can we change the protocols of the application to instead be a web-based or something like? If we can do that, then we use the primary network. If you cannot change the protocols, then we can see. Can we update kubernetes to actually handle this kind of protocol like sctp, is something that has been added like quite recently. I think, if we can do that, then we can use primary. If we cannot modify kubernetes, then we must use a secondary network.

B

Then there's other reasons. For instance, the operator the operator might decide that we are not allowed to use the primary network for like the public access and that, if you look at the previous uh slides or rossini, we have this low performance. High functionality network I mean, and that is uh the kubernetes private network will fit into that category as well, but sometimes we're not allowed to use it for for security reasons or whatever.

B

So the typical case here is that we are replacing a functionality that is already deployed at an operator and they have an architecture with a lot of networks already to other operators. So then we cannot like use shortcut all the networks. That's already out there. We must be able to terminate them, but then we have the question: can we try to migrate the legacy network architecture to a new flat network?

B

If we can do that, we can use the primary. If not, we have to use a secular network, so the other reason would be just the performance part. If we, if we need to like switch more than 5 million packets per second and with no latency, there is actually not much we can do, and if we are latency dependent, then we cannot do much. If we are banned with the pendant, then we could try to like distribute the traffic over multiple workers.

B

So in that case we, if we can like distribute traffic, we can use the primary network instead. But if we are very latency bound, we cannot do that. Then we have to use a secondary network.

B

So so this is the some of the reasons when we need to add a secondary network. I forgot, and then we are looking into different. I mean a very common. A very common cni to use at ericsson is the multus everybody's using it, but we are looking into this network service mesh. That is like a part of the cncf cnc extensive project as well.

B

Network service mesh is a novel approach. Solving complicated l2 l3 use cases that incubators are tricky to address with the existing kubernetes network mode. That sounds like perfect. What we need actually and then we can make network service and mesh maps the concept of the service mesh to l2 and l3 payloads as part of an attempt to architecturally transpose nfv in a cloud-native way to utilize microservice architecture.

B

This sentence is kind of condensed, but you can easily say that we have an network function, version and nfv, and we can. We can convert that to a micro service by using msm models, because we can model the the network service in a way that fits uh kubernetes, very good. We can use hair insurance to deploy it and and this kind of stuff.

B

So so I think that msm will be good for doing this to break down the functionality that we have and nsm is built around the distributed broker to keep track of all the resources and the set of integrated data planes to interconnect the resources in a coordinated way. So we as applications can actually implement a set of services on top of nsm, which and will nsm will keep track of those services, and we, on the other side, where we are consumers of this. We can. You say that we need to to.

B

We have dependence on this service and nsm will connect us, so we have a defined api to implement the services. I talked about recently like firewalls load, balancer different external devices, and we have defined api for applications across segments on services.

B

So then we have a demo because we have done a couple of pokes with msm and this demo routinely show a bit later. I will start here with the logical view of it.

B

What we have is an application, that's the purple box and that's a pod, and then we have a controller. That is a the controller, is like an nsm controller and what it implements is a load balancing service and we have another controller. That is a front-end v-loan termination service.

B

We have, we have modified asmr bit here, so we are running with the with the linux kernel, data plane or forwarder, as it calls nowadays. So the load balancer is actually the ipv. Yes, the linux kernel load balancer and then we have a cross connect and we have also broken that because usually in nsm, the controller and the data handler are like the same container.

B

But here we just control the resources from the nsm pods and we actually, the data path, is going through the corner. So this is not a. This is not extremely fast, but it's kind of flexible and we can then we can model our functions. So we will come in to the cluster with the v alone. We go to cross connect and then we we will use like a vxlan to reach the load balancer function and that one will point to multiple application. Pods. I think next screen will show the more physical realization of this.

B

So we have a kubernetes cluster. We have three nodes. One monster two workers we start by having like a port sitting in the worker working one or two. This this port is a distributor. I mean it's the same function as just in two places and that's normal and there's no distribution. Then we start adding stuff here. So we add the network service, mesh manager and we add the network source mess forwarder this folder we have modified ourselves.

B

So then we add the service called the front-end network service endpoint, the one: it's it's actually the control receiving a worker too, but it creates an interface on worker, one that will go out to the external world next step. We add another, we have the load, balancer network service endpoint and that one will then be connected.

B

We can see here so we get the packet coming, because the h1, which is the associated with the front-end network service endpoint, that one will send the packet to the load balancer the load once we select one of the or in this case it selects both, but we will select one at the time of course, and then we have, and then we have it now we have the selected we have. I need to go back here again to this and then I will leave the rossini, so she can actually show the demo running.

A

Thank you antash. So now we will show the same setup uh in the live demo, so the external client will be called as a test machine and I have a master node and two worker nodes. So this is the my master vm and I will show you the notes. So you can see. I have a master node and a two worker nodes and I will uh show the ports. So when you install a network service mesh, you will get the control plane, which is the nsm manager, and uh you get the kernel forwarder.

A

uh You can choose uh the kernel, forwarder or vpp forwarder, based on the available uh data planes in the network service mesh that you have installed. So here we used kernel, forwarder and also we did some uh modification uh to make use of a friend and a friend and a network service logic.

A

uh This sftp ports are my application ports, so I'm running uh an sftp server uh so that I can uh test using an ssh from a test machine. So, as you see, I have two ports running on two of my worker nodes and uh front end uh network service. Endpoint uh is the network service that, uh as uh anders has explained that we have uh designed and added as a plugin on top of network service mesh framework uh which will allow us to connect uh access.

A

The controller and connect to the external uh network, so load balancer network service is where we have the load balancing functionality, which will do the load balancing of the external traffic to the back-end ports.

A

So maybe it's interesting to see the network interfaces of the application ford.

A

So when you install when you deploy uh an application uh on kubernetes et 0 is the default uh interface that you receive in my system, I use calico as a cni plug-in, so this is the calico network and uh since in the application board, I am requesting uh a network service uh endpoint. uh So that's why I got an additional interface that is nsm which which is actually handled by the network service mesh.

A

Maybe it will be interesting to also see the uh interfaces of load balancer. So this is my load balancer network service. So, as usual, you have the uh default interface and nsm interface that I received from the network service mesh, and this is the network uh interfaces towards my application board, and this network service interface is the uh one that is facing towards the external network.

A

With that, maybe we can uh try to do a testing from an external uh client, so this this machine is actually running outside of my kubernetes cluster. So from this test motion I will try to do ssh towards my application board. So let's try to so. This is the uh virtual ip that I'm going to use every time. So, every time when I try to do ssh, I should get different host name. So here I'm trying to print the host name of the application port.

A

So let's try every time what is the host name that we receive so each time when I, when we do the ssh we can see we are getting different host name. So basically, this is the host name of the application board, so this load balancing it's still using a single whip and the load balancing is happening uh behind and this is uh being done by the load. Balancer network service, so uh maybe it is also interesting to see how, in the application that we are requesting the network service endpoint.

A

So in this is the deployment uh manifest, and here you can see that I have a an annotation that is appointing uh that is requesting uh to a specific network service. So you can implement your own network service and can deploy it on network service mesh and can refer the network service uh in this annotation.

A

So this is the only change that you have to do in the application. So if you are using voltage network uh multisolution, uh you still get additional interfaces, but then the service, the same uh service, discovery and load balancing has to be handled uh here. Network service mesh handles uh everything from application point of view. You just need to specify which network service you need to use.

A

So with that, I have concluded the demo and I will stop sharing and we can go back to the slide.

B

So, thank you, everybody, and if we have some key takeaway from this presentation, that would be that our old temco legacy cannot live with only like the primary web interface network. We we actually need a secondary network and we think nsm is a good way forward, because we can model the my network as microservice modulus. I think what you say.

A

Yeah and and application specific network services can also be uh packaged and can be deployed on a network service mesh and when we go to customer cars, application doesn't need to request for special privilege just because that is being handled by network service mesh itself and also all the networking solution is handled by network service mesh. So it's kind of a cloud native kubernetes way that application can just request for the network service and remaining logic will be handled by network service mesh.

A

So with that, uh thank you. Everyone for participating in this presentation.

A

B

A

Everyone thank you for coming, uh so we have got uh some interesting questions so we'll go through them. um Shall I read the andres.

B

Or we can take the the first two questions from marco antonio gonzalez ortiz is about how kubernetes can handle the throughput requirements and sriv infra design requirements. I I I assume the questions are very similar to me. What's what again say that kubernetes cannot handle that you need to have multis or all nsm or something that can provide you with the user mode. Like networking interface, I have, I think, well what do you think you're feeling.

A

uh Yeah so basically kubernetes primary networking cannot handle it. That's why we need to have similar kind of solution. It could be multus with or nsm or any other solution which can which needs secondary networking anything outside kubernetes, networking, okay. uh So that was uh first question. uh Then uh there was a question from sergey uh in sorry. If I pronounced it wrong has sctp been added to kubernetes.

A

uh Cni uh sctp uh has been supported by kubernetes from uh 1.12 release, if I remember correctly, but then uh multi-homing scenario is not supported by kubernetes networking, so that will be um a challenge uh requirement. Yeah.

B

And it is the cue proxy that needs to track the sessions. I think it's not the cni itself. I assume yeah.

A

And there was another question: will you use vxlan for 5g, cnfs or vlans.

B

A

B

B

Yeah and what we showed here is the poke on the I mean and our 5g implementations today.

B

I think they use mul to send v loans most of them, and this is what we are looking into like to like him, to make this more model, driven and and in our poke that we show the demo racing show we use v loans for the external connection, but internally nsm is using vdx loans for them for different uh streams or.

A

B

Yeah yeah, so that's at least what I know. I think that our current implementation in 5d uses long but most doesn't he don't think? Do you have any other information.

A

No, uh I agree what you said, so we can take the next one. Why was maltose not enough for this use case? So this is what we summarized in our key takeaways that um um when we have to base our solution on multus, then um the services like uh load, balancing service, discovery, ipam and so on, which is handled by cuban at this networking by itself, has to be implemented as separate services, and it has to be bundled with the application, and you have to carry that in different uh deployment.

A

Environments and some might require special privileges and so on. But when we are using network service mesh, what we need is to have this additional network services that will fulfill the use cases. The remaining whole solution or the networking is handled by network service mesh so which is similar kind of uh uh cuban at this cncf kind of way.

A

So this is why we think an assam could be like a potential, uh um a solution uh for secondary networking uh and, having said that uh of obviously, if you're going to a customer cars and if nsm is installed, there, then uh from v from like an application, doesn't require any special privileges, because that has been already been carried uh or taken care um understand. You have anything more to add.

B

More that's the same as you're saying, but I think that you can say very easy multitudes actually giving you an interface into the port and with that you can do like every. You can do anything in the world with it. But everything will be an ad hoc solution with nsm. We see that the possible way to like make it model driven and have like a standard wavelength, so yeah.

A

Okay, uh so uh next question is: are you using calico without ipip in your demo? Is this running on a bare metal uh with bgp? This is the question from adele look uh so in in our demo. uh We have been focusing on the secondary networking and, of course, our primary networking is uh using calico with ipip, and uh this is run on uh vm missions, so with bgp, of course, but uh our focus was mostly on the secondary networking.

A

That means a network other than the kubernetes calico network, and if do you have anything to add.

B

A

Yeah uh there is another question from francisco: castello is the demo code available? uh We uh probably push this uh changes to uh I mean the services we might push towards a uh nsm community and uh andersha.

B

Yeah we want to do this because the main thing here is actually the the breakout of the front: temp. Isn't it and what yeah the heck in the? What we need to do is to use the new msm architecture to make our own formula and that we could probably make.

A

B

But this code, if we make it available, it has to be made available on the side of it of the nsm standard reaper, because it won't work together with it.

B

Do you understand that yeah I mean we, we have messed up, then. I said we're. Reading the.

A

Code yeah, um shall we go to the next question.

B

A

Okay, um it's a question from killian muldoon. Is this a better overall solution at scale than multi-spaced solution? So the question is about scalability.

B

And multiple does it scale, it just gives you an interface per node. So actually, if I say I have a multiple solution, I don't know how that scales. Really I mean it depends. You will put cables into a switch, but how it's working. I cannot say- and this is the problem.

A

With multiples, I think amultus.

B

Is used like a way into the pod, it doesn't say anything of the overall architecture. What do you say.

A

Yeah so I mean, if you want to get an additional interface, you can use maltose with additional cni's, but then there are much more that we need to handle like the services and that will uh uh that.

B

A

B

This is the thing.

A

Yeah exactly so, once you got, uh uh I mean an extra interface that won't be enough. You have to do bit more, then that has to be handled within your application code, uh which is like uh a bit. uh I mean it's a bit more work uh per each application. Then each application have to do individually and that will be uh again a bit different and and here from nsm point of view from application side, they just need to request for uh in service in their manifest like uh in their deployment, uh manifest.

A

They just need to add an annotation, as we have shown in the demo uh to explain like I need a specific network service that will handle load, balancing or whatever, then the rest, everything the connection and everything is handling candle uh by nsm uh using network services and other mechanisms, but uh from application point of view, uh we are kind of uh no need to worry about what is happening in the background similar to kubernetes. So in kubernetes networking we don't normally write a load balancing mechanism or uh service discovery mechanism.

A

It's it's handled by kubernetes. So this is something similar that will be done uh with an nsm frame. It's a framework where you can add a network services as a plugins, so which kind of network services you want. That is up to you to write and um so yeah, so for in our example, we have shown like load balancer as a network service and friend industrial network service.

A

I think I talked too much. I should go to next question. uh Do use it's a question from adrian braunder.

B

Do you see, don't have friends, say silence.

A

Okay, sorry, I um yeah.

B

How are you coming challenges of pods that require local affinity as a limitation? Yeah I mean that's. We have to depending, like everybody, there's no way around this. I don't know I mean I I have seen, because we are not come so far. You have to see something where in kubernetes, where you can put that you require a number of cpus and you you put the limits the same as the amount you have or something, and then you will mores with that.

B

We need to fix it with c groups or something, but we need to have them. We need to have the process standing on the core, the top of one core and the same comes to scaling of the forwarding plane. I don't know that it has to be done with the application in mind. I mean we need to find some way of dividing it into sessions or something like this. We cannot just split the stream up.

B

I don't know, if that's a good answer, do you know more graffiti.

A

Yeah, I think I have the same answer as well, but if you have more questions, we can get clarified in the slack channel later, so we will be available for 15 more minutes after the time is over.

A

So next question from adrian bronder: do you see a need for even more networks or does secondary means just any other kind of network except the primary? It's the.

B

Terminology that we are using.

A

B

You understood.

A

Right, it's any network other than primary networking, so it could be one or more.

B

Or five secondary.

A

B

Your is your secondary from kubernetes.

A

Maybe the terminology we used is wrong, but uh yeah, so we called the second tree yeah.

B

Yeah we are looking at the nsm and it's not the cni at all. But if you want to have cni, then then it's the intel dpdk, I mean srv suite that people are using most. I mean, I think, that in our place, most people use those as well.

B

They have survey ones with the dk and what's it called to get the user mode? The pci.

B

Do you know that vfr you something you're seeing.

A

Okay, so next question from atul deshpande: are you exposing the cluster node outside world with the public ip in this architecture, when external client is connecting to lbnse, so it was a vp address, virtual ip that a routable virtual ip? This is what we were exposing in the demo uh and the external client was trying to connect to the vip.

A

So that's why, even though behind we had multiple application ports, we could connect uh to either of them uh using the same veep. So that was the uh concept that we used to unders. Do you have anything more to add.

B

Now, what is this we've done? This different, I mean you can use, you can put the public ip on the load balancer and then you use nat to get into the system. Sometimes in amazon we actually use uh the server return, so we use delete buttons on the actual clients as well.

B

So so you don't you, don't have any net in atlas, load balancing- and I don't know in this case- I think it's a nat. Isn't it yeah.

A

B

A

B

Example, where we run moonlight as well, we have to be sitting on all the pods.

A

And next question um uh some teams say: nsm not ready for 5g core. Is it feasible that we deploy 5g code without nsm then evolve? Is it possible yeah? So these network services that we have mentioned has to be developed and designed with telco grade uh yeah quality, and this is something that has to be uh initiated and uh and of course it is possible to evolve from any other solution to nsm. As I mentioned, you just need to have the annotations in your application, where you mention which network service that has to be selected.

A

But then this network service uh needs to be uh there in the nsm has to be deployed in the nsm. So this this is something that has to be worked on, uh and uh yes, uh I mean it is feasible that you could evolve unders. Do you have any other opinion.

B

It's not feasible today, I would say, but otherwise.

A

A

No, I am trying to maximize my screen. Can you read.

B

What about nfv and its standards when it comes to msm and this new model? I don't know that really, because nfv and its standards is more or less just we have we use nsm to like implement something. Then what package comes in and out of the protocol handlers I mean that, of course, must be part of the standard. I assume.

A

uh Maybe you need to mention.

B

A

B

The day come up: okay, yeah, maybe justify the.

A

Okay, uh so next is uh from francisco uh castello. Can this be applied in uh hosted kubernetes like uh google cloud or amazon eks?

A

uh Yes, I mean in some community they have been doing testing with these uh public clouds as well uh as far as my understanding is right under.

B

A

uh This I mean uh in community, they are doing testing on uh public clouds as well.

B

Yeah, what I don't know here, because some of the stuff we are doing still needs to be like the services needs to be, have priority or something like this and.

A

B

Maybe what we would like to see is that all the services we really need or like standard components, so they could be like installed in all this environment and and be approved by the operators or the owners of the platform. I mean, because I don't think that they would have, does come and put in the data plane ourselves.

A

Okay, so we are now five minutes past uh the uh command a a lot of time, but we will be available on uh slack to kubeco networking uh to continue question answers. uh We got a lot of questions which we couldn't answer in this short time, but uh thank you, everyone once again uh for attending the session and we will be available for 15 more minutes in the cubeco networking uh slack channel.

A

Anders to say something: yeah.

B

We have a lot of questions really. I don't know if how we can bring those into this lecture, but thank you, everyone that comes.