Kubernetes IoT Edge Working Group, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: PCEI at the EDGE - Amar Kapadia, Aarna Networks, Oleg Berzin, Equinix

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

PCEI at the EDGE - Amar Kapadia, Aarna Networks

The Akraino Public Cloud Edge Interface (PCEI) Blueprint is advancing its maturity and is poised to become a key tool in the developer’s toolbox for deploying K8s in a diversity of environments, including bare metal. The purpose of PCEI is to develop a set of open APIs, orchestration functionalities and edge capabilities for enabling Multi-Domain Interworking across the Operator Network Edge, the Public Cloud Core and Edge, the 3rd-Party Edge as well as the underlying infrastructure such as Data Centers, Compute Hardware and Networks. In this presentation, we will be showing how to use Terraform to provision Infrastructure on a Bare Metal (Equinix Metal) Cloud, install K8s on Bare Metal, and configure Azure Cloud (Express Route, Peering, VNET, VM, IoT Hub). We will then interconnect Edge Cloud with Public Cloud (Equinix Fabric), and deploy Edge Application (PCE), which includes a dynamic K8S Cluster Registration, dynamic onboarding of App Helm Charts, instantiation of composite cloud native app deployment, and end-to-end operation. What's new about this presentation is showing a SaaS Portal GUI to facilitate end-to-end application deployment, configuration and interconnection.

A

Hi everyone: okay, let's jump in.

A

The title of this talk is pcei, which I'm sure none of you know what that means. So I'll explain it at the edge and uh this blueprint is part of Linux Foundation, Edge akraino project and the blueprint is led by Oleg berzin he's the project. Technical lead he's a fellow of technology and architecture at equinix. So this is really his brainchild.

A

He couldn't be here so I'm going to try and do my best to to represent the blueprint and, as Tina said, this was submitted to the Etsy Mac hackathon, and it was the winner about two two and a half weeks ago.

A

So what you're going to see is a really practical implementation of edge computing.

A

So what is pcei PCI like I, said as part of Linux Foundation at jakraino. It stands for public Cloud, Edge interface and really the idea of PC EI is to specify a set of open apis for cloud Edge instances and the cloud Edge is defined as an edge that is closer to the cloud than the user. So you you've seen all kinds of definitions of the edge right. This is sort of closer to the cloud.

A

Not really device Centric, but in a data center or a Colo, and the cloud Edge exposes apis towards the public Cloud, so public cloud service provider, instances at the edge and the whole idea is that the edge and Cloud are complementary. So it's not like the cloud can replace the edge or the edge can replace the cloud they're complementary, and for that reason the thinking is that the cloud Edge deployments offer many opportunities for collaboration by exposing the network capabilities to provide value-added services and, on the right hand, side.

A

You see the two pictures so Cloud Edge and Cloud workload, so your application can straddle all the way from the edge to the public cloud and same thing on the infrastructure. Your infrastructure can straddle from the cloud Edge to Cloud infrastructure and we are going to see examples of both.

A

So this blueprint has been progressing for almost two two, maybe even longer two years now- and this is a logical representation of the blueprint and then I'll get into sort of more details and show the physical structure.

A

So we have two Edge sites, Cloud Edge sites and one public Cloud site.

A

So what you see at below is the physical infrastructure, so the physical instructor structure constitutes of equinix metal, which is bare metal as a service equinix network Edge, which is networking through equinix Network Edge. You can get things like firewalls routers load, balancers on demand and fabric, which is on demand, Cloud connectivity and then Azure express route is the other side of the cloud connectivity on the Azure side. So that's the infrastructure that was used and on that two clusters were created.

A

You will see one in Silicon Valley, one in Dallas and then there was also a service used on the on the public Cloud on azure and cluster. One ran 5G so what's running on cluster one is the 5G Ram 5G core and location services and iot an iot device, and some things are simulated like the core. Is real but everything else is actually simulated. The radio is simulated as well site number two is running the multi-access edge Computing application, the mech application, which is iot Edge, and the public cloud is running iot hub.

A

All of this is fully automated. The orchestration is fully automated, so literally with a click of a button. The whole thing spins up normally this might take you sort of a regular user a couple of months to set up so by automation. We are cutting those couple of months into a couple of hours.

A

So the same thing in sort of words, we are using the PCI blueprint and the mech location API service to demonstrate orchestration of a Federated Mech infrastructure. It fed rates across two maxites and a public Cloud site.

A

We are going to show bare metal interconnection, virtual routing for both the max side and the public Cloud sites, IIs and SAS, fully automated, and it's across two providers: 5G control and user plane functions and then the deployment of the mech application in which, in this case is iot, so the iot application making use of 5G access and it's distributed across geographical locations and across hybrid Mech, Edge cloud and public Cloud SAS infrastructure.

A

So with that I'm going to I'm going to dive in into some details, so maybe I'll just come here easier.

A

So what what you see? On the left hand? Side is site, one: that's in Silicon, Valley and Silicon Valley. You see three metal servers.

A

One metal server is running the UE and gnode B simulator, and for those of you who don't know 5G, that's just um the user equipment which could be a phone. So it's a simulator for a phone and gnode B is the radio area network. So it's a simulated user equipment and gnode B the iot data generator. The simulated iost device is also running on that server.

A

Then we have another metal server and all of these are running kubernetes by the way. So the second server with kubernetes is running the 5G UPF, so I'll Show a slide on this, but 5G can be broken up into 5G core can be broken up into a user plane and a control plane and the user plane is the element through which data flows and control plane is is, as the name suggests, it's a control plane.

A

So this runs the UPF, the user plane function and it runs the location, API server. So because in in 5G one of the key things you get is you can get the location of your your device or in this case iot device, which can be very important. All of that is connect. All the metal servers are connected over an IP network. The third metal server with kubernetes is running the 5G control plane and we'll see more details.

A

So that's site one site 2, which is connect site, 1 and site 2 are connected over equinix Fabric and we are calling that the mech Federation data plane and that's not really been specified in the standard. So Oleg is trying to push that back into the standards.

A

The second site, the site in Dallas, is running one metal server with kubernetes and that is running Azure iot Edge, which in this case is The Edge Computing or the mech application, multi-access Edge Computing application.

A

That in turn, is connected over equinix fabric to a public Cloud site Azure in this case and Azure is running um iot Hub. So, as you see fairly complex environment, actually you know quite representative of a real life environment and at the top you have the orchestrator, so it it's either called the mech Federation broker or orchestrator. However, you want to call it and it's orchestrating the left hand side the fabric in the middle, the right hand, side everything fully automated, like I I mentioned.

A

um So that's that's this blueprint. You know.

A

What what does the use case do it does three things? The first thing it does is setting up the sites, the infrastructure. So it's setting up the 5G provider and in this case another thing I I didn't mention is you could have the 5G provider and the edge Computing application provider be two separate companies? They could be two separate entities, so you have the 5G operator and what we are doing in the orchestration stage is we are setting up bare metal servers. We are setting up kubernetes fully automated.

A

Then we are setting up the mech Federation Internet connect provider, that's a equinix fabric and that creates the private mcfederation data plane connection, and then we set up the right hand, side site, which is the mech provider. There we set up bare metal, we orchestrate the bare metal we orchestrate kubernetes. On top, we orchestrate the Virtual Router that connects both to the 5G provider and to the public cloud, and then we orchestrate express route to Azure. So that's step, one step: two. We orchestrate the network functions and the application.

A

So, on the left hand, side we orchestrate 5G functions, control, plane, functions, user, plane, functions, Mech, location, API, server and iot Edge and, on the right hand, side. Sorry, the iot data generation on the right hand, side it's iot, Edge, uh Gateway and Azure iot, Azure IIs with iot hub and then, when all of that is set up, then we actually run the service. So that stage we register with the 5G Network, the UE, the user equipment. With the 5G Network send encoded iot sensor data.

A

Then we enrich it with location services, data and then finally, we um send it to the iot hub on Azure, where you can actually see it. You can visualize it.

A

So this shows 5G and those who are familiar with 5G. This will look very familiar for those who aren't I'll, maybe just spend a minute explaining it.

A

So what do you see? On the left hand? Side is the user equipment and the g node B. So, like I said, the user equipment can be a phone, it can be an iot device. It can be a hololens anything generating connecting to 5G the G note. B is the radio area network, that's the radio part and the lower level protocols that go along with it.

A

That then connects to the UPF. Upf is part of an entity called the 5G core. So it's the core Network that connects you to either Mech applications or to the data Network to the internet or whatever, as the case may be so the user plane is, is there and that's how the data flows the Green Arrow and then the purple Arrow um on the top. You see the 5G control plane, the 5G control plane is quite Rich.

A

There are lots and lots of functions I'm not going to go through them, but they all need to be you. At least have to have What's called the AMF and the SMF to be minimally operable, and then many of the other functions are required for full functionality.

A

So all of that was orchestrated and you see the network some of the details of the network as well, and that was then sent over equinix fabric to the Dallas site to the Dallas server, where the mixer Mech application resides and the kubernetes stack that was deployed on each server. You can see that as well. You have, you know, kubernetes with maltes and flannel, and all of these applications are packaged up as Helm charts.

A

You do see a couple of virtual machines here and we, as you know, kubernetes and virtual machines are not incompatible. You can use Virtual machines with kubernetes using Technologies such as cubework, so um yeah. We don't use it different Technologies, it's all through kubernetes, so that shows the the 5G deployment.

A

This shows the flow of data from the user equipment- the iot device, in this case all the way to Azure iot Hub iot Hub. So what it's showing is that the sensor data is sent through the UPF through the Mac Federation data plane, which is the connectivity between the two sites to the Mac application, which in this case is iot Edge, so that data is sent then the iot edge application then says: oh I need the location data, so it then sends a request to the location.

A

Aps server gets the location information and then enriches the data with the location information and sends all of that so temperature, humidity pressure and latitude longitude. All of that is sent to the iot hub on Azure, where it can be displayed.

A

So this uh I'm, you know, don't be scared, I'm not going to go through the whole thing. um This shows the orchestration how the orchestration is done. So what we are using are Linux Foundation, networking, open source projects for the orchestration piece.

A

One element we are using is called Emco, Edge, multi-cluster orchestrator, it's a way to orchestrate Network Services, using Cloud native Network functions or Mech applications using Cloud native applications across multiple kubernetes clusters. So that's Emco. We use another piece from Linux Foundation networking called CDs controller Design Studio, it's part of a project called own app and that's used for um infrastructure, orchestration and configuration management. So those are the two pieces we use.

A

We use kamunda on top for sort of service orchestration and that talks to either apis on top or to a GUI both are available. um The combination is able to do everything I mentioned. If you attended my earlier talk this morning about nephew, we are actually going to start integrating nephew in this, because nephew provides a very nice declarative, intent which, frankly, is missing from this blueprint right now. So anyway, the orchestration is agnostic of the domain. So you can have different different domains.

A

Some domains may need you know: hem charts like the iot edge or the location services component or the 5G core. Some may need ansible like to deploy kubernetes we use ansible and to bring up metal and to bring a fabric we use terraform. So we use different different southbound Technologies depending on what's needed, and the cluster config Helm charts, ansible playbooks terraform are all in a git. So it's all devops CI CD flow.

A

So I think that covers the orchestration.

A

Yeah, so the idea is orchestration using infrar as code, so it's uniform it's model free. The orchestrator does not have to know the intricacies of the underlying domain. The state is kept external in a git repo and through git. It's all devops driven.

A

The Mac location, API implementation is as a kubernetes application. It's through a Helm chart and the home chart is onboarded using the orchestrator and then orchestrated um onto kubernetes.

A

So that's what this shows and finally, it's uh just a summary again of the n2a and flow we saw this picture already. But what it's showing is that the yellowish Arrow goes all the way from user equipment to iot device to iot Edge.

A

The Green Arrow is where the iot edge queries the location, server and enriches the data with with location, uh the latitude longitude and then The, Black Arrow shows black or purple I'm, not able to use a black shows how that information is then sent to iot HUB on azure, and then the blue blue, solid line shows the bgp peering between the two sides.

A

And just to sort of complete the picture, you can see the color-coded key at the bottom. The metal, Network Edge fabric are all uh equinix. The orchestrator is from our networks the place where a company where I work, the green, is from Etsy Mech. The gray 35gc is an open source. 5G core and the yellow is azure Microsoft Azure services.

A

All of this is been contributed to the open source. So if this is of interest to you, you can this. My presentation is up uploaded already. So if you are interested, you can go to these places and and see. What's going on and now I'm going to fire up a demo, the demo does have Olex voice, so I'm going to play it and and just sort of let him show the demo.

A

Oops all right all right hang on. Let me.

A

A

It's not letting me.

A

Do I have to do something on the audio.

A

Okay, yeah I can.

A

It's Mac airbook speakers, exetron scalar, D, USB, audio device, the USB.

B

An edge cloud provider.

A

B

The other one is the 5G access. What you see on the screen is the use case where the two providers, one is the Mac provider, um an edge cloud provider and the other one is the 5G access provider Federate their resources by way of providing services to each other. The Mac service operator provides access to a Mac application and Mac resources, and the 5G operator provides 5G access and location API service from Mac. Both are interconnected, using the Mac Federation data plane across the global fabric.

B

Our use cases proceeds in through in three stages: the infrastructure, orchestration stage, the 5G Network function and Mac service and application deployment stage and the end-to-end operation of an iot application. First, we orchestrate the topology used in our Network's uncap system and we examine what the created topology at the next step.

B

We will examine the input into that topology G, so the intent based input describes various components of the solution that we're going to create and those components are described in this XML file, Json file and then we'll launch the topology creation and then we'll see that the topology has been launched and has been orchestrated in order to see the orchestration we'll first look at the Azure side and look at the creation of the express route connection. So you see the express route connection is being created.

B

You also see that it is in provisioned has been provisioned and both the provider side and the equinix and the Azure side, and we see the private bgp peering has been configured for that connection. Next, we'll look at the equinix side to see the the status of this connection to Azure. It is active and this connection is enabled on the virtual device, the Virtual Router. So we'll look at that. The Virtual Router is the vnf that equinix provides and then it is a Cisco csr1000v, Virtual Router.

B

It has been assigned an IP address and we can see the connection to Azure. Express route is active and provisioned, which is the parameters and the description of this connection here in the equinix portal. The next step for us is to look at the status of the bgp peering after the creation of the vnf and creation of the private viewing on the Azure side, We examined the routing table on the Azure cloud and we'll see all the prefixes that are advertised from the vnf at equinix to Azure and the prefixes that Azure advertise is back.

B

This is the components of cloud interconnect of the Mac service provider, Next Step, we'll look at the what orchestration created for the Mac Federation data plane. So this is the the connection that links the Silicon, Valley and Dallas together, where our providers are. The next step for us is to look at the bare metal service. So part of this orchestration.

B

We orchestrated bare metal service and equinix data center in Dallas, so you'll see that this server has the properties that I'm showing from memory and CPU, and then it's connected to both connections, one to Azure and one back to the 5G service provider to in Silicon Valley. So here we can log into the server and look at its configuration, so we also as part of the orchestration deployed kubernetes on this server, and you see the um basically the system pods running on the server for for the for the for the kubernetes part.

B

The next step in our uh orchestration is the deployment of 5G functions in Mac service application. So we'll see this next and this step consists of creating the 5G functional deployment with the control plane with the AMF SMF and all other functions, including the web UI.

B

In order to log in and to see the status of the of the connections, we also see the you, the UPF cluster on which we deploy is a a standalone UPF, distributed here in the separate location and also also a simulated UE, so to examine that we'll log in into the control plane cluster. We see all the pods, but all the control plane functions up and running.

B

We next connect to the UPF cluster and in that UPF cluster we also look at the UPF pod that has been deployed successfully. So you see this here at the 35gc namespace. We see the UPF pod and also the Mac location, API path that we deployed as part of the orchestration. Next we'll look in our simulated UE just to make sure that it has connected so it has deployed the simulated.

B

Ue is realized in a kubernetes pod, we'll look at the status of the UE connection and on on the 35gc web GUI and we'll see that that UE is connected. It is correctly associated with the IP address in this case 10 1 0.9, and this is what we can verify on our UE.

B

If we connect into the container and then look at the address assigned for the mobile tunnel and we'll see the same address on the mobile tunnel interface within the container, we can also verify um uh just basic traffic that we can send from the UE to the mech server.

B

So we we do that by um specifying The, Source IP address of the mobile interface of the UE and then the max server IP address, and we see that little latency here between Silicon Valley, where the UE is in Dallas, is approximately 33 milliseconds on the equinix fabric. One other interesting test is that we can ping the internet directly from the UE from the mobile interface and because the UPF is distributed, our local breakout delivers very low latency under two milliseconds um next thing.

B

We go to the max server to make sure that we have deployed the uh IIT application. This is the Azure iot Gateway, which is the cloud native ID application that runs in the paths, as you saw. Our next step is to actually show the MTN operation of the iot application itself, with the ability to to enrich this application with location information. The first step is that the UE and the iot client will send the measurement data.

B

Just the sensor data, the IIT Edge Gateway will go back to the location, API server, obtain the location information for that UE and add that location, information into the iot data and then the the newly constructed message with the sensor data and the location data will will be posted to the cloud here. You see the end-to-end traffic pass so from the UE to the to the iot edge Gateway from my th gateway to the location, API for location information and then post it to the cloud.

B

At this point we will go back to the UE itself again looking at the it's its IP address and we will launch from within the UE container our reference iot client. So this is the python script and we just basically launch it in order to launch that traffic, we need to specify the max server IP address where the iot edge Gateway is running and the port on which it's listed listening and that Port is is shown in the in the display here.

B

So once we specify the port, the iot client was periodically start sending the data, and we see the first attempt was successful. It sends an encoded pdu field which encodes temperature humidity and pressure information into the message. Then we can look at the Mac server across the Mac Federation connection and look at the log of the listening, iot Edge pod.

B

You see that it receives the same data. It then goes to the location. Server obtains the location information for the UE. It then adds that location information into the message with the sensor data and then post the enriched message with temperature, humidity and pressure, uh longitude, longitude and altitude information to the cloud on the cloud. We can see that it is received by the cloud iot SAS backend, which is called the iot Hub in Azure, and we'll see the messages that are being received in the metrics statistics.

B

For that um for the device Shadow you'll see that the counter messages is increasing, as we expected all the way through this infrastructure that has been deployed and activated.

B

At this point, we will also show the end-to-end path that we realized here. So we constructed the Met Federation environment, with two providers connected via global.

A

Okay, so that concludes the presentation, and you know, even if you sort of put aside the 5G and and some of those complexities, I hope that that you, the big takeaway from this, which I hope you got, is that there's a lot of talk about using hyperscaler clouds and private clouds and networks in between and here's actually a practical implementation. That shows all of that. So with that I'm happy to answer any questions.

C

A

C

Yeah so uh I discussed with the volcano engine. People volcano engine is one of the cncf project, so they are using the t-talk application they try to. So, if you go back to one of the slides, you have the Azure there and the iot applications if they want to replace the iot application as the t-talk application, but they found it's now quiet. They can apply directly the volcano engine Edge Cloud, to replace the Azure Edge cloud.

C

So is there any tips or uh tutorial or otherwise to the team how they are going to slide in this piece of blueprint.

A

So they want to replace Azure with so.

C

There are two things they want to replay the iot application here as the t-talk. Second, they want to replace the Azure with volcano engine Edge cloud.

A

Yeah, it should be possible. So what has to be changed? On the left hand, side, it's a Helm chart change and, on the right hand, side. Currently, it's a terraform and, depending on what volcano requires. Maybe it's something different. So if they attend the next PC EI call we can, we can go through it technically and.

B

A

Can sort of give them I'm sure we can yeah.

C

A

Can give all like a heads up and we can. We can resolve those issues. Yeah.

C

It should be possible yeah. This relates to my other question. If you were here when I talked about the average catalog, so we try to do the kubernetes one click the deployment, but we don't know whether the kubernetes deployment fire here is too complicated. We cannot do one click like how many kubernetes extensions the installation file you need. Do we actually can do one click. We.

A

Can do one click so in the orchestration there are two steps: design, step and deployment. The design step has many many steps because you have to onboard the the CNF or the CNA. You have to put intent on how you want it orchestrated, but for deployment it's one click you just click on it and it deploys so yeah.

C

Yeah, so I would suggest you to double check of el fetch catalog to see whether the ham chart there.

A

C

Be identical to this hangout, oh any anything like a joint things could be done. Okay,.

A

That's a great idea: that's.

C

A great idea, yeah yeah, then it will be. This blueprint- will be the Pioneer to use that it'll.

A

Be a lot more valuable than yeah. Okay,.

C

Thanks yeah I just saw a bricks for the questions and to introduce the Jade that you may have.