Cloud Native Computing Foundation Kubernetes on Edge Day EU 2021, 14 May 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: A GitOps approach to Zero Touch Provisioning for the 5G vRAN Ed... Ian Jolliffe & Yolanda Robla Mota

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

A GitOps approach to Zero Touch Provisioning for the 5G vRAN Edge - Ian Jolliffe & Yolanda Robla Mota, Red Hat

Traditional approaches for deploying software at the Edge don’t scale. For example, consider the effort to deploy thousands of nodes at the Edge of the network for 5G vRAN DU’s at the base of cell towers, we need a new approach. Zero Touch Provisioning (ZTP) and declarative frameworks can be leveraged to drive down OPEX and deploy software in parallel to multiple sites, with no human intervention. The declarative framework leverages many open source projects including Ansible, Kustomize, OpenShift, and Open Cluster Management. This is a diverse environment, using declarative frameworks, we can define the infrastructure as code and manage it via GitOps methodologies. These approaches are required so that the operator can bring a diverse ecosystem together to address their business problems at scale.

A

Good morning, good afternoon welcome to cube on the edge.

A

My name is ian joloff and I'm here with my colleague yolanda, and we're really excited to be talking to you today about a a git ops approach to zero touch provisioning for the 5g vran edge.

A

Well, let's maybe start off with introducing each other. Maybe you go first yolanda and then I'll all do the same.

B

Hello, I'm yolanda robla, I'm from spain. I've been working in red hat since five years and I've always been dedicated to open source and clouds. In the latest times, I've been working in kubernetes and openshift, like especially setting in telcos and well, I'm working in like using openshift for 5g and run deployments.

A

Thanks yolanda and again my name is ian joloff. I recently joined red hat, I'm located in ottawa, canada. I've been working in open source projects for the last 10 years and most of my career in in telecom. I really interested in virtualization, openstack, kubernetes and other open source projects.

A

So, let's dive in so today. What we'll discuss is some of the 5g rant topologies and challenges really trying to take those concepts and apply them to infrastructure's code at the edge go through some declarative frameworks and workflows for these deployments and then wrap with a demo.

A

So one of the technologies that we're leveraging is openshift, it's really a distribution of kubernetes that comes with things like a registry and many other very important components like networking, overlays.

A

So, let's look at start by looking at some of the challenging topologies at the at the edge of the network. um Let's start at the bottom and and work our way up. So uh starting with a three node cluster, where we have supervisor functions and worker functions on a set of three nodes.

A

This would probably be located in a regional data center, potentially running cu workloads up next we've got where the supervisor nodes are located in the in the regional data center again, but then leveraging remote worker nodes at the edge of the network, and this is the configuration that we're leveraging in our demo and the third third topology is really uh where supervisor and worker nodes are both at the edge of the network, uh and this is a really our single node edge server configuration this really helps with low bandwidth or completely remote sites, and that's some work.

A

That's going on right now, the key technology that we're leveraging is open cluster manager and it really allows you to manage many clusters, as if you were operating in the in the core of the network, helps with life cycle management, policy, governance and advanced life cycle management, concepts like deployment and updates and upgrades.

A

So some of the key considerations at the edge of the network are really about latency from the du to the radio unit. Extremely tight tolerances are required for latency in this modality, and so you really need to make sure you're getting the maximum performance out of the servers that are located at the edge of the network.

A

Things like real-time kernel, multis and srov are key components that we need to think of and deploy at these edge sites, so we're really providing a common and automated infrastructure that allows you to manage all these nodes as if they were in the data center. We've also looked at how to deploy um with small iso, so it's easy to bring up and deploy your servers at the edge of the network.

A

So it's really all about making sure that these servers that are in very disparate geographies can be configured and managed as if they were part of a normal kubernetes cluster.

A

So, let's shift the focus of the conversation today to get ops, and you know this is the deployment topology that we'll be following, and I thought we'd start with a an industry definition for get ops. I won't read at all, but uh the keywords that really resonated were for me are really it's: a set of practices um traditionally focused more on application deployment, but we're using git ops in the context of infrastructure deployment.

A

The other major concept and and word for me that jumped out was really declarative, so we'll be using uh yaml in a set of well-known defined patterns to provide a declarative framework for all the infrastructure that we need to deploy and it'll all be stored in git and be able to uh be leveraged by the open cluster manager technologies that that we've been talking about earlier.

A

So how do you break this? You know very complex, rand topology down into into a git ups and have it all defined as code. Well, it really starts at a high level from a site plan that a telcom operator would have for their mobile network, we're going to break that down into a set of manifests that can be stored in git and then we're going to deploy with zero touch using the open cluster management technology.

A

So it's really important to understand some of the players in this workflow, and you know some of the key personas that we identified were in the telecom operator side, the uh planning organization, who's responsible for network design, vendor selection and capital planning, and then on to the installation and deployment teams where they're building out the sites, hiring contractors for the installations making sure power and physical plant infrastructure is available to these new and interesting servers that are now living further at the edge than they ever have then last.

A

Lastly, it's really the maintenance and operations team, that's really responsible for the day-to-day operations updates and upgrades and and the deployments so really uh those are the three personas on the telecom side. So let's look at the overall ecosystem that is enabling uh this new paradigm for uh for getting compute to the edge.

A

We have the system, integrator really pulls together a set of partners and and the overall ecosystem integrates orchestration and brings together the lower layers of the stack together with the application layers, and they can help the operator consume the site plan and represent the infrastructure as code.

A

The infrastructure provider persona is really the uh people providing the platform technology such as openshift and ocm, open cluster manager, and and also deploying the high performance profiles to make sure that the workloads can operate uh these strict vran workloads and then, lastly, we have the application provider, that's really providing a 5g ran application or a 5g core application, so very diverse ecosystem is now coming to play to enable these new 5g ran applications at the edge.

A

So that I'll hand it over to yolanda.

B

Okay, so I will explain how the workflow for the digital approach is working.

B

Our source of truth is just a set of git repos, where we are containing what we call a site planning and a site planning is just a definition: a set of ml files that are containing all the different settings for the clusters that we want to deploy.

B

So we will include things related to the cluster, like the cluster name domain like ip ranges, and we also have like definitions for the hardware like for the for the environmental host. We can just provide details from the bmc static network definitions as well, so we have this data with the site plan, and then we are combining that with uh other para other manifolds that are defining that the coordinates manifest operators that we want to deploy in all the clusters.

B

uh Like we give the clusters a set of rows, we could tell which monitors manifesto. We want to use everything uh with that is combined with customize and we are producing using like the final output like it will be a yama definition of all the the settings and all the things that needs to be applied on the clusters.

B

When we have this yama, we are just applying passing that to ocm to advanced cluster management. This is consuming it and as soon as the clusters are joining and rolling into acm, it will be just reapplying all the different operators, the different configurations that we have just told in our site plan.

B

So we begin with clusters finally configured and deployed with our settings.

B

Okay, so I'm going to explain a bit the operators and all the layers that we are doing for our cluster provisioning and definition. So we start with the base components.

B

We assume that we are starting with uh servers that don't have any operating system in standard that just are bare from the factory, so we start by provisioning them with operating system that this core os. On top of that we're installing the openshift container platform.

B

So when that is completed, the the nodes, the worker node, will join into an existing cluster once it happens, we'll start applying the 5g run profile that is composed by a set of operators.

B

So we start with machine configs that are just for enabling like system, definitions or low level settings at system level like we enable ntp, http things like that. Second, as the 5g, it's very important to have good performance we're using the performance addon operator that is taking care of that. So it will install real-time kernel on the nodes that need it. It will just define the number of cpus the cost. We reserve the number of huge pages as well, depending on our workloads.

B

After that, we also need to just interact with low-level network functions, so we want to consume directly. The the network features from the cart, so we are using the salary operator.

B

This will allows us to just use the physical function and the mutual futures for every nic, and also on the views where we need like very, very good time, synchronizations. We are using ptp that is precision time protocol, so open cluster management will be installing that as well.

B

Okay, uh so, finally, I'm going to explain a bit how the deployment process looks like so we start as we were telling with everything in git like we have the site planning it in what we call the regional data center.

B

We have a cluster with open cluster management installed there, and then we can apply our gtp and it will be used to deploy different types of clusters. So we can just deploy like clusters with a control, plane and remote worker nodes.

B

So that will be one use case. Another google use case will be single node, where we can just uh use it for the use or it's just a server with masters and workers at the same the same host, and also we can have a ceo pool that we are using three masters that have also the the roles of workers, so this sctp is able to just deploy clusters with these different topologies, depending on our site definition.

A

That's great yolanda thanks very much for uh walking us through that. I really like how we can start with uh the infrastructure site plan apply. The customize overlays uh then wrap those uh that overall picture in policies that can be then leveraged by the open cluster manager. So, let's, uh let's dive in and see uh see the demo that we have here.

B

Hello, so, let's start with the demo, as I was explaining before everything was starts, with a site plan, a site plan is just a set of yammer files that are restored into a game and that contains the information needed for deploying of massive number of clusters. So you can see here what I'm talking about for each clusters and we have their settings the name of the cluster, the domain, the version some network settings as well, like the english and the api bit.

B

uh Well, then we can define the control plane entries for the control plane here, where we explain the number of masters the mac addresses of the masters as well, and then we define the workers as well. Workers are the remote worker not that they're going to join after the control plane is deployed. So for each worker we define the bmc details like the the address user and password.

B

So with that, everything is translated to just someone civil inventories and from there we start the deployment of the cluster okay. So next thing I want to show how to configure static networking for the workers.

B

What you need to provide is just a file, a yaml file that will contain all the network configuration in nms state format for all your worker nodes. Okay, so you can see here that we have a list of interfaces. Each interfaces is belonging to a worker node as it's consuming any state. You are free to define the kind of network that you want like bilan bonding.

B

So next thing is creating a cluster. You can see that simply we're just going to run a next stack in the playbook.

B

It will just create a cluster in using the openshift assisted service. Okay, you can see here. This is the assisted, installer web user interface, and you can see that we have just created a new cluster with the settings that I pass it on my mentory. It is using for seven.

B

You can see it matching the cluster name, the cluster domain, okay, so the step finally just finish it and you can see that you could have three nodes, just enroll it into a system installer and with once we have just three nodes. We can start with installation of the openshift cluster. It will just set up some settings and just the order of the blue in the cluster, and it will just wait after some time the cluster has been installed.

B

You can see here that you have the three masters showing us installed with all the details and now it's time to just worker nodes, I'm starting to show you how the server is reacting, it's okay, so you can see that it has been powered on and we are telling them instructions to boot from virtual media.

B

It starts to use to mount the image that it has downloaded and it will just complete the provisioning step. Now. Okay, you can see that the nodes start to report up to assisted installer. We will have two worker nodes that will be joining as you have seen. The nodes are already enrolled in the system installer. So next thing is to deploy the cluster to you have seen that the workers are starting the installation.

B

Okay, so you have seen that the finally, the worker nodes aren't rolled into the cluster, so we have a working uh kubernetes cluster with a master control plane and to remote worker nodes. So we have this ripple that is called gtp acm manifest, and it contains all the objects that are going to be applied into advanced cluster management.

B

Everything is here, is working in a github way, and everything is controlled by just coordinates. Manifest and crds.

B

Here is how our basic layer is composed. So we have common common is just the base base manifest that are going to be applied to all clusters. Then we have profile that we are going to use it to configure clusters depending on the function they have like. We can have an additional layer that is hardware type, that we can configure clusters depending on the type of hardware. And finally, we have the size. That is a specific configuration at site level.

B

Everything in acm is working via subscriptions. A subscription is just an object that is telling icn that it needs to listen to a specific repository or to a specific folder. Okay. So here, for example, you can see that we have the subscriptions for common base.

B

Everything is composed mostly of namespace channels subscriptions and applications, and you can see that the subscriptions are just pointing to another ripple, our ram manifest that is actually having the source of truth of all the configurations.

B

And we have the same, you can see that we are matching the structure that we had in acm subscriptions with https and managers.

B

Okay, you can see here that everything is based on a policy. A policy is just an object definition of htm that is informing how it needs to copy which object. That does it needs to create into the cluster, and it is just simply creating the operator. Okay, yeah, I'm telling I want to create this object in all clusters, and I'm just going to create an object that installs srw operator after a policy.

B

There is also what it calls placement buildings and placement rules, so the placement rule allows you to define to where do you want to apply this policy? So in our case we are applying the policy in all types of clusters.

B

In this case, for example, we are just applying all the configurations to all the clusters that are small and let us do and then we can have the side policies as well. That are specific things that needs to be configured at side level like, for example, vlans name of nics, so things that needs to be configured at state level, for example, are the linux selectors like you need to exactly match the name of the interface here? So it needs to be at cell level, and everything in acm is just configured into this in this way.

B

This is just a default acm installation and on this installation I have applied all the manifests that you have shown in the gtp acm manifest repo. You remember that I have been showing that we have different layers and we have created subscriptions based on that, so you can see here the run, subscription, hardware type, the profile and the sites when a subscription is applied, it's actually creating all the objects that we are pointing to into the chat and to the the folder of the git repo.

B

So you can see here the final outcome of that we have the common run subscription. That is the the main object that we created into acm. This was pointing to the gdp rank, manifest matching folder that contains some policies, placement, buildings and placement rules. So you can see here how acn is just applying all the different objects. What I'm showing now is how a cluster is enrolled into htm.

B

This is pressing. You can also see that the cluster has some labels. That is profile duo. It means that it will apply all the workloads that are matching with the view profile, and this has been the finance for the placement rules that I created.

B

Okay, and you can see that everything is installed properly. If this acm is monitoring the status of the cluster, it shows the version name of the cluster, the profile, the number of nodes. You can see that it doesn't have any application, and this is anything you can see that there are policies that have not yet been applied when a cluster joins acm it the icm automatically recognizes that and tries to start reconciling all the changes that are needed to configure the cluster automatically.

B

This is you remember that this is managed via policies, so we have several policies that are defining the kind of objects uh manifest that needs to be applied on the cluster and then acm is matching those policies with the rules that we define it and decides where to apply. This is prone to change and we are just working on a final version.

B

Everything where everything will be integrated, so no need to run money manual, answer playbooks. Everything will just be defined on the git repo, including the cluster definition worker hardware.

B

The hardware settings for all the the clusters and the procedure will be just automatically. Writing all the information from git and applying the procedure that you have seen here with assistant, installer and scn to configure all the needed.

A

A

Thanks yolanda, that was a great demo, um really enjoyed that great job really had a great time uh doing this with you and let's open it up to questions thanks very much. Everybody.

B