Cloud Native Computing Foundation Cloud Native Telco Day NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Automated Policy-Based Service Assurance in Cloud-Native... - Sandeep Sharma & Sundar Nadathur

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Automated Policy-Based Service Assurance in Cloud-Native Environments - Sandeep Sharma, Aarna Networks & Sundar Nadathur, Intel

Application orchestration across multiple Kubernetes clusters is becoming increasingly important, but policy-based service assurance and closed loop automation for applications continues to be a gap in the industry. We have developed an open-source solution to address this gap, using OPA, the popular policy engine, with EMCO, the multi-cluster orchestrator from Linux Foundation. Users can specify an action (and the corresponding actor) that should be taken on the application if certain conditions are met during the lifecycle of the application. The action could be re-configuration of application, scale out/in, migration to another cluster, etc. So, when deploying an application, the user can define declarative intents in EMCO that specify the OPA policy for that application, metrics to collect, the action to take on triggers, and the actor for that action. EMCO automates the policy configuration, metrics collection and policy enforcement. In this talk, you will learn about EMCO, OPA and how they come together for policy-based service assurance.

A

Hello, everybody I am Sundar nadato from Intel and I'm happy to co-present. This talk with Sandeep Sharma from all our networks. In this talk we will first introduce MCO and describe how it manages workflows. Then Sandeep will explain the work that Arna has done in using MCO workflows to automate policy-based service Assurance for cloud native workloads, The, Edge and multi-cluster. Orchestrator is an open source project to deploy and manage Cloud native applications and network functions across kubernetes clusters.

A

It is designed to handle jio distributed microservice based workloads which need to be deployed at scale across jio boundaries in public or private clouds in Telco or co-location, edges and on-prem data. Centers MCO is unique among app orchestrators, in that it can not only deploy apps scan charts but also automate the configuration of the environment around the application. For example, if the kubernetes Clusters Ingress Gateway, it may need to be provisioned with the HTTP routes for the application.

A

Also, the Enterprise deploying the app may have its own certificate Authority. If so, an intermediate CA has to be set up for the cluster and certificates for mtls and other needs need to be derived from that MCO automates. All these scenarios MCO has been designed to be extensible. It has a micro service based architecture and can be extended by adding more controllers or custom workflows.

A

Mco can be used as a building block for larger Stacks, such as the 5G super blueprint from Lux Foundation.

A

Emco has been a separate open source project from 2019 since its origination as a module and own app. Since last year it has been a part of Linux Foundation. We have participation from several leading industry players, including telcos commercial support is available, for example, from RNR networks as part of their amp car platform.

A

Emco is a part of the 5G super blueprint from Lux Foundation networking. We have an active ecosystem and there are ongoing engagements with some telcos for deployment.

A

Mco can be extended by adding new controllers that execute as part of the app deployment life cycle, but you can also extend MCO functionality with workflows for situations where actions need to be taken outside of application, life cycle events or when you want a limit access to the MCO shared resource and databases.

A

Mco can handle workflows using the temporal workflow engine since mco's 22o3 release, you can launch workflows, monitor the status and cancel them all using MCO. Apis temporal is a widely used open source engine that offers resilience and proven scale. We can think of temporal workflows as a set of distributed processes, Each of which execute several complex. Stateful steps these workflows may run for days or months and may even be cron jobs that repeat periodically the temporal framework provides many benefits.

A

One is resilience since the temporal server records the state for each task and restores the state in the event of a failure. It provides a standard way to set timeout values and retry policies for workflows and their constitute activities. It also helps with observability of workflows with status, checks and queries, and it is known to scale to a large number of workflows and tasks. These workflows can be written in many common languages.

A

Let us now look at how temporal can be integrated with MCO in a cloud native deployment, the temporal server will be deployed as a container running in one or more kubernetes parts. The worker processes will also be deployed, as kubernetes Parts, perhaps in a different cluster.

A

They may be deployed with the help shot across one or more clusters, possibly by MCO itself. A single worker can execute many tasks from different workflows.

A

The workflow client will also be deployed as a kubernetes part, and it may be in yet another cluster. One part may contain more than one workflow client and each client may possibly handle more than one workflow. Mco needs a way to invoke this workflow client. So we expect the client is packaged with the HTTP server.

A

This HTTP server takes an HTTP post call to trigger the right workflow client, the workflow parameters are included in the body of the postcard M course composed of many microservices MCO may be deployed in a different kubernetes cluster than workflow clients.

A

We have added the workflow manager as a separate microservice that allows MCO users, to Define intents for workflows, including the workflow client location, as well as timeouts and retry policies for each activity and for the whole workflow. The workflow manager initiates a workflow by adding a HTTP post to the workflow client.

A

The workflow manager can also perform status checks and execute queries on a workflow by directly calling the temporal server. It can also cancel or terminate a workflow I now hand over the floor to Sandeep. Thank you.

B

Thank you, sunder hi. All my name is Sandeep and I am from Arna networks and I am going to demonstrate a service Assurance use case using MCO. For this purpose. We have introduced a new controller in Amco and it is called the policy controller.

B

This controller will enable users to define the closed loops and Via Amco intents and deploy the closed Loops in the Target clusters.

B

The policy engine that we have used in this controller is the Oppa open policy. Agent. Opard enables users to Define policies. Declaratively and opar separates the policy code from the application code from the micro service code code flow. So with that brief introduction, I'll div I'll describe what the Amco policy controller is about.

B

So Amco policy controller is a new microservice that is introduced in memco and it acts as the policy informants enforcement point and, as I have already described, the policy controller uses oppa as the policy agent. So there are two parts to this policy controller. One is the policy agents. These agents are deployed in the Target clusters and their sole responsibility is to collect metrics from different resources.

B

Prometheus is an example, and in our demonstration we are using Prometheus. This agent is implemented as a kubernetes controller and we Define custom resources for when we Define custom resources specifying the metrics the kind of metrics that we want to collect the in the intervals of metric collection, etcetera.

B

These agents collect metrics and they enrich these metrics with with very specific information which the Amco controller requires in the central Data Center. In order to in order to complete the closed loop, the other part of the controller is the the main policy controller which sits in the central Data Center.

B

Its main responsibilities are to serve the policy intent apis, which the user specifies, and it also is responsible for calling the the policy engine itself for uh for the evaluation of the policies based on the kpis and metrics that it receives from the agents in the Target clusters and after the evaluation, the policy also uh and after the evaluation.

B

The policy controller also calls the actors, and, as we know in a typical closed loop scenario, there is a component which executes the policy and then there is a component which takes the actual action of the uh of auto correction.

B

So in our case, the controller that is doing the auto correction is the temporal workflow engine in MCO and in our use case, the auto corrective action that we have to take is to migrate an application from one cluster to another.

B

So temporal workflow as the temporal workflow engine as the actor is configured in the policy controller that I will show in the demonstration. So these are the responsibilities of the agents which sit in the central Data Center.

B

So this slide explains the policy controller and temporal workflow integration.

B

So, as I have already explained, the actor in this case is the temporal workflow engine in MCO and the the policy controller after the evaluation of policies will call a specific workflow in the temporal workflow engine for performing the auto corrective action. This diagram explains the end to end closed loop flow.

B

As we see there are policy controllers in the central data center, which are consuming events from the agents which sit in the Target clusters and policy controller is consuming these events and evaluating the policies against the metrics that are present in these events and if the policy condition is met, the policy controller is then calling the actor, which is the temporal workflow.

B

So, with this explanation, I'll jump on to the demo and demonstrate the service Assurance use case using MCO. In this use case, we are going to consume the memory memory utilization As, the metric from application, and the condition that is fed in t policy is that if the memory utilization is above a certain value, we migrate that application from one cluster to the other.

B

A

In this demand,.

B

Demonstration we are going to show the policy intent specification and the workflow intent specification since workflow is the actor in the service Assurance use case. So we will start by showing the the custom resource that is created for collecting the metrics. So add ahead. As I had explained, the agents which sit in the Target cluster are are implemented, as the kubernetes controllers.

B

So at this point we can look at one of the custom resource that we have created.

B

A

Is very simple.

B

All we are saying is that we want to collect CPU and memory related metrics and with the specification of this CR, the controller will reconcile to it and it will start collecting these metrics from all the parts that are created via Amco, and these metrics are collected through Prometheus and after collection of these metrics.

B

The the agent is going to enrich these events and for in the enrichment is done by adding the deployment ID of these pods, so that MCO in the central data center could relate these parts to the composite applications and the entry deployment intent groups, because the the intent, the policy intent and the workflow intent they are specified at the deployment intent group level. So this is how the whole relationship is getting resolved and maintained.

B

So after this, we are going to prepare the actor and so by by that. What I mean is that we have to specify the the temporal workflow intent, which Sundar already has described.

B

This is the uh this is the workflow definition, so it's a very simple workflow. All we are doing is we are going to migrate an application from one cluster to other cluster, and this will be triggered via the action that is, that will be taken by the policy valuation.

B

Now we can start applying the uh the intent.

B

So this CML defines the intent, so we are applying the intent, the workflow intent.

B

So after the intent specification now we are going to uh specify the policy intent.

B

So this is the definition of the policy intent, so the policy it has to the intent has to know the endpoint of the policy controller and the uh the workflow name which it has to trigger.

B

So this is the uh this is the policy definition and dynamically. We are going to change this number, which is basically the memory usage bytes. So we will make it a more logical number so that this condition is hit.

B

After changing the uh the value of the memory bytes, that is to be compared with, we see that the policy has hit and- and it has already triggered the actor to take the action and, on the right hand, side of the screen- we see that the the application has already started migrating, so it it is terminating in the cluster one and it has started to come up in the cluster too, so yeah. So that concludes my demo. Thank you.