Cloud Native Computing Foundation GitOpsCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Managing Apps Dependencies and Kubernetes Versions with Kraan and Flux - Adrian Vacaru, Fidelity

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Managing Apps Dependencies and Kubernetes Versions with Kraan and Flux - Adrian Vacaru, Fidelity Investments

The Fidelity platform team supports internal users on different Kubernetes versions and gives cluster admins from different business units permissions across EKS, AKS, and on-prem (with Rancher). Their OSS Kraan project helps create this seamless experience in a highly regulated industry by using Kubernetes and CNCF Flux. This talk covers Fidelity’s enterprise use case and how users can use Kraan to provide modular and reusable components across different Kubernetes clusters. Platform teams can spin up K8s clusters and install application “layers” using Kraan. Eg. an “IoT layer” comes with Prometheus, Grafana and InfluxDB for storing data from IoT sensors. Cluster admins don’t have to deal with complex deployment/upgrade logic because the kraan-controller, helm-controller and source-controller take care of that. Kraan also has a versioned layer dependency system that simplifies dependency management between addons. Add-ons can be certified and mapped to different Kubernetes versions. Other possible Titles: - GitOps in a Highly Regulated Industry - How To Solve Complex Apps Dependencies on Kubernetes - Kraan - The systemd for Kubernetes - How To Make GitOps Work for You - Layering Addons on Top of Kubernetes

A

Hello: everyone, my name is adrian vaccaro, I'm a software engineer with fidelity investments and I'm working in the public cloud platform team. We are working towards enabling our internal business units to safely and securely adopt cloud at scale I'll, give you an overview of why and how the crown project started, we'll deep dive into how it works and we'll also have a look at some of the features it currently has.

A

First of all to better understand the crown and the need for it, you need to know that fidelity is embracing a multi-cloud strategy and the approach we took to achieve that was to build a multi-level platform on top of kubernetes and other cscf technologies.

A

We started to work on building conformity across all the platforms that a business unit may consume the goal of doing that was to achieve a consistent developer experience, no matter what the platform, what platform the developers are using. In other words, if they are using kubernetes, they play on top of kubernetes it's kubernetes.

A

So we work to eliminate some of the complexities or differences between running kubernetes on premises or running kubernetes managed by a specific cloud provider.

A

In this way, when an application developers one wants to deploy to the cloud they shouldn't have to modify their applications to suit, depending on the cloud provider they are trying to deploy to.

A

While the focus is on the user experience, which is the application teams, the platform developer, experience is also improved when working in layers. It allows for reusability of components and collaboration between different business units and teams to improve the platform experience.

A

For example, here the security layer can be developed and tested by a dedicated team of cyber security specialists.

A

Once a new version of this layer has been released, the platform developers can integrate that in their systems as a dependency, they will have the confidence that that layer was tested and certified to be working on a specific kubernetes version and on all the cloud providers before we move on to the next slide. Let's take a moment to look again at the big picture here. So, with the help of the color coded boxes, you will notice that, as a user of the platform, your experience would be very similar.

A

No matter what the underlying cloud provider is, your application can leverage the same tools, the same services and on each platform, and you can easily migrate your applications from one to the other, with minimal or zero development effort.

A

Now that we've seen what the objective was, we needed to implement a solution uh to find a solution to implement that. I think you might be wondering why do we need another deployment tool to achieve that?

A

So, in the beginning of our cloud journey at fidelity, we found ourselves constantly adding new features or, as we call them add-ons to the platforms we were developing so managing the deployment of those add-ons became more complex and harder to maintain as the system expanded using helm and customize at the beginning works very well, you get a quick mvp, you have. You have fast development lifecycle and ultimately you get working software.

A

The problems start when you need to handle all the dependencies between add-ons all the preconditions before installing or deleting another and all any and implementing remediation retries. If something goes wrong and doesn't work as expected, for example, you might have applications or services that are depending on cert manager being installed before they can deploy correctly or you might want your opa constraints to be in place before you deploy any other applications to your clusters, and so the list can go on.

A

Crown comes into play when you need your suite of application to be deployed following some very specific business logic. It can simplify your deployment uh logic, but you can also speed up your deployment process by grouping those add-ons in layer layers before we move on from this slide. There is another thing it's worth mentioning here, so the difference between customized, helm and cron is that crown is a kubernetes controller, so it's deployed inside the cluster and all the deployment logic. It's happening in a reconciliation loop.

A

Let's have a look at this example, so your deployment script could turn from what you see on the left side. Here, a lot of ifs, while loops uh to one single line with cron qctl apply dash, f, layers.yaml chron will then take care of all the prerequisites you defined in the layers manifest and only deploy your applications when they are ready.

A

More than that, it will also reconcile your deployment application for further upgrades. All you need to do is define the layers define the dependencies and let cran and flux take care of the rest.

A

Let's see what's inside the layers at yaml file, so two of the most important configuration options here are the source and the prerequisite sections the source will reference, a git repository which contains the layers contents.

A

These are hammer list files inside the git repository all those helmer list. Files will be installed as part of this layer. The prerequisite sections section is kran's brain. It is what is driving crown in applying those helm releases.

A

They will only be applied. The hemolysis will only be applied after all. The dependencies here are fulfilled.

A

Other thing to notice here is the the version of the layer, but also notice that the version is using the depends on property as well. This is what we call the version dependency system.

A

The common layer here depends on the security layer at version 120., so unless this layer here it's already deployed to the cluster, the processing of this layer won't start like by crown at this stage, all you need to know is that managing the prerequisites section wisely, combined with defining meaningful layers in the git repository, can help you achieve even some of the most complex diploma scenarios, we'll deep dive into how everything works in the following slides.

A

So, let's see some technical details about cron in order to understand how the magic takes place. Cran is built on top of the github stool toolkit. It is using two of the key components of the flux family of projects. Those are the source, controller and the hand controller.

A

This means that, on top of the features crown brings to the table, you can take advantage of the previous experience of using the flux projects when deploying crown to a kubernetes cluster you'll. Get the source and help controllers out of the box as well.

A

They all work together to bring your cluster to the desired desired state during each reconcile. Loop.

A

To better understand how all these components interact, let's have a look at the following diagram, so the dotted lines here just separate the controllers from the resources they manage. We start from the bottom up. The crown controller reconciles add-ons layers. The add-ons layers construct is the only the only manifest file that a user would add would directly apply to the cluster.

A

Everything else happens automatically.

A

So as a user, you apply the the manifest that addons layers manifest file to the cluster and that add-ons layer contains the source reference which is, in fact a git repository that is managed by the source controller inside the git repository in a source control management system like github, for example, there is a path that contains a bunch of hand, release files, those are the manifest files that crown will need and it will apply to the cluster.

A

Once crown obtains a copy of those files from the source controller, it will apply those files to the cluster and, from here on the hand, controller will react to this change and it will work together with the search controller, to get the hand charts from the helm repositories and apply them to the cluster.

A

Let's have a look at the git repository and github's reconciliation so because crown is built on top of the flux projects, the source controller is the one that's handling all the gitobs operations when defining the git repository source for an add-ons layer. We have a choice of how we want to source the source controller, to track that repository, for example, if we decide that our add-ons layer will track the develop branch of a git repository every time, there is a new commit to that branch.

A

When the source controller reconciles the repository, the crown controller will pick up the changes and decide if there's something to do to bring that layer to the desired state. In this case here you've seen there is a new add-on added. There is a new hand, release added to the layer and it will be deployed to the cluster.

A

Everything happens automatically, starting from the git commit to the develop branch. The source controller pulls the latest changes from the git repository. The crown controller gets a fresh copy of the files and notices the difference. It then proceeds to apply, move or delete an add-on from the layer in order to bring it to the desired state.

A

If you want to use, if you want to track stable, released versions of a layer, then a tag or a commit hash should be used. You have probably noticed how the version of the layer is tightly coupled with the reference. The repository is tracking, besides playing an important role in the version dependency system. It also gives the visibility to the cluster administrator into what versions of each layer are deployed.

A

The next slide will describe how an add-on can be configured familiarity with the hand. Controller would be an advantage if we simply put it this way. An addon is just a hand. Release here are some of the features the the layer developers cannot take advantage of so the remediation retries. When installation or upgrades fail out, they can automatically run hand tests after each installer upgrade.

A

They can provide dynamic values to the chart using values from a config map inside a cluster, and they can also define dependencies on other hem releases just to clarify once again, when I'm talking about hand releases or when I take when I'm talking about an adam. This is what is actually behind the hood, in other words, all the information, all the metadata about how an application should be installed.

A

The application itself has to be packed as a help chart that is developed and maintained separately.

A

uh Let's now talk a bit more about the business logic of crown and we start with uh how cran handles the dependency relationships, so the main construct that cran defines you know is an add-on layer and inside that add-on layer there can be a number of different add-ons.

A

But the most important piece of the business logic is that you can define dependency relationships to from a layer to another layer or from a layer to a specific kubernetes version.

A

Let's take an example, so we have layer x and layer y and we want to only install layer x if player y is available, but also we don't want to install layer x if the kubernetes cluster version is lower than 120 and we don't want to install layer y if the kubernetes cluster version is lower than version 121.

A

This helps with reducing deployment, complex diploma script complexity and also gives developers confidence that the add-ons layers have been tested and certified to be working on a specific kubernetes version. One other thing to notice here is the intra layer dependencies that can be added. With the help of hand controller, you can see them on the screen on between add-on, two and three and between add-on, four and five.

A

This adds to the flexibility you have when defining your layers and helps with the confidence that all the atoms will deploy in the expected order, with minimal minimal remediations.

A

Further down into the dependency system, I would like to talk about a small but very powerful feature of cran and that's the version dependency system, we'll make a comparison uh between the version dependency system and a non-version dependency system.

A

We have as an example two layers where layer x depends on layer y. The dependency is needed because all the addons in layer x depend on cell manager from layer y between the two layers. There is a just a simple dependency relationship that allows their x to start deployment only if the layer y is already deployed to the cluster.

A

Please also notice the versions of each layer, both of them being on version 1 at the moment. Also, another thing to notice is that cert manager in layer y is at version 1 at the moment.

A

Given that the both layers are applied at the same time as soon as layer y is deployed successfully, then layer x can start deployment as well and it will come up. It will complete the deployment successfully because addon, 1, 2 and 3 were able to create all the certificates they needed because search manager was already installed.

A

Now, let's, uh in the scenario of an upgrade, we develop add-ons one two and three to use a new version of cert manager. We they are they're gonna, be dependent on version v2.

A

Okay, if we put those new add-ons in uh the version two of layer x, layer y, we're gonna, leave it as it is uh that it will have deployed cert manager, v1 and that's it. It's already deployed it's in a deployed state so because the dependence relationship doesn't indicate a specific version, the layer deployment will still go ahead, so layer x will start deployment because layer y is already deployed and, of course, the layer x deployment will fail at deployment time add-on.

A

One two and three were trying to create a new kind of certificate that only becomes available in cert manager v2. So how can we avoid this kind of errors?

A

If we go back to the initial state, we can avoid the previous error by defining a version, dependency relationship as follows. The first step is to to say that layer x depends on layer y at version v2, with this condition in place. Now crown won't start processing layer, x, v2 until layer y v2 has been successfully deployed to the cluster.

A

The current in the current state layer y v2 doesn't exist in the cluster, so layer x won't go ahead with deployment, won't go ahead with the upgrade we'll have to. It will have to wait until we deploy layer y version 2 to the cluster.

A

Once that is done, you can see here uh set manager v2 once that layer is applied, it will complete successfully uh deploy the deployment successfully.

A

Only at that stage, layer x will get the green light to deploy and complete successfully all because now, cert manager, version 2 is deployed by layer, y and add-on. 1 2 and three were able to create the certificates they needed.

A

Another interesting feature of cron is that it is possible to change your deployment strategy and you can move an add-on from one layer to another without having to remove and reinstall the helm chart doing this, you can avoid any unwanted downtime of your applications or even worse, potential data loss.

A

An example for data loss would be if you want to change your diploma strategy. So with one of your add-ons, you are deploying, has crds deployed as part of the hell chart. Removing the add-on could cause the crd to be deleted and all the related custom resources will be lost as well. So you don't. We don't want that. uh How can cran help in this scenario?

A

So, let's say, you've defined you've defined the layer y to contain only two addons instead of three, so layer sex layer, six, it's uh now outside that layer, but you want to move it to layer x so, instead of just deleting that atom when it's removed from layer y and instead of adding the the hand, release file applying the hand release file again because it's defined in layer x, uh kran is waiting for a period of time before taking any action on this add-on.

A

It's waiting for a period of time and it will observe that okay layer x is now referencing. This add-on, I'm not gonna, apply the file. I'm gonna just change the referenced owner of this add-on, and it will simply do that and we call this the orphan and adopt feature. As I said earlier, this helps with unwanted application, downtime or potential data loss. While allowing you the flexibility to change your deployment strategy, wait what about uh deleting an atom, so it works in a similar fashion.

A

Only that, if, at the end of the grace period, no other layer is adopting the addon, uh it will be marked for deletion and whenever that grace period finishes that will uh now this will be the the desired state is going to be reflected in the cluster. The add-on will be deleted yeah. So now, last but not least, I'd like to mention that cron is an open source project.

A

So we'd be glad if you check it out on github at github.com, fidelity, slash crown, that's where you can find all the risks, the resources or the source codes of the crowd controller and also a hand chart that can help you deploy cran the source, controller and the hand controller.

A

So you can start building layered platforms as well.

A

I'd like to thank all of you very much for joining me for this talk, and I hope it has helped you in some way I'll see you in the live q, a section and thank you again. Bye.