Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OLM: A Declarative Way to Install, Manage, and Upgrade Operators - Dan Sover & Alex Greene, Red Hat

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

OLM: A Declarative Way to Install, Manage, and Upgrade Operators - Dan Sover & Alex Greene, Red Hat

This presentation will cover the Operator Lifecycle Manager (OLM) — a suite of production-ready operators and tools that help manage the complexity of installing and upgrading operators on a Kubernetes cluster. OLM provides the following features out of the box: ▪ Over-the-Air Operator Updates and Catalogs ▪ A Rich Dependency Model ▪ Operator Discoverability ▪ Cluster Stability This talk will focus on the following topics: ▪ Introduction to OLM and its APIs ▪ Discovering and installing an operator via OLM ▪ Features that OLM provides on-cluster ▪ The operator bundle format ▪ The future direction of the OLM project Interested in joining the OLM community? We have weekly meetings every Thursday — check out https://github.com/operator-framework/community/blob/master/wg-olm/README.md to get started!

A

Hello: everyone thank you for attending our kubecon eu 2021 presentation where we will be covering the operator lifecycle manager, project commonly referred to as olm.

A

My name is alexander green and I'm a senior software developer, working at red hat, primarily focusing on the operator lifecycle manager project. My teammate dan is also a developer for the olm team.

A

In today's presentation, dan and I are briefly going to introduce the concept of a kubernetes operator, discuss some of the challenges associated with managing a kubernetes operator, as it evolved over the course of its lifetime and finally introduce some of the features that olm includes. That makes it easier to manage these resources.

A

So, typically, we rely on kubernetes for deployment scaling and management of applications, but at the end of the day, we really just want to have a service available to our engineers or our customers.

A

Take, for example, the classic mysql application on kubernetes mysql is a relational database management system that consists of both state, full and stateless applications.

A

If you take a look at the resources at the bottom of the slide, you'll see a deployment channel a service, a persistent volume and a persistent volume claim.

A

As a developer with limited, my sequel experience, it's fairly easy for me to get a my sequel service up and running on my cluster by just creating four kubernetes resources. Unfortunately, there are a few shortcomings with this approach.

A

First, each of these resources are independently managed. I don't really want to manage each of these resources on my own and when I have a disruption to my my sequel services, I won't immediately know where to look. I might have to look if the deployment came down or the service itself was deleted. I might have to see if the persistent volume or persistent volume claim encountered an issue.

A

It would be much better if I could look at a single resource to identify what's going on with my mysql application.

A

Perhaps more importantly, I don't actually know how to safely perform an upgrade to a mysql database.

A

Now certain, I could devote a lot more of my time to understanding my sql to understanding how the upgrade should be performed and make it safely happen, but I would much rather leave it to domain area experts to codify and handle this upgrade path.

A

And this is where operators really come into focus operators allow domain area experts to codify operational knowledge and software life cycles into an operator. This operator exposes a kubernetes api that allows me to define what type of application I want installed. In this case, it allows me to define the mysql database that I desire and based on that information, it's going to create all the kubernetes primitives, such as deployments stateful sets and everything else. You can imagine so that that service is available. On my cluster, you may be wondering how this is done.

A

Operators are composed of just a few components. First, you have a controller which is just an application that runs on your cluster, typically as a pod. That includes a client. This client allows it to communicate with the kubernetes api to understand the current state and the desired site, as well as creating updating or deleting resources.

A

Once the controller is created, you have the custom resource definition. The custom resource definitions are responsible for expanding the kubernetes control plane by introducing a new api. These are cluster-scoped resources that define the api's name, version, scope and supported fields.

A

The crds are effectively blueprints for resources that the custom customer can create.

A

The resources that the customers create that implement the the customer resource definition are typically called custom resources.

A

So, with this information in mind, a typical approach to using an operator would be for a user to create the mysql cr. In our example, the operator would notice that that event occurred would look at the mysql database described by the user. It would try its best to create that app that that mysql application and then it would write the status of the mysql application to the status of the custom resource I created.

A

This gives me a single window for me to define what I want on cluster and then a single place to look to see the status of that resource.

A

Hopefully, that makes it fairly clear why operators are valuable in the kubernetes ecosystem. It allows you to build kubernetes native applications, but let's discuss why om comes into the picture for the management of these operators.

A

So when you decide to use olm to manage your odd operators, you'll automatically gain access to an extensive ecosystem designed to make expanding expanding the kubernetes control plane. Very easy.

A

Olm immediately grants users access to a catalog of over 100 operators, and the contents of these of this catalog is updated in real time as new versions of the operators become available.

A

When you install an operator using olm, you can, that operator can place dependencies on other operators in the ecosystem. Doing so allows operators to take advantage of services offered by these different operators. So I could write a front-end website application that relies on the mysql operator to store database store information in the database. It provides allow me to build a very good, complex application without reinventing the mysql database operator.

A

Finally, once these operators are installed, users are capable of identifying which operators are running which custom resource definitions are available to them in each namespace.

A

So, despite these features that I meant that I introduced earlier, you may be wondering well why can't I just use a different tool to install and manage the operators on my cluster.

A

This is mostly derived from the fact that the management of crds and operators, as they evolve over time is far more complicated than that of typical kubernetes primitives.

A

By installing a kubernetes operator, you are expanding the kubernetes control plane and as soon as the crd is introduced, users can take advantage of the services offered by the operator by creating crs changes to the customer resource definition associated with the operator could invalidate these existing resources on cluster. If they don't adhere to changes in the schema, such as when a field is removed or a new required field is introduced to the schema.

A

Perhaps more troubling is the fact that, if a crd is removed from cluster, every cr on cluster is immediately deleted, potentially causing customer data loss. This is extremely concerning when you think about our previous example, where we discuss a mysql operator. Imagine if you had workload operator installed and a user accidentally deleted the crd, causing all the customer data to disappear.

A

That's not a good time so outside of the possibility of rendering existing workloads invalid and losing customer data. Another big issue with the using operators without olm is the fact that kubernetes eddies does not offer a concept of crd ownership.

A

When a crd is introduced to a cluster, it is not tied to a specific controller that consumes the resource. Rather, it simply introduces an event stream that any operator can watch and reconcile.

A

If you introduce the crd, there could be multiple controllers that watch for events on the cr d.

A

In the same name space. This can cause race conditions where the two operators constantly go back and forth updating the resource and the resources they create.

A

Finally, there are other complex scenarios that make upgrades ding an operator potentially dangerous to users when you don't have a system like olm in place.

A

As a user of an operator, you deserve to be provided with the opportunity to upgrade an operator safely as new versions become available. In my opinion, one of oem's greatest features is the ability for operator authors to define safe, upgrade paths that are pushed to customers clusters as they become available.

A

This approach allows operators, authors to ship bug, fixes and new features in real time to customers all over the world as they become available.

A

You may be wondering which operators are available with olm and those can be. This list of operators can be found at operatorhub.io.

A

These operators are developed and maintained by domain area experts that are trying to get their product out to you in a safe and controlled manner.

A

Now that I've discussed the impact of operators and the number of features that they uh that are supported by olm, my teammate dan will provide a bit more context surrounding how catalogs are built and made available in your cluster. Thank you.

B

Thanks alex so I'd like to discuss today, the operator bundle format and some different ideas around packaging and delivering kubernetes manifests and kubernetes data onto the cluster.

B

So, to begin with, um let's look a little bit about how we package and deliver content in the kubernetes space. Today, generally, we have solutions that evolve around helm, charts, customize or git ups, get repositories, flows or simply raw yaml. These are all the different ways that people commonly take kubernetes manifests template them and push them to a cluster.

B

There's also. Another variation on this, which is to use container registries which are generally available from within the cluster or easily externally, via hosted registries like koi and docker hub to store, manifests historic, kubernetes data, and this is a very attractive sort of system, because the kubernetes cluster already knows how to talk and pull down and push images from a registry.

B

So if we take a step in that direction- and we start thinking about how do we package kubernetes manifests in container images, we have this manifest bundle, kubernetes enhancement proposal that we merged recently, and it basically describes how we can associate a set of kubernetes yammer manifests with one or more runnable container images to provide a standard way to transmit data between clusters. If you think about some of the features you get for free, when you tie in your kubernetes manifests with a container image, you get things like identity.

B

The container image is hosted in a certain repository on a certain registry, so you get identification with it. You get distribution in the sense that the container registries are highly available and accessible from within the cluster. You get replication uh deduplication and signing.

B

For example, you can sign continue images to give more information along the software supply chain on where that image came from its provenance, and you also have cluster ingress generally, as I said, uh because everything kubernetes essentially rolls around a pod and a pod has an image and its spec grains is a lot of tooling around dealing with container images and vulnerable, cleaner images in particular.

B

So this manifest bundle cap is an is a way to try to standardize how to package and associate kurena's manifests with container images so to make it more clear on the next slide here I have an example of an operator bundle, so this is the uh scd092 operator bundle and we see that at the top level, there's two directories slash manifest and slash metadata.

B

The manifest directory contains the coupe ctl appli appliable yemo manifests so essentially everything that you would would say as your crds, your core resources like config maps and services, your olm specific custom resources like cluster service versions. Everything that you can apply to a cluster via group ctl apply dash f.

B

That would be under your manifest directory, whereas the metadata directory is a directory that only contains application metadata, and this includes things like name version packaged information, including dependencies and a whole host of information, that any sort of package manager like tool would need to know to be able to install and manage your manifests in a more robust way, and so this is an example of an operator bundle, and this bundle fully defines a particular version of the lcd operator package, and so conceptually which, which you can think about, is that if you have a cluster, you may want to install an operator bundle onto it.

B

You may want to install multiple operator bundles onto it that all associate themselves with different services, like the mysql operator that alex mentioned earlier, the lcd operator, uh et cetera, et cetera. So you want a way to sort of group these bundles together and expose them within the cluster. So on that note, that's where the idea of composing bundles comes in being index, for example, here on operator hub. We have an example of a couple of operators which which are associated with bundles.

B

So what we can do is we can compose these bundles into an index which is itself a runnable container image which has a set of bundle references inside of it. That knows how to provide bundles to install via an api. You can basically package up these different bundles, like let's say the istio and the chiali bundle into an index, and then you can ask the index hey. Can I get the?

B

Can I install this istio operator or istio mesh inside my cluster and the index will be able to surface all those manifests all the data that you need that's stored within those container images to actually install them onto the cluster, so what the index does is.

B

It enables one to have a content, addressable store of operators and services from within the cluster, even in a disconnected environment, and where you can mirror all the content that you need ahead of time into the internal cluster registry image registry and then install things that way so index is basically a composition of bundles and is very powerful tool for using and installing different operator services onto your cluster.

B

So that was a brief introduction into the manifest bundle format, and with that I think I like to talk a bit about the future of olm and some of the things that we have in store in the next year or two so going on the same topic of the bundle format currently olm supports uh certain bundle resources uh several of them, certainly, but not all of them, and one of the big things we want to do is we want to support bundles with arbitrary resources, including custom resources and pod security policies.

B

So you can imagine that their bundle, uh let's see the operator bundle packages, a certain version of an operator. If you allow the user to package a custom resource into that bundle. Alongside the other manifests, you can basically have an init configuration for your operator on startup. There could already after the crds are installed on the cluster, there can be a cr already there, so the operator can use it to have some more advanced configuration than it would. It may be like a config map, so that provides some um interesting use cases. Also.

B

Pod security policies are something that generally associated with security and uh overall robustness. There's a big ask for being able to lock down the workloads associated with operators and ensure that there's no privilege escalation. So that's so we, but we want to open up this bundle format to basically everything, so all that will will be able to gladly accept any uh any manifest that it finds in a bundle.

B

Another big thing that we're working on right now is around declarative indexes so indexes today, as I described earlier, as sort of a composition of bundles.

B

They uh package all the bundles into a sqlite database internally and then expose the api, the server bundles api over a grpc connection and that works well, but it generally produces some edge cases and pain points, and what we want is to have a more get ops friendly design in that we have a declarative index config file, which is essentially just a big json blob that users can use to edit and replace and update in place the different bundles that compose the index and basically have a really nice experience in modifying an existing index or building an index from scratch.

B

It's just a json file that you know that an api can serve in front of it, and part of this is also using on other open source projects like qlang to help validate that the index json file needs certain conditions and basically makes it much more easily extendable and usable by other projects. So this is another exciting initiative around declarative indexes.

B

uh Lastly, we have v2 apis, um v2 olm is being actively developed on sort of in a prototyping stage right now, I think one exciting thing is that we are planning to allow users to unpack and run any standalone bundle image without an index. um So you can just say I want this version of this operator on my cluster and boom. You can install it no problems.

B

Another which is very interesting is the idea of using pluggable content provider controllers so right now all them sort of monolithic in that it's a series of controllers, but we could sort of make it more modular. So more similar to the existing persistent volume. Persistent volume claim storage class design, that's already in native kubernetes, so that different content types can be supported via different controllers. So the old lambda bundle type, which is sort of akin to the bundle format that I described earlier, would be described by the olam controller.

B

But you can imagine there being another uh helm.chart type media type which the helm chart. The helm controller knows how to basically when it sees one, it knows how to interact with it, how to unpack it and how to put it onto the cluster and then manage its life cycle through updates and through deletion as well, and so basically becoming more modular in what kind of applications media types that we accept, and maybe us have more than just simply.

B

Operators right now is very focused on operators, but we want to support different types of applications in the future, so this is sort of the olm v2. Another interesting initiative we have is around search manager.

B

We want to integrate with the cert manager project, which is an outstanding uh covariance project to issue and manage certs when installing operators via olm olam, currently handles ca, server generation and rotation using self-signed certs for all operators, web hooks and api services that it manages, and so delegating to cert manager would naturally be very attractive to users who want sort of you know a highly advanced and configurable uh circ managing when they install operators and other operator artifacts.

B

So that's that's about a bit. That's a quick run through uh here is some information on the online community. Olam is part of the operator framework, which is a cncf incubation project and we have a public working group or issued charge meeting every week. Every week we on thursdays. We have a meeting. uh Please check out the operator framework, all mdev mailing list uh for more information sign up to the mailing list, and you get in meeting advice to the calendar events.

B

uh Then we also have a slash channel on the kubernetes slack uh hashtag lmdf. So please stop by there and feel free to ask questions and, lastly, on the bottom, there's our community page, where you can find links to all the meetings and all the information. So I'd like to thank everyone for coming to this meeting and learning a little bit more about olm and now me and alex will be around and happy to take any questions or comments that you guys have.

B

Thank you. Everyone.