Red Hat OpenShift OpenShift Commons Briefings hosted by Karena Angell (Red Hat), 25 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OCB: Deploying Crossplane Providers with OLM - Krish Chowdhary (Red Hat)

Description

Crossplane is a project that strives to bring cloud infrastructure, services, and applications closer to your Kubernetes cluster in order to create a hybrid control plane. This goal is primarily achieved through the use of providers, which are standalone controllers for a specific API group. The Crossplane project itself manages the lifecycle of these providers, from installation to cleanup. In this briefing, Red Hat's Krish Chowdhary will briefly discuss what providers are, the Crossplane architecture as well as how we can repackage a provider installation via the Operator Lifecycle Manager (OLM).

A

Hello, everyone welcome back to another openshift commons briefing and today krish is back with us. uh You've seen him before uh when we've also talked about crossplane, and that was a few months back and crossplane is- has applied for graduation from the cncf. So that's exciting news as well, and chris just had a blog post, go out that I will send to chris for those of you watching on all the openshift tv channels.

A

um You can go open up his blog post as well, and I will let krish go ahead and introduce himself and talk about um well, I will just let you go krish. Thank you.

B

Thank you karina, um so yeah hi everyone. My name is chris. uh Today we're gonna be uh talking a little bit about crossplane providers um as well as uh their recent olm repackaging effort from from the office of the cto.

B

um So who am I uh first things? First, so I'm a software engineer and red hat office of the cto, more specifically in the uh emerging technologies team, I'm also a maintainer on the cross, plane, sub projects for provider aws and the provider in cluster, I'm also a sick storage, member and maintainer on some sub projects.

B

So my twitter is is christo underscore. If you want to follow me there, my github is just chris chow, um so uh our agenda at a high level uh we're basically gonna, go over the. What? Where, when? Why? Who uh for cross-plane providers uh as well as then the how for olm repackaging uh in between there we're gonna go over the main components to help provide some more context for folks who aren't familiar with crossplane.

B

So, let's start with the one. uh So so what is crossplane? Let's quickly go over the project at a high level, uh so the first you know main pillar for crossplane is really provisioning right. Crossplane allows you to provision cloud resources from within your kubernetes cluster. uh It also allows you to manage the entire life cycle, so not only creating you know cloud resources, but also then updating those- and you know eventually deleting them if you need to.

B

uh And lastly, the specific providers expose information on these external resources right, so you can.

B

Over time right so something happens to your your rds uh postgres instance. um You will be notified on the custom resource through events uh and so from the crossplane docs. We have that providers are packages that enable crossplane to provision infrastructure on an external service right. So again, the main goal of a provider is really to provision things outside of your kubernetes cluster or your openshift cluster.

B

So these providers bring crds or managed resources that map one-to-one with external infrastructure resources, as well as controllers to manage the life cycle of these resources. So something key to note in that sentence is this notion of mapping? You know one to one of providers really being focused on fidelity for these external resources. This is going to be a common theme, but what are providers really right? Definitions are all well and good, but but what does it mean?

B

So providers are similar to operators right, uh but the main difference is that you know we're for operators, we utilize olm for providers, we use the cross plane provider to handle installation and and all other management right, so updating removing providers- that's all handled by the cross, plane operator, as providers use a lot of the same tooling as operators such as qbuilder controller, runtime and controller tools.

B

For this reason, a lot of folks in the community often consider providers to be an opinionated form of an operator.

B

But again the main difference is that providers are designed to reference some external resource right. So for a developer. The really nice thing is, you don't have to worry about our back deployment and and all the bootstrapping since there already exists a lot of tooling for for new developers and folks that want to create you know new providers so the where. So, where are providers located right? So cross-plane providers are open source and you know most will be available on github under the crossplane or crosspin contrib organizations.

B

So some examples. We already talked about the provider aws and provider in cluster. There's also, you know the provider sql, which allows you to orchestrate sql servers by creating users, grants roles, etc. All your all, your favorite sql resources, as well as the provider helm, which allows you to manage and deploy helm charts using a custom resource and, lastly, the provider aws that we talked about, but there's also many others that are not listed here.

B

My personal favorite is the provider pizza, which allows you to order pizza from domino's, using using custom resources and so they're all available under under the cross, plain orgs, so the when so, when does it make sense to create a provider right? uh When should you create a provider versus when? Should you use some some other solution right? uh So it really makes sense to create a provider if you're consuming external resources.

B

Right we've mentioned this before, but if you have some external resource and you want to manage the the crud life cycle for these resources, the provider is a really good fit.

B

uh Similarly, there's a really high emphasis on on high fidelity right, uh really really important focus, um and so you should create a provider if the resources that you want to use are well defined in granular right, so it doesn't really make sense to create a provider um for for managing abstractions right, since that's not the goal.

B

um And, lastly, um a special feature in crossplane is this notion of a composition engine and so the best uh with an asterisk. The best way to to utilize the composition engine is to create a provider that manages your resources right. uh The reason that there's an asterisk here is uh it's technically possible to utilize any resource within a composition, but it's really recommended that you only use resources that are exposed by uh by a provider.

B

um So the why, right? uh Why should you create or contribute to a provider? Well, so crossline is a cncf sandbox project um as as, as karina mentioned, uh they're applied for graduation, so hopefully they'll be on the next steps in the cnc.

B

Soon, yeah, and so all the providers are open source, so anybody can contribute. um So this comes with quite a few benefits right. Not only is there shared development and maintenance of common resources right if there's 10 companies and all of them, you know want the ability to provision resources on on azure or ibm cloud. They they can all share a common code base to achieve that. Instead of creating their own, you know bespoke solutions and for cloud vendors they can choose to expose their api uh in kubernetes through a common interface right.

B

So this is not only good for the cloud vendors, but also good for users right, I would prefer a standard interface or a common interface, at least for creating resources on azure, gcp, aws and crossplane helps to make that possible.

B

And, lastly, there is a very streamlined development and consumption process. Right crossplan handles a lot of the messy parts, uh so you know cluster cluster administrators and developers can can can get started on doing the work they need to do so. Why might you want to repackage a provider right? We've talked a lot about the cross, plane, design and structure, but why might we want to take a provider and structure it as a standalone operator?

B

So uh one of the one of the first issues is has to do with proxies right, we'll go into more detail about this shortly, but something that's really integral to the cross. Plane operator is pulling an op, an oci image right, so the operator itself pulls an image, um and so this can cause quite a few issues uh when there's a cluster that's running behind a proxy.

B

um This is not like an unfixable issue, but it's definitely one more thing for uh administrators to note and it's not immediately clear that this is how a crossplan is designed um similar. A similar problem has to do with credentials if you're using a private container registry um you'll need to supply credentials separately to crossplane and the cubelet right.

B

So can I, as you can imagine, this, can cause issues with credentials going stale or or or diverging between craftsman and the cubelet and and lastly- and I think the most compelling reason has to do with resource limitations right or or or essentially you might not always want to install the cross plane operator and allow crossplane to manage.

B

um You know the entire life cycle of a provider. um You know with with with the olm repackaging effort you can choose to only install the providers, you need and directly provision resources from those providers.

B

So, for example, I might just want to use uh the provider aws to create s3 buckets, and so I can do that uh with with the with the olm repackaging, and I don't have to worry about all the other setup for crossplane or um or if there's any other issues so um who is actually creating and maintaining these providers.

B

um Well, there's three main groups from my perspective, uh there's folks from the open source community. uh So, as I mentioned almost all of the providers that I know of at least are open source and there's many contributors and active members from the community.

B

There's also contributors for organizations like upbound, red hat, squiz and accenture and lastly, there's there's engineers from organizations like alibaba, ibm, aws and equinix that contribute to the development of their respective providers.

B

And so, let's get into the components of a provider and olm operator. So uh let's do a quick primer again on the provider aws.

B

So this is one of the most popular crossplane providers, with support for dozens of aws resources and for our purposes it's also a really great example of a provider to to dissect and then from the olm side, um we're going to be looking at the memcached operator, which is a common operator, that's used in guides and examples like this one and it's also a great example of the standard structure of an olm operator.

B

So what are the core components right? So, let's take a look at the memcached operator, so we can see here. First, we have our api folder. This is where we define all of the api resources that this operator exposes.

B

Then we have the controllers directory, which contains all of our controllers and reconciler logic for the aforementioned api resources and lastly, we have the config and bundle directory together. These allow us to describe all the metadata and deployment information for our operator uh and, as for uh provider, um we have uh uh we'll see the structure is pretty similar right.

B

We have the apis directory here which, just like the api directory, contains all of our our our types that we want to expose, um and then we also have our pkg directory, which uh contains all of our controllers and reconcilers. So, as you can see at this point, uh the difference so far is is pretty minimal and lastly- or I guess most importantly, is we have a combination of folders.

B

Here we have the package folder as well as our cluster and build folder, and so together these three directories kind of handle everything around deployment and packaging that we'll get into in a second, uh and so at this point um I think most people will have noticed that the difference here is mainly semantic right, but the last point that we talked about around deployment for providers is really key and we're gonna hone into that a little bit more here.

B

So how do I actually then deploy a provider right? There's no deployment, there's no cluster rule, no cluster or binding service account. Nothing that I can use that's clear here that I can that I can deploy with um so this is. This is something that's that's missing.

B

Right and so let's take a quick aside on deployment, uh so, as we mentioned before, within the crossplane model, you have to have the cross plane operator installed already and the operator the craftsman operator exposes a few different uh api groups, so, namely the pkg package that crossplaned group contains a provider resource a provider custom resource, and this is one of the most popular ways to install a cross pin provider.

B

All you have to do is define the uh image and tag uh that you want to provision so for this example, uh we're gonna use the provider aws image with the alpha tag, so this is straight from you know. The crossplane docks uh one of the other ways to install a provider is through the use of a configuration right.

B

This is installed the same way as providers, but configurations are designed to expose xrd's or composite resource definitions that we won't get into in this talk, but that's a that's a whole another a whole other topic in itself, but users can install providers through the use of dependencies on this configuration.

B

So you can see within the spec there's a depends on array where we can define the provider as well as the version, the minimal version that we want to install, um and so what is the delta? Then right? uh Essentially, uh this boils down to where are the missing components related to deploying the provider?

B

So um you know my favorite way to try and figure out something like this is just to start hacking away and break things down. So let's actually take a look at what our package contains right so with a tool like undocker, you can actually browse through the contents of our image without digging through the entire build process. Right. I personally would rather look at the end result then take a look at every step of of the build process and dig through a make file.

B

So let's pull our our image with the 0.18.1 tag, um we'll run on docker, which allows us to unpack the image layer by layer, uh and then we can see that uh within this directory, uh the only the only file is a package.yml file in the root of the image. This is interesting right. um There's, no controller! There's there's nothing! Nothing here that stands out: it's just a yaml file.

B

So if we take a look within this cmo file, um it starts off with a long list of the crds that this provider exposes uh and then all the way at the bottom. In the last you know 143 lines, uh we'll see, there's a meta.package.crossplane resource here and at the bottom of that we have within our spec uh an image referenced here, which is the provider aws controller right.

B

So this is the actual controller that uh we'll be using and and everything inside of the package.aml is basically just metadata and crds that we have to create.

B

um So, let's so, what's the rest of the puzzle, then right so based on this, we can deduce that there's two separate steps here right, there's our actual metadata image right, which is the provider aws and then there's the actual controller, which is the provider aws dash controller, and so it turns out if we dig through the cross plane operator code, that the operator parses, the package.amo uh installs, all the crds uh and configures the rbac for the provider at runtime right. So it parses all the crds and creates our service account.

B

Our cluster rule cluster role, binding.

B

All of that happens at run time when we try to install the operator or when we try to install the the provider, and so for brevity here uh we're not going to go over the cross screen operator code base in detail, but there is a link in these slides if anyone is interested in digging around there, uh and so at the end, the operator, uh the craftsman operator, creates our deployment uh using the tag referenced in the spec.controller.image within our package.eml document, and so this is where that aforementioned issue, we talked about a little while ago.

B

Around proxies comes right. So if the main cross plane operator is pulling an image, we have to make sure that it not only has the appropriate credentials, but that it can also even access the container registry we're using.

B

um So this is kind of the the crux of of that issue, as this is notable right, because olm opts to define all of our back and deployment at build time, uh whereas crossplane ops to do it at runtime.

B

So, let's get into the actual process of of olm repackaging, then.

B

So there is a project, a repository that we can use to help out with this process, um namely there's the olm repackage repository within the red hat et github org, and so if we clone the olm repackage repository and examine the contents, we'll see quite a few files here that we're going to be using so from the top. We have the docker file and make file that we're using.

B

So the docker file just contains everything required for building the controller itself uh and the make file uh just has uh all the targets. We need to actually uh build our operator and build our bundle that we'll see in a little bit.

B

We also then have our project.boilerplate.txt file, and so this just contains some boilerplate for our project file. This is used by the gen project script here at the bottom, and so this project file just contains a lot of metadata about our operator.

B

uh Lastly, or I guess, we also have the config directory so, as you can see, this has quite a few subfolders, the contents of which has been emitted, but we know that the crd directory contains all of our generated crds.

B

The manager contains our deployment manifest defines uh what our csv, our cluster service version, looks like, and lastly, within the rbac directory, we have all of our generated. Rbac manifests, which is related to the gen rbac script, so this script will create an rbac.go file which contains all of our cue builder, annotations that are used to generate the rbac during the build.

B

Process so we use quite a few different tools during the repackaging process as well, namely we use yq, which is a tool for querying different fields within our ammo document, the operator sdk cli tool, which is used to handle generation and validation for deployment manifests and, lastly, the custom we use customize and controller gen, so the the former uh of yq and operator sdk must be installed manually uh prior to repackaging, and the latter two uh customize and controller gen are both uh automatically installed by targets in our make file.

B

So not pictured here uh is golang and uh docker or podman uh golang. Of course you need, for um you know, actually compiling code uh and running, you know different generation and a docker and podband are needed, of course, to build and push up the image.

B

So what is each process then uh of repackaging? So part? One is all the setup steps right, so you have to clone your target repository uh in this example. We're going to be using the provider aws, then we're going to set up all of our dependencies right. So this was everything that we saw on the previous slide and we also have to make sure that we have credentials and access from those credentials set up for a container registry like key dot, io that we're going to use here.

B

uh And, lastly, you have to clone the olm repackage repository and then copy the contents from the olm repockage repackage directory into the contents of the root of the provider. If that doesn't make sense, we're going to go over an example now so uh in this example, we're going to clone the provider aws as mentioned before, and so we're just going to be using the the master tag. So we can get all the most recent changes.

B

And then we're going to cd into our provider aws and clone our olm repackage repository. So my preferred way to do this is just to create a hidden folder. I like to use dot work, and so once that's done.

B

We can just recursively copy the contents right, so this will override our provider make file as well as bring the config directory, as well as our docker file.

B

So now we just have to set some environment variables, uh more specifically, our container registry, user or organization, as well as the operator image that we're using so under you know our org we're going to use the provider aws tag and or the provider aws image and the master tag.

B

So now we can just run docker build and docker push with our new image. So within our make file, now we're going to pull controller tools and controller gen generate all of our or cogen all of our files that we need.

B

And in one second, we're going to generate all of our crds, the next step is running the gen r back script, which generates our rbac.go. So the actual contents of this go file is a bunch of annotations. Basically, um within our within this file, we have comments uh which describe what the rbac needs to look like uh and then we'll just quickly build the the image. So for me that was all cached, but if you're building this yourself, it might take a second uh and so we've gone through the build process for the operator.

B

We've added all of our environment variables run the cogen and push it up to our container registry and the last step of the build process is actually creating our bundle right. Although we've created our operator and that's been pushed up, um we still have to create the bundle and this bundle create contains all of our manifests and metadata, and this was put into a single package, that's installed by the operator, lifecycle, manager or olm.

B

So now we can start the process of building our bundle, so we're just going to have to set all of our.

B

Environment variables again so again with the image, the user, the repository and the tag that we're using.

B

We're just going to set the operator image again as.

B

Well uh and now we can run the gen project script right, so this just defines, as I said, before, our project file, which contains a bunch of metadata about our operator right, although this isn't strictly required. This helps to provide more detailed information for folks that are trying to install our operator.

B

This will finish in just a second.

B

Good chance for me to grab some.

B

B

Yeah, so there we go. That's done uh now we're just going to run a few commands here to template in some values into our config directory. Of course, we could manually go through this, but it's easier to just just run a command here and now we're going to rename our cluster service version dot yaml to include the name of this operator right. So we chose to name our operator provider aws, so we're just going to make that change, and now we can actually create our bundle.

B

So this again runs all of our uh code generation for our crds uh as well as generating our our back again. This will take another second uh and so just to talk about something we did kind of behind the scenes. So when we templated in all these values into our our config directory, uh this just updates a bunch of references and and metadata, so our bundle has been created. Now we get some warnings here, but but no issues.

B

Luckily, and now we can set our bundle image, which is the uh image reference for what our bundle is called. So I like to follow the convention of image dash bundle. Basically, this will effectively just be called provider, aws dash bundle and it will have the same tag as the provider aws.

B

So now we can just build our bundle, uh which is just a docker file, uh and this should happen pretty quickly since we're just copying a bunch of files around and now we can push that up.

B

B

B

So now, let's actually run the thing right: let's run the provider aws operator.

B

So again, here we're going to set our bundle image.

B

And now we can use the operator sdk to run the bundle. This is not the way to run the bundle in production, but for our purposes here uh this, this should should work.

B

And so the operator sdk will start the process of creating our catalog source, our operator group and subscription.

B

So these are the main resources required to not only create a new uh operator within openshift, but also it allows us to create our operator group, which defines name spaces that we can target as well as the subscription which actually requests uh the operator to be created, uh and so now uh we're just waiting on the cluster service version to be created, uh we'll see that that's pending now it's installing and it succeeded, and so just like that we now have our operator running, so we can check to see that all of our crds are defined here uh with all of our aws resources.

B

So now we can get started with creating you know an s3 s3 bucket or rds database or anything we want.

B

So that's all for this presentation. Thank you for your time.

B

My linkedin and twitter are here. If you want to connect or chat or feel free to bring me on slack, uh I'm on the kubernetes slack as as chris or uh the crossband slack as as krish as well, I believe, are there? Are there any any questions.

A

Thanks chris uh hey chris, do you see any questions in twitch.

A

All right, we do have a question: why is the repackaging necessary.

B

Yeah so let's, let's go back a few slides here.

C

B

There we go um so there's a few there's a few reasons why you might want to repackage a provider right uh if you're looking to get started off the ground as quickly as possible.

B

It might not be the right fit from my perspective, one of the main benefits of repackaging a provider is, you can choose to just install the providers you need and directly provision resources from those providers uh while building your own abstractions on top of them right. You might not necessarily want to utilize the cross, plane, operator or composition engine.

B

You can achieve something similar with, let's say helm for example.

B

So that's one reason, aside from that that the first two issues here, um you can either work around these issues and I believe, there's also fixes that are in active development to kind of handle these.

B

So, for example, uh one use case that I can talk about myself is another another project that I contribute to is the container object, storage interface project under the sig storage, and so for that project we essentially are trying to create uh drivers or provisioners for object, storage, vendors right, so we could use uh the provider aws operator uh as a driver for the uh cozy process, cozy project right, and that makes things easier for folks that want to get started since they just need to install the provider aws operator instead of you know, having to install the cross plane operator and handle everything through through through that operator.

D

Yeah- and I guess I mean, if you're strictly an olm type shop, just running operators, um you may not want to have another layer of abstraction from you know, cross playing where you can just kind of repackage it and plug it in um to work with all of your existing olm workflows that you've already established.

D

So um I do have another question chris that I don't know if it was clear to everyone, but um so will this repackaging like the package, the repackage repo work with any of the cross plane providers.

B

Yeah, so um not only will the old repocket repackage repository work with any of the cross plane providers out there um there's also an automated version of this process. So what I showed today was kind of the manual way to go about it, but under the red hat et org there is a github action that can be used uh to kind of automate every step of this process and repackage a provider for you and push it to.

B

You know a target registry um so that github action basically goes through the exact same workflow here uh except you know it can be automated and applied to any operator. So let's say you know you always want to use the most latest version of the provider gcp. uh You could set up that action to run on a schedule. Let's say once a day you know at 12 am, and it can you know repackage the most recent version of the provider gcp and push it up to you know your favorite container registry.

C

So, chris, I have one more question for you um and it's a leading one. Does this uh cross plane operator? Have you tested it up with okd, yet.

B

Yes, so um uh we've primarily been using um openshift, but we we haven't directly tried it with with upstream okd. Yet.

B

But if you, if you wanted to use, you know okd directly or kubernetes, vanilla kubernetes, um all you need to do is install the olm operator and then you can get started so that that you're able to achieve that pretty quickly. Okay,.

C

So it's not limited just to open shift. Ocp! Yes, perfect.

A

And chris you've been working mostly with the adf aws um provisioner right and you're. The main maintainer for that provisioner is that correct.

B

uh I I would say I'm one of the maintainers, I wouldn't say I'm the main maintainer yeah, but I've been contributing uh to the upstream provider uh aws for for quite a while. Now.

A

I was mentioning that because I was wondering if you had any tips as your as anybody else is looking to help write the providers. What are some things that you've run into some gotchas. Some tips that you'd like to share.

B

um Yeah, so I guess uh from my experience I think uh the biggest tip that I could have for someone that's looking to get started with contributing to um either the cross, plane operator itself or any of the providers. um I'd say the biggest thing is just to reach out and and talk to people. um So the the crossland communities is, I think, one of one of the best open source communities out there. Folks are always willing to help out and answer questions uh and there's community meetings that run, I believe, every thursday now.

B

um So that's always a really great place to start uh just to talk to people and get a sense of what there is to work on and more generally, if you're looking to contribute to an open source project, I think cross plane providers are a great place to start.

B

Not only is it a good chance to, you know, learn the fundamentals of contributing to an open source project, um and also you know everything about kubernetes and go. um It's also a good chance to learn about. You know some, a cloud provider right. You might want to learn about gcp or aws or azure, and you can use um you know your experience from writing a you know: gcp provider or resources in the gcp provider to kind of learn more in-depth information about google cloud itself in terms of gotchas.

B

I think the biggest thing is just staying up to date. The crossband project is still, I guess it's more stable now, but when I started contributing the project was definitely still uh in in the earlier stages, uh and so there would be. You know, breaking changes once in a while and it would be important to stay up to date, but uh now that the project has you know, matured quite a bit. um I think the api is pretty pretty stable. These days.

A

B

Hopefully that answers your question. Yeah.

A

Yes, I'd love to see different repackaging scenarios and you know using okd as a playground for that.

C

I think that I'll take that to the okd working group- that's kicking off in about 30 minutes um so yeah. Hopefully we can get some testing done um from the okd working group on this and and give you some feedback there as well so um and that you should be able to find a blog post if we get that done on okd.io sometime in the not too distant future, so um yeah again really great work.

C

It's wonderful to see um the use of olm and operators um to repackage this, and I'm looking forward to getting some feedback on this and watching the journey that cross playing takes um in the cncf too, in the not too distant future. So.

D

My my only critique chris would be. We should have used provider pizza because it's lunch time.

B

Yeah, you can get everybody uh a slice of pizza.

C

It's lunch time somewhere here we're still drinking coffee on a west coast, that's good, so yeah, scott and and and chris really. This is the great work that you guys have been doing and I'm looking forward to seeing a few more providers out there so um we'll we'll get those tested and and give you feedback from the okd and other arenas and uh karina. As always, thank you for for organizing this and making this happen.

A

All right thanks again krish and chris. If there aren't any other questions, can you please see us out then thanks.

A

A