Red Hat OpenShift OKD Working Group, 2 Feb 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OKD Working Group - Ansible Collections for OKD - Fabian von Feilitzsch and Timothy Appel (Red Hat)

Description

OKD Working Group 2021 02 02
Ansible Collections for OKD
Fabian von Feilitzsch and Timothy Appel (Red Hat)

A

uh So I'm tim mountain, I'm a senior product manager on the ansible team. I was actually with the original ansible company and I've been along for the ride to red hat and then to ibm and I've been working on how ansible can integrate with and help automate things happening in the container native space. So one of the things being from you know, working for red hat is that we looked into uh once. We started this effort, how we could start to help automate, what's happening in okd and what's happening in openshift clusters.

A

So what I was presenting here was like the 1-0 that we came up with uh initially and how it came together, and I want to try to just speed through this um there's a lot of went on here and we could go a whole lot deeper, uh finding uh put together a demo if we have time for it. um If not, we can provide the code. So I like so that's that's my background.

A

Where I'm coming from there, I was a programmer one time used to develop and then became a pm, and then they took my keyboard away from me and said you are no longer allowed to code, um um but I get to work on this type of stuff uh bobby. You want to say a few things.

B

Yeah, um I'm fabien vampirelic, I'm a uh software engineer in the openshift org. I work on operator framework.

B

um I've been involved in like the ansible kubernetes integration space um since pretty much right after I got to red hat five years ago, um so I've just been working on building out like the python clients uh and then integrating those python clients with ansible modules and things like that.

B

Just have like better sort of like full, like application to infra level integration, uh so that people who are using ansible in there like traditional I.t things, can kind of more easily transition to the kubernetes space without having to up end all their tooling logging monitoring, etc and I'll cut it there. So we have time for the whole presentation.

A

All right thanks bobby yeah, though so. The first thing I should mention, if it's not apparent by who's presenting to you, is that this was a joint effort between the ansible team and the openshift team. uh Fabi was one of the engineers that um came over and worked with us uh in developing this, and we had some of our own people from the ansible team working together on this. So uh this is truly a joint effort that happened out there. uh So just diving in uh I'm gonna like say I'm going to speak through this.

A

So this thing that we're talking about community.okd is an ansible content collection for automating and managing the unique capabilities of okd and openshift systems and the keyword there is unique and I'm going to come back to that now.

A

I know I'm not talking to an ansible um group here, so you might be wondering what what's a collection uh you might be familiar with ansible, but in the last year or two we've had this huge effort going on to separate the core engine that that ansible's known for that command line tool and the what we call the content so that they're separate in that they can move independently.

A

One of the problems we ran into with our batteries included approach was that you had to wait for the next release of ansible to come out to get new features for um a cloud service or some type of other application or api change, and, and it was just getting way too bogged down.

A

So we came up with this thing called uh ansible content collections that we've been moving towards we're most of the way through or it's just short for, our collection and it's a new format for organizing ansible content, so that it's independent of the engine and can be added and installed and updated independently of what's happening in that.

A

So uh what we're talking about here is one of those collections that is specific to working with okd uh and openshift here so, like I said uh just to review that then, and so so uh this is to focus on the unique capabilities of okay openshift systems. We also have another collection, which has been now renamed kubernetes.core and that provides the baseline, kubernetes and helm3 automation uh capabilities out there.

A

So if you're, working uh with okd and and openshift you're, probably going to use both of these collections together in your your playbooks and the stuff, that's baseline, you would um work with the stuff, that's in kubernetes.com and then, when it comes to the things that are specific, that okd adds. On top of that, then you would, you would pull from uh the community okd collection, a couple other side notes.

A

uh If you go out and start researching this, that you might become uh confused or or wonder about is uh uh camino okd is the upstream uh uh uh collection called redhat.openshift and that that is the supported offering that we put together and put out there to customers. So it's one in the same. It's just one once the the downstream and once the upstream of that content.

A

um Another quick side note is originally our kubernetes content started off as a community effort and was called community.kubernetes we're going through the process of changing the name uh migrating the repo things like that, so they're they're, essentially the same, but the community.kubernetes is going away for marketing and business reasons and is going to be called community uh kubernetes core all right. So that was just a little background. So you know what you're looking at here.

A

So let's talk about what is in this collection, uh so so what we did when we, when we pulled together this effort uh last summer, to make something that was supportable, that we put full-time resources on we worked together with um um is we we looked at the what was in that community, dot, kubernetes collection and said all right? uh We need to break this into two parts, because what had happened is it was just done through community contributions coming in and it was.

A

It was um mostly uh baseline kubernetes, but some openshift specific features had rolled in and then we were getting kind of uh complaints from both sides. People that were trying to use opd, openshift and saying uh hey this is missing, and then there were people on the baseline, uh kubernetes crowd coming to us and saying hey. What is this stuff?

A

That's in here that it's operating different than it should so we decided the best thing to do- was then to to split this stuff out into their own collections, so that they could both move and focus on each other's communities um better, rather than trying to find this like um middle ground. So uh that was the one of the first big things that fabian and other engineers uh took on.

A

uh The other thing that I mean it was very very helpful in was getting proper ci testing, uh including prow integration into this, so that all of what we were doing got run against the latest uh builds that were happening there. That was something that, unfortunately, wasn't happening in the previous uh collection and work, uh so we migrated a whole lot of community uh community content over uh uh that was open, shift, specific, an inventory, plug-in and oc connection plug-in.

A

There was a uh an openshift auth module that was called case off at the time, we've renamed, and then we created a a module specifically for working with declarative um resources, but it gave it the added logic for working with things like um I'm, trying to remember some of them deployment, configs and projects and things that are specific to openshift, that the kubernetes core module would sort of trip on um there.

A

So uh uh one of the things that was uh a little interesting that we went through is ansible's added name spaces and we decided to make use of that, um and so we had the case module that, like I said, handled the baseline kubernetes declarative apis. So, rather than create a a totally different named one, we decided to use the kate's name again, because there you don't have to do it fully qualified like like I've shown here.

A

um It would make it a lot easier for people to move or or port their playbooks between baseline kubernetes and then moving to open shift um in that regard, because then they would just have to switch what name space they were pulling that module from. So there's a little side. Note more advanced thing, uh and then we created a few modules. So this is an area that we were. We went did a quick survey, and so what are the most common things?

A

People are trying to automate with openshift right now to figure out what is in the 1l and the and the two things that came up was here was the the ability to uh expose a route uh which which is sort of like the um exposed in kubernetes, but the added stuff that you can do uh in openshift and then the other was was the uh templates that came up the ability to to render and optionally apply those to what you were doing were also things that we were seeing.

A

A lot of people that were trying to use ansible with openshift were were trying to do and struggling, and we wanted to make that easier. So we created those two modules there. um So I'm going to stop there, like, I said I sped through a lot of stuff. um Do we want to take the time for a demo.

C

I think we actually have the time where there was one question in there: okay,.

A

C

I can't see the chat, unfortunately on my that's okay, but I think you might have answered it. James, you are asking will play books written for community okd um work without changes when used with redhat.openshift.

A

uh Yes, as long as yes, there should be no no issue there. You just have to be put a little bit of care into how you're managing your name spaces. If you do it fully qualified, like I showed back here, you would have to do a search and replace, but you don't have to do it this way um and I would recommend not doing it this way.

A

If that's, what you want is the ability to go between the two easily um there's a there's, a way to create like a namespace search path at the beginning of your playbook, uh and then you don't have to do this stuff. The fully qualified stuff in your in your in your plays and your roles.

D

Is there a reason why uh the red hat open shift whatevers wouldn't also provide the community okd name if they're going to effectively be identical.

A

um That that was more of a marketing issue of our of customers getting confused to what is supported and what is not, and so we decided to do it through the naming to make it apparent.

D

A

um Yeah, there is also legal reasons that we couldn't use uh open, shifter, red hat in a.

C

A

I almost lost my mind trying to deal with this. I did not want this, but unfortunately like I'm, not a lawyer either.

D

A

Get it I'm annoyed.

D

A

Yeah yeah, but me too, honestly, yeah.

C

Things in there oops, okay, that james is ansel, is pushing very hard to get people to use the fully qualified name. This is going to make broken playbooks and roles when trying to switch to okd and oh ocp,.

A

um Yeah, that's more of an we've done it for awareness and also clarity when, when you're dropping in a single task to document something you don't see all the other stuff, you could have done at the command line or in the playbook declaration, and then it the you me. People may get confused over time, as you have modules, with the same name appearing in different collections entirely um that it was it's just for clarity of that type of documentation.

A

We're not recommending that that everything you do should be done in with fully qualified namespaces in your actual work. It's more of us getting this concept across and also in the clarity of when you're, only taking a snippet and dropping it in you could lose someone.

C

A

Don't that makes sense james.

B

Alright, so I'm going to demo some of the features that we've added in the newly released community.okd ansible collection, which contains a variety of modules and plugins that make it a little bit easier and interact with the openshift api.

B

So uh first things. First, let's look at what we did around image streams we added some special handling so that we can kind of better handle resources that are referencing image streams without causing unnecessary redeployments.

B

um So this is what the basic module invocation looks like for the case module the community.okd.cades module is very similar to the kubernetes case module, except that it has some of this special handling. For example, projects, if you don't have permission to go, to create a project it'll instead issue a project request and handle that whole api flow, which the core kubernetes module cannot.

B

So this will just go ahead and create a project named testing. You know kind api version, it'll just send it right through to the api, I'm just going to kick it off on the right here.

B

So if we look, it returned a lot. A lot of that is because of this managed fields, uh field, that's returned, but this is what the api returned. When we issued a create with this definition, you can see we have a project here. uh It's got the name of test and that's pretty much it. um So next, let's go ahead and create an image stream.

B

Let's look at what that image stream looks like so we can see here it's just a regular uh kubernetes, manifest instream.yaml and we're going to be pulling in the python docker image, and you can see here you can just reference a file directly much as you could, with qctl or any of the other.

B

You know common utilities, so let's go ahead and create that image stream, which should import that python image all right. So we can see that it was created. We see the spec, that's returned here and the status which gives us the new, uh the location of the container in the uh openshift image registry.

B

um So next, let's create a deployment config to reference, this image stream we just made. We can look at what this looks like real, quick. It's basically going to just it's going to use that base python image. It's going to spin up, but just a basic http server and we can see we have some environment variables set and then it has this image change trigger.

B

That will basically say you know if, if, if the, if the image stream gets a new tag, then automatically update our deployment to use it, and it will also mean that in this spec, open shift will update that python. That image that it's referencing to be the image from the local registry. Instead, so let's go ahead and create that deployment config.

B

And you can see here we have this weight and weight condition. This will basically look in the conditions array on the status of an object and it won't end the task until this condition is true, so we're basically creating this deployment config and we're just going to wait until it reports that it is available.

B

So here we see it's ready. If we scroll up, we can go past all the managed field stuff. So we have a deployment, config, hello, world dc and the testing name space, and we can see here.

B

This spec does in fact have the image replaced by the image stream reference and it's a specific sha um as opposed to what we put into the spec initially, which was just python.

B

So now we're just going to create we're going to run this exact task over again we're going to recreate the deployment config now because the deployment config is already there, the modules will will see that and rather than issuing a create, which would obviously fail they're just going to issue a patch instead, so that any difference in this deployment, config definition from what is currently in the api server, um would then should result in it to be redeployed and we'll note, because we have this image field from the api changed to the registry, whereas in the deployment config we just reference it as python.

B

So there is actually a difference between our local file and the file in the api. Currently so, but if we run it, we can see that, even though there's a difference in the local file in the file in the api, uh it reports no change.

B

uh It issued a patch, but it issued the patch using the trigger because it knew that this image field is once managed by another controller and there's no sense in wrestling with that controller for control of that field, and this is especially relevant if you're writing an operator or something like that, where, if you're referencing these image streams from an operator, it would then lead to infinite reconciliations as the deployment continually emitted. New events and your operator continually went and tried to change it back.

B

Okay, so now we can do the exact same thing using a regular deployment you can see here. Deployments can actually be made to use the image streams using this special annotation, which tells it what container to pick and what to replace.

B

So uh if we go and we go ahead and create this deployment, yes, so that will deploy it, it's using the same base image. So hopefully this will be pretty quick and then in while we wait for that- and you can see here- wait wait wait for that to finish here. We're issuing only a patch request to replace only the fields that we specify. So we only want to replace spec template spec containers. We want to replace the container named hello world with the python image again.

B

This should result in a change except because these are the openshift modules and not the core ones. We know that we don't want to update that, because it's not it's not going to stick. So there's no point in doing that.

B

B

Okay cool, so that's the end of the industry. So uh next, let's look at what we added for routes, so um routes are basically uh open, shifts method for creating ingress. So here let's go ahead and just create this simple deployment. It'll deploy a hello, openshift container that runs a docker container that basically just outputs hello openshift in in over http. So we go ahead to that. We'll create the service as well to expose that port, and then we will look here at the open shift route module community.okd to openshift route.

B

Openshift route is approximately equivalent to oc, create route. It can expose most of the same stuff. So here we can see we reference the service that we just created that exposes that container, that we just spun up um and we're going to create every all of the stuff in the default namespace. So when we create that route with the fewest possible arguments just giving it service and namespace, we can see that it returned this object, which includes this url.

B

We can go ahead.

B

And hit that ourselves and we can see well, I can see, let's that it outputs hello openshift when we visit this route now, and we can verify this from the announceable task by attempting to hit that url and we can see that it also returns the content.

B

And here you can look through the code to see all of these different methods. I'm not going to go through all of them now, but basically there's a lot of different arguments. You can pass throughout again it's roughly equivalent to create openshift, create or oc create route.

B

So you can do custom names, you could allow. You know tls or disallow tls or make tls redirect all different kinds of things all exposed through the module without needing to you know, get in there and and do this sort of definition uh by hand alright. So let's just skip through the rest of this route.

B

B

um All right, the third thing I wanted to look at we've added the ability to interact with the openshift oauth server directly through this module called community.okd.openshift off. So first, let's go ahead and create the secret which will contain the information for this user, a username of test and a password of testing123.

B

Let's go ahead and configure the ht password identity provider so that it uses the secret that we just created to verify users and we'll go ahead and create the test user and we'll mark it saying that it uses the ht password provider and it's the user test there uh and we'll create a cluster role binding this cluster row binding will give our new user uh cluster reader access.

B

So it should be able to see all the things in the cluster.

B

So next we're just going to use this community.kubernetes.cates cluster info module, which will return information about how we're connecting to the server what the host is, what authentication parameters we're using, etc, etc, and we're just going to store that in this cluster info variable. So, let's go ahead and get that api. Url.

B

So you can see here all the information that that module returns uh returns. All the information about the connection it returns, information about what apis are supported by the uh server that you're connecting to and a bunch of stuff like that, as well as client and server version.

B

um So next we come to the actual indication of the openshift off module. So you can see here we're just saying: login is the user test with the password123 to this host that we just pulled from uh the cube config that we're using to connect right now? So let's go ahead and obtain that access token, and there we go so you can see that we ran that and it returned this api key uh that we can use to authenticate um and now that we have that information.

B

We're gonna use that api key stored in this off variable here and we're gonna use that host that we picked up before in order to see if we actually can list pods which we should be able to because we have cluster reader access.

B

So here we go here, we can see all those pods and- and we listed them in testing namespace. So these are all the pods that we created during that image stream demo just a little bit ago,.

B

Okay: next, we also added the ability to interact.

B

B

uh Open shift templates, which are basically a way that you can either locally or in the server um set up some some basic templating or without you know, using helm or the ansible templating language or any of the other uh options that are out there right now. So um the nginx example template is one that's included by default in an openshift installation and it pretty much does what you'd expect, which is create an nginx deployment.

B

So that example lives in the openshift namespace. And then here we can pass the parameters. So we want to deploy it to the openshift namespace as well, and we're just going to give it the name test123. So, oh, and also we are putting it in a rendered state, which means that we don't want to go ahead and create the resources. We just want to see what resources would be created. So let's go ahead and render that template.

B

So we can see here this: these are the resources that would be created. It would create a service that exposes our nginx pod. It would create a route that exposes it.

B

On the you know, that would expose the ingress for it and uh a build config for building the image from a git repository and the deployment config for actually deploying it and and the pods, um and it's all hooked up to the uh nginx example. uh uh I'm sorry and it's all.

B

Apologies left turn I thought for a second yeah and it's all hooked up with the image stream tag. So now that we have these resources, we're storing them in this result variable here we can go ahead and create those rendered resources by looping through them, and the supply parameter means that we will be using. um You know uh the basically, the equivalent of cube ctl apply in order to create them.

B

So let's go ahead and do that um and we can see that it made all of those resources uh that we had rendered before build config, et cetera, et cetera.

B

um So let's not delete those resources, we'll give it some more time, and then you also have the option rather than create like rendering and then create in the manually that you could also process and then create them directly in one step, and we should see pretty much nothing change here, because it is the same resources all right.

B

um And finally, the uh information, uh the the last little bit that I wanted to highlight um was the openshift inventory plug-in, which gives you the ability to use openshift as a dynamic inventory. um And basically, this means that the the plugin will go. Look at the cluster, look at all the pods in the cluster and add those pods to your ansible inventory as targetable hosts. So you can see here this second play targets the namespace testing, pods group.

B

This is a group that is automatically created by the openshift inventory plugin and basically it will target all of the pods that are in the testing namespace. So let's go ahead and run that, so you can see the first task, ignoring this meta one. The first task that we run is setup.

B

Now it is important to note running modules other than raw does require python on the container. So we can see here this test1231 deploy deployment container.

B

um Does that doesn't have python installed, and so this is not going to work on it, so it was failed, um but we can see these test 123 nginx example, the hello world dc and the hello world deployment pods that we spun up earlier all were found and setup was run and we can verify the setup runs successfully.

B

By looking at the value of the test environment variable because if you remember in our deployment configs, we added that environment variable just test with the value of test, and you can see here it output in the hello world dc that the value of test is test and in the hello world deployment. The value of test is test and, of course, in the nginx example. It does not have this environment variable defined, so um we had a failure there and then uh last uh you can.

B

As long as there is python on the pod that you're targeting you can copy files to and from the host um from from ansible making it so that it is as long as there's python installed in the pod, basically or in the container there's. Basically, you can do anything that you could normally do with ansible there all right- and that is all that I wanted to demo. So thank you.

A

Yeah, so um thanks thanks bobby uh it was, that was great. Hopefully, it got across, like all the different things that you can do to to automate and cut down on all the command line stuff. You would have to do in manual, work and repetitive, work. Every time you would deploy a cluster or do anything the the the big question that we have for for, for you is besides using it trying out seeing how we did in our our 1-0.

A

Is uh you know what could we do next? What do you want to see in this next there's there's a lot of areas that we didn't touch on and the question we kept asking ourselves was well. Would that be useful, well, which one of these is a priority, which one is not that's the type of feedback that we're looking for? We got a good core set of feedback, mostly from from red hat consultants and a couple other people.

A

We knew in the community that were doing work with ansible and openshift, and they gave us that initial batch of use cases and we've essentially covered them all right now, so we're we're. Where do we go next? So that's the feedback we'd love to hear um for those of you who are interested like. I will give this deck to uh diane to to send around, uh but uh these are some of the repos uh of the content.

A

You were looking at starting with fabian's um the demo code that he was just running through and and the repos where we're developing the collections we've been talking about and then there's a bunch of uh blog posts. If you want to go deeper and and read about this stuff uh in in more detail, maybe more elegantly than I've been speaking uh about it. So uh that is all that we had.

C

Well, that sounds good it. um I think we could use some feedback from the community on um maybe some uh uh maybe more uh useful examples. The example was good fabian that that was bad, but just um how people um would use this in in production and so joseph and other folks who are doing that um your feedback will be most welcome.

E

One thing I'd love to see, and maybe it's already part of the playbooks, um it's kind of um standard operating procedures for admin tasks like uh key rotation. um I'm not sure we probably don't have that yet, but something like uh snapshotting and backing the backup of cluster states.

E

um So I'm not sure is do do we have a playbook for key rotation, for example,.

B

um No, so the content there was mostly focused on just kind of like providing those basic building blocks um kind of the foundational work in order to allow us to build more things. On top of that, but collections do allow you to currently distribute roles and I believe in the future, it's planned to allow you to distribute playbooks as well, um and so the hope is that, as we've sort of filled, this out get more community involvement, get some. uh You know better. Subject matter experts I deal a lot.

B

You know working with operators with kubernetes api and low level components, and things like that. I have less experience with sort of higher level cluster administration, um and so that's definitely like we. We would love to have playbooks and roles in or that would enable users to like very easily automate um those tasks. But um you know that's sort of like now is the point where we go out to the community and and, like you know, look for people who who know about that.

B

Don't necessarily need to do all of it, but if they have, you know, requests for features like that um documentation. Maybe some like getting started places like that. Those would all be very useful things for us to see pop up in that repo in order to help us prioritize and also in order to help us understand what exactly those cluster administration tasks are and how we can help automate them.