Red Hat OpenShift OpenShift Commons | Red Hat Livestreaming, 10 Nov 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OCB: Index Image Build (IIB) Service for OpenShift

Description

Join us for an overview of the Index Image Build (IIB) Service, a REST API to manage operator index container images and some bundle images.

A

Hello, everyone welcome to another openshift commons and really excited. Today we have brian cook and luis carvalho, and I am karina angel. I'm an openshift product manager and brian is as well, and he runs the red hat container factory program he's the pm for that so and luis carvalho. He is the primary or one of the primary engineers, and he can correct me when he starts talking for iiv, so brian, please take it away. I see it's not index image builder.

B

Thanks karina, so, as she said, my name is brian cook, I'm a product manager on the openshift team and I work primarily with what we call the red hat container factory and the container.

C

B

Is really an internal program and our responsibility is for building all the container images and operators that we end up shipping out to end users. So we have lots of images that we build lots of operators that we build. We only ship a subset of them.

B

Our collective teams are responsible for all the build infrastructure and testing pipelines and it's really a very collaborative program.

B

One of the tools which we have developed recently is focused on sort of operationalizing the testing of the new operator index images at scale and the tool called iib.

B

Originally iib stood for index image builder, but then, when the scope expanded, I think the team saw an opportunity to add a snazzy recursive acronym to it. So now it stands for index image or now it just stands for iib image builder, so they get to mark off their recursive acronym merit badge for themselves.

B

um So iib is a python 3 based api running python flask. We.

C

B

It internally for for actually two pipelines, one for building our own operators and also we use it for our isv operator certification program. It makes it much easier for us to manage testing operators at scale. It also helps us manage our production changes to the index images for the catalogs that we actually ship for open ship for openshift. So when you open openshift, you see in the operator hub panel, the certified operators, red hat operators, community operators and sometimes the red hat marketplace operators.

B

So these um these catalogs are actually the changes to those catalogs are managed by ib as well. But in this talk, we're primarily going to focus on the development flow using iib because we figured there are less people trying to manage production indexes and more.

C

B

Trying to actually build operators to ship for the existing indexes.

B

Okay, next slide: blue.

B

So we have a number of issues trying to test many operators at scale here at red hat, and we try to contain the the sprawl that we have to maintain across all the different teams that are building and testing operators.

B

If you want to build an operator using operator index or operator bundle, images first thing you need. Is you need an index image to be able to install the operator on openshift, so you need an index image that has that bundle in it, so that you can install it on the cluster.

B

If you want to create that index image first, you need to know how to use opm.

B

You need to know which version of opm to use because it can be different, depending on which openshift version you're actually creating the index for, and you also need to know which version of opm you want to embed in the index image again that can change depending on what version of openshift you're targeting after you would build your text test index, you need to get it hosted somewhere, so that then you can attach it to your openshift cluster and, if you're doing multi-arch testing this gets.

B

um You know quite a bit more complicated, because now you need to test or you need to build n indexes, depending on how many architectures you're targeting you need to assemble them together into a manifest list, and you need to get that manifestless push somewhere where you can access it and you need to be able to basically do this many times, possibly very many times, depending on how many customers you have, how many different products you're shipping and how quick their iteration cycles are.

B

In addition to this, the operator, bundled images need to have shaws in them. They need to have digests inside them to really be accurate about what you're testing.

B

However, each time that you're trying to build a new bundle, image which could be like on every commit, for example, in a ci pipeline, updating all these shots becomes, it can become really tedious.

B

So we want a way to try to automate this and not have to have people trying to update shots continually for sure inside the operator metadata, and we want an easy way to automate everything. We want an easy way to automate all the jobs and to troubleshoot issues and most of all, we want our developers not focused on how all this works. We want our developers focused on building and shipping their products, and we want to try to contain all these issues into.

B

You know the smallest space possible and not try to fix it in 10 different places. Excellent.

B

So iiv for us, we build as a centralized service and it can create the index images needing for testing all of our operators in this cict driven environment. So we have two different entry points to iib. You can either submit new requests via the rest api or you can submit new requests via amqp messages.

B

The new indexes that it would build can have one or more operator bundles added to a new index digital output. So you could be testing a single bundle, a single operator in isolation or you could be testing an operator that has dependencies interdependencies on other operators, and you need to make sure that you know potentially n unreleased operators in testing are added to the index so that you can install all of them on a cluster and do some kind of like integration testing.

B

The iib service will output a new testing index, which is it takes an existing index in most cases, and it will add the one, the new bundles that you want in there. It will add them to that index and output it to a pre-configured registry location, and then the progress of this can be tracked by looking at the so. The user interface has like a we'll show some of this later, but you can.

B

You can check job statuses by polling if you've submitted via the rest, api you're, probably going to pull your job status, that you got a job id back for or if you submitted via amqp messages. You can just wait for a return message on the bus, with a status update for that job id and then, at the end of this, the location of the new index can be returned to the ci pipeline.

B

So the ci pipeline can continue um on with the process of like attaching this new index image to the openshift cluster and then launching testing jobs. The iib itself determines which version of opm is used and which version of opm will be embedded in the index. So this doesn't have to be worried about too much anymore.

B

We we make continuous updates to ifv now so that uh like, if we know which version of openshift's being targeted and essentially we will try to figure out which version of opm to use in the case where multi-arch testing is needed.

B

This is an interesting thing, so we have ib, has the ability to construct index images for amd64, for power and for assistance s390x, and then we will do that all on amd 64 systems, and um we can talk about the details of this a little bit later, like how we do this, but essentially it removes the need for you to actually have iab workers that have access to s390x power systems, so it makes it very nice it will create the multi-arch index for you.

B

It will assemble the manifest list, push the entire manifest list out to the pre-configured registry location. So you can launch functional tests on clusters of all the different architectures. I think in the future. We will also add arm support to this, as we roll out arm support for openshift.

B

um Going to slide five, so as you iterate on operator metadata, we talked about a little bit before managing the digest.

B

So there is another feature of ib called regeneration, and one of the things that can be done during regeneration is, if you have, for example, a container image specified and you have it set to a tag of one two 1.2.3 iib will resolve the tags in the digest and update the csv metadata so that now it's actually pointing at a shot.

B

So you could be pointing at like tags for a specific branch or floating tags. That's like iterating quickly in ci pipeline and each time it will update it to the sha so that you can tell exactly which thing was tested.

B

It will also populate the spec out related images block for you. So, if you're trying to support the offline use case- and we try to try to populate this for everything now, but effect.related images needs to point at the specific digest for all the containers which are used by this operator and iib will basically look through all of your metadata and find all references to container images that it can find and it will take all of them.

B

If there are tags it will resolve them to shaw's and it will build up the spec.related images block with all the references to containers from the rest of the csv metadata from scratch. There are a couple different options for how to make that work.

B

What I just described is sort of one workflow around it, but there are. There are a couple other ways you can also use that which are documented it'll it'll, make those changes to the csv file and then it'll rebuild the bundle image and push the bundle image the new bundle image out to its output registry.

B

So we call this regeneration.

B

You can batch this so that you would take like an original image that you want to, pin and have it regenerated and then, once it's regenerated, you can take the output image and then you can add it to an index and then have the new index be output that can go into your ci pipeline.

B

There are a few other things that can be done during regeneration and probably lots of opportunities for contribution if people have specific needs there, but uh one thing we do during regeneration sometimes is change registry host names.

B

This can be useful if you want to include like registry host names that are like development registries and at some point you want to switch them to pointing at production registries.

B

We also have the ability to inject arbitrary annotations into csv files during regeneration.

B

um Slide six.

B

B

This is a sort of just a high level overview of a process, a very simplified version of the process, so ib can parallelize all these operations, so this could be happening many times at once.

B

It'll act, item-potently on ad requests, so consider the case where you have like operator a and operator b and a depends on b and both of them are in a development cycle. You need to make sure you have both of them in a test index to run functional tests, and you need you know which versions you need for that test: you're going to submit a request, iib to add a and b of specific versions to a test index in a previous release of iib.

B

If you tried to add version a with a specific, like version, a of the version you're asking for was already in the index. That would cause a problem. It would say I can't add this, because it's already there so ib will now sort of um transparently just say: okay, that's already there and it will make sure that the other one is also added with the same one. So this reduces the number of disruptions that we've had in our pipeline. uh Quite a bit.

B

I already talked about the batch request, so we can batch request. This can be a number of like iterative ad requests or it can be regeneration, followed by batch requests. Just basically any any. You know serialized number of requests you want to send to iib, uh and then this shows sort of the workflow of sending a request. So I have my imaginary ci pipeline requesting a new index with the bundle foo added. That's gonna include a pull spec to the bundle image booth.

B

So it's going to be a specific version of food represented by a bundle image in a registry that goes to the iib service via the rest, interface or amqp.

B

The iab service will initiate the build and it will tell it will assign it to some worker. The worker will pull the index image from the registry. It doesn't have to be the same registry. Often our index images are our production index, so we would have a production index on register.redhat.io and we want to add a pre-release bundle image to the production index and then use that for testing. So we'd pull the production index from red hat dot io.

B

We would add the package foo to it, and then we would take the new index and push it out to like a temporary location. We use a private output organization on quad.io for all of our output from iib. So that's what that registry 2 represents once that output index image is out in registry 2.

B

The results of this job would be available from iib, so you would either get that back from the job status by polling or you would get notified on amtp message and you would be able to then know what the full spec of the output index image was. And then you could move on to attaching that catalog to the openshift cluster for testing.

B

Maybe often ask if there are any questions at this point.

A

I don't see any yet in the chat.

B

Okay, cool then, at this point, I'm going to hand it over to louie, who is one of our senior developers and also as primary designer of the iab service and he's going to go into some more examples in technical detail.

C

All right, thank you, brian, um so I just have a couple of examples uh here of some of the requests that brian's mentioning- um and this is you know, there's different ways to trigger this.

C

uh The simplest way for the purpose of the presentation is to show a curl command, um so in this case I'm making an ad request, which is adding the specified bundle to an existing index, and it's important to notice to note here that that index would not be modified, but a new index image will be created which contains whatever was included in the from index, plus, whatever bundle images were provided notice that bundles is a list, so you can provide multiple items here.

C

It could be different versions of the same operator or it could be completely different operators. They will be added in the order that up that is provided. So in some cases that's important.

C

Okay, I'll move on to the next slide, and this is a a response uh once you submit that curl command iab will track that request and as it progresses, you will populate with additional information. uh Eventually, the request completes and you get an output like this, uh and you can see there's a quite a few things here. um You know, starting from the top, it tells me which architectures uh the new index image was built for uh there's some batch information here, uh even though it wasn't, uh this request was not created in a batch.

C

um Ibu will generate a batch anyways. It will be a batch of one item, it's just some due to its internal implementation, but it will give you information about which binary image was used and the binary image is the image that contains the opm binary.

C

It's essentially the it contains the binary opm which will serve the index information, uh and it also tells you um which what was the digest of that binary image that was used, uh and it would tell give you a mapping of the uh the operator name to which bundles were associated with that operator, and it gives you a lot more information here too, such as the logs with all the commands that were executed to build this, and the important parameter here is the index image which is about halfway through the screen, and that tells me the pull spec of the new index image that was just built by ib, and one thing to notice.

C

Here too, is that there's a state history which I collapse for this screenshot, but it does give you, as the task is progressing iab, will add, update information to this. So, if you're, following along anxiously, you can see what ib is doing at a high level and see that it's proceeding as expected.

C

Another example here is for regenerating a bundle, so in this case, in the simplest form you just pass in a json which has a an existing bundle. Image and ibu will process that bundle image and create a new bundle image with the different modifications that brian was mentioning earlier, including pinning images that are mentioned in the csv from a tag to a digest and also populate the spec related images section.

C

So, based on this request, uh you have an output like this, uh which is very similar to the previous output. I showed, um but this has information, that's relevant to uh the regenerate bundle request.

C

An important value here is the bundle image key, which tells me what's the pulse back of the new bundle image that iab regenerated and, as always, iab will always record uh the digest of any of the images that it has used in the process.

C

So in this case, I made a request with an existing bundle image by digest, though, from bundle image and from bundle image resolved have the same value, but I could make this request with a using a tag, and in that case the resolved version will have the digest and the known result. Version will have the tag, and this is pretty useful for auditing purposes uh and as the other requests there's a logs that are accessible to uh users, as well as a state history.

C

And for batching operations in this case there's different types of batch operations. You can do a a batch operation with regenerate bundle, requests or you can do batch operations with that add, remove requests. You can't mix them the batches. Don't allow you to do that, but in this case it's an example of.

C

I have a two bundle images that I want to regenerate, so I can make a single request to ib and if you will split that up into two separate requests, but you will combine them into a single batch, so the batch id will be the same for both of those requests which is uh useful for tracking.

C

It also allows you to provide some an arbitrary set of annotations to be applied to the request, and this is useful for tracking tracing the progress of um the requests across different systems.

C

um For example, uh whoever is making this request to uh iab could be an external system that has its own id tracking, so the annotations become very useful. In that case, once iab finishes practicing a batch, it provides information via message, queue notifications, but you can just listen. uh You know subscribe to those uh topics and be notified when, whenever the batch that you requested has completed- and the annotations will make its way to the uh to this message, so it does give you, um hopefully all the contacts that you need to react on those.

C

Messages so I'll I'll dig into uh the architecture of iab a little bit. um I think in this case um in case you you're interested in contributing to ib or, if you're um want to spin up your own instance of iab.

C

This might be useful to you, so here's what the one of our deployments looks like, so you have different components of iab and if we center, if we focus in the middle of this picture, there's the api front end the broker and a set of workers, and currently all those are running in a single vm.

C

So far, we haven't hit any resource limitations, but it does provide. You know single point failure issue which we are working on addressing, but outside of this vm outside of this core components, there's a three other components. One of them is a database.

C

Iab relies on database for tracking data, all the different requests being made to it, and we also have an external message: queue which is used for notifications of status, changes of requests and, of course, the container registry that is used by ib, which is used for storing the output of iab requests.

C

On the technology side of things, the api front end, as brian says, it's just a flask application. We use apache to serve that up and it's also iab itself. It's a salary application, which is a python framework for performing asynchronous tasks.

C

So in this case the api front end is the producer for that and in the broker. It's it's used for the communication from the front end to the workers we use rabbitmq and in this case it's the salary backend for the workers. They are salary, consumers, nothing special about them.

C

They essentially have specialized code to run all kinds of container tooling, like scope, you build a odd man and whatnot. The database will use postgresqlql.

C

We haven't tried using mysql or mariadb, but there's nothing specific about postgres data we use, so you could work uh with. Maybe a little bit of work.

C

uh Container registry uh is any container registry that supports the uh container registry api. So we use create.io. You could use an openshift registry whatever is needed, although the openshift registry doesn't support manifest list, which uh it would be. You wouldn't be allowed in this case, but any registry that meets the requirements uh should be able to be used here and for the external message queue we use activemq, and this is mostly used internally, because we do have an existing active mq instance, which is heavily used for as a pipeline integration point.

C

That's why there's the you know it's using both active mq and rabbit mq.

C

And if you're interested in contributing uh there's a development environment that you can, then anyone can create. Iab, of course, is open source, and it's upstream we use docker compose and podmin compose to create a complete, completely useful iab deployment from scratch.

C

It will deploy all the components of iab, not just the core components, but you will actually create a database for you it's on registry and whatnot, so you should be able to there's a make file that helps. You run this command and I believe the command is make run and if you just run that- and you have you know- docker pod man and the compose tool installed, it should just work for the production environment.

C

Of course, we have a more robust deployment, there's various configurations that you can specify for the api front end as well as the workers. I won't cover them all here I put links in here. I don't know if the slide deck will be available for viewers later. But if you just go on github and search for iap, you should be able to find the project and it's right there. In the with me file.

C

um You can for a production environment. You can have a dedicated database. You can share a database if you already have something like that. uh The same for the register in the broker services um authentication um for the development environment, we don't have authentication enabled for obvious reasons for the production environment. You probably want to have some sort of authentication if or anything to at least track who is using your service internally. We support uh kerberos authentication, uh but adding support for certificates and any other you know, basic off.

C

Anything supported by apache should be uh relatively easy. Shouldn't require any changes to iov itself. It's more a deployment configuration at that point and you can one of the things that you can do is you can define in the configuration for ib which users can map to a different queue. So this allows you to you know, grant some users access to dedicated resources, and this is pretty useful to distinguish the different workloads. You may have uh accessing that same iab instance.

C

For example, in some cases we have requests that can run in parallel, but in some when we're managing the production uh index image, we need to do those sequentially to make sure that things don't get dropped on the way, and in that case we have a a dedicated. You know serial um queue where only a single request is done at a time. But in most cases you don't need that, but we have, I believe, six or seven parallel cues, so um iab can just handle that many requests at a time.

C

um We are also working, as I mentioned before, on re-architecturing um the deployment of iab a little bit, so we can uh add more resources to it, as well as reduce any sort of single points of failure.

C

um We don't believe that this will implies any code changes to iab- uh maybe some small tweaks, but this will be mostly deployment, related changes uh and one of the limitations, and one of the reasons why ib exists in the first place is that it's we have not been successful in running the opm commands to manage an index image inside openshift, um there's a requirement for using privileged containers in some cases.

C

um So there's a various teams have tried doing this and no one has been truly successful, which is why today ibm sorry iab uses virtual machines for running the container tooling and, as brian mentioned earlier, there's this question about multi-arch images, um index images and bundle images well, not necessarily bundle images but index images. They need to be multi-arch, so they can work on the different architectures that openshift supports, so iab has to be able to build.

C

Those and iob can do this on a single architecture which is uh amd 64., um and you can build different targets based on some of the builders parameters that we use so before I dive into that. I just want to recap a little bit about what binary images are. They are essentially from an ib perspective. A binary image is a container image, a traditional container image that contains the opm binary that is going to be embedded in the index catalog, and this is created outside of ib.

C

This particular step does require having arch specific hosts, to build this binary content, but once we have this, we can create index images from those and an index. Image is also, as I mentioned, is also a multi-large container image and it's essentially a combination of the opm binary image, plus the catalog data, the index catalog data, which in itself is just a sqlite file.

C

um It's not arch specific, so the way we we do an ib is we run the opm commands to generate a modified version of the database file, and once we do that, we iterate through all the arches that we want to build for, and we use build a override arch parameter to build to essentially change the specify, the actual architecture that we want to build the image for, and this works fine, because to build an index image, you don't actually need to run any of those uh run instructions that you'd see in a docker file.

C

It's not actually executing anything, it's just metadata. It's either metadata instructions or file system file, system operations and those are not art specific. So that's why ib is able to get away with this and then once we build a you know, art specific images. We just combine them all into a manifest list and that's the output that ib provides- and I just have to make this a little bit clearer. I just have a snippet of the logs from a request that I ran earlier today, um so you can see here.

C

The first thing that it does is run in the opm index. Add command, and this essentially the dash generate flag essentially tells opm to don't build anything just generate.

C

You know a docker file and a modified version of that database, and then we take the output of that and we run the builder bud command with the override arch for each architecture that we want and then at the end you can see that we add each one of those images to a manifest list and you can see it's the same manifest list here, the same tag and we make we push that to the registry and that's it, and one thing to call out here too, is that iab does add some specific labels to the um to the index image.

C

The first one is the delivery version, and this is used for us to help track. Make decisions of which open shift version this index image is being targeted, for uh there are some differences, just some logic in ib to treat things differently depending on the version. uh Again, it's all uh configurable as well, and the distribution scope is, as the name implies, is what's the target environment for this index image.

C

If it's in this case stage, it means that this index image shouldn't be released. It's meant for to stay internally for testing purposes.

C

I think I think that's it for me. I don't know if there are any questions in the chat.

A

We do have some questions and thank you both um we'll start with with opm. You can select to use a different container like padma. Is it possible to do that in iib.

C

Iab will force the same container tool to be used.

C

I believe, right now, it's using the uh there's podman there's a docker, but there's also the no container tool, which means that uh opm has its own um embedded logic on how to deal with um these images, and I believe, that's the one we're using today, um but there's no from a user perspective you can't select, which one to use this will be uh based on controls on the server side.

A

And do you have a final slide that shows your repository while we're.

B

Yes, there's more there's a couple slides after this actually.

A

Okay, um can you forward just so there's nice, slides.

B

Yeah skip forward get forward to slide.

B

A

C

B

Have yeah I can and then there's one more question in chat I want to get to, but um so there are a couple things on the on the roadmap. Louis already talked about the architecture, changes to just make sure we have no remaining single points of failures.

B

So that's in the works now. Another thing we're also working on is a way to visualize the update graphs that are constructed in the temporary indexes, so imagine you're, building a temporary index with a bundle image that has a bunch of logic about what it's skipping and replacing in it.

B

This can get pretty complicated and what we want to do is produce a method that allows someone to see a visual representation of the update graph. With respect to the current version of the bundle image that was just added, so you would have a much easier, visual way of verifying that all the all the yaml stuff that you threw into your csv actually resulted in the graph you intended.

B

So this is going to we're working through a lot of design on this right now. This will ultimately end up as like a some kind of web interface on ib, where you can select an iib job that was run and view the graphs or the packages that were added to the index, using like a pattern, fly interface.

A

I know a lot of people are going to be excited about that and I'm sure they're also wondering- and so I'm going to put words in their mouth, um how to give feedback on the design as you're doing it right now is that do you have external calls that they can dial into or is there a way.

B

We do not currently, but this uh the visualization has actually been a collaborative design effort between interaction designers in openshift engineering and in the container factory program. So we could bring preliminary designs for the visualization to another commons briefing or like an upstream operator, sdk meeting.

A

Awesome that sounds fantastic, um we'll schedule that okay.

B

The next slide should have the resources you're looking for so there's the source code, the api docs and some of the messaging integration there. So.

A

Thank you and then.

B

There's one more question and chat. I that says what modes does iib support from opm uh like replaces somber or summer skip patch.

C

Yeah he uses replaces today there's some ongoing conversations um to potentially add stamford and samversky patch mode as well.

B

I think if there are people interested in.

C

B

That functionality, it would be great to talk with them about what their needs are, so that we could potentially collaborate on.

A

That so, would that also be in a operator sdk call, perhaps or.

B

um That's a that's an okay entry point if it gets to the point where you know I would like. I would love to see this get to the point where we need to have an upstream iib meeting, um and maybe that's something that we can look at starting to do um we're looking for I'm.

B

I am looking for probably some renaming and in order to prepare sort of launching this as a more formal upstream project and potentially moving it out of the red hat release, engineering, repo and just keeping our current um like branch that we're running there or something.

B

uh One more question now.

B

Assume the tag pinning to digest is oh, assuming the tag painting is optional, I've already replied provided the request. Yeah there's a there's, a um annotation that you can set in the operator to tell ib not to try to pin the tags.

B

And see one more so ib call stuff replaces what I might do today via opm and dr push. Yes, it does do that that it'll do it.

B

It it also removes the need for you to care about some of the things around the opm thing, like what version of opm to even use, for example, for a specific version of openshift like it is somewhat non-obvious that that what might happen if you use certain opm versions against certain versions of openshift.

C

Yeah one thing too to add to it is that um right now, iob only supports the index, add and remove operations of opm. The regenerate bundle doesn't use opm at all, and other opm operations are not yet added to iab, but we're open to add whatever is needed.

A

So is that a request somebody can make in the repo to add additional information around that? What what do you suggest.

C

You say that again.

B

Yeah I didn't get the question.

A

Or it sounds like um clarification would be good for people that are using iib. So if they have questions around clarification, is that um something that they can ask via the repo.

C

Yeah yeah, finally, an issue uh in the github repo is probably the best way to gather arthur.

A

B

John and you can figure what version of opm are used in build in upstream versus downstream.

C

The opm version is per deployment, so, if you want today, if you want to use a different opm version, you have to have a separate deployment.

C

But if that becomes a pain point, I'm sure we can come up with something more flexible.

B

But that's not. I want to point out that's not the same as the opm version that gets embedded in the index image.

C

B

Which, which has some logic inside iib based on that openshift target, that was uh in the label to determine which version of ib gets embedded or you can override it, and you can point it at a specific version. If you want to use some specific version, embedding you can tell it what version to use.

C

Yeah there's essentially two versions of opm at play here: there's the uh version of opm that iab directly calls to make changes to the index database and there's also the opm version that actually gets embedded in the output index image so that second, one is completely there's some defaults, of course, but you can completely override that with uh something from whatever registry uh you want.

C

The my previous comment was regarding the opm version that iab itself is using.

A

And is that documented anywhere.

A

C

I don't think it's explicitly called out like that.

B

The the api docs do show how to override the version that's embedded in the uh the output index. It's just a api parameter.

A

And for everybody watching the api docs are pretty thorough. um You go look at them.

A

I want to call that out, as I was muddling through iiv myself, all right. Are there any other questions, so yeah kudos. Of course thank you for presenting on that. I know I know brian gets a lot of questions about iiv and opm, and it's good to have you here also um to answer these questions so last call for any other questions.

B

And I think it's great to have louie here, because I can't answer them all.

A

um So a quick, quick plug another thing- iib is really good for, is pulling, of course, pulling all of your operators into a single index, which is the point right that can also be used to deploy a custom catalog.

A

So I want to call out that during day, zero of kubecon there's going to be a session on creating custom catalogs and an example, and that will be myself.

A

Brian gave a lot of input on that and then um andre goes from ibm, giving an example of how flat packs use that so again day, zero openshift commons um of kubecon.

A

Thank you again, everybody for joining us. The recording will go out with on the openshift blog as well and youtube channel as well as twitch, and thank you. Everybody for joining.

B

A

All right, thanks again.

A

C

C