Red Hat OpenShift GitOps Guide to the Galaxy | Red Hat Livestreaming, 12 Aug 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitOps Guide to the Galaxy (Ep 20): Multi-cluster management w/ ACM

Description

What happens when you deploy applications across multiple clouds? Wondering how to manage Kubernetes with GitOps with multiple clusters? In today's show, Josh Packer from the Red Hat Advanced Cluster Management team drops by to show off multi-cluster management with ACM. So join us to learn all about how we can step up GitOps with ACM and why ACM is the perfect tool for massive GitOps scale.

Learn more: https://cloud.redhat.com/blog/underst...

Follow us:
Christian Hernandez https://twitter.com/christianh814
Chris Short https://twitter.com/ChrisShort
Josh Packer https://ca.linkedin.com/in/joshua-packer

What is GitOps Guide to the Galaxy?
Every other Thursday at 3pm ET hosts Christian Hernandez and Chris Short sit down to discuss everything in the GitOps universe, from end-to-end CICD pipelines to creating Git workflows.

Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_c...

About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
Openshift.com https://red.ht/33Fh6uV
OpenShift on github https://github.com/openshift/
OpenShift on redhat.com https://red.ht/2o4ccDk

#RedHat #GitOps #Kubernetes

A

A

Good morning good afternoon good evening and welcome to another episode of github's guide to the galaxy, I am chris short host of red hat live streaming and showrunner. I am joined by joshua packer and christian hernandez. The get ops guy and we're gonna be talking about multi-cluster stuff. Today, christian.

B

Yeah yeah, I'm actually really excited about this. um I've been meaning to get um the acm team on um on the stream, and actually this happened completely by accident josh and I are wearing the same shirt um or.

A

I mean we planned that.

B

We're so tightly knit here. No, I'm I'm really excited um to get uh to get joshua um from the acm team out here josh. Can you um give kind of a brief introduction of you know what I mean? Who you are, what you do at red hat and you know what what your role is with the acm team.

C

Absolutely so acm is advanced cluster management or open cluster management. If you're looking in the community for us, we are all about cluster management at fleet scale and under that purview. I work on own, develop co, cohesively, connect and uh and lead the teams that do both our cluster provisioning and life cycling.

C

So everything from creating clusters to destroying clusters to all the pieces that go in between and then the application life cycle, which is where the get ops comes in and that allows me not only to sort of work with applications and work with you know what goes out into your fleet, but get into that space of you know.

C

How do we define what the infrastructure looks like this we'll talk about it many times the infrastructure as code, the single source of truth, and so you know being attached to these different pillars within the the product sort of allows a lot of this. What we'll see today, these sort of cross referencing, where we can use the single source of true skid ops for managing everything in your fleet? That's awesome! Yes,.

B

Yeah, it's what what I like to call um josh's what I like to call um the person I ask anytime. I have questions about acm, I'm.

C

The go-to guy yeah.

A

The go-to guy is.

B

The one I ask questions to is is what I what I see it here.

A

um And you might have seen scott on another show called advanced cluster management presents on the channel, so I will drop that playlist in chat. If you're curious about the topic.

B

That's right right, so if you catch definitely catch that stream. If you want to know all things acm, um I I invited josh really um to um to the stream, because you know we here on the show. We talk a lot about get ops right. We talk a lot about application, delivery and get ops as a practice, and um you know a lot of the time. So you know quick, quick story.

B

um This is an old story, because this is back when we can travel, um and this is back like around 2000 uh 2018, where um I actually did a red hat user group uh up in uh because in minnesota- um and you know I was talking about argo cd and the the you know. One of the questions is um that I got well. You know that I used to get a lot. Was it's like? Okay? You know like this. This is great like you know, but but what about like my day, zero stuff like?

B

How does how do I provision my clusters right in in a declarative way? And you know I I think um you know get ops as a practice right.

B

We all love, get ops, get ups as a practice, but I think acm brings a very interesting aspect of you know going beyond just like get ops and having multi-cluster management right fleet management and how to manage that with get ops right, because getups as a practice is great, um but you know now now we're talking going up a level right going up into like okay, like you know, I'm doing my getups practices. I have you know thousands of clusters, you know you know spread across. You know multiple environments. How do I manage that?

B

And um you know I think uh you know having a tool like acm having an offering like you know um something that manages the cluster with your get ops practices is a very, very powerful thing. So now you have an end to end right now you have an end to end of a uh of you know, I provision my cluster.

B

You know I manage it with get ops and I then I control the life cycle that that cluster I can destroy it. I can, uh you know, upgrade it. I can apply policies to it, so I don't want to see too much thunder, but I think it's very important to talk about. um You know when we're thinking about get ups, we're not only thinking about like the application delivery, but the whole infrastructure platform, your whole end-to-end solution as as a whole, so um so yeah.

B

So you know I I'd love to get your thoughts on uh josh on, like the whole end-to-end aspect of of get ops and and multi-cluster management.

C

Absolutely well so I'm going to touch back joe. Just on one thing you mentioned, and it was a question we get all the time and I'm sure it's a question. Some of the viewers on this have as well is red hat and we're wearing the shirts that say: openshift, openshift, git ops is red, hat's uh productized version of the argo piece, and so the question in advanced cluster management we always get is advanced. Cluster management had a way of provisioning, apps and doing github and helm. You know which do I use. Do I use argo?

C

Do I use the acm subscriptions? You know, how do I make the choice, and you know the answer used to be well dependent. What you were doing, that answer is still the same. It depends on what you're doing, but in both cases we want you to use both. We want you to use them together, and what we're going to show you today is how acm and argo are like this they're fully integrated the acm brings the fleet management that was sort of lacking from the argospace.

C

Argo is very much the you know contained for the the cluster that it's working on or if you've gone out, and you have a bunch of clusters, you still have to bring those in acm. We're gonna see, brings all those pieces together and gives you that that visibility into argo that one number one we never had before, but number two allows you to operate. Argo and through openshift, get offs at scale across the fleet. So without further ado, maybe I'll grab a screen share and we'll we'll get rolling.

C

So hopefully it pops up. What we're looking at here is the acm. This is the 2.3 release. It literally went out the door like three.

B

A

Ago I think it was fresh, fresh.

B

Yeah yeah exactly new.

C

New here right out, the pipeline, this is exactly as new as it gets. So what we're going to do.

A

Is we drop the vine.

C

Absolutely new, no, um and so you know, as christian mentioned, you know doing everything from day zero on through day infinity with your cluster. um I, if those of you familiar with acm, I just navigated in from the welcome screen down to applications, and so what we see here, you see some type, some labels that say argo. We see some that don't have it, I'm not going to get into that just yet. But what I want to target on here is there's this application.

C

That's deployed in acm, called infrastructure build out, and so the back story in this is that I got an openshift cluster. I ran an open shift, install to be honest internally. I just requested one and it came up, but you deploy an openshift cluster. You go to hub, so you go into openshift's operator hub. You say I want acm, you hit the install button in seven minutes. You have an acm cluster at this point, I'm at this stage I have no applications. I have no other clusters.

C

I want to create my first application in acm, and so I go through. I click the create and what I do is I choose get, and I pick my infrastructure git repo and in this case it's labeled get ops fleet samples.

C

So I pick my repo that has all the definitions for my security for my configuration for the clusters. I want to provision for the applications that need to go out to it. So all that with just this single infrastructure build out and let's take a look quickly and get here, we'll get it to pop out.

C

I was able to build out a bunch of policies provision. Some clusters apply policy to those clusters. In this case we actually go out and we deploy openshift get offs, which is argo on all of those clusters as well. So what does that look like? So we see there's a couple of different directories, the one we're going to be interested in seed and we were having a little taking a little liberty with the naming.

C

So this is the directory that seeds all of the infrastructure pieces for your environment, but here it starts to get pretty simple. Anything under common is something is configuration that is going to be common to all of my clusters. So one example is the acm policies, so things like I wanted openshift get ops or orgo on all of my clusters, so I have a policy that's defined in git. That does that I wanted to enable etcd encryption across all of my openshift clusters.

C

So again I have a policy and I'll click in one, it's a bunch of yaml, but again, if you've seen any of the acm stuff or you can go back and watch the twitch channels on it. When I go into the ui and say generate a policy, I choose all the settings I want in the ui. I click that yaml button and I've got that yaml that I can just paste directly into git and then that becomes part of my source of truth.

C

Part of my configuration that it gets applied by acm to my entire fleet. But so again, you've got a split up of these seeds. We've got common configuration and policies.

C

We've got those specific to our acm hub, so configuration apps, special cron jobs again, because this is a bit of a demonstration cluster to show a little bit of everything. We've got specialized, cron jobs that in the evening put the manage clusters, so portions of my fleet to sleep so that I'm not paying aws fee. Gcp fees.

A

As we're fees shut down those dev environments when you're not excellent, exactly.

C

When we have a lot, we were just talking about a long weekend this weekend. It's the same ideas. We are able to turn those things off and you know if it doesn't happen from the get ops it can always be overridden from the ui and so again in 2.4.

C

We finally bring- and I'm diverging here from git ops for just a second, I promise, but I selected a cluster and you see we have the hibernate and resume now as part of the user interface that you have access to to uh to control power states on your provision clusters uh again, but so back to the the original discussion is I created a single subscription. It brought in all the configurations and it brought in all of these other applications, so it deployed acm.

C

We use this term called subscription is our version of an application in argo that is able to go to git and pull in the hello world app um here again it did a mortgage app and we pushed it to one core one system, but it also again. I remember I mentioned it deployed git, ops everywhere, so I have our git ops. Openshift get ops, which is argo, so all of my managed clusters have a copy of argo on it. So if I click into this so number one acm knows: hey, there's an argo app.

C

So not only does it see the acm mortgage app, the hello world app, but it finds apps that are deployed but deployed by argo. Argo introduced with from the red hat team has this concept of an application set. All an application set is: is grouping a single application to many clusters, similar concept that acm has with our application, which is why we're able to display it this way. So we click into it, and so we see we've got this standard topology. We have some warning symbols and we'll talk about that in a second.

C

But so this is an argo app. It's deployed by argo. It's deployed with an application set from argo. So if we click on this it'll just take a second to refresh hopefully, and we get the application set we can see. I know it might be a little small. Let me oh it's not zooming in, but you can see the argo project reference there. So again, acm building that topology, but this is strictly, for all intents and purposes, and application coming out of argo.

C

Now I just want to show for comparison purposes, I'm going to click on the mortgage app and again we can see exact same topology. Everything looks the same, except that little reference link I had for application set, that's missing now: let's take it a step further, so we go back we're going to click into the engine example again we have the application. So we can take a look here again. We see all of the different clusters where it's deployed and we also see why there's a warning.

C

There is a problem with the deployment on a single cluster. But if I'm going to click this you can see it says, launch the argo editor it's going to push me out, and this is the actual application definition in argo that is running running on that same openshift hub. So all that data that argo has about the applications it's brought in to me about the deployments. So let's hop back for one second. So again, that's actually.

B

Pretty cool for like so you can imagine um an administrator, um you know administrating, you know, you know a fleet of clusters and the information of what the the developers are doing right: their ci cd process, the the the deployment process right in argo, cd bubbles up to the admin, so you can actually see kind of like like a like a single pane of glass. um So you don't so as an admin, you don't lose that information right as your developers are, are doing things with argo cd. All that stuff is coming up. That's right!.

C

And exactly, and- and this is the piece we we always push about, acm- is that this topology looks the same, whether it's the seven clusters here or a hundred clusters or a thousand clusters, but you'll see where you're having a problem, and so let's click it. We click on it. It takes us to the argo, and this is the one that is having the problem.

C

But let's the problem exactly and actually the problem is, is I'm trying to deploy it on my hub cluster in a name space that I don't have permission to, but I needed. I wanted an example to be able to show how we can dig down into it, but let me just quickly show a comparison, so this is that same. Oh, I pushed enter too quick. This is that same list, and so it's it's readable at seven. So you can see there is the problem, but then let's say we get to 10. The page is full.

C

Let's say we get to 100 you're now scrolling through now yeah you can start to play with synced and unsynced. But again, the point is: is that acm brings that fleet view when you're talking about hundreds of clusters, even 50 clusters, even 20 clusters?

C

You know you're not scanning through that pages and, as christian said, it's that that much easier to just look at it and see where the problem now. This was an entire deployment problem where it didn't deploy any of the pieces to the cluster.

C

But imagine it was just a route you'll see just that one little icon you'll know it's a route problem, you'll click on it and then list on the right you're going to be able to tell exactly which cluster it has the problem on and then click that and you get the argo editor same thing as acm. The last piece, though I want to show here is: we've also demonstrated this for acm before is an acm app.

C

You can click on this without leaving acm at all to see the yaml and the logs that go with it. That still applies to argo applications as well. So this is this we're still in it opened a new tab. That's why the screen flashed but we're.

C

This is the yaml definition for that deployment there, and so you can scroll down and take a look at the status etc and see if there's a problem as well as you can see, the real live logs from the system as well, and that's without leaving the pane of glass of acm. You didn't have to jump down into argo. You get that information immediately and then, if you need to do something deeper, then you click on that application. Icon.

C

You choose, take me to argo and boom you're in there, and so this is that scenario where you've got argo running on the hub and it's deploying applications to all of your well two sets of your manage cluster, and so the question is well. How do we make that easier, and one of the ways is, is if you've ever done it before you need to run run a command on the argo cli to import clusters into argo acm in 2.4. Does that for you automatically in 2.3 we had us, we had a slightly different approach.

C

We would import them as well, but you were limited to if you had only a single. If you had more than one argo running, it would only work with one argo on your hub, but so now we use- and we've talked about this on some of the other streams. But we have this placement concept, which is really I'll. Keep it simple. It's just label matching is. If your cluster has the right label, then acm will import that cluster into a given argo, and so you can have two argos.

C

For example, you can have one for configuration and that could get all clusters you create. Doesn't like all openshift clusters, you create doesn't matter whether it was for production for test whatever it goes to. You know that one gets imported to the argo that is for configuration, but you could have two other argos, one for development and one for production and in those cases only the ones with the usage label production, for example, go to the production, argo and all that happens seamlessly when you provision you don't have to.

C

You have to do anything special, there's, a resource we introduced um here. Let me pull it up quickly, there's a resource we introduced that allows you to specify that. So, if I go back here to the hub under configuration, we see this get ops cluster kind. This is just that it uh you specify the argo name space where the argo lives and you specify again. I said this placement rule which, if we scroll down here, is just the label match so any cluster. I provision that have the label usage and it's either production or development.

C

Argo gets installed on so again that import step that used to be the manual piece. That's all automatic if you have a bunch of open shift clusters and you have argo and you want them imported into a new argo, because you just deployed acm and you're, making a new hub as soon as you import those in if, with this resource, created those get imported into argo and can now be targets for argo.

C

So again, any cluster you provision you import into acm can be automatically, can automatically become a target for argo and that's a big game changer from a fleet perspective, because that was a manual process up until this point, with a bunch of cli commands, you have to find the coupe, configs, etc. Now that's automatic.

B

That's really cool. That's really.

C

B

Cool now it's like one button. um You know push button, get ops right. It's just like one button, there's essentially one button and it's it's all there for you. So that's really really cool exactly.

C

Yeah, no, it's it's this it the argo integration, it's it's gives us a lot of different possibilities and it seems to be at least for the time being resonating with a lot of folks as well, which is, which is what we always want to see. So now, maybe we'll dive a little more into argo integrations because it didn't it doesn't just stop with the basic application sets. We wanted to bring these things.

C

There's two more points, sort of major points I want to make, and then we'll do some cool provisioning et cetera, doing get up through get offs, but one is application sets we were bringing acm together with application with argo application sets fits the mold of sort of the what you create one application set to many cluster concepts, and so but again it was limited to clusters and having a knowledge of clusters that are imported into the argo, argo server, and so what we did was we worked with the argo community and we integrated our acm placement rules into application sets as well in argo.

C

So that's an example here. So this is a.

B

Which is really cool by the way the um you're talking about the the generator right, the um absolutely yeah yeah. So I I don't want to see your thunder there, but that was that's.

A

C

Cool I saw, I saw.

B

Well because you know like I, I was a really big fan right of of application sets and you know, as as it's evolving I saw a song git I saw on github. I saw like a bunch of commits from joshua. I was like. I wonder why I wonder what josh is doing in you know doing all these commits application sets. So this is actually really cool of how the integration between argo, cd and acm is happening through these generators. So um I'm.

C

B

You're gonna you're gonna go into that um as well, so.

C

Absolutely well, that was what I yeah just pulled up, so we talked about placement rules again. Placement rules. Is this automatic always reconciling way of defining a set or a group of clusters that you want to target for an application in acm for a policy, but now for an application set in argo as well? And so we got that in. I want to say a couple months ago, so it'll be in the next release of ocm um or sorry.

C

Openshift get ops, which is the brand naming for argo, but it's already available in the 0.2 community release of application sets. But what this allows you to do is we introduced this cluster decision resource so before they had they had their own labels. So if you had specialized labels on there on an argo secret, you could match it, but again that doesn't fit well with or too well. It doesn't fit that well, with the I've got a bunch of managed clusters.

C

I want to target, and I'm already using placement in policies and other spaces, and so we introduced- and it's called cluster decision resource and not placement, because we were trying to you know not just be acm or ocm pushing our way in somewhere. This is actually a generalized approach, and so you see it has this concept. This idea of a config map reference. You define a config map that kind of says what resource should I read for a list of servers? What parameters within that resource?

C

Should I read for that list of servers and then it takes the name of that actual resource, and so what that allowed us to do is number one. It may meant it could work with our placement rule, but it also meant that you know the folks at flux.

C

The folks at any other integrator that builds a fleet management solution would actually be able to take advantage of this change in application, sets and use their version of placement, and so it was kind of our well our homage to you know trying to help the community along a little bit while also building an integration point for us, and so what we do with this to ease the ease the work along is you don't have to do any additional work when you're using argo with acm acm builds these config maps there's actually two, because we have two types of placement rules they build them automatically.

C

So literally, all you have to do is create an application set resource. Tell it whether you want to use an acm placement rule or the new acm placement there's. So we have two types: give it the name and you don't need the req is sort of how often it checks it's technically optional, but so just two parameters and the placement rule created and boom you're deploying to your clusters, those same clusters that were imported into argo using our other resource.

C

You can even share the same placement rule between the two objects, and so for us this was big and we're not stopping there, and I don't have it. I don't have it to show you today, but I will plug a future. Is that soon enough? We're going to have a application set, create ui within the acm space because we're so behind it as a capability, and there is no application set ui anywhere else at this point either.

C

So we're hoping to be the first we'll see it's coming in the it's coming up in the october time frame. So we'll probably be back to show that as well.

B

Yeah there you go yeah.

A

Yeah make our show planning easier. Thank you.

C

And so, but so yeah, so the point being, though, is you can see it's a deep integration? We really anything we do with an acm subscription, which was the native capability we brought and that we have in our community. We also wanted to extend, because argo is now such a part of the red hat strategy.

C

We wanted to make a part of red hat and be able to do whatever we do in acm be able to do through acm in a fleet with argo, which will lead me to the last piece, at least around the argo pi that I'm going to talk about, and that is, we've talked about acm in the past, our application strategy and I'm going to shrink down here a little bit. It was always a pull strategy in that you define a subscription on the hub.

C

That subscription is propagated to your managed clusters based on the placement and then each of those subscriptions on those manage clusters reached out to git and pulled it down the idea being the hub goes down. The subscription is able to keep processing. You can replicate that with argo and and some of the recommended configurations are actually in that format and to replicate that what that means is you put argo on each of those managed clusters, but then the question becomes number one.

C

How do I get my application definitions for argo to those clusters to do that, pull mentality as well as how do I see that, because argo doesn't have a growl-up across argos today and that's where acm again comes to we'll say it comes to the rescue for today, but yeah tries to tie that tries that.

B

All together, it's the peanut butter to the chocolate right. It's the things it goes. It goes together. It's the peanut butter to the chocolate. It's the it.

C

Makes the nutella in the no.

B

I guess that's right: it's the nutella to the two pieces of bread right.

C

Exactly- and so that's this last one- and so here again, because I'm acm and we're working with argo- and this is a demonstration platform as part of this infrastructure build out it created an acm subscription which is again the subscription is able to go to git and bring things down. It creates an acm subscription using placement, so that again means to all my managed clusters.

C

That brings an argo app, so an argo application definition is being propagated to all my managed clusters using an acm subscription, but that could have easily just been an argo on the hub, pushing the an argo application object to all those managed clusters. But what happens is when that argo application object is created on those managed clusters, the local argo that again at the very beginning, you guys saw me at a policy that said put argo everywhere. So I've got argo on every single one of my openshift clusters that argo on that remote cluster says.

C

Oh there's an application object for argo. I am going to deploy it, and so, in this case, it's again another ng nx app, and this is the special, the special piece, the the piste resistance. Someone would say is that, because acm has these managed clusters and we've got this search collector, that's out there grabbing data, it's able to detect that there is an argo application deployed on a remote argo. So there's no other in there's.

C

No other information that there's an argo remotely there, except for that fact that a policy put it there, but our search is able to detect, there's an argo application that argo application deployed and hey. Look that same argo application. It's actually deployed across all six of these clusters, and so it builds a group and puts even though they're individual argo applications.

C

It builds a single topology because it knows it's from the same source. It's from the same, get repo it's from the same branch in that git, repo or tag, and it's from the same folder in that git repo in its definition. So, even though these are six distinct, argo applications on remote clusters, acm is able to detect that information and bring it back and display it, just as we saw with the other one. So we can see all of the different clusters it's on. We see the deployment under the replica set.

C

We get the access again to look at the yaml and the logs, and so this is also when argo. We talk about argo everywhere or that's the way I like to call it. When we have argo everywhere, you may have developers that are using it. So as what you can take away from this is as they're off they're creating apps they're, not under the covers it's not going under any radar. Acm is going to be displaying that for your operations for your sre folks in its application list, so you'll be able to tell that hey.

C

You know christian's over there, and you know he built the miss pac-man app instead of the pac-man app that I showed earlier on the remote cluster and he thought well, I was on the dev cluster and nobody was going to notice kind of acm.

A

C

Going to bring those details back so all again this application, although it's being realized and shown, as christian said, on a single pane of glass in the acm, it's actually only deployed in argos on remote systems. It is not in any way on the hub instance itself and that one is really the kind of cool piece. So it covers all of our possible scenarios. It's our we've got. Acm works with argo running on the hub. Acm works with argo everywhere. Acm can help you put argo everywhere.

C

Acm will detect what argo is.

B

Doing what argo is doing, yeah exactly exactly.

C

And bring that back, but also in presented in a meaningful way, because you know those again. Let's say you know christian and I both had access to three of those six cl the different three of those six questions. So christian had half I had half. If I had put pacman and I again you same github, same branch, same piece on one, those, three and and uh christian uses on the other. Acm is going to say: hey look, we've got six copies of the exact same thing deployed across all six clusters.

C

Is that really what I want to do? The sre is going to be like. Why is it on six? You know knock knock christian. Why are you doing this knock knock josh? What are you doing over here? I see you guys are deploying you know. So, there's a bunch of visualization help you get here too, and I say six, but this works for a hundred or you know when we start to talk edge scenarios.

C

We start talking about a thousand and that's really where you expect to find not argo on the hub targeting a thousand clusters, but argo on each of those single node clusters everywhere and acm again is going to bring back your application status. The fact that they're, you know there are argo applications there and- and so you know that makes sort of the trifecta of we've got argo on the hub, multiple largos, on the hub or argos out in your manage fleet.

C

Whichever way you want to slice and dice it, and it can be any combination of that, as well as using acm subscriptions in there we're going to display it they're all going to look the same, it's going to be the same, look and feel there's no new learning curve, etc. It's it's all just there, and so as an example.

B

It's kind of like what uh to steal what kind of andrew.

C

B

uh Sometimes says it's, it's not a or it's a yes and right like yes, and you know you can do all those other other other cool things, and I I like that. You brought up the the edge um kind of use case as well um and to take that kind of like a little further is that um you know leveraging that multi-tenancy that you get with openshift, with argo city of multiple developer teams being able to bubble that up right into acm.

B

To, like you said you know, to have that that policy driven to have the sres or or have the ops guys uh be able to look and see like hey, like you know, we're having all these applications bubbling up and oh and by the way I can see that and you know, take action if I need to so it's it's really. It's really cool. It's a yes and right! So it's like you know you can have it on the hub. You can have it. um You know everywhere you can bring it.

B

uh You know you can bring it in from um from other clusters and see that information, meaningful information as well.

C

And the cool part is: there's nothing special like you, don't there's no weird like I don't you don't have to switch flip f, flip switches, anything fancy if somebody goes out and deploys argo on a remote, managed cluster and starts deploying apps we're going to detect that and bring it back. It didn't have to be a policy that I used just to put it everywhere. I just used apology, so I had a single source of truth.

C

That said, you know I want to put argo everywhere, so I had to find a policy and my placement rule said put it everywhere and therefore it is, and so you know there's that detection mechanism there's nothing, there's no, no special switches. Anybody that's running acm will get this and we'll it.

C

You know, gets this for free and uh and we'll start to see it coming up and so just stay, because you know I like to prove that I'm really doing what I'm doing I'll click on this, which is again we're looking at the remote one. So it took us to the argo. This is the remote argo running on the other cluster. So if I click back here before remember, we saw I had lots of apps and it was even in a different filter mode.

C

There is only the one app because this is a completely different argo on the remote cluster. I'm not going to show urls since we're recording this, although they'll be blown away tonight, anyways but but you'll have to. But you know I figure it's enough to see that there are no other apps like I'm clicking here, we're not filtering it's it's literally it's on the remote system versus if we scroll back through my tabs enough. Maybe we find the other one I had. Maybe it was this one.

B

C

Here you see, we've got all of them. This is so. This is the argo that was running on the hub itself, and so again it's you know it's all these interactions and we're just creating you know creating more and more of these pieces and all of the details you see here again I mentioned it came through search it's available and you can. You can deal with it through the acm search, which is just amazing for remote management and sort of keeping an eye on your cluster. Oh.

A

B

B

Cool see here, there is a question that that um you know that chris answered, but maybe you can expand on it a little bit uh josh about um the difference kind of like the history between mcm and acm. um I actually still get those questions at all like.

A

C

B

Mcm is mcm a replacement for, like they they're they're a little confused on on that history, and you know you coming from ibm, um could probably give us just a little brief uh history of in the differences. If there's any.

C

Absolutely so it seems like ages ago now, but I guess it was really only about two years ago.

B

A

C

Was multi-cluster manager and that is actually what became the open source code base. That is now advanced cluster management, although I think, as always to confuse things, acm or sorry ibm still has an mcm product, but it is now acm based and it's really it's, I think, the they call it a. The cloud pack includes mcm, which is really just acm. They use our community builds, and that is why we've, actually you know not the plug here, but we've expanded acm onto power and the z platforms as well.

C

So if you're running openshift, z or openshift on power, uh you can take advantage of acm's management platform or manage two platform as well. At this point in the game with 2.4, which is amazing,.

B

So basically acm is mcm. Yes,.

C

Acm acm has replaced, is the is the child of and has now become, the new mcm cool.

B

Cool. Thank you um sweet here. uh There's no questions. People like my metaphors, which is always cool.

C

B

And peanut butter.

C

Then maybe I will expand a little and we'll do a little more gidopsy stuff, because this.

A

C

Ops call so we've talked about the argo implementation. Argo very big on git up, but also does helm acm does get, does helm. You know some of the different targets. Let's just talk about for a second, you can toward bit bucket get t um all of these can be get enterprise. These are on premise, car or versions of get that you can run as well that you can target both these platforms too.

C

I guess the piece the takeaway before I get into some of the cluster provisioning cool stuff is that uh to take away from this is they're very interchangeable. We're highly integrated with both we fully support both and will continue to develop both because there are different places for each of them to to be used and they work great interchangeably as well, which I hope I demonstrated here, especially we did. You know the remote argo apps.

C

They were deployed by an acm subscription, so there's all kinds of different possibilities, and you know we'd love to see what people come up with as well. But you know we try to show at least that there are a number of different ways to do it, one that hopefully fits everybody's need that they'll be able to pick and but they'll get a similar experience regardless, which is our goal.

B

Yes, yes exactly, are they.

A

All right are they following the same versioning acm with a mcm kind of transition.

C

And they should be documenting as well under the covers where which version of acm they specifically are pulling in, which again would be the standard releases, so they'll be picking up at 2.3 shortly as part of it and in the cloud pack for in ibm they use the the acm releases as well cool good to know all right. So let's switch gears a little and we're going to talk a little bit about get ops provisioning of clusters, and so that's great. I've done some demos.

C

I think they're, mostly they've been internal, although we've talked about it at least a little bit on some of the previous twitch streams with the provisioning as well, but provisioning, a cluster and git is a pain and the main reason it's a pain. Is you got secrets, galore that you need to be able to make this thing happen? You've got a big set of configurations and anytime you're using the word big that doesn't actually do well with git ops, because somebody needs to review big and the more yaml.

C

There is the more the chance that there is a sticky finger or a nefarious set of fingers that are trying to get something by, and so what you really really want is what is the simplest way? We can provision a cluster and something you get for free and what I'm about to show. You is it's fast as well, and so I think I actually talked about this with gurney on a previous twitch, the acm integration of cluster pools that come from hive.

C

So the background there is a cluster pool is literally a pool uh one or more clusters that you pre-config, the administrator pre-configures that is sleeping in hibernated state and that as you need, you are able to provision, and so in 2.4 I shouldn't say provision you're able to claim, because it's already provisioned as the idea and so 2.4 we introduced uis. And so here you see, I've got a couple of different cluster pools created across a couple of different cloud providers, one in amazon, one in azure, one in gcp.

C

You can see that each one of those has a cluster in it asleep. This is a demonstration, so I'm using single node clusters, but you can have three node master clusters that host workload as well as six node clusters or twelve node clusters. The sky is kind of the limit.

C

The cool part is, is you can have a bunch of different pools as well, and the only cost is the storage that are assigned to some of the drives in the in the providers, because otherwise they're off and they're asleep, but what that also means is within seven, maybe five. Well, five to ten minutes I'll use that number within five to ten minutes. You can have a functional cluster with snow. It's a little clo, it's a little faster when you've got six or twelve.

C

It takes a little longer for all the nodes to sync together, but literally it's the time it takes for the thing to power up and re-synchronize back together is how long it takes to get a cluster. But so why is this big news for get offs? Well number one? Is it's fast again, five to ten minutes for a cluster, pretty cool? What else cluster pool does, though, is if I don't have a cluster in waiting? It will start to provision another one. So, even though I've got one in waiting, I can request two.

C

I can request three. I can request four and the pool will just provision those additional clusters, plus one more that it'll just put to sleep for the next person that comes to order, and so there's no limit, except if I hit a quota which is we won't get into today. It's a whole other discussion, but um but quality is real.

A

You want to limit.

C

Somebody access, but you can claim a cluster and so the key here again I keep going on and on about simplicity. So we look here. We see there's a cluster that complete claimed in aws one in azure, one in gcp. I think we'll try claiming one in gcp, so we're gonna do it from git. So I'm gonna come back here. Get the list up.

C

uh I don't know if I'm gonna be able to do it as a true split screen, so I'm just gonna we'll do the change and get and then we'll flip back and we'll see how quick we can we can get the screens to sync. I thought I had to get.

B

There it is that that's one of the things with with getups is that um sometimes like it works too well, I.

C

B

I like I've, always it's almost a complaint right with with us that, like that, do a lot of demos and talk with customers. Is that I'll do a demo, I'm like oh wait like it already, did it, but, like I promise already did it it actually just works too well, like I couldn't pause it so.

C

Yeah exactly so, I was gonna split screen this, but my screen's, pretty small, that I'm doing the demo, so I'm not sure how we'll flip back and forth but anyways. So I'm back in my same folder that I was before, except this time I'm going to go to a clusters directory and I'm actually going to switch the branch also to cluster management and so again, all git ops terms.

C

All things you do in git ops, you define yaml, you define it in different branches in a repo and then the github story is you've, got argo or you've got an acm subscription. It points to that branch. It points to that folder. It points to that repo. It's always syncing. You can choose when it syncs, you can manually sync it you can time window the syncing and that's key, especially when you talk about provisioning, because maybe I want to define it. I want to make the change so here you can see.

C

We've got a bunch of clusters, but if we look at the customized, we've actually only deployed the zero one, the for aws, the zero one for azure and the zero one for gcp. And if we flip back I'm just going to close as I go, so that this becomes quicker as we do it, you get carried away. You get too many of these windows all right.

A

Because it always happens on a live stream, we've.

C

Got the aws the azure, the gcp, so these were all provisioned via git ops. These three I built up myself outside, but these three came from the repo from the git ops because, as we saw a second ago and we'll go right back, these were the ones uncommented in the repository. So I'm going to do an edit I'm going to do this live just straight in git, so we're going to do another gcp. So I'm going to just comment it out here or take the comment out.

C

I should say- and I'm going to put in my comment that says except I have to use my bluetooth keyboard because my other keyboard died on me.

A

C

Jmp gcp02, and so I'm gonna commit this, so this is now committed, so the customize has now activated this additional piece of yaml. So if we go over here and we look again- I was uh talking about simplicity. This is the simplicity. I have the name of the claim which becomes the name of the cluster that we start to see. In acm we have the name space where the claim is being created and so from an administrator point of view.

C

This is a name space where I've been granted access to create, claim objects, so you can see exactly who's got what where and then also I got some labels acm. Any label you put on here will transfer to the imported clusters.

C

So if I want to define it again as development as production as it needs open shift, get ops installed, I put the labels on it here and that becomes part of my manage cluster, that that becomes my possession but then gets pre-configured by all the get-offs I already set up and then the last one is. I picked the pool and again this is going to be based on the namespace you've been given access to. There may be one or more pools I like to think of pools in this.

C

This regard really is templates of a cluster. So from an administrator perspective, the administrator took care of what this looks like and just gave it a name, so he defined it or she defined. If it's going to have one node, is it going to be an extra large or a 2x large, all those kind of things, and then they could you know, and so then you commit this to git. I just committed it directly. In theory, if you're doing true get ops I would have had, I would have had this forked.

C

I would have done a pull request and my administrator would have approved it which merged it in and so once it's merged in the acm git ops is going to apply it automatically. Now we don't see it here, because I have it on a subscription and we talked about synchronizations and time windows.

C

If we go back to the subscription I'm using for this, which is the snow cluster provisioning because there's snow, which stands for single node openshift, I have synchronization off. And so, if we look here, we can uh reconcile off.

B

C

We have the reconcile rate off because I didn't want it to automatically apply but again subscriptions argo automatic reconcile. You have the option of both whether it's acm or argo time windows that we call them time windows. So even if it's committed to get, we can have.

C

It only apply on sundays at one in the morning or any day of the week at one in the morning, when maybe nobody's using it that's or using other systems, and that's when we want the provisioning to happen because it's going to hit a bunch of repos to deploy a bunch of apps. So again you can use time windows, sync controls on top of the fact that you've got the git as that source of truth that you, you know, you're, monitoring, you're, controlling and remember.

C

There were like three things we had to set in that resource. So it's very easy to see. Is the person making the submission choosing what we really want them to choose and should I be letting them- and you know, did they use a name that makes sense and they didn't have access to try and sneak? In you know, I wanted to go to forex large, because I was going to run some netflix servers on there as well yeah or netflix containers on there as well. So you know it really simplifies that process so anyways.

C

I am going to hit the manual sync button here for it, which should then give us a time update if everything is working come on.

A

So we got a question chat here.

C

A

Can you do transformations to application like if you move them from one cluster to another? Do the routes change the storage classes get updated? How does that get managed here.

C

So it can and it sort of depends on what.

A

You're doing right, yeah so.

C

um You can do it based on, so our placement determines where it goes, so you would use place you'd, be able to use placement to move it from location, a to location b and then you've got access to things like customize, etc, to modify names, pvs, etc.

C

When we get into pvs, I don't want to give away the soup of some of the stuff that's coming, but we've got a bunch of stuff around replication within the app so that you, you know you're able to move an app from location, a stateful app from location to location, but in the most most cases today, if you're, if you want to deal with stateful, it actually has to come from a pv in a container that is separate to the main application.

C

If you're going to move it and so coming, but coming are there's a number of different replication strategies, both from odf, which is red, hat's storage, as well as the it's called volume sync from the coupe back organization that uh we're integrating with. So. If you stay tuned, there is going to be soon we're going to have a bunch of examples where we've got databases flipping from cluster to cluster type of thing and keeping the physical volume or the uh the pvs. In sync,.

A

C

A

As we do that yeah it's, I wanted that 10 years ago.

B

Yeah exactly go back in time and go like that. I was uh any time yeah. Actually, chris espana, you mentioned like any time. uh um You know I talk about uh get ups with someone. It's like well, like getups is not really anything like like new per se. It's like now. We just have the ability to do the things. We've always been wanting to do exactly.

C

A

We actually can do it right. So it's not like this.

B

Like the idea is new or anything we've been wanting to do this, but now we can so it's really cool yep.

C

And so again talking about things that happen, so I flip back to my cluster list and we can see that I'm getting there's my gcp02. This is the actual underlying cluster that was sitting there hibernated. We see that it's resuming and coming back, but again we got that usage development pieces pulled over so once it comes online and the manager goes to ready state. It's going to get my policies that cd is going to be encrypted.

C

It's going to get open shift, get ops installed on it again via the policy after that, deploys all the applications that openshift git ops knows about and is referencing those also get applied. So you know if we come back in about, I don't know about 15 minutes or so I will have all of these things that we looked at before in application. That said, six plus local will now be seven plus local, and you know all we did was define. We wanted a new cluster yeah.

A

We've been talking about turning off environments for decades to save money.

C

B

It's just you know you didn't want to commit. Well that and that's exactly it what's really cool is like just taking it back step back is that you have a um uh a get ops way to pro, not only to provision a cluster but to apply everything you want to apply to this cluster right. So then I you know I you know either as like an administrator or, like you know, a project, you know manager, right or even a developer.

B

I can go and make a pull request to not only give me a cluster but then apply all the manifest that I want to apply and then also that that whole clusters managed, you know, has has a life cycle management aspect to it all via get ops workflows, which I think is just it's just like you said chris. This just blows my mind and that like how how far we've come to where, like now like we're like where we we've always been wanting to be right for the past, like 10 years.

A

But I mean even the past year with just hard.

B

C

And everything else.

B

A

The advancements that have come so quickly have brought us to this point where it's like. We can do all that now. Wow.

C

Yep, no, the biggest challenge I think is now for us, as the dev team, especially in the github space, is, is the simplification piece of it is you know we don't want it as six yamas or, and we have to deal with. How do you deal with you know, there's a whole different discussion around. How do you deal with secrets with git ops? You know the cluster pool for us.

C

Is that perfect way of easy provisioning, obfuscate all of the mess of what the configuration needs to be it's all predefined up front, which is what we're usually asked for the developer, doesn't really care. They just want either the large the medium or the small right, and they wanted it yesterday. Well now there is the large, the medium and the small t-shirt size, and you get it in five minutes, which is as close to yesterday as I can get yeah. That's.

B

Right at least today, yeah.

C

Yeah, I don't, but so it's already it's the cluster's now online, it's in the pending import, which means it's being imported as it came out of sleep into acm and then soon enough. The applications start to will start to roll down onto it and before long it'll be configured, and so it's it's. This, for me, is the real game. Changer and just I I know- we've talked about it on this before but pools for certain type of development scenarios. Where you need clusters all the time. It really is a game changer and.

A

It is, I can't say that enough.

C

And if it sounds like I'm excited, but you know I'm pushing it, I use it in my daily. It's it's changed. My workflow like I come in on a monday, and it takes me about 15 minutes and I've got the latest and greatest acm deployed.

C

It takes me under 10 minutes to get a six node cluster three worker three master and then takes me under 10 minutes to deploy the latest acm build snapshot on it and then I'm good and you know what, if I find a problem on wednesday and I need to trade, try the latest stone out. It takes me another 15 minutes and I've got a complete. I start from. I get a completely new cluster in less than 10 minutes.

C

I deploy acm to it, so any kind of development even and we're now applying it and we're going off and get offs a little bit here. I I'm, uh but uh you know we use it in our build strategies, etc. Like it's getting that clean cluster is like the nirvana and getting it quick. It's it's! It's a game-changing experience for developers that at least that work in multi-cluster spaces.

C

All the time and you're doing dev or you're doing tests, and you you know you need to spin things up and down at different versions or you just need a bunch of clusters. Having that on demand is, is huge and there's really no other way around. It.

B

Well, yeah: well you talk about like changing your day-to-day workflow like fundamentally change your work, like that's kind of um where uh what get ops did with me, whereas you know I used to be an sa and you know- and you know I build demos and things like that where it's like now, I just need a bare kubernetes cluster and then I can you know apply. I can basically go from like kubernetes install to my demo working like in a few commands now this is actually taking that even a step. Further.

B

Now, like you just now, you automated the building of the of the cluster. Now I don't have to build it and also by the way in in the application of like my stuff right, like, like my actual absolutely.

C

B

So that's like you, you take that you know like an extra step further right now I don't have to you know, run openshift create cluster. Now, that's all that's done with me. I don't have to do the oc apply dash k to apply my dem like you've, already done that um in.

C

No you're you're absolutely right and it's and then I mean and that I don't even include that in my calculation, but it's actually there is after I deployed acm. I logged in I clicked into application. Just like I showed at the you know. We come further circle at the very beginning. Like you know, I write in my infrastructure build out, I choose get I paste in the url.

C

I choose the branch as so here I'll, even I'll go through it. I'd literally do this. I choose github fleet sample it populates the branch which I choose main it populates, and I so this is just pulling it out of git. I pick seeds, I choose so this has been created already, which is why I was able I didn't have to paste it. I was able to pull it from the drop down. Usually I use uh I'll use the low reconcile rate because I don't need it to check that.

C

Often I just needed to check the first time and then ever so often it's every hour, it'll go out and look I hit save and that's it. I walk away and you know I go and do my daily development, but then, when I need a cluster I'll have three, you know it part of that git ops flow is it went out and it pushed out the three three clusters from a pool. It deployed a bunch of applications.

C

You know my pm or christian, comes to me and says josh. I need you to do a demo. I didn't set anything up this.

A

Is the environment because it builds it's just, as you said, I filled it out every time.

C

And you know, as we get new stuff, I need to patch something you can add things in it. You know you just yeah. I check it into git and my source of truth. Just you know, for my infrastructure code just keeps growing, but it's it's there and it's always the same. It makes it the same every single time and I need to make a change. I commit it if it doesn't work and I see something I'm like how did that get there? I go back and look at the commit history and go. You know.

C

Why did my buddy hear john check this in you know make this change that that source of truth, that's the whole. That's the git ops mantra, and you know it's it's it's wonderful when it when it's working, it's wonderful and it it definitely improves setup, configuration time and there's a whole discussion on the side of like recoverability. If I need to rebuild an infrastructure, all I need is that single piece to start and then once we start to bring in chris, as you were talking about with with the pvs and stuff, it's the whole solution.

C

It's like you know, built from sketch one command type of thing. On a new. You know I just provisioned a new cluster. I installed acm from the hub with three clicks. It's literally three clicks, maybe four and then I apply a subscription or I you know all right or I click in the hub and add argo and then I add an application set that points to that. Git that had that is my seed, like argo could be pointing to that same seed directory as the acm application set. It builds everything out and that could be.

C

You know my six clusters, my 60 clusters acm, will attempt to do reimports if the sky's, the limit, like it's it's. The infrastructure is code when it's realized and especially as we try to sort of keep it simple. So it's readable, it's it's it's game, changing in a lot of ways. No.

B

It's yeah, it's it's, it's definitely game-changing, and it's also, um I mean a while ago. It changed my mind, the the the you know not now, I'm I'm the preacher of like well. Let's keep your your um your clusters as cattle right.

B

So like long, I think long is the days where people thought like you can install kubernetes and treat it like like a vsphere instance right and it's like that's not what it was made for and now like now we have the tools and the abilities to like treat them as treat them as cattle right like clusters, as cattle like we, um you know uh um washari here asked about. um You know, moving um applications and it's like well, it's not this. You know it.

B

Your applications will move with everything and it's not really necessarily uh a move in in the literal sense, because it's actually it's it's a it's a copy and destroy right. It's kind of like the mv command on linux. It actually, what it's actually doing is just copy delete versus um an actual like moving of of your application. So.

C

There's actually.

B

There was actually a question about support for vsphere for the vsphere ipi. um I know we showed a lot of cloud stuff um that these. So um what? Where are you guys at with the the vsphere ipi.

C

So you definitely can provision with vsphere. um We don't have the pools yet in vsphere. So, okay, vmware and openstack are the two that are in the works for the cluster, pooling um it's they're, so openstack and vsphere both provision a little faster than the cloud providers and sort of storing on local is not always it's more of a just. The pool just becomes a speed thing at that point.

C

There really isn't the same kind of cost savings, because even if you're, using a cloud provider for vsphere like packet or something like that, you pay a fixed fee for the esxi, whether it's empty or it's full. So you know suspending your cluster. Doesn't actually do you any good, but uh but we're bringing it, and it's more so just because it makes these flows so much simpler. And you get this idea of sort of templating and being able to define what you need your club.

C

You know your administrator defines what the clusters the developer just says. I want the big one or I want a small one. You know I want the one that's in vsphere versus the one that's in aws etc, but so yeah, if you looked at well so number one. I just wanted to point out like we can see. Things are already starting to float over with the numbers. It did actually provision. But if we go back to the clusters and we do the create, so this is the current uh current stack of provisionable pieces.

C

So we have the aws, the gcp, microsoft, azure, vsphere, openstack, bare metal, there's also tech preview for the assisted installer, which is the iso based for bare metal, but also, I think, can be used in virtualization and then um you we also support imp, the import, so I had click click the create, but you can import your um aro, your rosa, your aks eks iks.

C

I know I missed one anyways all.

A

Of them azure what microsoft.

B

There you go yeah, it's it's, uh um you know talk about hybrid cloud right, we always talk about hybrid cloud and you know all the.

B

Chris, you said the get ops is a holy grail. I think this is like the nirvana of yeah of devops right. It's like yeah. I got to have truly true hybrid cloud um and be able to see everything with a single pane of glass.

C

And it's it's yeah and it's not just like. Even if it's you can do a combination, you can have git, ops and githubs for your configuration definition definitions, but you can manage all of your clusters manually in the acm, but as they come in the git ops apply. You know all the pieces of github supply because it's all just labeling, and so you can import those star ks's. I this is another thing that was added in the 2.4 discovered clusters.

C

You can you know if so, if you're in and you already have five or six open shift clusters today that are not under fleet management, you can add acm to one of those clusters, and then you use this page. You define your ocm credentials and you'll get the list and you can pick which ones you want to import.

C

You can provision a new cluster put acm on it. As I described you, provision ocp go to the hub. Acm goes in in five to eight minutes, and then you can discover all of your clusters from the remote ocm, open cluster management online and then pick which ones you want to import into acm. So there's a number of different ways to put it all together, but then, once it's together, you can start leveraging git ops, and you can do that for everything.

C

If you want like we demonstrated, and this demonstration showed today, which was all the way from provisioning these things to configuring them to loading out the applications on them or you can just use it for configuration or you can just use it for the application management and you can manually manage each of those pieces from the uis from the clies as well. There's a bunch of different ways to slice the pie, all of them.

C

You know all of them supported all of them working together I mean we did everything in git ops today, but we viewed it all and looked at the topology through the single pane of glass and that's kind of the key is sort of bringing these two types of technologies together that manual and they get it get in such a way that it's it. It allows you to manage your entire fleet at scale.

B

Perfect, that's! That's amazing! That's uh that's a that is amazing, so we're here at the top of the hour. um I don't see any more questions coming in um so um so yeah so josh. uh I don't know if you have any any final thoughts. Any final words um um you know before we close out here.

C

I was just going to say: if you haven't given us a try, give it a try if you're using argo, you know, don't be afraid of acm anymore, it's not in this or that it's a together. We are better, and so you know it's everything we talked about today is in operator hub. If you want to find the get off samples.

C

Look me up on github, which is jn packer, and it's everything we do is in the public here in red hat, for the community and for the open source, and so you can see how it was done. You can try it out yourself.

B

Cool cool thanks um so yeah so cool. Thank you, everyone for uh for joining in chris! I don't know if there's another another show coming up. um There is not so tomorrow, red hat. We have a recharge day, which is really great, um so yeah. So we it's. You know we have tomorrow off. So it's a long weekend for us red, hatters, um there's other companies that do something similar. I think amazon has has something like that.

B

So um you know for you for those who have a recharge day, um please enjoy it um and uh yeah so next week I don't know what we have next coming up next week, chris I'll. Let you take that.

A

B

Yes, because I mean just remember: I'm ready.

B

There you go so um so yeah next next next show. um um I talk about our back. With, with uh with argo cd, uh we tried that once we failed failure is not an option. I'm gonna get back.

A

On the keyboard again, yeah.

B

We're gonna hit on again, so um so look out, uh look out for that uh in the next couple weeks, so um so yeah other than that. That's it! Thank you for everyone for, uh for showing up.

A

Yeah, thank you so much josh and thank you to the audience and stay safe out there. Folks all right.

B

Cheers everyone.

B