Red Hat OpenShift GitOps Guide to the Galaxy | Red Hat Livestreaming, 2 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitOps Guide to the Galaxy (E38) | Exploring Argo Notifications

Description

In this week’s episode of GitOps Guide to the Galaxy our guides will give you a practical demo of notifications with ArgoCD!

Argo CD Notifications continuously monitors Argo CD applications and provides a flexible way to notify users about important changes in the application state. Using a flexible mechanism of triggers and templates you can configure when the notification should be sent as well as notification content. Argo CD Notifications includes the catalog of useful triggers and templates. So you can just use them instead of reinventing new ones.

Want to learn more?
https://argocd-notifications.readthedocs.io/en/stable/
https://developers.redhat.com/e-books/getting-gitops-practical-platform-openshift-argo-cd-and-tekton?sc_cid=7013a000002qG1BAAU

Try Red Hat OpenShift: https://red.ht/3sMaW7O

Follow us:
Christian Hernandez https://twitter.com/christianh814
Hilliary Lipsig https://twitter.com/Int2Caffeinate
OpenShift https://twitter.com/openshift

Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_confirmation=1

What is GitOps Guide to the Galaxy?
Every other Thursday at 3pm ET host Christian Hernandez and guests dig in to discuss everything in the GitOps universe, from end-to-end CICD pipelines to creating Git workflows.
Learn GitOps https://red.ht/3saqlzS
What is GitOps? https://www.redhat.com/en/topics/devops/what-is-gitops?sc_cid=7013a000002qG1BAAU

About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
OpenShift on redhat.com https://red.ht/3LGCIeD
OpenShift on GitHub https://github.com/openshift/

#RedHat #GitOps #Kubernetes

A

Let's get into the big topic of open source, it's something that we.

B

Actually, this.

B

Is that process that a developer or, let's say.

A

Has the kubernetes.

B

B

Hey I'm going to uh steal the chris short intro and say good morning good afternoon good evening, wherever you are, we are live with the get ops guide to the galaxy episode 38, uh which is what my it's, my fourth or fifth one here right christian. I.

A

Think yeah, I think it's it feels like you've always been around. So it's probably it's probably the fourth one I would say officially.

B

A

It only feels that way because, like we've just been friends for 10 years, right.

B

A

For those watching we we have a background, so um we we worked it together. In the past. We became friends now she's here um but yeah. I think it feels that way, because you just I just always known you yeah.

B

I know it's been, it's been forever um so yeah and, speaking of it being forever and being friends right we've. We also have done all these. um These co at least joint talks together right. We did the helmet back again, talk forget devconf uh and um we just got back from uh kubecon where we spoke at get ops con, and I see you've got your your shirt.

A

Yeah yeah, I got my shirt for it, uh getupscon kubecon. We were there in valencia. We I I met a few of you. um It was, it was actually pretty cool. I don't know about you. Hillary, but like people would come up to me is like hey. I seen your youtube stream, so it felt kind of weird to be kind of a like a z or a wide level. Celebrity. I don't know what you would call it in it. That.

B

Did not happen to me that.

A

One person.

B

One person came up and was like hey. It's me from the street yeah.

A

B

Oh, my god, I'm so happy.

A

B

Person like that happened to.

A

B

Time, I guess I'm not very um I'm not pregnant.

A

Now yeah, yet not yet not yet you'll you'll get there, uh especially since the talk we did so it was. um We got a lot of good one-liners. It was uh you know we we, you know we were um conference, buddies right but like we kind of separated and then we kind of hung out, we kind of separated. So I'm curious to what you wait. What was it your first kubecon yeah yeah? Okay? So what was your impression of uh of uh of kubecon.

B

um So let's see.

B

Was I found that most of the talks I was most interested in and this comes from being an sre right this? This really comes.

A

B

An sre that a lot of the talks I was most interested in, I was like I'm gonna. Go to that one we're either one virtual. It was like dang it or two, oh we're given by other red hatters, and I was like okay.

A

B

Did not fly all the way across the world to go talk.

A

To people I work with yeah yeah co-workers right. It's like.

B

Email them for their slides later right, yeah.

A

B

uh I ended up going just kind of browsing around in and out of talks in another vendors hall. I I got to meet a lot of people in person uh both at red hat today and who have left red hat, um and I ended up spending a lot of time at the booth answering some questions about openshift and um I will say that one thing that happened repeatedly was people were coming up to me asking me where things were like.

B

I would know, and I recognize that red hat is an executive like sponsor or whatever everything I know or diamond.

A

Sponsor, I guess, is the term yeah like.

B

I I don't actually know how to find things. I I literally have dyslexia. You should never count on me for directions or math like just.

A

No okay! So when you so so when you say make a left, go right.

B

Yeah no yeah, I will do this, I'd be like which one is an l. It's that one yeah.

A

Which one is the other? I guess this one's left yeah um unless you put your hands like this, then you're you're messed up at that point.

B

A

You're really in trouble, so uh what's really cool for me, um so uh it felt like it was coming back so for those that have gone to um uh kubecon in the past, san diego was insane. So uh you said this was your first like kubecon in san diego, which was the one before you know the pandemic.

A

It was like shoulder to shoulder people. It was just it was just nuts. It was just full. So compare that to like when it happened here in la where it was really really small. It felt really um weird because it was mostly vendors, so you know cool. It was just mostly us walking from booth to booth talking to each other, but this one this one was a little different. I felt that um there was more end users there. So I had a lot of really great conversations.

A

So for me these these conferences um as much as as much like, we like submitting talks and as much as we like going to some of these talks, I'm really about the networking aspect.

B

A

People person, love talking to people they'll, be like, oh, so what happened when you were down for three hours and yeah like and and what was you know? What was the the post-mortem like and you know like? I, just love love that conversation being part of also about the um get-ups con was fantastic. It.

B

Was really a good con yeah.

A

It was, it was great because I'm part of the planning committee, so I'm kind of you know um to you, know uh tuning my my own horn or, I guess our own horn here of just like looking out and just seeing the sea of people it was. um You know I felt a little sense of pride right being like. Oh, you know all this planning and hard work.

A

You know that we did um all the cfps I I read um were it was worth it because we had like 300 ish people there um we had two rooms, two tracks and the second room was too small. I think because there was people out in the hall. I remember during our talk. Oh you're you're, the site. I remember during.

B

A

um uh You um we had to like sit in the hall for our turn, because there were just so many there was no standing room only yeah, um so that was that was pretty. um That was pretty cool for you to witness. I'm, like okay cool see her first get. Ups. Con was an actual good one. It wasn't like the tiny room that we had in l.a with like 60 people. um I.

B

Mean I will say uh there was it was standing room only right in the room where we were speaking at the talk before ours, and rightly so. That was a very fascinating talk. I went back and watched the recording later, um but like hey boy, it emptied out. Like half half half of the people left for I was like I mean I get it. There's probably.

A

Yeah yeah there's, there's uh yeah because there's you know it's that's another thing. It's hard to pick the track right, um especially in kubecon. You were. You were joking that um uh you know. One of your talks was one side of the building and the other one was on the other side, and you had like you know your with your head up. Sprinting.

B

The third building- I never found it by the way. I have no idea where that you had to go.

A

B

I never figured out where that was.

A

Yeah, that was uh that was the um the the event in valencia right. The conference center valencia. There was like three: it wasn't like a conference center. It was like the conference center had like three or four buildings or whatever, and trying to figure out upstairs downstairs in the other building upstairs downstairs in that building. There's a third building. No one knew where that was. um I finally found the third building, so I do.

B

I should have asked you.

A

Yeah yeah, I did find it. um It took a while, but yeah I found the I found the third building, so um that was pretty cool. um So anything else we haven't covered for uh getobscon kubecon.

B

I think the cfps for get ops con are they still open for detroit north america.

A

So, oh yes for kubecon, yeah, kubecon cfps are still open up until friday, so uh so for those who haven't submitted anything for kubecon, it's still open, cfps are still open. um You have uh you know a day and a couple hours left. I guess so so try to submit those those are, um it's always cool to get accepted for kubecon, because it's so competitive. So um you know, do the yeah tomorrow, someone said uh william um in the chat said yeah to a cfp's tomorrow, so they close tomorrow so get those in.

A

um So those are it's a tough it's tough! I don't. I think I've only ever gotten one cfp for kubecon accepted. So that's like my badge, my badge of honor, I'm like oh, okay. I got one now.

B

A

Yeah would I say for the rest of my career. I don't have to you know. I don't have to worry about getting inquiry. I see at least gotten ones.

B

Yeah people have asked me if I was submitting to that one and I'm like I am still tired from valencia.

A

Yeah, that's right. I can't even think.

B

About that right now,.

A

B

Submitting one to ansible fest, though, which will be in chicago later this year. Oh.

A

Awesome awesome, cool yeah, that'd, be awesome, yeah, um so yeah all of these there's so many um uh yeah for the next so self. Our hope nine said for the next cube con. They should have a cube ctl describe venue.

B

Right so, let's say describe.

A

Venue- and it kind of gives you a tree- I want.

B

An ascii art of like a map, yeah yeah, definitely.

A

An esky art of uh of the map and or maybe like a you, are here. Yeah, that's that's useful! So that's something I noticed that they didn't have in the in the in the map. They gave you the map where everything was cool, but they didn't give you uh like on the wall, but they didn't give you a you are here so I spent there were. There was two pavilions and I was wandering around in one pavilion, not knowing that what I was looking for was in the other one.

A

So that happened to me a lot um and so uh eventually.

B

Yeah yeah, I imagined yeah yeah I'd rather not actually yeah.

A

I'd rather have yeah overflow right you'd get a buffer overflow you'd get just some sort of memory memory uh issue there. um So one thing I do uh argo con is still a thing the so those cfps are closed. So um we got 127 submissions, speaking of of conferences so 127, which now I'm I'm already getting tired thinking about reviewing these, because I had to do 80 for get ops con, but now for argo, con, there's 127., that's a hybrid event.

A

I put that in the chat um uh here I'll give it to producer stephanie, so she can um maybe put it up here for for those who are watching um so argo khan, uh this will be the first in-person event for argo khan, so um that'll be exciting. This will be a hybrid event. So, if you want to do virtual, you can um do virtual for argo con. So, um let's see here what else you were on um uh we're on on open access, openshift admin uh right.

A

This uh was it this was it this week or last week it.

B

Was this week it was yesterday.

A

Yeah, so you um I heard uh you were uh quite popular on that uh um telling telling andrew and johnny um that you'll you'll replace them with a small shell script. What was the? What was the thing that came out of that at this point? Well,.

B

I I I told uh I I did actually didn't say it in the in the stream. I told them this in chat afterwards. I told johnny that I can write regex smarter than him. Oh.

A

Yeah, that's right.

B

Yeah yeah regex is exactly.

A

There sorry so like if you're talking uh um the uh yeah.

B

Andrew said, like.

A

uh Ask openshift admin is the best stream of them all. So um that's uh uh yeah, so hillary was yeah, so you were ruthless. I like the chat, that's going on so if you guys are are are on twitch or on youtube check out the chat. There's some funny things going on there um and.

B

A

Johnny said still yeah, it still hurts he's. Like my, I think what you said also. There was another thing that came like. Oh.

B

He called me an alpha nerd. He said uh yeah.

A

Yeah yeah, like pearl.

B

Or alpha nerds, and since I love pearl like I'm an alpha nerd and I was like I'm really not sure how I feel about that. My best friend pointed out. She was that just means they're beta nerds.

A

There you go yeah, so there you go. Thank you pearl. So, if you're having the right pearl it, you know my hat's off to you on that. Well, what pearl what's funny about pearl is um that it works accidentally. Sometimes we're.

B

A

Know why that worked almost.

B

All of it, almost all of those xkcd memes, actually are about pearl right. He's.

A

A pro programmer.

B

And so it's kind of like that meme about like. Why does my code not work? Why does my code not work and you dig into it digging you're like how did that ever work? To be honest,.

A

That is pearl. That is perfect yeah. That is that's. Why.

B

It's part of what I love about pearl. It's such a like. It's the programmer pleaser of languages. I heard somebody say golang was that and like I hard disagree, it's pearl because you can do anything with pearl, regardless of whether or not you should.

A

Yeah yeah exactly worth it right. It lets you shoot yourself in the foot and.

B

It really does.

A

It and it's unapologetic about it, and it's really: it's like you, can throw anything at it and it'll work and it'll.

B

Do this and like you.

A

And and like you said, you're like, then you read it and you're like I don't understand how this ever worked like. What is this? You know um yeah. So pearl williams said pearl is the best way to appreciate any other scripting language, so um so yeah yeah so the uh so that was another one-liner. That's pretty! um uh That was pretty fun. I've been on that stream as well. um That's always fun to to go on with those guys, um so so hillary.

A

This is officially my last stream here um with uh get up's guide to the galaxy. So for those who haven't uh see my twitter, twitter, post and um my uh or my linkedin post here, I am, um uh I accepted an offer outside of red hat, so um I'll still be in the get up space. So you'll still see me around. um I begged and pleaded to to have hillary come come.

A

uh Have me come back on as a guest, so she begrudgingly accepted um to see, uh see for for me to come back um as a guest, but yeah. So today will be my last stream. It's uh been kind of um uh yeah, so yeah producer stephanie. I like I like what she said. It's more of my last episode as a host, because I I will be coming on. um I do have to I guess: postmate hillary, some coffee uh to to bribe her to to allow me to come back on.

A

But um it's been, it's been an absolute pleasure uh doing this stream, uh starting this stream from nothing watching it grow. um You know watching it evolve. It's it's been. It's been great, um uh really appreciate everyone who's who's liked subscribe. I know stephanie likes when I say that so still like still subscribe, because I will still be subscribing and I'll still be on at least in the chat, um so um so yeah. So it's uh it's been great. It's been great.

A

um I'll uh I'll, definitely see everyone around um and um I'll leave you in good hands with hillary cause. It's I.

B

A

So well, if you haven't seen her so you you come up with these great one-liners. If you guys haven't seen our talk on uh uh get ups con, we should probably get that link out um so so they can uh so they can watch it, um and so uh you come up with like the great one best, the best one liner. I think from from that talk is, like you always said: choose your tech debt.

A

Don't yeah yeah, don't don't choose the best tool for the job because you're going to have tech debt so choose your tech debt and I think yep.

B

A

The technical debt.

B

That you can pay, I actually put it on a t-shirt right. Some feel like. I want that on a t-shirt. I.

A

B

Put it on a t-shirt there.

A

You go, I did. I did.

B

The thing, oh, I didn't actually do it. I told my husband, like man, that's like the fourth person who said I want that on a t-shirt. So he's like here's, your t-shirt, store.

A

Yeah here it is.

B

So he actually, actually I won't. I won't plug that. That would be super inappropriate. It's it's it's most sort of a joke so that you could yeah, don't actually buy the t-shirt. Folks, um yeah.

A

But uh it's more of a joke: it's a joke, don't buy the t-shirt.

B

Please um so yeah.

A

B

Yeah it's uh but yeah. So it's not a t-shirt. Choose your technical debt wisely um along and also sre, is not your system right. These are the things that I say a lot now: they're they're one-liners that are out there on videos, so people realize I say them.

A

B

Yeah and- and uh chris short he had said to me- it's such a perfect thing, because I was literally just telling somebody that conway's law always applies, and it's like those two statements, those two one letters are very complementary. So I guess that's gonna, be your uh get ops con detroit uh t-shirt right.

A

Yeah yeah exactly choose your tech that yz so yeah this t-shirt actually was was widely popular and for those who attended I I apologize. I didn't order enough. There was, I ordered 100 and there was 300 of you. So hopefully we'll get we'll get better shirts, and I think I'm going to use that that that quote on the back of this one, because this one says um has my my catchphrase: if it isn't in get it's only a rumor um and so uh maybe maybe choose your tech debt wisely or you know, that's.

B

Gonna be my catchphrase, I'm gonna close the stream with that. Remember: choose your tech debt wisely.

A

A

So um so yeah so pretty cool. So um I don't think there's anything else.

B

The old orphan statler, sorry honorary balcony. I actually started in that balcony right.

A

Waldorf settler: that's right.

B

When I was uh when you first started the stream I was watching it, I would give you grief about doing things like not updating your git ignore file uh yeah.

A

Yeah yeah exactly so. It's really.

B

My turn honestly for you to do that to me.

A

Yeah yeah so um yeah, waldorf and so and actually and for andrew who's watching right now, um we kind of feel that way about uh about a lot of things. So it's it's! It's really. um uh You know an honorary balcony in the in the in the future, so I'll like that I'll bring popcorn and then I'll I'll uh I'll I'll, be the heckler now and instead of the instead of the host. So um so I think uh with today's topic is well talking about argo notification.

A

So um let me I'll just share my screen here.

B

Say I hope you remember to actually prepare the demo, because that was on you not me.

A

Correct that was on me and I do have it ish um here uh there we go screen three I'll, never remember how to share a screen. I think everyone always fumbles, sharing a screen right everyone's like how do I share my screen and then it's uh there. We go um so argo cd notifications, so this is kind of um argo.

A

Cd notifications is essentially what it it it does, what it, what it's, um what the name implies, and so um the idea being is that anytime, something happens with argo cd it'll, send you a notification somehow right, either via via slack ops via um you know, like slack or email, or you know, whatever um it's uh it'll send you notifications, based on certain things right so before um before getting into this argo notifications used to be like its own thing, like you have to like install it as part of separately, but now it's included as as part of argo cd so like if you install openshift, get ups or go cd now in any latest version, you'll get you'll, get argo cd notifications.

A

So um it's uh um it's there for you to use right off the bat. So it does this. um You know again in enough in a few ways, sort of kind of like like like with everything in kubernetes. I feel like it's a um I I was noodling about this um the other day.

A

It's like everything in kubernetes just has to be like a series of like little things together right, um so just like just like anything in kubernetes like you need to, like you know, deploy like five or six crds or in crs to get it, get it to work, but the idea being that um uh it's based on triggers and templates.

A

um So I think I'm gonna start from the from the from the bottom up. First talking about the notification services right, so the notification services is like okay like what to do. um If you know what, where to notify right. So like things like slack, it has uh just straight up email right um matter most, which is interesting. So you can tell some europeans wrote that, um because he has a matter most uh telegram, uh you can hit a web hook.

A

I think webhook's a generic generic way of there's teams integration right, so I'm pretty sure you know send something to grafana. So there's there's, um um there's certain integrations and actually so for those. Let me put this in chat, actually um so put here um a uh the slack integration here and it's essentially, I followed this like blindly and it and it works. So I mean so. This is kind of like a very explained to me, like on five directions. So whoever wrote this so I I'm always lamenting about argo.

A

Cd notifications, uh sorry, argo, cd documentations just in general. Just because um uh to put it nicely, you can tell an engineer, wrote it um usually so like so. You know when something's good, I'm like pleasantly surprised.

A

So I just kind of just followed this on the slack right and then there is um the the trigger template right so triggers and templates, and then um the the the triggers is basically like, like what triggers a notification right so like when something's created it'll send something uh when something's deleted something's deployed when the health is degraded right. So this is, I think most people would want like like on sync like the bad things right, I think, um uh but also like. I guess you would normally I guess, as an sre.

A

I guess I would ask you hillary, like you, would want to know if something is created right on a cluster. I would think that.

B

You would want like that.

A

Sort of notification, so it's not necessarily only bad things. I would imagine.

B

So yeah it does depend a little bit on on what it is. It's it's more. So if we care not necessarily if something's created, although we could, it depends on some of the capacity things and so forth, um but we would we definitely care if something's, deleted or.

A

Yeah changed changed right. I would think or.

B

You know, of course, failures always but yeah, something something being uh created as well. It depends on on the on the situation. um Certainly uh you.

A

Yeah, potentially.

B

Want to know that information, so there's a couple questions or maybe the same question a couple times: I've lost track of who who's doing the asking um yeah. Multiple people cannot do pagerduty.

A

Pagerduty, I don't think there's a pager duty integration, but there's email integration and I'm pretty sure, uh pagerduty offers an email from what I understand right, like an email address or something that you can send.

B

I think it also has a web hook. uh Oh.

A

Yeah, if agent duty has a web hook, then you can like I'm pretty sure. I.

B

Don't remember, uh I think you and I have said this before pagerduty architect should be a single like that is somebody's job.

A

B

And it should pay them very well to.

A

Play them very well, yes, exactly I. I got into pager duty like right when they like first started, and I thought man. This is great. This this thing is going to be big and I guess it is um when um but yeah it's a web book, so there's no actual pager duty like integration, I'm pretty sure the web hook is a generic way of doing that.

B

That should that should work. You should be able to map it to services and have the services have the escalation policies and the escalation policies page the people and um I'm telling you from an sre perspective. If you send off my pager because something was created, I'm going to be yeah.

A

I'll be really mad. Yeah.

B

All right, I expect that to happen. Yeah.

A

B

Unless there's some really amazing use case, you've come up with to track when something has been created that should never be created, and you want to.

A

B

S3 for that use your page or duty alerts wisely.

A

Yeah, just that's another thing on a shirt use your page or duty alert wisely, don't cry! Wolf!.

B

We actually audit them in my team. We audit them.

A

And our alerts and.

B

Our warnings um twice a year.

A

Oh there you go. I we used to um uh for for alerts. We used to like flag, bad apples and so then, like so like, if someone would fire off an alert and like if they cried wolf enough, we essentially they they would firing off an alert and they would get like the notification that the alert was successfully fired but, like it didn't actually page anyone.

A

So, yes, you do don't don't cry. Wolf use your page or duty alerts, wisely, you're.

B

Nicer than we are, we reroute them to engineering.

A

Oh okay, there you go there, you go.

B

Yeah, you you, you exceed the the toil threshold, the alert threshold.

A

B

Service, you actually start getting the pages yourself. Congratulations.

A

Yes, it was, you know, what's funny is see, I just send them to dev knowledge. You're. Actually, that's you know, like you said, I guess I'm a little nicer.

B

A

That's a good that's a good idea, though I like that.

B

Yeah, we do we actually it's in our agreements with our our tenant engineering teams, like if you're.

A

B

Our toil thresholds, your service, is not demonstrating to be resilient enough. You go back on the page of yourselves and unfortunately, fortunately, we've never had to do that right, but that's also like I said it's part of that that sre like contribution right in the life cycle, so something happens in production in a problem with the code operations, we feed that back into engineering and help them ensure that it gets fixed right. So.

A

B

Helping to to you know, push the continuous improvement of the product, so we've we've never actually had to do it. It's just like and honestly, I don't even think we'd actually have to threaten these people right. These are other red hatters that they desperately want their code to be awesome. uh Yeah.

A

B

It out there like just in case it wasn't clear like we do say this.

A

Yeah, so um the see here uh templates it's just basically what what gets sent right. So, like message, hey something has been created. This could be you know, um as you know, as you notice, there's like um you can get stuff from the payload right like app metadata name has been deleted right, so you can kind of grab some of these. It could get um pretty complex.

A

As you see here right, you can like even do attachments and um it gets pretty robust, um where's the templates here uh yeah, so the templates- uh that's I just went over and then the triggers yeah. So when so so, the triggers here is like when app status, sync equals unknown. Do this right? um uh It's it's basically um pretty self-explanatory, except like it's all over the place, it's it's kind of, like I said before it's kind of like there's like rat droppings everywhere in kubernetes. It's like how do I change something?

A

Well, you have to do it in five different places, um which is why operators are important so.

B

Why operators are important.

A

But you know you can write that into the operator, so I do have a kubernetes cluster here, um it's in argo, cd yeah. So let me clear this and then do uh oops ergo cd.

A

uh So here they have the notification controller um and all the um get go to the config map hold on. I think it's a competitor yeah, so.

A

There's a config map, this guy here called argo cd notifications, config map, and it's essentially it comes already with, like things already pre. So this is this is like. um I didn't actually um create this. This is this was created by the installation itself, so it already comes with like pre things that you can just use out of the box. You don't have so like. I don't. You know when I first started like researching this. I'm like, oh, my god, I'm gonna have to write like all this stuff.

A

um You know all you know it's not like it's hard, but it's just like I'm trying to like okay. I just want like a simple demo um to just see how it works, and um but it already comes with with uh things. So that's that's pretty it's pretty cool when someone gives you something and then you just kind of modify it for your needs. Instead of starting from scratch,.

B

um My favorite thing: that's that's. What is this saying? uh Good developers copy great developers pace right.

A

Yeah yeah yeah exactly exactly like. Why write something? That's already been done right.

B

Yeah, no, you just you just go copy that from stack overflow, you rename the variables so that they make sense with your application. Yeah.

A

B

Save that right, whatever whatever your save command, is a control s. If you're in an ide, you know escape.

A

B

X, if you're in vim and uh you know commit that.

A

That's how all of us code, isn't it yeah.

A

There's an o'reilly book right: uh let's see if I can find a copy and paste um stacker, overflow or yeah. So there's like.

B

A

There's this guy there's there's, which is really funny copy and pasting from stack overflow that it's essentially unless, like that everyone has this all good coders, have that in their drawer. um We've all done it so so then you have that right, so notifications. So then here I already, um uh I already set up the slack secret. So if you go into the um first, where was I notification?

A

So first I did this right, so I, like I said before this was actually pretty easy uh to follow copy pasta and then you have to create a secret with the slack token- and this is what it looks like- I'm not gonna show the slack token, because we're going to be using uh the red hat uh slack channel that I created.

B

A

So um so the sac secret is staying secret, but trust me. It looks like this. It's just a regular token like you think it would be, um and then you do um the actual see so then uh you have to add this. So I put a note here. It says: don't apply this edit, the config and edit manually, um because I overwrote the argo notifications and like nothing was working like why nothing was working like. Oh because I ever wrote it. I didn't, um I didn't add it.

A

I overwrote it so um so yeah so first, let's create the secret um with this basically create the token and then uh okay apply slack secret uh in argo, cd, all right so the secret's configured to now.

A

um We have to add this guy to the um what we do at it and the notifications and then basically you just put it in here like that and so basically you're you're telling the configuration it's like hey for your for the slack service um use this token- and this token is in a secret because that's what the dollar sign means in argo in argo land. So, for those those that don't know- I don't know if this is common from between different operators and how they do it. But um at least for the argo projects.

A

If there's a dollar sign in front of something, that's a flag to argo, to know to look for a.

B

Secret, so um that that is how it's done um and other operators can confirm. Yep.

A

So uh sometimes they do like you know, secret ref and they put the name of the secret, but I think since since most most people um put things in config maps, this is like kind of like the the the practice there.

B

Yeah it it depends on the implementation. It depends on several other things. It depends on uh like where the secret comes from. If it's getting rotated through some automated process, so you're going to make decisions on your secret management, I think was it andrew block, 101 ways to manage secrets right.

A

A hundred thousand wastes.

B

Five thousand ways to manage secrets right and he talked about a sidecar model. That's what I've also seen so, but.

A

B

A very common implementation, an operator looks just like this.

A

Yep, yep and um and I'm I'm actually being so uh game bounty, said it's common to use the dollar sign to refer to secrets. So uh thank you. Thank you for that. That's um uh and I'm like, like I was saying I was a bad bad citizen, because I actually just loaded the the secret in plain text into the kubernetes cluster. Don't do this please.

B

Yeah yeah, like hillary.

A

We did a past episode about secrets as well um to uh so pretty sure to check that out and then that's pretty much it, and so then, when you create an application. So let's look at the application here for um how you annotate your application to send um these notifications right, and so here um I have notifications.

A

It's an annotation, basically saying um you know notification. I am going to subscribe to this notification and anytime a sync happens. Send a slack to uh here is we're showing kind of my roots here, um get off happy hour, which was was the name of the show before we rebranded.

A

um That is the name of the slack channel, so uh without the uh without the the hash sign, um without that right like that, that would be in slack, but obviously without because it's a comment in um in in uh in in computer language, so that that's why you um leave it without the hash and then just like the application.

A

The way, uh the way it is right, and so um let's go to so here. I had some. Let's have this uh happy hour channel here and uh let's do a sleep two and then okay apply of the welcome application.

A

Let's see right and then here um sync will uh will start in argo cd and then it would send me a notification that the sync had happened right, and so you see here it says the application.

A

You know that whatever I named, it was successfully synced at this this fun time here, computer time, um and then uh it says here no value because I didn't set up a route for this, for this instance of argo, um but you can actually see it'll actually send you the link of where to see it right, and this kind of gives you cool stats this here you can modify this. This is completely modifiable, um not sure editable.

B

A

It's mutable there you go. I like that. I, like that, it's it's! It's mutable this this configuration. I just took the standard. um uh Let's go back here, one! No uh yeah um this! You know template here. um I wonder if I see slack yeah, if service is slack blah blah blah it'll send um you know you can edit this so that way, it'll um it'll show you whatever whatever you want here, and so, if I do uh argo cd, we can clear this here. Argo cd, um app.

A

Yet no is it app list. I rarely use the cli.

B

Argo cd, app man right.

A

Yeah yeah. Well I mean because like it's get up so like, I don't really use the cli. I just put everything from git, but I know get okay. So, let's, let's clear this again app get uh and then it's just the name, no uh unauthenticated. Oh, maybe I should authenticate uh there. We go my setups yeah, my my setup. Script wasn't um uh wasn't helping here.

A

Oh okay, so you're gonna see my my setup script because it messed up. Let's try this again, I'm just gonna rerun. This.

A

I'm like, why isn't it working I'm like? Oh, okay, there we go all right, all right, uh our go up! Yeah.

B

Your bash orchestration failed.

A

Yes, yeah, my batch orchestration did fail. I should have just done it with uh with arco um so yeah. So here we go, it says sync, so you can do like. um You know, do app uh sync and then I think I think I could just force it and then this here, while that's going, you see that it came again uh at you know at the 40 mark here it syncs again.

A

So every time this application, syncs it'll, send um uh it'll, send an error of a notification here, but I haven't done yet, which would be nice to kind of maybe hack at a little bit. Is how do we?

A

How do we make it send it when something fails? So let's go here. um What do we want.

B

A

Have to put in.

B

In your little, what was that little area with the dot notation, which I love, dot, notation.

A

Right here we go there, you go yeah, so uncreated undeleted.

A

Undeleted, that would be nice.

B

Yeah that one's nice.

A

Let's touch that that one out.

B

And things should never go away right, you that would be.

A

Yeah so let's see- let's- let's let's test this here so um so here, let's think well, here what we want. Is they assume undeleted yeah? Would you add another annotation see that I don't know see I haven't I I'm I'm flying I'm going rogue here. I haven't.

B

We're learning together.

A

We're learning together so um so the on, let's just do uh k, uh replace.

A

uh Let me k replace.

A

So pop quiz- let's see if anyone in chat knows um what's theirs to replace and uh apply.

A

Does anyone know, let's see here, john jonathan, I'm not asked johnny so johnny, do you know the difference between replace and apply? Why am I not applying it?

A

Let's get that here? Let's, let's see here, I don't know rogue one also they're talking about rogue one here.

B

Oh because you said you were going rogue and then.

A

Rogue I thought they were. I thought your poster was still there so.

B

No, my poster is not still there. It's just a throwback to last episode. I would hold my answer: apply we'll try to.

A

Create a new yeah, so um so so the reason I'm doing replace is because replace deletes and recreates it, um whereas an apply will try to do a weird merge and then I've, especially like with objects or even like arrays uh in yaml. You get weird um weird, uh unexpected consequences right, and so I just what I want to tell argo is I want you to foobar this configuration and replace it. So let's try that and then that should fire off another uh here.

A

B

Yeah there's an immediate field field would not apply.

A

B

There's actually, I seen this happen with um uh install plans actually, so there are.

A

ah Install plans for operating.

B

The tecton operator, actually the the tecton operator, install plan in between one release and the next. They changed a field, so it used to be immutable and then became mutable and.

B

The the method of uh of updating or of using the stall plan was, was to apply.

A

B

Wouldn't work because it couldn't reconcile that that field having changed right because it wasn't supposed to be so. You actually.

A

Did have to recreate in order.

B

To fix the issue that was, that was, that was a real sre thing that really happened.

A

Yeah, I remember you telling me the story, yeah, you actually that there was like a big outage because of it. It.

B

Was very frustrating um because it took us, it took me and I'm getting alerts for the managed workloads because what's happened is the uh process attempting to upgrade the tech dot operator which is called olm was chewing through. It was filling up lcd chewing through all the cpu available on the cluster, because it could not reconcile the situation it kept repeatably trying forever and the api server went down. So I'm getting alerts about the api server. The customer is complaining. They can't access their cluster like that's, not a good idea.

A

B

um So we figure that out we we do the thing um it wasn't even in the manage you know this is this: is a community operator wasn't even one of the managed operators so we had to? We had to go into a namespace, we don't ordinarily touch and fix the problem, and um so I was actually. This is one of one of hillary's super secret open source contributions is, I went and I talked to the tech talk team and I was like: can you fix that yeah.

A

There you go yeah by the way.

B

I was like, can you? Can you fix that? Because it won't just be my customer's cluster, it will be others as well.

A

B

um And I will say this: you know the techton team uh they heard about the incident they were like super.

B

uh I only have good things to say about the way they responded to that situation, about, like taking in the feedback immediately responding immediately prioritizing um and trying to make sure that the techton users were going to have only the best experiences yeah.

A

Awesome, that is, uh that is very topical story because of the replace and apply I like this and uh and waleed said, uh keep keep sharing these stories right. So that's. Why um that's why I I thought hillary would be perfect for this for the stream, because she like lives this stuff. It's not like um you know. A lot of times and and uh andrew can can attest to this um being in the bu. um You know a lot of times.

A

It feels like we're just you know, just on the soapbox pontificating, so it's nice to have someone who's like yeah, no, I'm living and breathing this stuff. um So.

B

I had to write that rca man.

A

Yeah exactly yeah rue's cost analysis. I have immutable, mutable field became immutable and all hell broke loose.

B

A

The notification came through, so that's cool. Let's try deleting the app right. So let's do okay, okay, get app and then it's okay delete.

A

I wonder what on on delete, so I wonder if I delete the app or if I delete, let's just try deleting the app how about that find.

B

Out, let's find out the hard way um I hate ourselves. No rcas are amazing, I love rcas. Why do you hate rcas? You- and I have to talk about this- that's a problem.

A

Yes, you should never hate an rca. Yes, you should never hate an rca. So let's see here so I was I'm wondering because um how would it note the fire off? I think I think I'm misinterpreting the.

B

Yeah, there's probably it's probably uh something deleted that that argo actually manages.

A

Let's do yeah, that's what I'm that's what I'm thinking too, let's go to. uh Let's change the subfield to false. Let's do that um because we don't want it to self-heal. So, let's go to uh this time I'll do and apply.

A

I will create it too. If it doesn't exist. That's another thing about apply. um Yeah.

B

A

So, let's do uh that should have yep fired off another one. The little red line tells me and then let's do get pods.

B

uh Yes, we have to add time stamps, but uh we have a very shiny feature that audits everything we do on the cluster. So that's that's a copy pasta exercise.

A

There you go copy pasta from slack overflow stack overflow.

B

A

Got here uh kgb pods, um so that's running. Let's do um uh so, then. What I did have here said undeleted right. So how do we, I guess, just delete the namespace.

B

Yeah delete github's happy hour right because that's the object.

A

Well, the the namespace that it of the applications created here, so I did that and that okay, so that's the sync.

A

B

A

You have to actually.

B

Delete the github's happy hour thing because that's the thing you've defined in your.

A

Oh here we go. Oh I I it messed up, though it said, wait, uh sync operation. It said out of sync somewhere no value successfully, sync successfully synced synced out of sync sync status. Right now, it's out of sync.

B

A

So send a notification, one is out of sync, but it says.

B

Look at your config: what did we tell this thing to do.

A

Yeah so here so it is out of sync. So let's look at the config here. This is subscribe on deleted. Slack get us happy hour.

A

Yeah is it on delete when argo cd deletes it? What says out of sync, and it did send a notification? It's just not the notification. We wanted um a bonus, that's that's the bonus. Well, at least you got notified um huh see here. What did we do wrong here?.

B

All right, yeah, that's right slack is that's the slack name in the in that thing.

B

I don't expect that to to be the slack channel name when I read it.

A

Or this right here, yeah.

B

I don't expect it to be a section, it's not natural to me to expect that I'm expecting it to be like uh okay. We want to do this when we do to this thing right, just just because of the way it reads, but yes, obviously duh. You explained that earlier, I'm just not awake, she's, not awake.

A

You need the postmate coffee.

B

I need that postmated coffee yeah, I do exactly rca- is a company. uh Yes, it is. It.

A

Is it is a company rca wow? I haven't heard that in a long time, okay, um someone said try doing on sync failed.

A

Then we would have to how would how would why? Okay, I mean we could. I just don't know.

B

How to get links to these documents? Let's go pull this up too we're learning together. I love learning together, I hate learning alone, I'm not good at it.

A

Look at that I'll sing failed. Let's do prune equals false as well, so.

B

See on delete when.

A

B

Deletes it, I would have thought.

A

Yeah, I would have thought that as well.

B

uh What is it argo, cd, notifications, docs? That's what I'm googling.

A

Yeah on sales, okay, and so then.

B

I'm like trying to look at your thing subscriptions to figure out how to catch up.

A

Status is synced, okay, all right.

A

How would I, uh let's, I think, it's like the config map I have to uh so. Let's look for oops slack. Let's do slack.

A

It's uh was that r2.

B

No, it's a chocobo.

A

Oh okay, sorry, my.

B

Ringer is apparently on, I usually turn it off during work hours. uh So, oh, oh, my ringer is off. That's that's a prioritized contact who gets to bypass all silences that makes sense.

A

B

um Yes, uh my my text message: chime is a chocobo chirp from final fantasy and my ringtone is uh the axle theme from kingdom hearts. 2.. That's.

A

B

So how nerd is hillary like, let's just like rank this from you know, maybe omega.

A

B

And I guess it's alpha right.

A

I guess it's alpha. That's right, unlike johnny, um so.

A

B

Never going away that joke! That's never going away.

A

So it says here it says: white check mark it says exclamation for is unknown cut out of out. I think it's like in this config map that we have to set. I just don't know white.

B

A

Is now running version blah blah blah.

B

The hard thing about trying to speed read for the piece that you missed in this situation is that it's just everything is like wall of text. Have you seen that there's a thing that went around? They called like bionic uh type type setting or whatever it's like the first three letters of every word are bolded to help.

A

B

Continue to work through.

A

B

A

Yeah I've heard of that yeah.

B

Yeah that actually works really well. I love that. I think technical documentation should implement that because it really can get like eyes glaze over. What did I just read: yeah.

A

It's just like a wall of text and you're just yeah. I.

B

Was I was trying to figure out a way to programmatically? Do this to my documentation. The other day I was like, I could probably write a script that does this I'll. Do it in pearl.

A

Yeah I'll do it in pearl and it'll accidentally work.

B

It'll accidentally work I just just, but that I mean that, isn't it like pro slurp already, that's the first half of this drift right, just ingest all the stuff.

A

Yeah do everything so here it says synced uh and.

B

Then the rest is regex.

A

And the right is just regex. My regex makes more sense than you do um talk about a nerd, a nerd burn, uh see sync synced. It should be an out of sync pretty soon. I would think.

B

I don't know I've lost it at this point now I'm trying to read this docs and catch up.

A

Yeah it well, it's it's uh it's hard to read, because it's a green exclamation point yeah.

B

A

A green mark so that doesn't isn't help isn't helpful at all.

B

Examples customize.

B

Oh, did you see this thing with customizing argo, cd notifications, that's cool.

A

uh No actually interesting, it's.

B

In the troubleshooting section of the docs.

A

Maybe I should look at the troubleshooting section.

B

Yeah, that's what I went to um and then I found customize on accident. I was like whoa nice uh good thing about path. File containing obviously, notifications. Big map, if not specified the command loads, are good notifications, the secret great okay,.

A

This triggers some.

B

Examples: okay.

A

I didn't even know there was a cli interesting.

B

A

Do I have it on maybe cd?

A

No, I just have autopilot.

B

I do like jeffrey's theory, I think, he's right um since we're manually deleting the app the undeleted won't fire since argo syncs, it back so it fires on sync. Instead,.

A

Never delete itself or undelete, it won't fire, oh interesting, um but what I so let me replace this here. Do it again.

B

That might be why the out of sync fired instead. Is that that exact theory, oh.

A

Manually, the app the undeleted won't fire yeah.

B

A

I'm thinking too so if it fires on on sync, fires are going to do the same yeah. That does make sense. It.

B

Does that that tracks the logic checks out.

A

Okay, I wonder what, if I just do this, let me comment that off and then do a replace and then let's do get app cargo cd, so synced all right, beautiful! uh Welcome!.

A

And oh yeah see what is um so. I only have on deleted all right that tracks all right. Where do we have where's my there? It is uh so it's out of sync. It's interesting that, but whatever um it bugs me, but because I don't know what's wrong, um let's do an app uh sync.

B

Oh, I just found this. There was. There was a question earlier about pagerduty right, um so argo, cd notifications controller serves prometheus metrics on port 9001.

B

So if it serves prometheus metrics, then your alert manager can scrape it and it can do page.

A

Integrations that way, oh that's, pretty cool yeah! That's pretty cool.

B

So I did not find the answer to what you're trying to do here. Truthfully. You've lost me, but I found that.

A

B

Oh, the web terminal operator thought we showed that off yesterday on ask an openshift admin. Yes, if we had the web terminal operator installed, uh we could have looked at the the things.

A

All the things, okay, all.

B

A

All the things so, let's delete uh the name space here uh and see if it fires.

A

I wonder like I wonder if you want to subscribe, so what I had what I didn't find- and maybe I should uh um maybe we should look at here about um the number of subscriptions right so uh so here subscriptions.

A

A semicolon separated list of recipients. Okay, so you can do multiple channels.

B

A

uh Multiple channels in in slack, for all application, obviously by adding the same annotation to project crd, okay, okay, so you can do it for all your applications and you only need to annotate the app project that it lives under.

A

um What did I'm just wondering if you can have more than one notification uh set up here, so I deleted the.

B

You have to have more than one notification. Surely.

A

Oh surely yeah like I'd, be surprised if it didn't I'd, be surprised and mad if they didn't um so, health status is missing out of sync okay, so then it's or the actual get. uh Let's try. This okay delete uh oops, delete app, let's delete and let's just see, if something fires, I don't think so, because the annotation is on the object that we're deleting. So how does argo? No um and it didn't send a notification?

A

A

So we only got on sync work, so we got one thing to work, so that's cool um yeah. I think I'm gonna try. I think I know we're. On top of the hour. I'm gonna try to think one more thing here.

B

So, hang on there's a.

B

B

Nope, that's not helpful, that's not what I wanted from you. Documentation.

A

uh So here I have onsync succeeded.

B

A

To follow up on twitter, we'll.

B

Have to figure this out before.

A

B

We're like, oh, my god, we figured it out. Look how uh how great we are.

A

Hey look: how awesome we are it.

B

Took us 40 minutes, but.

A

We did it, but we did it. I actually I've done that before.

B

A

Yeah, where it's like, I finally figured out and check it out on twitter.

B

It's something silly we're just not seeing it because of the like.

A

Yeah shoulder surfing shoulder.

B

Surfing exactly.

A

B

Trying to type when it's really important and people can see your screen, I can't type with an audience. My typing is just garbage with an audience by myself, though, I'm like nonstop.

A

Yeah, like um you're flying right so.

B

A

Okay, so let's do here and let's do sync where's. The sync sync happened that shouldn't have said anything all right: cool, uh let's delete uh namespace and let's just see if this fires off something else, I'm.

B

A

Something else for the for the last ditch thing: yeah.

B

Sure the last room can't end until it's done and committed right, yeah.

A

Yeah, that's exactly exactly it! I'm trying to try to make things.

B

A

That's right because it'll take me forever to figure this out. um Okay, so I have successfully got it to notify you when it's successful, but it looks like it might be so like if I do.

A

Status, no no status uh list yeah out of sync. This is out of sync. Health is missing and they didn't send anything.

B

A

um This would have to be a a twitter update afterwards um because uh we're at the top of the hour- and I have run out of ideas. So um I.

B

Don't know the documentation's not helping me a whole lot here, I'm like I. I need like what what what what thing are you expecting like clearly like there's something I'm not passing you that you are expecting to see in order to do that.

A

B

But you don't you're not telling me what you want, so I can't give it to you. That's that's where we're at with this.

A

Yeah so then it's um so.

B

The question is: why is your app out of sync with argo right now.

A

I why is it out of sync, because I deleted the namespace and so argo? um I basically deleted all the object, argo's keeping track of, um and so that's why argo said it's out of sync: it's missing because that's right, I deleted it. um So, let's try to. I was trying to force an error here um so cool.

B

Yeah, well, you would have thought if you deleted the name space that argo's looking at and it's not there that are gonna, be like your namespace is gone. But I guess.

A

Yeah exactly it would just.

B

Know it's gone not that it was deleted per se.

A

Yeah, but if I delete the object.

B

Okay, so what else.

A

B

A

B

Not looking at the screen, what all things can you delete? Nargo.

A

Go ahead and see here or.

B

Delete that argue would be watching.

A

Yeah, so the I mean I could so william did say like if, like I removed, maybe if, like the repo, wasn't there the git repo that could be like it as well, um but I'm not gonna do that. I mean I guess I have a local copy. I guess it doesn't matter, but um it's just a dumb, app anyways but the the uh I'm trying to think.

A

I think it really has to do with the configuration on the config map, there's just something that maybe I need to add or something that you need to add, which is why um good documentation is important. I think if I did a if I looked at the app health degraded portion of documentation, I could probably figure that out, but me sitting here in silence. Reading by myself, isn't um I mean it might be? It might be interesting to people.

B

Might be everybody's idea of a good time, it's yeah.

A

Exactly that's exactly my idea of a good time: yeah yeah! I was like.

B

I'm actually on the pager. So if you've seen me pick up my phone every now and again, there's a certain chime. That means something has happened on certain work channels.

A

And I'm double checking to make.

B

Sure they're, not the ones I get paid for.

A

Yeah yeah exactly yes, also in real life, so it's like uh sre camera right, sre, uh um sre, vision. What.

B

A

Day in the life of sre is like looking down on the phone making sure.

B

I have to keep that thing on me. I can't be more than 15 minutes away from my computer at any given time during my own call shift. So nights and weekends mean that my phone is does not well, it does go into silence. Pagerduty can bypass my silences and the ringer is turned all.

A

The way up, um I will say I've.

B

Only been paged at a weird time, one time it was 1am on a saturday.

A

The tecton thing it.

B

Was not the tecton thing that was terrible, though the tecton thing happened at like 4 p.m, west, coast time on a friday, while I was.

A

Trying to leave.

B

For vacation and.

A

I was actually.

B

Not on call anymore, but what happened was is the person who was on call was in ireland, so it was the middle of their night and it was still technically my work hours. So I was trying to take it over before the pager woke them up.

A

B

I already knew what was wrong and why. It would page yeah.

A

B

It didn't, I was unsuccessful by the way.

A

B

Woke him up and he got in and fixed it.

A

Oh okay: well there you go what all right.

B

A

Sometimes yeah you have to to too slow on the draw here so yeah um yeah. So I don't think there's no um other questions. um I would. I would say that um now that argo cd notification is part of argo cd, you can go and uh set up uh set that up for um you know for for pager duty.

A

I guess you could be like hillary and be woken up in the middle of the night uh when your applications out of sync um or just for just basic metrics right, you can set it up for basic metrics. There.

B

So that's true, your pager duty stuff does not actually have to wake anybody up. You can just have it exist um or, um like I said, I saw that it. It sets the uh the prometheus serves prometheus metrics, so you can do things with those prometheus metrics. So that's always exciting.

A

Yeah, it's it's pretty cool because, like certain notifications isn't as simple as on or off or black and white like it has to be like threshold right like if it's within a certain threshold from for a certain, you know amount of time, then send me the notification.

B

A

Like if it's something's like flapping on and off on and off like okay, maybe send me an email, but don't wake me up for that. You know yeah.

B

I don't know it's it's one of those things where you have to make your best judgment based on your software right, sometimes if an alert is flapping. That is this. That is the biggest sign that something very very terrible has happened if cd.

A

A

B

um So and there's the api server comes on and off as well, so uh yeah honestly, I I would rather be woken up for something flapping than for something happening once. Oh.

A

Something you can prevent. That's very true. That's a good point! That's that I'll, amend that, because yeah because like I would rather be woken up with something that I can fix right away and go back to sleep then be woken up and then just having to stay up, because it's it's fubar yeah, it's fubar.

A

It's definitely fubar all right, uh so yeah! So it's been um again. It's been a pleasure uh to be to be on the stream to be your captain. I guess in this guide, um but you're you're in good hands with uh with hillary um she's. um uh She.

B

A

Yes, she's continuing the show and she's gonna actually have some pretty good ideas. We're actually spitballing some ideas. um So this is my last stream as a host, but I will come back as a co-host. I am I'm threatening as a threat. I don't know what you call that.

B

You're, coming back as a guest.

A

As a guest, I'm coming back as a guest I'll be a guest. um When I come back next, but um it's bittersweet, I love um you know. um You know I love doing this. I'm not be going far check out my twitter um or my um my linkedin.

A

If you're following me there to see where or I'm ending up not too far away, so um I'll still be in the get off space um and um you know you'll you'll, you know you can follow me there I'll still always always a red hatter, I'm keeping my red hat right, that's not going away. I.

B

Spoilers, yes, actually dude. Actually you can just come on we'll do it together. It'll be great. There.

A

You go william.

B

Exactly what I'm going to say as well.

A

So uh so yeah so um yeah, so they say uh once a red hat or always a red hatter. um You know um well we'll uh um it's a bittersweet, but um you know exciting things are are coming up, so I don't know if you have anything else, hillary uh to say, do you want to see us out? Do you want to um do you want to do you want in the show on your quote now since I've always, uh I always send it. I ended it on my quote.

B

Oh man, I'm not sure if it's, uh let's have you one one last time one.

A

Last time to see me yeah, yeah, so I'll tell you the situation with get ups right, just like I always say unless it's in it's only a rumor. Thank you. Everyone appreciate it cheers.

A