Red Hat OpenShift GitOps Guide to the Galaxy | Red Hat Livestreaming, 7 Apr 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitOps Guide to the Galaxy (E35) | GitOps with Keptn

Description

Dynatrace talks about Keptn (https://keptn.sh). Keptn – pronounced captain – is a control-plane for DevOps automation of cloud-native applications.

Keptn uses a declarative approach to build scalable automation for delivery and operations which can be scaled to a large number of services, built on the concept of SLO and SLI CI/CD.

Try Red Hat OpenShift: https://red.ht/3sMaW7O

Follow us:
Christian Hernandez https://twitter.com/christianh814
Hilliary Lipsig https://twitter.com/Int2Caffeinate
OpenShift https://twitter.com/openshift

Subscribe to Red Hat's YouTube channel: https://www.youtube.com/redhat/?sub_confirmation=1

What is GitOps Guide to the Galaxy?
Every other Thursday at 3pm ET host Christian Hernandez and guests dig in to discuss everything in the GitOps universe, from end-to-end CICD pipelines to creating Git workflows.
Learn GitOps https://red.ht/3saqlzS

About OpenShift:
Red Hat OpenShift is an open source container application platform based on the Kubernetes container orchestrator for enterprise application development and deployment.
OpenShift on redhat.com https://red.ht/3LGCIeD
OpenShift on GitHub https://github.com/openshift/

#RedHat #GitOps #Kubernetes

A

Foreign, let's get into the big topic of open source, it's something that we.

B

Actually, have.

B

Actually, is that.

C

Process that a developer or let's say.

D

Has the kubernetes ecosystem really boom.

A

And we are live. Welcome everyone to another, exciting episode of get ops guide to the galaxy. It's a very special episode today. It's very I'm very exciting, very excited very happy uh to introduce our co-captain. We get, we finally get a co-captain, so so hillary I'll, oh wait! You are on this side of me, um hillary I'll. Let you introduce yourself kind of um yeah to tell everyone who you are what you do at red hat background, all all that fun stuff, I'm pretty sure we'll talk about your shelf as well.

C

Oh yeah, my shelf right, everybody likes my shelf, um so hey guys, um gals and undecided. uh My name is hilary lipsick. I am a principal site, reliability, engineer and global team lead um for uh some of the one of the managed services uh teams running on openshift dedicated.

C

um So if you've heard of our offerings of uh manish kafka or red hat open api manager, uh those types of things that is uh the team that I am leader leader of, um let me see I've been at red hat for about- I don't know year and a half now ish um and actually prior to being an sre. Here I was a quality engineer for 10.

A

11 years, yes,.

C

I like to say that I dedicate my life to writing software that impersonates people. Basically, that's what I do.

A

I think it's a very, very apt description.

C

More or less uh god yeah, so that that's really that's really it! um You know at my time, outside of red hat, it's still technically part of red hat. You can find me at the whit network, where I run a community of that nonprofit for women in software engineering um and are actually kicking off this. um This month.

C

We're kicking off our coding coach coffee hour, uh which is specifically uh an effort designed to help us particularly women, but anybody uh do well on code tests to get past the code test and just in general upskill as engineers.

A

Yeah, I actually just dropped the link into the um into the chat, so for those of you who aren't familiar with the wit network, um it's uh all the cool things and everything that hillary just described, probably more in detail in the in the link than than than I can. Then we can spend spend the hour on. So please, please check that out. It's really really cool um and I'm very excited I'm very excited to have actually someone on who, like actually does things right.

A

So a lot of us at red hat, um a lot of people that I bring on you know, solution architects, um people here in the bu we like we work a lot on theory, so I was really really excited to bring an actual sre onto the show and be part of the um the you know the the permanent hosting here. So we finally have a co-captain here at um genops guy to the galaxy. um You know I've worked with hillary in the past she's.

A

She you know she's she's, more technical than I am I I guess, if that's even possible now, um I guess now being in the bu is like they say: that's where all engineers go to die, I guess is, is isn't the bu so being there um is. uh um You know I'm very excited to have uh someone who does this from day to day so um so cool. So welcome. Welcome hillary welcome.

A

um So if, if you guys have any questions um um again feel free to drop it in the uh in the chat, we'll be happy to to talk about it here, so I'm excited for our guest. Today, it's our guest today, um you know we were kind of hillary. We kind of decided like well, you know. When should we bring you on now? You know we actually made a decision a while ago um about you know who the permanent co-host is and heather he's like. I.

C

A

To jump on this show because we're talking about slos and slis, and that's like your jam right, that's like you is is, is what you love doing, um and so um so yeah, so our guest today um they've been kind of hanging out in the in the green room, so I'll we'll bring them on. So if stephanie the producer can bring him on is from dynatrace right.

A

So um you know talking about so I used. I I used to pronounce it uh ketten right so like trying to pronounce it like the german way, but you know you know andreas told me it was uh it's actually captain so um talking about captain, so um so andreas or andy, I guess um oleg I'll. Let you guys introduce yourself, um you know who you are what you know. What what are you working on? uh What you do all that cool stuff.

B

Yeah, oh like, shall I go first or do you want to go first.

D

B

Okay, so well thanks, first of all for having um us on the show uh thanks hillary for doing the great work that you do in your wit network um and yeah. What do I do? I I feel a little connected with you hillary now now, knowing that you have worked in quality for uh about 10 years. I think my my screen just went black, and now I came back. Hopefully I'm still yes,.

A

Yeah, I know we can see you yeah perfect.

B

um Because hillary, my background, is also in software quality. um I started my career. Yes,.

B

Because who needs software that doesn't work well, so this is why.

A

Quality is very important.

B

You know what the fun thing is. I started my career as a software tester on a testing product, so I.

A

Did performance.

B

Testing on a performance testing tool back, then it was called silk performer. It was uh when I started the company was called segway, then got acquired uh by borland and then became microfocus, and I think now it's part of I don't know, um but I was a performance uh test on a testing tool.

B

uh I did that for eight years and then 14 years ago, almost to the day, I started with dinah trace and there I'm kind of you now now I call myself a deaf rel or an evangelist or whatever you want to call me and try to help people understand what we can do with the data that observability gives us, especially as we are using it to drive automation around deliveries or making decisions on whether the quality is right, based on metrics to push stuff from there all the way into prod and then use data to make better informed and automated decisions to bring systems back to a healthy state to keep systems reliable.

B

That's kind of what I mean yeah, that's about me.

A

It's like it's like what don't what haven't you done? um No.

B

It's just, I think it's a very small thing, but I've done this consistently for a long long time, um but I don't know a lot of things that I should know, but I'm happy that I know what I do know.

A

Awesome, oh, like oh, like so, uh go ahead and uh give your quick introduction, or so.

D

The test guild uh actually my background yeah. I also started from quality assurance, but quality assurance in hardware and embedded systems. So first five years of my career to be rather about testing devices, then I started more hardware development, mostly digital hardware. I have a phd in hardware design for what it was, but uh yeah when you do quality assurance, you need automation. Hence you need tools. So at some point I started using hudson jenkins and you know at some point: if you fix something qci, you become a ci guy.

D

So this is exactly what happened to me. So I started making uh hudson then jenkins. I started contributing to open source, somehow ended up in a junkies core team. So if you use the jenkins in the last decade, probably you have seen my name on jira on github and starting from 2019. I started following captain because it's a really nice project.

D

It has been exciting to follow it, and this summer I started contributing a bit and finally, I joined the dinan race to help with building a community around captain. So now my job is rather community management and whatever is related. I uh call myself uh wanna be captain obvious and yeah. This is a part of my job.

A

ah So you know, what's really really cool is that we we now have like talking about like qa. We have like qa from from, like front-end engineering, all the way to like embedded systems so like, if you guys so those of you watching. If you really have questions about testing almost anything, I think this is the stream uh to do it in so yeah.

A

So speaking of captain, so what's interesting is that um you know when I first thought about bringing you know when I first heard of captain and thought about like this, I want to bring someone on um to talk about captain and then you know I jumped on a call with with andreas and oleg, and it was completely different from what I thought it was right, and so you know they they kind of give me a good education and good um good background on it.

A

So I think um I think maybe we should just start there right. Let's just let's talk about what captain is, and you know, let's, let's start the the exploring process there. So um so I think I'll hand it off to to you guys, andy and oleg, to talk about. First of all, what is captain.

B

Yeah uh is it okay? If I quickly share my screen, let's do it yeah, yeah? Okay, let me figure this out screen number two I'll move this out of the way, because you know you know how you look like.

C

Yeah, I'm just.

B

I'm just using one or two slides, I'm actually, I'm very fortunate to to be able to travel again. I mean keeping fingers crossed that on sunday get on the plane to north carolina, because I will be speaking at uh devops days in raleigh next week and I I think I want to just use- maybe the intro slides, because also there uh there's three things I do in my talk. What is captain why we build captain?

B

I think that's also an important piece, because why yet another tool, maybe you think, there's already a solution out there and then how captain works? I really just want to get the first uh question addressed what is captain and with the segway also into uh the whole topic of slice and slos, so you're, the first ones now to actually see this like this deck. um Oh we're.

A

B

A preview awesome.

A

Exactly but don't tell anybody all right.

B

So what is kept in a nutshell: um captain enables you to do on the one side, multi-stage delivery. Captain itself is an event driven orchestrator of what we call sequences of tasks. The first use case is for we call them.

B

I think the devops engineers that are reliable or responsible for pushing artifacts through a pipeline multi-stage, where captain can orchestrate tasks like deploy, test, validate and then release into the next stage, but captain is not just yet another tool that can push stuff through a delivery pipeline, because the second use case is around orchestrating day one and the two operations, and for you know the way I define it. This is everything to make sure that uh you can successfully.

B

You know, release your canaries so making sure that the systems, the canaries get scaled out if it's good or scaled back, but also when there's a problem in production, it can automate the orchestration on.

B

Let's say you know, fixing a problem, so it's assessing the situation and then is triggering based on the problem uh that has been identified is then triggering the right actions and always validate if the action brought the system back to a healthy state, hopefully that's it, it can go through multiple iterations and eventually, if it solves it good, if not, it can escalate now the key magic piece here- and this is, I think, one of the interesting topics we want to talk about today- is that embedded and kind of baked in at the core of every sequence.

B

It orchestrates captain makes data driven decisions based on slos service level objectives and we'll go into more detail on how we treat slos and how we defined it, because I think there's a lot of people that talk about slows and everybody has a little different understanding of it. So we'll show you how we do it most important thing is that the data obviously comes from your observability platform of choice.

B

The first integrations we have built is obviously we're a cncf project. So prometheus was the first integration. Then we built the dyna trace integration, because most of us contributors are from dynasty. We already have a data dog integration that one of our community members built and there's more out. There integrations that people have asked for now.

B

On top of this is all of the tools that actually do then the individual job, because cabin itself doesn't deploy it, doesn't test, it doesn't roll back, it doesn't scale, it doesn't notify it really orchestrates all of these tools and it does it through an open event standard. What that means is we're really taking away the pain for you to have, to you know, write lengthy automation, scripts that call the tool for deploy then validate the deployment happens successfully then called test veritative test was successful, then reach out to your observability platform and figure out.

B

Your s line is the low status and then do the next thing. So we take care of the whole orchestration, we're using an open event, standard that also oleg special is driving through the cdf foundation, the continuous delivery foundation. So we standardize on a way we communicate with these tools and at the core after every task, captain makes an informed data driven decision to figure out how in which direction the process should move.

B

So this is kind of captain in one slide, but I think this is important to understand uh kind of what the use cases are and before I kind of open it up. One key point, and I think this will will help us drive the conversation around slos. What really happens there is every time captain has this evaluation stage. There's a core component in captain that we call the lighthouse service.

B

The lighthouse service is driven by your slos, so you can define slos in the ammo that you store and in the config repo that captain holds for you, I mean for every captain automation project as a git repo, so you put in your slo where you specify what are the slides? What are the metrics like you? Can you can see here, error rate, jvm memory, count number of database calls and what the objectives are, but captain itself is not with is not or the lighter service doesn't do anything with this.

B

Yet because the first thing it does through the same eventing mechanism, it's reaching out to your observability tools and is asking them to query the individual value, because in the slo we have a separation of concerns. Here, the slo just says: error rate. How that error rate is captured is depends well obviously on the observability platform.

B

So for prometheus, you would specify an sli gamble with your prom ql for dina tracy would be our manager with metrics expression, language, and so that means that the we call it the sli provider reaches out to the observability platform pulls in the data, puts it pushes it back to the lighthouse, and then the lighthouse compares the values that come back against the criteria.

B

We are evaluating them every single metric, every single solo and then on the very top. We calculate a total score between 0 and 100, and then this is then the deciding factor on green, yellow or red. That then decides how the automation process goes on, so this was kind of all of the slides that I wanted to show. But hopefully this gave a little bit of an overview of what captain is and and and why these slices and slo's are so core to captain.

A

Yeah, I think, um and if anyone, if you have a question, go ahead, feel it for the drop drop it in here. I think, out of the three of us, I think I'm the most ignorant one, um because my my background's, mainly in operations right and so I've. um um You know the the importance of of data-driven. The deployments right, I think, is what captain is trying to focus on um and so, um and actually we just got a question here from rockhound. It's a rockhound welcome um a ci cd rapper of all other solutions.

A

Question mark with a test dependency at the end is that is that what.

B

I guess I I guess I did a crappy job in explaining what captain is because if the first thing that comes to mind is it's yet another cicd tool wrapper, then I I really may have you know, missed the point. Captain is not another cicd tool.

B

What captain is it is helping you to orchestrate the tools along what we call a sequence, an automation sequence and yes, obviously, we always bring the example of pushing artifacts through a delivery process, because that's why you know I have to deploy test and validate here, but the other big use case is for handling problems in production. So if you are prometheus raises an alert, if you get a prometheus alert that something is wrong, then you can also orchestrate all the remediation tasks where you can say.

B

I first want to notify my sre team that there's a situation going on in the same step, then looking up a list of sequences that should be orchestrated kind of like your run book based on the problem that comes in and then captain is then triggering your individual tools. And now the thing is, I: I really have a hard time with the word wrapper, because what we are doing here, we have uh it's an event-driven system. That means there's no hard-coded tool, integration that you build. Where you say in my particular project.

B

I want to call jenkins or I want to call helm to deploy, or I want to make a notification call to slack it's all through event subscriptions, and maybe that's actually another good point.

A

I would like to, I would like to think of it as so, and this may be a really simplistic view um from my point of view, I really like to think about it as almost like an event bus right, so.

C

A

There's there's there's certain events that happen right and you want certain things to happen based on those events, it's just it's just so happen. It's just sitting. You know at I guess at the heart of your deployment.

C

Yeah, if I think the powerful thing here is that there's essentially a rules engine right so.

D

C

Not just that there is the the messaging and the events going through, which is super important, but that you also have a rules engine so that that is the power. In my opinion here.

B

Exactly I think, that's the power and the other power. Is there simply it's it's loosely coming like what? What did we do with all the monoliths that we broke into smaller pieces? uh We in the end, you know we stitch them together through an event bus that also has a rule engine. You typically have like your business process management, where you then say I know exactly what needs to happen, but you can always replace individual pieces because you just subscribe another tool to handle a certain piece of activity and that's the same thing.

B

What we do here right if you look at, I have a multi-stage sequence here. It starts in staging and then in production, and what you see here is we call, I call it delivery here. This is the delivery sequence and I have multiple tasks. Monaco stands for monitoring as code. This is where I basically say I want before. I do anything make sure that my monitoring system is configured now, in my case, maybe for my project I use datadock.

B

So I have datadog subscribe to the monaco event or I have another tool, let's say dynatrace that subscribes for it and then does the configuration or it's prometheus right I mean, basically, you can, through event subscription say which tool should handle, which type of task for which particular project in which particular stage and the events that are sent here. So you can, you can see the little icon here and I'm just pushing on the the deployment.

B

This is the events that we're standardizing. So this is not just yet another proprietary, uh let's say protocol that we're introducing here, but we're really working with the community to standardize this, because the challenge that we have seen is that every tool out there, whether it's deployment tests, observability tools, notification, everyone has their proprietary api. This is also why it is so hard to build and maintain integrations between all the different point tools and we have introduced or introducing here a standard way of how these tools communicate and.

A

They almost want to take away yeah. It's like um you know, like you know. For me, my world is argo right so like. If you have argo right and then you have information coming from dynatrace since you guys are on um right, like I have like the dynatrace, you know feeding those metrics to me like. How do I make argos speak dynatrace, and how do I you know and and like instead of writing, that integration right there's like an event bust, that I can basically loosely couple.

A

You know what my deployment mechanism is. Is you.

C

A

uh I guess decoupled from the the information the metrics that I'm receiving.

D

You're totally right, so there is a standard technology called cloud events which is exactly designed for delivering various kinds of events and what you see here, captain and many other tools actually built on the top of it as producers or receivers of events.

D

There are many standards like cd events, which was referenced before by india, which actually extend this cloud event specification and allow to standardize particular areas, but ultimately, it's a universal way of did inter exchange, which is extensible, and if tools know how to speak to each other, it unlocks a lot of possibilities for your systems because, as we build modern applications as microservices the same for delivery and operation services, we also build them as microservices loosely connected with each other.

D

Through events, and this actually communication layer enables us to build that efficiently and to use many building blocks like captain to assemble intel operation system. You need for your applications.

B

A

Yeah, I wonder um so so you're saying you can also use this information for rollback right.

B

A

If, if you know, I guess you're monitoring your system, the information is coming that this latest deployment broke, something that, for whatever reason- and this will make hillary cringe past in um past, it passed qa for whatever reason, and but it's so like something in production is breaking. um You know you can have those same. You know rule set, I guess hillary. I think you, you said that um I think the the best way to say that is you have this rule engine that can then roll it back to a previous version.

B

So the exact so the way maybe- and I also see some comments like- is this just another? If then, do that kind of tool for cloud ops, I I guess I still need to do a better job in explaining it's, not just if anything else. So maybe, let's have a quick look behind the scenes, so that people understand will behave behind every automation project is everything starts in the beginning, with what we call a shipyard file? The shipyard is really now this definition of the process.

B

What you see here is we are grouping or you can define so-called automation sequences like I have my delivery sequence for my staging environment, and here you specify your tasks. Monaco monitoring is code deployment evaluation we can have different tasks in a stage like here is a delivery sequence. That also does an additional test or here's just a delivery with test, and what we also have here is in production.

B

I have another delivery process where I say I want to trigger that delivery in case staging finished successfully or staging delivery with test finished successfully. Now I can also go down here coming to the rollback. What you're basically doing is you are specifying sequences and you connect them together, so a little bit like. If then else, of course, from a sequence perspective.

B

However, the magic. What I see here is you don't see any tool? I I don't care here in the definition of the process who is doing the rollback, because I maybe that defines that process doesn't know what's the target tool in my environment, and this is now where the event subscription comes in. So if I go back to my captain ui, we call this the bridge and if I go here on the left side to our settings, then we have what we call the uniform.

B

The uniform is kind of like all of the tools that captain has to steer the ship, and this is exactly where all this event subscription comes in. So, for instance, I have my helm service here. I could also have here my argo service, my jmeter service, my jenkins service and, basically, I say hey. I am now subscribing helm in my case to certain events for a certain project stage and service. That means I can control you through event subscription for to which tool this event should actually be forwarded and then handled, and this allows me.

B

I think this is the nice decoupling between process and tooling. I can easily switch tools as long as these tools obviously understand the event, and I can also subscribe multiple tools to the same event now. Why would I want to do this?

B

Well, because maybe I want to push notifications to a slack channel every time a deployment failed or a rollback is triggered, but the rollback itself is obviously done by another tool, and so I can have multiple subscriptions to an event and I I can easily switch tools because in the end I just need to make sure that I have tools to fulfill the process, and hopefully this explains it a little a little better, and you you tell me if I, if I did.

A

Yeah, well, I I think you're doing a great job. Also oleg has been feeding me links so for those that are following in chat. I've been posting some of these links um to find out. You know more of the information more more about these definitions here so uh oleg um he doesn't. um He was gonna sign up for a twitch account, but um you know I'm kind of feeding that you know he he uh he said. Maybe next time next time, you'll you'll come with your twitch account.

D

You just don't want to to see my twitch account.

A

Yeah yeah, so there you go yeah yeah.

D

And the maybe we'll rate it.

A

Maybe we'll rate it afterwards, we don't know.

D

I mentioned union form. uh Actually it's a fun fact about. Captain that we have a lot of special terminology comes from nautical terms, so from shipyard to captain itself uniform this web ui, you can see on empty screen, it's called uh captain bridge and.

A

There's a lot of.

C

Terms like that.

D

So that is internal lore in captain community.

C

I love that. I love that um I was actually wondering if you were going to mention um and andy. I was wondering if you were going to mention sending notifications to slack channels and so forth, because chat ops was going to be one of my questions um and for those not in the know, chat. Ops is sort of an idea of that.

C

When something happens, you get a chat message that you can interact with via slack and you can actually like the power is so so much so that even from like my phone slack application, I can see that chat. Ops come in and do things like run a a playbook based on based on the alert and then sometimes can be multiple options depending on you know, various certain pieces of data, um so that was going to be my question.

B

Exactly so that, basically, you know, I'm pretty sure actually can find captains like webhook.

B

Yeah I mean whatever whatever favorite, uh whatever favorite um search engine, you have, it's typically pretty pretty easy to.

B

To find all of your different integrations that we have and how we work uh there's also a lot of ex like jenkins is another example. Obviously oleg been very active in the jenkins community, everybody knows him there. uh You know how we can subscribe to um to captain how to do jenkins and send information over, but essentially what I I'm not sure.

B

If maybe it was a little too fast, but in captain on the integrations in the uniform one of the things we can do is we have a web hook service where you can select, for which task you want to call an external web hook, and every task has three stages triggered is when captain says I need this test to be executed.

B

Who wants to do it and then the tool that is actively doing it will then send back when when it's starting the job and when it's finished, but I can also subscribe as a notification tool perfect that if, let's say the evaluation finished, then I want to make a call to um to the slack api, and obviously we have secrets here right. You can configure secrets. Like your token, you will obviously put into a secret here.

B

You can define your custom payload and then you can pick any of the data elements that are in these cloud events that are that we've shown you earlier. That means you can obviously reference right, the dashboard link or you can reference that was used for the slo evaluation. You can also reference the evaluation result. The git commit the indicator, result the total score, and then you just push this over to whatever tool you want.

C

And so I want to touch on something else that you mentioned. I have so I have so many like things I want to touch on here. I think this is such a neat thing. Yeah.

A

This this is your uh your favorite subject, so.

C

Right like this is amazing yeah, I'm I'm, I feel like uh I feel very much with kindred spirits right now, um especially with all the joint qe background, like that's amazing uh but anyhow, um so there's something you mentioned, and I wanted to ask a little bit more about it. So, to my mind, as I'm evaluating this, um it looks like this technology's. Really, I guess I'll use the term. It's non-linear right. You don't have to just have like one set of things happening.

C

You can actually start spreading out through a branch of things which is is neat and I'm wondering kind of. um If you have any examples of that, if you have any strategies for doing that in a maintainable way, because the problem with making things spread out too far is, of course, losing track of them.

B

Yeah, so from a spreading out perspective, what we can do when I go back to my to my actually, let me give you another example here: the demo rollout example. That's I'm using argo rollouts here for uh for canary deployments.

B

um What you see here basically, is not not a spread out but kind of like a sequence of sequences. So I start in staging and I am delivering um a new version just in the regular blue green deployment uh using uh using uh our helm and argo rollouts to actually push it out like a blue green deployment. When it's when everything is passed at the end, so tests have been executed. I think in this case I'm using low cost.

B

You can see here another thing right where the testing tool is kicking it off and running tests. Then we do the evaluation.

B

Then I'm going over when everything is green to production, and here what I've specified is a delivery sequence that is then followed by a canary rollout phase, one phase two and phase three- and this is, if you remember earlier- and let me just show you what we the the shipyard file behind the scenes, because behind every project there is a git repository and in this case it's the delivery demo and here's my shipyard file again, so the shipyard file.

B

Remember that's the the initial definition, so you can specify, as we mentioned earlier, the sequences and then the way you connect them and kind of said spreading it out is through the triggered on the trigger all means. If one sequence is done, then you can trigger another sequence: uh it's not that you can then kind of branch out into multiple areas. It's always like one sequence. What, then, is what, when this is completed, uh either with a successful status or a failed status?

B

You can then say: are we done or are we kicking off another sequence depending on the finished results of the previous one? So this is how we do it. We like what you're meaning is completely branching off. um I mean it's also possible, but I think from to this extent, where you would say, one sequence is done and you're kicking off 50 others. I think this is not where we've gone so far, and I'm also wondering, um as you said, this would probably be.

B

I mean it's doable right because we would then show it here in our ui. We probably need to figure out how to better visualize it, even if it branches out completely, but this is not the way we have we've seen it been used so far.

C

No, that makes sense, um and I think that that's that's fantastic. uh You can always count on me to figure out some way to break your tool.

C

I right absolutely use it wrong. That's.

A

An sre sre life right like it's like I'll use, I'll use your tool wrong. This is what.

C

It is yeah, I I frequently say that sre is a type of quality department right, we're driving continuous improvement through the observation of of the production realities, um although I usually, I think say it better than that. But this is.

B

C

So this is what we get today um right. So um I wanted to ask you because it was a kind of a quick thing and and if there's other questions I'll see to the actual audience uh who were here to entertain instead of myself.

C

um But um I wanted to go back to um the the slo uh components and some of.

B

C

Definitions again.

B

C

Because, like I said it was just really quick earlier and I was like oh I want to. I want to spend more time there.

B

Yeah, so what you see here is kind of in the end, the visualization that we have right. So this means every time I run a sequence, and I have to evaluate a step in there. Then captain will reach out and to the monitoring tool that has subscribed to the give me your sli values. The values will then be brought back and then every value every sli will be evaluated against the slo the objective and then a total score will be calculated. So this is just the visualization of one of these. You can scroll down here.

B

You can actually see the individual values that came back. One of the cool things that I like is that we can not only compare or specify an objective based on an absolute value. So you can say I don't know.

B

Host disk q length should obviously be you know smaller equal than zero, because otherwise you have a problem, but you can also combine it like the here's, a metric called service, throughputs per minute, you can say I want the throughput to be smaller than 200, but I also want to combine it that it should not increase by more than 10 to the baseline and the baseline is then calculated based on previous results. So you can combine absolute objectives with.

B

I call it relative objectives, I'm not sure if there might be a better english term for that, but this is what it is now. How can you define all of this there's two ways how you can define it remember what is behind every project.

B

Come on it's a question.

C

For you, what's behind every.

B

Automation, project.

C

Oh, I I don't know yeah json code infrastructure. I've got a long list, I'm not sure what what leading question you're trying to ask.

B

A git repository so all.

A

Of the computer.

B

Repository all of the all of the actual configuration, because so far we only looked at the shipyard. All of the configuration then really lives in the git repository. That means I have my shipyard here, but then, depending on how many stages you specify so per stage, we allow you to then put in all of your configuration files, either your helm charts. If you want to use helm for deployment, I had low cost earlier, so I would put in my low cost test script here or my slos would also be here.

B

So, for instance, this is one of the of the samples that I run. I have my service called tnt. Aapl service um just quickly go back here right in this project.

B

I'm currently in my service called tnt, aapl service, the way they're structured is project stages, and then you have services where you can execute automation on this is why my git repository in the production stage, I have a subfolder called tnt, apl, svc and here are all the relevant config files. If I want to run automation for this particular service in this stage in this project, so for instance, here is my slo file and in the slo file.

B

You exactly have this configuration that you've seen earlier in the ui, so yaml, where you specify as an sli with a unique name. Svc rtp95 can also have a display name so that it actually looks nicer in the ui and not just using this acronym. So you can say service response time, 95th percentile, but here's the thing you can specify pass and warning criteria. We also allow you to change the weight by default. Every sli- and I have multiple here by default. It has the same weight to the total score, but you can change it.

B

You can say this. One here should have a weight of five, because it's five times more important. You can also specify if individual sli should be a key sli meaning if this one fails, the whole evaluation should fail. A good example would be if failure rate is higher than one percent right. I don't care how fast it is, then it should be considered as a failed evaluation. Overall, so that means you can change the weight, you can change and mark one is key and you can specify pass and warning criteria now.

B

The question is what is behind this here? What is behind svc, rtp95.

C

B

That is in a different file because it depends in my case I use dynatrace. I could also use prometheus or datadog and urelic, but in my case I use dynatrace. Therefore I know when dynatrace kicks in my integration.

B

It looks in the dynatrace folder and in here I have my sli yaml definition, and this now defines the actual query behind the scenes to query a particular piece of data.

A

ah Okay, so the um so again, I'm coming from this from I'm just a sysad, just think, I'm just a cis admin trying to understand this. um The payload is is, is um it's where the payload is generated, doesn't necessarily matter right, it's like where, where the information that you get it from and how you, basically, you you're, just basically passing the payload from tool to tool to make exciting for informed decisions.

B

Right exactly and the thing is, and not only that we're that's what we're doing in general, the orchestration of captain is, we are triggering, we are triggering a task and in the end, in the beginning, and the captain doesn't care who is doing the task. We just know here's a test that should be carried out and whichever tool is subscribed to do this task. Here's all the information you need to know the one side, there's information. If I go, let's say again to the deployment there's information in the deployment event that says hey. I need.

B

Deployment triggered so the task is deployment I want to trigger it. This is now part of the project delivery demo. This is for this particular service. This is for this particular stage. By the way I have some additional information for you. This is the artifact that should be deployed, so whoever can do a deployment. This is all the information that I give you now.

B

Additionally, in my case, the helm service is subscribing, so helm then says: hey I want to do it, so it says I'm starting the task, so captain can keep track of you know what's going on right now. Additionally, however, every service we integrate here also knows that behind every project there is a git repository and in that git repository, you have additional config files that might be there for this particular tool. So I'm going to the delivery demo project. Now, if I go to staging, then again, I have my sample projects here.

B

My sample services, like tnt, apl service, and now I see here I have my helm sub directory. So this is now where all of my help charts live, and that means this particular point tool can then get the configuration file, for instance from here.

B

So it's a separation of process and tooling, and the configuration that a certain tool needs can is also fully version controlled in the git repository that is organized based on the way you run your automation.

C

I love this so much what just.

A

I like that, that you're using get tea as well. So that's that's pretty cool yeah.

B

I just use kitty for my for my demos, because, obviously you can you can specify any upstream git I mean you can.

A

Yeah mirror yeah, but which.

B

Is we call this? We call this captain in a box, so we have some of our tutorials where you just run and install right. I mean this here. It just all runs. Actually, in my case on on my, uh I have a k3s here right and then the captain itself is a uh it's a container based event driven solution and it runs on kubernetes. So when you install captain initially uh you you install what we call the control plane, so that's kind of the core of captain, the the the orchestration engine right.

B

So, for instance, what do we have here? We have the shipyard controller that is responsible to actually control the workflow that is specified in your in your shipyard. We have certain out-of-the-box services like the lighthouse service. Remember, the lighthouse is the one that is then evaluating all the all the results you have the web hook service. You have the secret, I mean you see all this stuff right and then, in my case everything is on one machine, so I also have installed some specific integrations.

B

We have like the low cost service to execute locus test the dynatrace, the helm service. We have a job executor where you can just let captain uh deploy a container as a kubernetes job to do a certain task and then wait until the job is done similar to a github action. So this is all there, but the nice thing for me is I didn't. I don't need to care about. How do I call a particular tool and wait until it comes back with a result?

B

We've automated all of this. For you, the only thing you need to do is you specify the process and you subscribe your tools to the to those events and.

A

And that's it so we got uh an interesting question here from rockhound says: can I define an r back which to which steps are available for which repo commit user, for example, or how do I.

B

I think captain are back, and probably all of you probably know even more than I do about the status.

D

Yes, so there is an enhancement proposal for captain cap60, one nd screen shares, so basically it addresses user identification and authorization.

D

Currently captain started as a developer tool where the basic approach was just isolated through network and inside once you get access, you would have full access to in principle like an influx db, db works and many other projects. Now with captain growth. Of course, there is interest to have user model inside and the current state that we have user identification so through open id connect, etc. You can authentify users, but the permission model is not fully integrated inside.

D

So what you have now you have airbag for services. So, for example, you have a service which needs specific permissions, like data trace integration, service or, let's say gmeter integration service for performance tests, and these services can run with special permissions. So on infrastructure level you have some model and some resolution, but not on the user level. It's coming soon, maybe in uh well.

D

I would expect the preview in three months or so, but then it will take some time before all of this permission model stabilizes, we close all the gaps, because it's a long story, so maybe six to nine months and we will have uh user authorization support.

A

Awesome I actually threw that through that in the chat, so uh for those of you following the chat, I've been throwing the links in there as they come so um check that out sorry to cut you off hillary, but there was a. There was a question that came up so I didn't I didn't. I didn't want to uh uh miss the question that.

C

That's exactly what I was going to bring up was that question.

A

Oh okay, all right, gotcha, gotcha I've been.

B

Doing it for a little longer so.

A

I'm on autopilot so yeah and.

B

Isn't that one more, I think, there's one that talks about and you don't have to write any of the integrations. I think this is something don't you have to write the integration.

A

Like programming.

A

Well, rock count also said: the great thing is that all event driven base product that it's all event based driven products, so it just.

B

A

Care you can just do anything as an event.

B

Exactly the only thing, obviously you need to like, for instance, and this is the reason why we have like, let's say a dynatrace service here or a low-cost service.

B

We are sending events, you can subscribe to events, you will receive those events, but there are certain tools that might not be. You know that they don't have the capability right. So if we give you, we give you a couple of options. The first option is, if your tool that you want to integrate has an api that we can call for a web hook, then you can use our built-in web hook service and that, just as I showed earlier, you can subscribe here to an event, and you can say hey.

B

You know what I want whenever a test is triggered. uh In my let's say staging stage, I want to make a call to my testing tool and here's an api and it it reports back when it's done. So that's one option: if you have a tool they want to integrate and it doesn't support these subscribing to the cloud events. We haven't understands it natively. This is one option. The second option is, we have the so-called job executor service.

B

That's the one I mentioned earlier. The job executor service and I.

A

Was gonna ask I was I was gonna ask about that? Is that, like a job for like a kubernetes shop, particularly.

B

Okay, cool actually, sorry, um because.

A

I was gonna say that would be my catch. All right be like if.

B

I can just create.

A

A kubernetes job like I can wire anything. That's.

D

Exactly that's exactly silver bullet of automation, so captain itself is orchestration engine. But if you sure you need automation, you have an option for that.

B

Yeah and then here you can say well again through a config file, you put in the in the git repository that again behind every project, there's a git repository and then you put in a a job.config.yml file, and here you can say if this event comes in then please run this particular image as a job and there's obviously more options here. You can pass in all the parameters and and stuff, but this is the second option, so the first one is web hook.

B

The second is through this and the third one is, and you still have the option you can also write and develop. If you want to a a custom, what we call captain service like these captain services, they are basically just containers that you just deploy like I have here right deployment is a pots like the. uh I have a low-cost service actually here. So this is nothing else than a container that um has a sidecar container, which is called the so-called distributor.

B

This is a component that we actually deliver and that distributor is actively registering itself to captain saying, hey I'm here I can handle the test events and then every time a test event comes in then it just forwards it to that container. That sits in that same pot and just calls a particular end point. So so we have with a couple of options. We call them captain services and I believe, if I do my if I use my google phones yeah, if I can google correctly then.

B

uh Let's go to the doc.

C

I've commented several times that my my real skill, my actual life skill, is my ability to navigate a search engine. That's really good.

B

Exactly so write a cable service here we go right. This is how you could do your own. We also have templates. We have a go template. I think you have a python template um and that's exactly with me.

B

I know oleg is probably already on it, but now maybe I'm faster than olek, and I'm posting this into this. Oh.

A

There we go, you got it, you won.

C

We do have a question uh from from rockhound again uh thank you, rockhound, um so trying to remember the exact context of the timing when hands asked this question here. She asked this question, so um are those events queued with amqp, for example, or do you limit the maximum amounts of executors on the global level.

B

Yeah, so what we internally um I mean that might be too much of technical detail, but we're using nets internally.

B

So the the shipyard controller remember the shipyard controller is there is the controlling instance that is always aware of in which state is my is my current sequence and then it sends out the appropriate event. So, for instance, when a new sequence is triggered like a delivery sequence, it knows based on the shipyard file. The first thing is, it needs to send out a monocle.triggered, so it sends that out and then everybody that subscribes to it will receive that event and then the shipyard controller waits until it hears back from at least one service.

B

That says, I can do it and then also the shipyard controller waits until all of the tools that have said I'm doing something actively also finished, with a pass or a fail result, and then the shipyard controller takes over. So the shipyard controller really manages everything and in terms of limitations, I don't think there's any limits on maximum there's. No, no, no, no limits on maximum number of of executors yeah, at least not not from an architectural perspective. No.

D

As long as your cluster can handle that it's fine yeah exactly and one important thing to mention that uh captain itself it has control plane, but uh its execution planes can be actually located on remote kubernetes clusters. So, for example, if you need to have uh run heavy payloads, uh one of the base examples is load testing, because the quality gate, which was one of the first use cases for captain. So with gmeter and with other services.

D

There is key six integration and you can basically upload this heavy execution to another kubernetes cluster and captain can just manage it for you as external execution plane. So it also increases scalability of as a system.

C

That actually um kind of anticipates one of my questions, which was actually about to be um scalability here. um So I guess you know, knowing that it can. You know, have a singular control plane executing across multiple well execution planes. um I guess the the real question I have is sort of um like multi-tenancy in this model. um For example, red hat is a huge org and we have a lot of we'll call them cohabitating engineering teams who might not necessarily want to see everybody's pieces but would want.

B

C

The same tool so yeah, I was going to ask about multi-tenancy and scaling to accommodate significant numbers of teams.

D

Okay, captain I initially started rather as a developer tool, where the approach was that you have multiple captain instances for relatively small steps, but now it changed and control plane evolved significantly. So now, for example, there is some going job to have a control plan, infinitely scalable also with features like uh high availability, zero time upgrades, etc. So, basically, a control plane is being redesigned now to support not only this scalability and high availability, but also support for basically non-stop operations, even if you need to update a captain. So this is just ongoing.

D

Quark uh beats landing uh incrementally, uh so there were some changes in the recent 0.14 release. There was some regressions uh which has fixed by now, uh but yeah. This is one of uh key highlights of uh this. Year's development is actually having control, plane, fully scalable and fully cloud native when it comes to terms of a single system. So you should be able to scale the captain much more than it was possible before.

B

And, and then also, I think, to add to this um kind of- if you really have multiple different teams that all want to use captain, they don't want to see the other stuff or not allowed to then two extras right now. You would have multiple captain instances. You can obviously run multiple captain instances also on one. uh I think there was one question in ocp: we would need to ins.

B

We would need multiple instances, so you can run multiple captain control planes also for that type of separation uh and with with with robust access with our back. I mean these are all things that are on the roadmap, so you can have team a logging in and only see their their staff and team b only seeing their stuff, but this is something that we're working on here.

D

Also, there are some quality of life improvements, so you have seen uniform ui with a lot of configuration there, but obviously everything can be configured as code and right now there is ongoing work to have official github separators.

D

Well, we have prototypes available for preview, but you will be able to manage captain fully, not just this code but in modules like githubs, so that you can share multiple instances, configurations between teams to apply templates and also rollout upgrades incrementally, as we mentioned with zero time upgrades uh just by having a number of repositories for your teams.

A

Awesome awesome and- and uh we um uh what's funny is- I can actually read that um so the the um here I'll put this I'll put this in the uh in the chat here. Yeah.

B

That was one of the videos and there's one of the the captain tutorials. We recently did um how to get ups with captain.

A

What's what's really what what I always always tell? People too is that, like, if you're a uta, condust, yeah, okay, um so the um uh you know like what what you're doing with um with our back right.

A

So now, it's like your ultimate flexibility right like if you want to leverage like our back and have like a traditional like workflow or like do something like get ops where, like all the interactions happening in git, so like you, actually almost just only focus what how your get is, our our back or your groups or your permissioning, or your pull request right like it's.

A

It's um it's totally flexible right so like if you're gonna do the get ops way, hey just use whatever organization you know like if you're in github right, like just use that model or.

C

A

You want a more traditional sense. You know, they're, you know, take a look at that um that pr there, I think it's a standing pr for now uh for for captain to to support our back. So here.

B

A

Checking here to see, if um yeah someone asked so this is, this is a good. This is a good question about, and I'd probably one of the last questions here, because we're almost that time and um I I unfortunately have another meeting um to go to so um someone's asking about uh openshift operator integration, so that'd be interesting uh kind of thing to write um about having uh uh operator integration with uh with openshift, so um not sure if there's a pr for that already or or or or an issue, but.

D

That would be really.

A

D

Right yeah like what.

A

Is the google foo on that.

B

Yeah, I mean that's exactly that's the the github operator that that falls into the whole thing. What we just mentioned earlier, uh how to keep up with captain. That's all operator based yeah.

A

Cool cool all right, so um so yeah, so we're almost that time, we're just about at times. So again, um oh, look at that andy already had uh the uh the link handy for you, um so yeah. So again, everyone thank you for joining andy. Thank you for joining oleg. Thank you for joining. um It's been great uh hillary! Congratulations! Your first show your um your first official yeah, our first official.

B

A

um You know again have uh excited to have you here so um yeah, so any any last words uh for from andy or oleg.

D

Second, question about captain at cubecon and.

A

C

We are going to.

D

A

D

A

Can meet us today.

D

We will have a project kaiosk. We will also have uh happy hour happy hour on may 17th, uh with some special agenda stay tuned for announcements subscribe to captain twitter, uh you won't regret it um and also on may 17th. There is cd events today, which is a conference again about all about events in cd etc, and the captain will be actively participating there.

D

A

Yeah I'll put that uh so I'll put the twitter twitter link there for the captain project, um so cool cool, awesome, awesome! So I'll see you at kubecon, so I'll be there as well.

C

Well, I you say that, like you're going without me we're going together. Oh.

A

That's right, my co-presenter, so hillary will also be there we'll all be there all four of us. So if all four of us be there so catch us there, um it's so awesome. So uh hillary any last words. uh But what what? What do you think about your first stream love? It hate it.

C

No, this is great. um This is this is going to be hard to uh hard to top in terms of uh favorite streams. um I say that now that could change but yeah. This is a great topic and I really enjoyed it and thank you guys so much for joining us. Yeah.

A

C

B

You again, you can say it was the best stream so far for you, because it's the first so right.

A

Because this is the first time.

A

You never forget your first, I guess um so um so yeah so again. Thank you. Everyone! Thank you for joining. Thank you guys um and um we'll see you next time. um I forget: what's hap, coming up um just look for twitter, we'll we'll we'll have an agenda out so um so yeah again everyone thank you for joining and, uh like I always say, unless it's in git, it's only a rumor so goodbye everyone cheers. Thank you for joining.

D

D