keptn Community videos, 21 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Keptn Can Orchestrate Your Favorite CLOUD-NATIVE Tooling | Ep 50

Description

Over the past years, the CNCF (Incubating project) Keptn has helped SREs and DevOps to automate SRE best practices around SLOs, Automated Testing, Automated Quality Gates, Automated Deployment, and Automated Remediation into their delivery processes.

Keptn quality gates provide you with a declarative way to define the quality criteria of your service. Therefore, Keptn will collect, evaluate, and score those quality criteria to decide if a new release is allowed to be promoted to the next stage or if it has to be held back and that's one of many other great features to be highlighted today by Andreas in this episode.

In this episode, Keptn maintainers will walk you through the most common use cases, show you how you can get started with your first project and how Keptn enables you to scale these practices across all projects in your enterprise. Grab your favorite ☕️☕️☕️ and enjoy the episode .

A

A

A

Thank you, everyone for joining us on cloud, fn podcast, and this is episode number 50, and I really appreciated everyone who will be in this show and share this journey about cloud native. So it's been a special day for me and it's a special day for my guest as well, because captain advances to cncf incubation and one of the best thing about this show that we're talking to the person who has been the co-founder co-creator of this tool, andres grafner.

A

Thank you very much for joining us and just share your thoughts on cncf incubation first and where, where you currently at right now,.

B

Well, thank you so much first of all for hosting me for giving me the opportunity you're right. It's been a major milestone and I think a lot of projects that start in cntf or even outside of cncf right. You always want to make sure that you are actually making an impact and you're growing it and you are providing value, and I think reaching incubation shows us that we are a definitely solving a problem, because otherwise we wouldn't have adoption b.

B

It seems that people like what we do because we also see uh people are helping us in kind of expanding captain, building new integrations contributing to captain. So that's great and uh yeah. It's just it's been. It's been a long journey. It's been a couple of years now, when we set together here close to where I am right now in a in a room, and we were kind of figuring out. How can we build better tooling for ci cd? How can we make this?

B

You know what's next and we actually took back then what we had available and then we figured out well there's great tools out there. It's very hard to connect them all, and then, with our background of observability, we try to figure out uh what the role of observability should be in the application life cycle and this kind of fast forward to now.

B

This is where captain is: captain really tries to solve the tool, integration problem right with our involvement in see in cloud events and cd events, and, as I always like to put it, we are putting observability into the driving seat of your devops automation and happy to be here.

A

Yes, absolutely absolutely, and I think being incubation tells you, like you- are working at a good pace, you're, solving the real challenges that the customer and the companies and the enterprises the small companies are facing and then having the incubation tell you like- and I think this is right now is the part we want to be contributing. His github link is listed in the screen. Github.Com slash, captain captain so start contributing to it like it's been, and I think we are lacking in the in the cncf and we lacking tools, you have sre and devops.

A

We got have a tool that gives you the both time, like, I think sorry, is all about service level objects, survey, level, agreement, objectives, indicators, what those are we definitely cover in the show today as well, but also the depth of pipeline, the dev, dev environments, the staging and the production environment.

A

These are the different environment and all you have the different toolings involved into it and you and if a tool can understand those stooling, that's a really enjoyable life for you, because otherwise you have to invest so many time in learning and learning still it can be a steep learning curve, but especially the cncf landscape. So so, let's first uh try to people who are really new to the captain, like what are the different features in it.

A

How you think about it when you're first starting about this project and what is the major like if you talk to the community, what do you think the most selling point of the captain or the most selling feature of the captain.

B

Yeah the question is: shall I talk about it or shall I show something? What do you say.

A

Absolutely you just talk about this. If you show something that could be really awesome.

B

Then let me show something because I have uh let me share my screen here share screen and, let's see I'm sharing this screen here now moving you away, because I think we know how we look like here we go. um I think um this is what made us really popular uh at the core of captain. We, as you said, right, we're looking at what we what the industry refers to as slo service level objectives.

B

I think where we have our little spin on on service level objectives, because what we are basically doing at the core of captain. uh We are integrating with europe's ability tool. In most cases, this might be a prometheus.

B

It could be a dynatrace, a data dog in your relic, a splunk wherever you get your observability from, but we thought what we do as an industry every time when we have a new build every time when we do a deployment, we always validate right. This is a good deployment. uh Is it still accessible from the outside world?

B

Does it use you know enough memory or not enough memory or what's happening right now? Does it crash anything so typical as humans? We then always look at certain metrics, whereas the world calls it sli service level indicators, and then we are comparing them against our expectation. Our expectation is either a certain threshold that we want don't want to cross like. We should not be slower than a certain millisecond amount, or maybe it shouldn't get worse than before.

B

Then it's classical regression detection by looking at historical data- and this is exactly what we do with captain at the core. So what you see here is what we call the heat map every uh column here is kind of representing uh an evaluation that captain did for me in the past, so every time, captain orchestrates uh automation, then as a keystone or a key task, as we call it we automatically validate. So what I have here in my last evaluation, as you can see, I have a list of slis metrics.

B

You can see them here, optics load average blah blah different metrics, cpu memory, and then every metric was pulled in from our stability platform was evaluated against a threshold, and then it gets a a grading, yellow, green or red, and then on the top. We do a top score. If I scroll down, you can actually see the raw values behind the scenes. So in my case I can see which sli, which metrics were retrieved for a particular time frame.

B

So every time you ask captain to evaluate a system, you also need to give it a time frame. This could be the time frame of the deployment in the production environment. Then you may be waited 10 minutes for real traffic to come in or in a pre-broad environment. You may execute some tests and then you say please analyze the testing time frame. So in my case I had here 30 minutes, but it basically then pulls in the data, the values and then it analyzes it against the past and the warning criteria.

B

You can specify- and you can see here it immediately says- does this particular sli meet your criteria or not, and depending on that you get a score and then on the top. We then calculate a total score that we normalize to a value between 0 and 100.. So that means every time you evaluate you, get a clear indicator, saying green, yellow or red, and why is this important?

B

And why do people get excited about this because I think a lot of people over the years have invested in automated build in automated deployments- maybe already some- maybe already some automated testing and also some performance testing, but where a lot of people have fallen short, and this is why this is kind of the first level into captain. Is the automated validation of your environment, based on indicators that you define, as this is a healthy environment right and at the core?

B

This is really what this is all about, um and then we bake this into something much bigger. So what you see here is just the evaluation piece, but we can bake this into a much bigger projects. So we typically see people start with just a validation or release validation. But then you may have scenarios where you are using like. I have a rollout example here, so argo rollout uh that I'm using I can use captain to do uh multi-stay I can.

B

I can let captain do uh orchestrate multi-stage delivery and what multi-stage means you can see. I have staging. I have production. This is our visualization of what captain is doing and behind the scenes uh we really are executing so-called sequences or we orchestrating them. So I'm moving around here a little bit, but you can see here I have staging and in staging my captain, whenever I I tell captain, there's something new for you to do. Let's say I built a new container. I pushed it to a registry and that registry then triggered a sequencing captain.

B

I kept will then execute certain tasks and then a key part of it. If I scroll down, you can see it's doing a lot of things, configuring my monitoring, deploying validating, releasing running some tests and then it's evaluating coming back to the the core component, so yeah. I think what got people excited, though in the beginning, is definitely the the slo piece, um because we really bring a core concept of site reliability, engineering to the cloud native world as an easy to use capability, but then we really center and and provide orchestration around slos.

B

And here what I really like right, we are really integrating uh with all of these tools. Like argo right, we have arg, we integrate with argo, we integrate with flux with jenkins.

B

The point is we don't want to replace these tools, because these are all great tools and they're doing a great job, but what is really hard to do is connecting them all together and then making automated decisions uh based on on observability, and that's really what we bring to the table so connect. We connect all of your tools, we orchestrate them in a sequence, we're using also a standard protocol or standard events, cloud native events, cd events, and then we always go back to your observability platform and validate for you.

A

B

Yes, absolutely absolutely.

A

And that I think that, like as you're chilling like people are thinking about the automated build the automated deployment, but what how do we prevent a small bug for going into the production? So we need some kind of indicator like I can push my app into the let's say a stretching environment.

A

Then I I then I put them under a load test of 10 000 request at a time and to see the matrix like how does is responding into the cpu with the ram spikes, how the data database is happening, and that is actually the heat map. The evaluation part yeah. I think that's really I like about, because I need to understand like if I putting something to production like how should it behave so.

B

Exactly and that's the I mean one thing: we learned right when we started with captain. um We wanted to make sure that everything we do is as close as possible to everything is code right. We wanted to make sure that that this actually scales so in the beginning, because some people may ask so behind the scenes.

B

How do you define all this and behind the scenes uh you can define your slis and your solos uh in in yaml that you put into a config repo that captain uses behind the scenes so, like you can see here, the slos are defined. We also visualize it here in the ui, but really what it is. If I go quickly back uh behind all of these projects and I'm looking now, let's say they: let's look at the at the the demo, roll up project that I had earlier.

B

So if I look at this here, then I have a git repository can be any type of kit and then I go to my staging branch and in staging I have my simple node service and in here I have my slo definition. So solos here really say: okay, what type of metric um response time sli and then your criteria pass warning. You can also give different weights. I didn't say that mentioned this earlier. I think this is also really important. What we learned not every metric, is equal right.

B

There are certain things that might be more important for you and what you can also specify, and I'm just editing. You could also say this is not not weight 2, but you could say key equals. True key means it's a key criteria. If this is not met, then I don't care about the rest. An example would be uh if you're pulling in slis. Let's say the number of uh detected security vulnerability. Security is really important for everyone right.

B

If you have tools that detect security vulnerabilities and then you have let's say a way to extract this through prometheus or dynatrace or other tools, then you can add this into your list and say I don't performance is important. Throughput is important, but honestly, if I have a security vulnerability, critical one, then I I don't want to go to this to go further, so you can specify this. So, as you can see here, you specify your slice, a metric, an indicator and then the criteria.

B

Now the question is: where does the the data really come from, and this is where, in my case I you know, I I'm using dyna traces from observability platform. So if a dynaj subfolder, if you would have prometheus, you would have a prometheus subfolder here and then you have an sli gamble where you specify the query language of the target tool- and this is also the nice thing we actually separated the definition of how I get the data to what I'm expecting from that data and the reason why this is important is right.

B

I, as an architect as a developer. I may know that I have certain response time and throughput and memory uh objectives, but maybe I don't control how the static data gets collected. Maybe another team is responsible for observability and they use you know.

B

Maybe they use prometheus, but maybe they use datadock or maybe new, relic or dynatrace, and so we are separating these two things so that one tool can define how to query that data, and then another team or the team that is responsible for defining their objectives, can then independently of the monitoring tool behind the scenes, specify the pass and warning criteria. I think this is also very important right and the last thing I want to say here this is really you know.

B

I think this works extremely well, but we also know, while kubernetes has been around for a while, there's still a lot of people that are new to this, and some of them are afraid of touching config files like this, and this is why what we have also allowed our customers to do with our customers.

B

And now I'm talking specifically about the dynatrace community, and I know that jurgen who, by the way, I want to say a big thank you to jurgen etzel stuffer, who was a founding member of captain as well, and he since moved on to microsoft. He's now an advocate there, which is great, but he but when, but what we learned is we need to make this easier right. We need to make this easier, uh because some people are just not comfortable with with yamls.

A

B

For our dynatrace customers- and I know that this is why a product you can, you can also started implementing something like this for graffana uh and elastic, so you can not only configure these things as a gamble, but in our case you can also configure this by just using a dashboard. You can create a dashboard now. This is a dynatrace dashboard. That means I can use captain and say this is my dynadress dashboard that defines the metrics that I'm interested in.

B

I can specify the path and the warning definitions on the dashboard, so I kind of visually start defining and getting comfortable with what I would normally manually analyze.

B

But then I say captain you take this dash, but this is your input definition and- and I think this is this- is this- is uh what we we saw, at least in our dyna choice community. um This led to a bigger, a much faster adoption, uh because we reduced the hurdle of having people to type in queries in in yaml. Even though I think the yammer-based approach is the one that you want to take once you really start to scale this out and push it as as far left as possible to to engineering.

A

Yes, absolutely absolutely, and I think the as you mentioned during the demos like we have slice, slo dot, yaml file, we have a ship rod, shipyard channel file and I think the all all the idea behind is that is the separation of concern between like in that. If I, if I take the example of of the multi in the delivery pipeline at different stages, we can have all these things like. What should I use in a build state that is written in the shipyard yaml file and who can do what is actually is telling them?

A

The captain is telling like say, help char to create this thing. Our captain is connecting with argo cd with litmus chaos to do the testing. So I think that's a really easier approach for the teams, like you have definition written in one file and if you want, let's say to a swap dynatrace with prometheus, you can go into the shipyard.yaml file and you just change the field in there and everything stays the same, and I.

B

A

That is really being the hurdle for me, at least like if we.

B

Have a different.

B

Go ahead. Sorry.

A

So that's really harder for me like in the past. We have a same depth of spike line and we want to use the same pipeline in a different environment. But I need to change, let's say the testing tool, but everything stays the same, but I have to maintain another type of pipeline because I don't reuse that pipeline.

B

A

The dyna trace- I I can see these- we have a shipyard ml file, the sli slo. These are the definitions and where these things are operating or implementing or in a different stage, so that I see the selling point of in itself and also the one thing or to be like I'm interesting to know, like you tell us, like you in the diner trace web dashboard, let's see if somebody come up with their own matrix of sli ammo. That is already not defined everywhere.

A

So do they build that one and then it deploy it on and then it's also fetching those matters as well like. I want to see like in the cloud environment how the aws lambda is operating or other services operating, but I want to have my own matrix in there, not just these defined by so how? What are the? What are the different methodologies of doing this.

B

Yeah there's multiple options that you have so. First of all you know you could, let's assume you know you brought up if we, if you stay with dynamite race, if dyna traces, your observability platform of choice, uh then you can push additional metrics from other sources to dynatrace and then dynatrace.

B

You just chart this metric here right, for instance, we mentioned aws lambda. We have a native integration where we pull in lambda metrics. You can also push any external metrics to dynatrace through and through an api, and then you have everything in one tool and then on the captain side. You basically just say: hey, you know, analyze this dashboard or use the yaml approach and in the yaml you can now also use a query to query.

B

Let's say the lambda metric, but the other option that you have- and I think this is also coming back to the separation of concerns we have. If you look at this here right, this is kind of the sequence for the evaluation and then um what the evolution actually does. The so-called lighthouse service is really the magician behind the scenes. The lighthouse service is the one that reads the slo and then basically says hey.

B

I need these metrics, because it's they are they find in the slo. So who can deliver me the values of these metrics for a particular environment for a particular time frame, and it basically sends a so-called get slime event and then your uh we call them sli provider or your observability platform of choice is then responding and say: hey here are the metrics that you're asking for which means what you could do is you can also have uh different. You can connect different data sources to captain as well, which is also great right.

B

You may want to we query for one project. You have prometheus as your data source for the other one. You may want to go straight to aws cloudwatch right, because maybe that's what it is right, because in a large organization, I'm pretty sure you have different teams that are working with different technology stakes, and that means they also make their own choices about the observability platform and therefore we allow you per project actually in captain right now to specify what is the data source?

B

What is the sli provider, and then you can switch just as you can switch testing tools. You can also specify uh which monitoring or stability platform is providing the metrics, which is really cool.

A

Yes, absolutely- and I think, and that's another use kit that I'm looking at like if we have a build stage, history or report like I have done 17 different deployments. This is my entire dashboard and in deployment one we break this service in deployment 2. I break this service in deployment 3. I break this service. Don't build process. We can actually make some kind of evaluation like if this service is breaking five times build stage or in a production environment.

A

I can slack a notification and tell the team member that this is the service is breaking too much. Let's do something of that.

B

Exactly yeah- and I think for this- that's a great point um you can and that this is what I wanted to say earlier, because you were, you said: you're a big fan of our separation of concerns with the shipyard file, where we specify the process, but in the process itself right. This is what we call the shipyard and we don't call it the process. We call it the sequences of tasks like monaco deployment, validate release test, and what we do here is, as you can see or not see.

B

Maybe we don't have any tool specification, because exactly that's the core of kitten. It's an event-driven system. Captain is orchestrating the sequences of tasks, it is sending events and then you can have one or multiple tools subscribe to certain events, which brings me to the points that you just made. I can go in here under settings and under integrations where you can see these. Are we call them captain services where you can install certain integrations that we have, for instance, this is where the lighthouse service comes in.

B

There's the helm service, the argo dynatrace monaco locust job executor. So we have certain services that are basically kind of you, install it on a target environment and they then connect to captain and say I'm interested in certain events. So it's basically an event subscription model.

B

What I want to say here to your example, with slack, how would you integrate slack with captain? Well, one option would be to write your own slack integration, but what we have done over the years and what we've learned we had. We learned we need to make this super easy. So one of the services that comes with captain is a so-called web book service and you can, as the name implies.

B

I guess you can add a subscription and in your example you could say hey if there is a deployment that is finished in a particular stage. Let's say production for a particular service that I might be responsible for. Then I want to post a message to. I think it's uh api.slack.com right and blah blah, and then you just say so you can. You can subscribe a web hook to an.

A

B

And there's actually uh different ways of how we do it. You can, as you can see here, for every task for every event or, let's say really, for every task, like a deployment like an approval evaluation.

B

There's three types of events that are triggered so initially captain sends a so-called triggered event saying I need somebody that can do a deployment again to your point. You may start with jenkins, but then, maybe later you say, I'm switching from jenkins to github or gitlab and therefore I'm just changing the event subscription.

B

So that means you can also do a web hook on triggered, which means uh triggering typically means. I want some external tool to do something active right, I'm triggering something and then I'm expecting also a status in return, and this is why what you can do here or if you scroll down you can.

A

Actually say: hey.

B

um What do you want me to do? Do you want to send a start event automatically and a finished event, because the way captain keeps track of is an external tool actually doing something, and when is it finished? It expects that external tool to respond with? Yes, I'm starting I'm doing my job now and in the end also finished and for web hooks. We also give you all the options right. You can say: hey, I'm I'm handling a deployment.

B

I'm doing this for you right. This might be. Let's say this could be your myjenkins.com right and uh you're sending it over, and then you say: hey, I'm sending the started event automatically, because I know jenkins will do its job right away, but I'm waiting for jenkins to call me back when it's done, because I cannot. I don't know how long jenkins takes maybe an hour or maybe 10 minutes, maybe a minute, maybe an hour so also for web books. We give you different options, uh which is really great, so you can handle on trigger.

B

You can be become active. You can also say I'm interested every time. Some other tool is starting something, and then I want to send a notification right so you're extremely flexible here, with what we've doing what you're, providing with the integrations.

A

Yes, absolutely- and I think on this point we should try to like is is captain is not just notifying you that it's a great tool, because it's just doing the remediation itself, like you, define like if, let's say if your deployment is failing and you've come up with the two parts, and you find like something happening to your application. Let's go out and tell your power to two to four and that's automatically happened to you and you just submit a file to captain and captain just automatically create a new file, and then it's reevaluate.

A

Those heat map like how is this. The issues has been solved and if it is all that's wonderful, otherwise they give you the notification. So I think notification itself, because people are telling you, oh wonderful, who people are saying this tool is telling me like there's so many bad things I'm doing, but do is able to tell me how I can fix those. So it is.

B

A great addition.

A

To on that part, as well, and also during the discussion, I think you mentioned the eventing part as well, and I think that's really a core into the what's really. I want you to elaborate more on the eventing part, because I think in the past, for how the build was architecture like if you can set up a build and build stakes one hours. You do nothing and just to wait to finish that build and then do the next step.

A

But that's eventing mechanism and the cause behind the scene is asynchronous and let's say you hit a call and it's telling like. I need to deploy the hand, charts and don't wait for it, but it's on the parallel. You can do other things as well and when it's finished is telling you that I'm done with it by notifying you now, let's do the other things as well. So I think the eventing is really according to the captain project as well.

A

So I want to elaborate more on this because you in the event I can say I can see that the captain is heavily investing in the inventing everything done behind. This is, is, is triggered by eventing.

B

Exactly that's what that's what it is uh before I go into this. I don't. I want to make one thing clear though, and I think this was a a misconception from the beginning, uh because we, our narrative, was always kept and deployed kept notifies captain itself doesn't deploy. Captain is triggering a tool through an event that can actually do the deployment right. I just want to make sure that people understand. Captain itself is not a build tool.

B

It's not a deployment tool, it's not a testing tool, it's not a monitoring tool, captain orchestrates all of these tools uh by executing and orchestrating what we call sequences of tasks.

B

The sequences of tasks are explained or described as we call as we see here in what we call a shipyard right and then now to really answer your question. um We're using events so you're specifying sequences like you, have a delivery sequence and that delivery sequence has certain tasks that you want captain to then orchestrate so now, let's take staging delivery. How would I? How would I tell captain to start this? I would do this by sending an event to captain, so captain itself is also triggered through the same eventing mechanism.

B

Let's go back to the bridge here. We call this the captain bridge. You can see the delivery sequence and if you we also show all of the events here, there's a little icon, the event payload. So initially I kicked off and you can actually see this here, andy while preparing for kubecon. I think this was the demo environment that I used back in may for kubecon.

B

But the point is you can, from your let's say from your ci, let's say: you're using jenkins and you're building your container you're, pushing it to a registry, and then jenkins can send this event to captain saying captain. I want to trigger the delivery sequence in staging.

B

It is for this project, it's for this service and again stage it's duplicated, but that's what it is so with this captain knows: okay, andy wants to trigger something in the delivery. Demo project delivery: demo for the service tnt acer. I can have multiple services defined and.

A

B

Wants to start this in staging and the sequence he wants to trigger is called delivery. So what captain then does it goes into the git repository behind the scenes says: ah staging delivery? That's the sequence. Andy wants to execute. So the first thing is monaco. Then deployment with some additional properties then validate with some properties, then release then test and so on. Now, let's have a look at how this actually looks like.

B

If I go back here, I trigger this sequence, I say for which project I want to trigger what type of sequence I can give additional metadata like labels. Labels is just things like who am I build id, maybe a link back to your tool that you that you are that you're calling now. You can also add additional config data, because maybe that sequence somewhere needs some additional data from the outside some well-defined data, like the image that you want to deploy right. This could be the image that jenkins put into the registry.

B

So what captain does with it? It takes this initial event, looks at the sequence and then the first thing it does remember monaco. That was the first thing in the test definition. So what it does it sends out an event and says hey.

B

This is all the event that I all the data that I have so far that the one that triggered initially, that sequence gave me, and I know that I now need to trigger the monaco event. So this is now the cloud event that basically tells that tool that subscribes to it. What to do and now the question is who is handling monaco? In my case, I have a service that is subscribing for the event monaco triggered because it can handle it.

B

Monaco stands for monitoring as code, so this then configures my monitoring, the monaco itself, like every tool that participates in the captain sequence needs to remember what I said earlier. It needs to respond back initially with a started event, so that captain can keep track of state and when it's finished it sends back a finished event. This here and the finished event needs to have a status succeeded or failed.

B

So if it's failed, it would stop here, but you can actually then also branch off and you can say if it's failing, then I want to do another sequence that maybe notifies somebody in my case it's successful, which means it goes into the next task, which is the deployment task. So what it does here it could.

B

It looks at all the information that it already has collected it's kind of like a trace from everything that happened so far and now says I need to trigger the deployment task and then, in my case, what do I have here? I have an integration that we built helm service. That is then basically saying. Okay, I can deploy and I have enough information to find the helm chart to do this and this and this started finished and so on, and then the evaluation kicks in.

B

So it goes on and on like this, but this is really the beauty of it. uh Maybe one last thing here because that's important the helm service is obviously deploying, and um you would you know, maybe you would call a jenkins pipeline to deploy. Maybe you would call something else. The point with the deployment here is that tool that actually deploys is also supposed to then respond back with a finished event, so deployment finished, hopefully with successful, but then also additionally, things like what is the url of that service that was just deployed now.

B

Why is this interesting? And why is this important? Because you may want to run a test afterwards, and this test needs a url to test against, and this would then be taken as an input all right. So that's kind of the that's really the the idea here on how the eventing works- and maybe I think I should have probably picked this one, because I believe in this particular in this particular example. Here.

A

B

This is exactly the deployment happens and then there's a test step. I'm using low-cost service and the low-cost service actually retrieves again as input the url of the deployment that happened earlier, and this can then be taken by the testing script to say: okay, I'm executing a particular test script that was uploaded to git repository behind the scenes that captain holds, and here is, for instance, the service url to test against, because that's the url that the deployment tool told me that it deployed to.

A

Yes, absolutely and that that's all the eventing part of it, and also, I think, before it's a question like do you when it's first, when you subscribe to the event, you notify the captain that is now subscribed to the event now during the notification part before the sequence, when the sequence, when the notifications start and when this event service send back the notification that this is done or the captain also watching intermediate situation is below just is just watching the first and starting an ending board between between the deployment or anything.

B

So what it does is it basically waits until at least one service says um it started okay, so it's at least one service needs to otherwise, at some point it runs into a timeout. We don't wait forever. uh Also what you can see here, that's an interesting one. Here we actually have two services that are subscribed to it.

B

I have the helm service and the argo service, so you can also have multiple tools that are basically saying I'm doing something and in this case captain is then waiting until all of the tools are reporting back that they're finished, so we're doing state management here we make sure that all the tools are are getting a chance to finish now, there's a couple of things that I know our user community wants us to improve. There's things like configurable timeouts. How long do you want to wait right? There's an option also what we already have implemented.

B

This is also nice. If a tool is doing a long running job right, go back to, let's say maybe jenkins and jenkins runs for a while until everything is deployed, you can also give uh send not only a starter. Then a finished event, you can also uh send it, so I think it's called an update or progress event where you can say: hey, I'm still here right, I'm still here, that's also important but yeah.

B

So these are things I think people have asked for is better timeout management, because right now, I think there's a global timeout that you can set on how long captain is waiting for a task to be started by a tool. I think from a retry perspective.

B

What? If? What? If something fails right now we fail the sequence, but people have also asked for maybe retrying things. So this is stuff we've. We have enough, we have in the backlog, but we also need to figure out, and I think this is the next major milestone.

B

From a captain perspective, we've learned a lot on what captain, what what peop, what what people get excited about in captain the slow capability, the automated validation, um and so we are currently also thinking about where the future is heading, and we really want to focus on things that bring true value to our community and that that also means right. If you focus on one thing, we may not do certain other things at least not right now, which makes sense.

A

Yes, absolutely, I think we discussed so much a lot and now the question people came up with like where the captain is actually running and where the tools are running. It is a question like captain running, I think a capital run in itself. The control plane is running in the kubernetes in the kubernetes cluster.

B

Exactly exactly.

B

Yeah, so exactly so, what I have here right, I in my demos I always use a linux machine on these typically use ec2 or just an ubuntu, and then captain itself runs on kubernetes. It needs a kubernetes. Cluster can be, kubernetes can be, openshift can be anything, I'm actually running a k3s lightweight cluster and, as you can see here, if I do this again right um creates to this k3 oops, okay, 3s cube ctl get ports a actually.

B

Then you see everything that is running in my cluster, then you can see that in this cluster I have my captain namespace with my captain specific uh my core services, so, for instance, api gateway, that's cool secret service, the bridge, that's the ui remediation approval. These are all captain core components.

B

A very important core component, for instance, is the web book service that I told you about, but then you can also install additional. We call them captain services, so kind of native or more tightly integrated services like the helm service, the argo service, the monaco, the locust, the dynatrace. These are all additional services I can install in case. I really need these type of of integrations, and why would I need them? If I, for instance, cannot integrate the tool that I want to integrate with a web hook, and there would be one.

A

B

That is very interesting is what we call the job executor service. This is also a service that you can install that you can download and install the job executor service is, as the name maybe implies, is allowing you to react on an event so to subscribe to an event, but then, instead of making a web hook, call deploying a container on that target.

B

Kubernetes cluster, where you install this one and kind of think of it like a like a github action, so we run a container as a kubernetes job, giving it all the input of the event that captain was sending out and then once the job is done, the job executor service takes care of like the start and finished event. Now what's important. I think- and this is the question that you wanted to get to uh the captain control plane.

B

So all of the core components they are typically installed on some type of kubernetes cluster right, whatever your tooling cluster, but then uh other tools that you like, for instance, the job executor, is a great example. uh You could have the job executor installed on different kubernetes clusters, your production cluster, your staging cluster, your dev cluster, and they would all in the beginning when they come up register itself and subscribe to the events and say: hey captain, I'm here and I am in production.

B

So I'm interested in events for production and then the other job executive service that runs in staging would say, captain I'm the job executor and I'm interested in staging events. And this is how you can use captain also multi-cluster, to orchestrate sequences that go across stages and then really target the tools in the target environments.

A

Yes, absolutely and I think all those like all the control planes, all these things that that's running in the kubernetes close to the captains are there. Are you deploying that, as part of the kubernetes crds, by extending.

B

The google, so we are that's a good question, so we actually have a project right now, um captain github's operator, so this is most likely and, as you asked earlier in which direction is captain heading when we started with captain initially, we did not think about making all of this configurable through crds right and as it is today, the captain core components, uh you install captain and then you configure captain not through crds, but you configure it through config files that need to live in the source code repository behind the captain project, and this is a thing that we want to address, because this was feedback we've received from our users.

B

Why don't you have an operator? Why can I configure dyna captain via rcds, and this is exactly kind of the first? uh The first approach is the github's operator from thomas thomas schutz is one of our colleagues where you can really.

B

You can see if a captain instance a captain project, the services sequences stage and all of this this was a great way to get started to think about how captain can play in the true githubs manner. I think what we're currently doing is we're currently thinking about. um How does this really need to look like in the in in cooler, because, right now, this is a sandbox project right, captain sandbox? That means we are it's there. uh It was great to get feedback, but now we are one of the goals for us for captain.

B

The next big milestones uh is to move towards that direction, that the captain is fully github's, complete ups compliant meaning you can deploy and configure it through. Just through cds,.

A

Yes, absolutely- and I I think, like another question like I think people might need to think about at this moment like when there is a only the jenkins the sea ice system is available. We can see all the integration that jenkins can do. They can integrate with the maven to test the code they can integrate with the other load, performance, testing, tool or load testing tool or other other secret or configuration management tool different toolings.

A

We see that jenkins community grow huge and we can see like the captain, is right now integrating with all of these cloud native tool, and there are more in the road map as of today now or very actually. How other do you see in the next time era that more and more these services being into the into the captain marketplace where we can see all of the all of the favorite kubernetes projects?

A

We have those eventing mechanism in place in the captain project, so we can hook those uh favorite cloud native projects and do some of the other things like, I think about the k native or other, like how you think about like. What's going to be happen next few years with the captain.

B

Yeah, so I think I don't think we will build a lot of integrations because we have a different approach to solving that problem, because building integrations means somebody needs to maintain all of these integrations, and then we end up with something that we see with other existing tools that you have a lot of integrations, but then they sometimes break, and then they break off them, because the tools, on the other hand, change and so coming back to the events again right.

B

These events here these are already based on cloud events, but what we are currently actively engaged in is in a project called cd events. I'm sure you've heard about it right see the events.

B

Cb events is a cdf project, so continuous delivery foundation, uh where we as it says here, it's a common specification for continuous delivery events, which means we're working with them to really standardize uh on events. That kind of allow us to map the life cycle of your applications. I mean right now the continuous delivery events, if you think about it right, they have, they specify continuous deployment.

B

Events like when an environment is created when a service is deployed, so there's already some of these events in there that kind of map to something what we already uh sending out or consuming uh we're working with them, because we think the future is specifying well-defined events from the creation of an artifact through its hops into the different stages until it's in production, even until it's retired, and if we have these events and they're standardized, then the integration problem will be solved in a different way, because then we and many other tools don't need to build point-to-point integrations.

B

We just agree on a common eventing standard and then tools that want to do something when, let's say a service gets deployed like notifying somebody can just subscribe to that event, and I don't need to build point-to-point integrations because we are standardizing the events we're standardizing the apis. Basically, and this is what the future this is also, when you ask me where is captain going, and I think we will see a significant change in captain because we are, we want to go full get-ups.

B

We want to go full city events, you want to standardize all of this, which means there's less of a need for us to build. A lot of custom integrations with captain, because this would defeat the purpose of what we've always been trying to do, defining a standard and that's exactly what.

A

Yes, absolutely, I think, that's for me at least it makes so much sense to choose the cd events like the same eventing mechanism that every dealing can do like. If somebody could give you the json response, somebody give the ammo response. If somebody gives you the scripting response, it's tightly a different story. If we come up with a standard, how the tools is sending back responsing for you and how you interact with them with the standard way, I think that's make the integration a lot simpler.

A

In that way, you don't need to invest a lot of time with what's happening in the ecosystem of that tool. What's because you have one one language is to trigger the event and one language to receive the event. So I think that's a great, for me at least makes sense for me, and everybody has some other ideas or other opinions about this approach, and andres is a great person to um being tagged with being friend. Being asked this question as well and before I.

B

A

And rest that, I think that's the last question from my side, and I think I really love the point thinking about. I see the captain lighthouse surveys. That's actually detects the all the what's happening.

A

What's going wrong all these kind of I'm thinking about, like I think in the cloud environment, especially the managed kubernetes clusters, there, we see some downtime on the outages in the aws, with the azure gcp coming and other other thing like I people down the road think about like for me, aks is not a good fit, but I started in the aks, but I want to go to the azure a aks side or thinking about like if something happened in the aks cluster, how do I remediate those issues?

A

How do we able to respond to that issue? How the captain requires a remediation in the devops lsa default? The sorry word like how do you think about those managed clusters and the issues popping up to that? What your thoughts on what is the best way to look about those managed clusters that are people running in large scales.

B

Yeah I mean first of all, I think the problem that you explain or describing is not uh constrained to manage clusters, because whether you have a managed cluster or whether you manage your infrastructure yourself, things can go down right. So, in the end, the question is really uh does captain provide an answer to.

B

How can we react to a system not being available, and the answer is this is the second big use case that we've always focused on, which is what you see here right. We are orchestrating a sequence of tasks that is triggered initially by an event that says, there's a new version available now on the other side, we can do the same thing, I'm not sure if I have it in here remediation or the remediation.

B

That means you can also trigger a captain sequence from a tool that detects that something is wrong. So I've come back obviously to dyna trace because I know dynasty is pretty well dynamic. Risk monitors your systems not only from the inside, but also from the outside.

B

We know your end users and if they no longer can access your systems because aks or something else is down, then we would create a so-called problem, detector problem and then we could either notify people or what we can do right. We can again say: hey captain. I want to trigger a remediation sequence in production, so the concept is completely the same: you're, defining a sequence of tasks that is executed in response to an event from an outside tool.

B

One event could be there's a new artifact, a new version, the other one could be there's a problem and then you can specify what should you do right? Maybe you're scaling up, maybe you're scaling down, maybe you're moving things over right. We had a lot of conversations at kubecon with people that are looking into tools like terraform cross, plane uh prelumi.

B

We can then move things and scale things up, but not only scale up but scale out and scale over right, and this is all what we can orchestrate so exactly. This is what it is, and maybe maybe this is one thing I I should. I would have liked to show, um because I use this slide. We use this at kubecon um and kind of kind of brings a little bit the bigger picture.

B

If, if your made, if it don't don't mind me showing this- um and it's also like a nice finishing touch to the whole thing today, so what we are doing with captain red, we are putting slos into the driving seat of of orchestration. We believe slos.

B

Are there because slo's allow you to describe what you exactly what you actually expect the system to to be to your consumers and is also why I call it the business desired state, because slos are not just cpu and memory or response time what I've learned over the last couple of months. You typically want to specify slos as close as possible to your end. Consumer is your system available to the end user?

B

What's the page load time when people are trying to access, do people actually navigate to your page? Do they buy something? Do they do whatever you want them to do? So you define slos to describe your desired state from your application perspective, but then also, obviously from the infrastructure perspective.

B

So that's what you do and then what captain does right? Captain gets triggered by two types of events. The one thing is when a change is pushed on purpose into the system, meaning a github exchange. Argo jenkins built a new artifact or something or production alert. Something is red and.

A

B

As you've seen me do today, captain can either orchestrate a sequence that is focused on the delivery promotion, assess an environment then trigger the deployment tool. Do some testing evaluate and then promote in a cycle or the other thing is there's a problem in production. Let's assess the.

A

B

Let's then, execute and trigger the actions that have been specified and then always validate, and then either try multiple steps. Hopefully everything goes back to green. If not at the end, you can always escalate, and now the missing piece of this is captain is not doing this themselves. Captain is an orchestrator that looks and make sure that your system is always green based on your slos, but it really depends and connects to your existing devops tools, and it's doing this through the open event standard right.

B

I think that's the the the kind of vision that we have we're putting rest of resolution to the driving seat of your orchestration and making sure that whatever you push out is green and whenever there's a problem in production, we bring the system back to a healthy state.

A

Yes, absolutely absolutely, I think I love this tool. That's give you not just a not build the infrastructure and then things how the, but also when the day two things are happening and they give you the remediation, because that's really challenging. We have ci cd tools. We have devops tools that could then can help you build automated deployments, like ci cd environments, dev staging production, but when something happened at day, two wait not bother about and actually for the devops people.

A

We really bother about day two because application when running, we need to understand what how is operating, how it's performing. So I can see these as we during the conversation talk about the separation of concern that really liked the approach and that suppression of concerns actually bring us to this direction. Where we have tooling in one place, we see how.

A

We can, or captain can orchestrate all those tools can and by but the cd events. If the community agrees to the one standard of connecting to the different tools, I think captain is going to be a standard in the devops and sre world, where I look at at this marquee and thank you very much andres, always when talking to you and learn about captain a lot and it's difficult to cover this all this concept.

A

In one one hour workshop I can see there is a lot ton to more about talk about, and thankfully we have wonderful trainer in our industry. I think checkboard with nana has a wonderful.

B

A

Videos I share.

B

A

With you definitely go check that out as well and also another one I should. I think you read about the cncf incubation by the league chef.

A

And you should read about this blog post as well, if you're thinking about what is actually the cnc incubations me. So it's uh it's been a pleasure talking to you as well. I will be in touch in the more as at the things happening in the captain board as well, and hopefully we can meet. Sometimes the kubecon happening this year in the north.

B

A

And we tell we teleport each other. The story in person live meeting is always been applied. Your friend talking.

B

A

Andres uh always he's been a good talking to you. Hopefully everybody who listened to us learned a lot and if you have question around any of these concepts that we talking about. If you have concerns here, we have the suggestion. The feedback address is always easily and approachable either on the cdf slack channel, or you can ping him on the twitter side as well. So both way you can connect with him or linkedin he's been a very easy approachable person on hanging out devops srd concept stay safe. Everyone stay healthy, bye, bye.

A

We hope to meet you again in the future. With another episode, bye, bye, everyone.