keptn Keptn Webinars, 6 May 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Workshop: Setting up Event-based Progressive Delivery with Keptn on k8s

Description

Keptn is an open source event-based control plane for delivery and operations of cloud native application. Keptn separates the process defined by SREs (Shipyard) from the actual tooling defined by DevOps engineers (Uniform) and the artifacts provided by Dev. With Keptn you can add or update a stage or replace tools without changing a single line of pipeline code and that saves time! Keptn uses a built-in deployment scoring mechanism based on SLIs & SLOs used to enforce quality gates and to automatically validate auto-remediation in production.

Join this workshop where Andreas Grabner will guide you through

1. Setting up a GKE Cluster
2. Installing your first Keptn Control Plane
3. Have Keptn deliver a cloud native app through a multi-stage delivery pipeline
4. Have Keptn enforce quality gates based on SLIs & SLOs
5. Have Keptn enable Blue/Green for production deployments
6. Have Keptn orchestrated Auto-Remediation in production
7. Have a lot of fun with cloud native

Workshop attendees have to bring a laptop that has

1. git, gcloud & kubectl locally installed
2. access to *.cloud.google.com, *.github.com & *.dynatrace.com

We will use Free SaaS Services from Google Cloud, GitHub and Dynatrace. In preparation please

1. Sign up for a Google Cloud Account if you don’t already have one. This will create an account with an initial $300 GCP credit. If you sign up for a corporate email you can apply for an extra $200 in credit by following step 1 in this page: https://keptn.sh/docs/quickstart/setup_platform/setup_gke/
2. Sign up for a GitHub account in case you don’t have one. We will create a workshop specific GitHub repo for our Keptn project during the workshop
3. As Keptn relies on Monitoring data to enforce SLIs & SLOs we will be using the free Dynatrace SaaS Trial, one of the Keptn supported monitoring tools. Feel free to signup for a Dynatrace SaaS Trial upfront but best is a couple of days prior to the workshop as the trial by default only runs 15 Days. Signup here: http://bit.ly/dtsaastrial

Social Profiles LinkedIn: https://www.linkedin.com/in/grabnerandi/

Twitter: @grabnerandi Slideshare: https://www.slideshare.net/grabnerandi

Bio:
Andreas Grabner (@grabnerandi) has 20+ years of experience as a software developer, tester and architect and is an advocate for high-performing cloud scale applications. He is a regular contributor to the DevOps community, a frequent speaker at technology conferences and regularly publishes articles on blog.dynatrace.com.

In his spare time he is hitting the salsa dancefloors of the world.

Learn more: https://keptn.sh
Get started with tutorials: https://tutorials.keptn.sh

A

And here we go perfect, so I know you have the slides, but still, I would be great if you kind of follow along. uh What I have prepared is a workshop just to give a quick agenda. I will see how we do with timing, because, obviously it depends on how this really works remotely, but I want to give a quick overview of what I've planned today. Then we do another check on the prerequisites that should be fairly easy because I know most of you know how to create a cheeky cluster.

A

All of you have said you have a dynatrace tenant, jose, I'm not sure. If you have a dynatrace tenant in case not go to dynatrace.com and register for a free trial, it only takes a minute or two. Then I will give an introduction of what problems we actually want to solve with captain. Then we walk through the installation where we are on boarding as a very simple micro service that I wrote then we're adding quality gates, which is a very key component of captain.

A

um The application that I have prepared for you actually comes with four different builds, so we're running more builds through that captain delivery pipeline, then, where we are going to extend the quality gates with additional slice and slos and then we'll see how, if time permits, we have some advanced topics for you. So that's kind of the idea um and also I know I gave you the slides on purpose. Normally people say don't give slides away because then people get get sidetracked. I assume you're all grown-ups. You know best what to do with your time.

A

You can obviously, if you feel bored go ahead a little bit um but yeah. That's why I think it should not be distracting still focus also a little bit on what I'm telling here. So what we have planned today and I can see- I think, there's more people joining now. That's great! uh I will do another quick shout out to everyone, but what I've planned.

A

So I have an application that I've pre-built for you already and there's four docker containers on my personal docker hub on the gretna and a simple node service version, one two, three and four. So I've pre-built them for you and pushed them up. I've also prepared a couple of cheap meter tests, because once we later on deploy our app, we will have achievemeter test executing load against the different endpoints of the app I've, also created a helm, chart and prepared a helm chart for you and slis and slos.

A

So what metrics are important for us to be enforced in the quality gates and against which objectives do we want to have them kind of validated what we will do? So this is what I've prepared and I have everything in the github repo which we will later check out. So we can use these files what we will do together. We will install captain our open source project on gke.

A

We will onboard my simple node service app for a two-stage delivery pipeline.

A

We will then I guess I didn't fix the animation completely. We will then uh have actually let me let me put this out here, because I guess there we go perfect. uh We will also have captain to automatically configure monitoring in our case today we're using dynatrace in case. You want to see the whole thing not with dynadris, but with prometheus. We also have tutorials online.

A

Captain will then also automate our deployment test, validation and promotion, and I think you can read for the for yourself. The most important thing is at the end, and this is, I think, the reason why we built captain captain does not require any pipeline code. Everything is declared, captain doesn't require any hard-coded tool integrations to actually do the deploy the test and the validation it's all event driven.

A

So once we have everything set up and we are going through my use cases where we deploy my app in version one two, three four, then uh what captain will do it will do the deployment for us in staging we'll, run the tests and then promote it into production?

A

uh What I've highlighted here in the animation later on we'll get to a point where we will deploy version number four, so version number four will, for instance, be automatically deployed in staging it will be tested. Then captain enforces the quality gates based on the slis and slos.

A

If it's good, it will actually promote it forward into production. We're doing canary deployments if it is turns out that production is bad. If the canary is bad, it will roll it back. Also. What we get in the end is a nice overview and uh and the feedback from captain on what is kept doing. Currently, uh how are my individual builds that have passed through, uh so I can go back in history and see everything all right. So this is what I've what I've planned today.

A

I just see. Let's see who else has just joined adil, I see you just joined welcome to the party. I am reposting a couple of things in the slack either in the zoom chat for you.

A

uh What this is is an invitation to the slack channel, so we have a captain's leg channel in general. I think it's great to be part of that channel in case you want to continue working with captain and then have access to the development team. We also set up a special channel today in that slack workspace.

A

That's called stockholm, gdg meetup may 2020 so where you can have site conversations that may not fit into the into the core part of the conversation here, and I have johannes and andreas helping me with these and then there's also the link to these slides.

B

Okay, people are joining in jose.

A

Now jose now I see you with a picture, so I assume now it's working.

B

All right, let's see if this is working.

A

Now, just do a quick check jose and adil. Can you hear me, can you see my screen.

A

I mean I need to unmute yourself. I cannot hear you right now.

B

And there's more people joining now, maybe.

A

C

B

The beginning with the.

B

People that are on.

A

Time in in stockholm anyway, I will resume, even though more people are joining uh because now we're coming to the part that I know those people that have been here earlier today already have, but let's just make sure we have all the prerequisites so folks prerequisite wise a reminder.

A

Join the community join the slick, so you can find all of the information on the captain community on github.com captain slash community. This is where you actually see the exact same information that I'm posting here uh get in touch with us by email, but you know very important. Join us on the slack channel. This link here also is the same uh registration link uh that I also posted in the uh okay. Jose has some connection problems. A deal works, okay, perfect, so adil and jose. If you have connection problems, that's okay!

A

Hopefully you can follow along we're, recording the whole thing, so you can also watch it later on. I am make sure you look into the chat window and download the slides. So in case you get lost. Everything is in the slides uh and we are now in the part where I will walk over the prerequisites.

A

So join the community. um There's also, uh I think, from a community perspective. This is also why johannes and andreas are here from the captain team. uh The captain can grow through its participation or through the community participation every thursday at 5 pm central european time. There is a captain community and developer meeting so where they are now discussing. You know what's currently going on in development and showing progress, and this is a great way to also give feedback, and if you also would like to contribute, there's different ways of contribution.

A

You can contribute your own sandbox project in captain sandbox, where you can extend captain, but obviously we are also very happy if you want to contribute to the captain, contribute captain core project and there's more information that you can find out on the community itself and the best way to get started is actually join one of these community developer meetings.

A

Good all right, let's see if anybody else has joined, there's another.

A

There's a couple of people that I think that have joined, but I will just continue good, so prerequisites. The first thing we want to do is we want to create a g key cluster. um I really want you to create a new cluster. I want this to be a sandbox workshop, tutorial, clustered we just use for today.

A

Why do we want to do this because, yes, we can easily throw it away in case something happens um also if we install on a clear cluster, uh if we don't have any side effects of any other software that you may have pre-installed and we don't have to work around it. All we need today is highlighted here so select a version that is greater or equal to 114. I will pick 115.9, that's the one. I've run my tests against give it the cluster a good name it.

A

We only need one node, the node should be ubuntu and uh the machine type is n1 standard eight. So that's what we're using this is kind of the uh allows us to not only install captain on this cluster, but also deploy some apps later on, and I will do this as well. So let me go to to my environment here, and actually this is the wrong browser window. Where is my browser window?

A

The one that I want here we go, so I have my my google cloud platform console and what I will be doing, I'm creating a new cluster in case. You have never done this before it's. It's really. You know intuitive, I would say, and also the slides contain, all the information I call this now. What did I call it in the slides captain tutorial?

A

I leave it in this region. As I said, I just use the same version that I've used for testing my tutorial, 115.9 gk. Now don't click on create yet there's more things we want to configure. If you click on the node pools on the default pool, we are changing the node size, the number of nodes to one again, we could go bigger, but it doesn't make sense to waste money because we just want to show you the tutorials and everything end to end so change the size to one don't check anything else.

A

Everything else is by default and then, if you click on nodes, so under node pools under nodes, there's two things we want to change here. We want to change the image type to ubuntu.

A

Here we go and we want to change the machine type from n1 standard, one to n1 standard eight. This is just it gives us enough cpu and memory to deploy a couple of apps later on right. It's not that captain needs all this footprint. It's a big machine yeah! It's not that captain needs all this, but it is also that we can deploy apps and, as I said you can, you can tear this thing down after we're done today.

A

Also what you can do is we have a special link where you can get 200 extra dollar. I think it's two or three hundred extra dollar from google credit.

A

If you are signing up, there's a it's also in the in the meetup page. um If you follow the links there, you can you end up at a google form where you fill out. Your email address be aware, don't use gmail. It has to be a business email because they're waiting that the emails are correct and not just somebody's email. But then you get an additional credit that you can apply uh to your account here, all right.

A

Let's click on.

A

Create and that's it this will take. Probably um you know a couple of minutes, so that's my my first, uh my first step in my prerequisites, so hopefully this you know very easy if you have done this before this should be really straightforward.

A

uh If I've not done this before it should also be not that hard. It's three steps and you have the slides that show you all the screenshots right. All right next thing is once the cluster is up. We can connect to it now, there's obviously multiple ways to connect to it. I will be using the cloud shell, um which is really convenient, because google takes care of everything. I don't need to connect for my local machine, but whatever you feel comfortable with, I will later use cloud shell, but I will do this.

A

I will come back to this in a minute because creating that cluster will probably take you know two three minutes so I'll just jump to the next prerequisites, and this is something I forgot to do- the the animation. Oh thanks, johannes, for the voucher link, that's perfect! So, johannes, just posted the link to the voucher in the chat window.

A

I will walk you through this, but we need three pieces of of uh of data in order to let captain connect to dynatrace because we're going to use dynatrace as a monitoring tool. As I said earlier, if you don't want to use dynatrace later on, but if you may want to try the whole thing with prometheus, which is also supported, you can go through the prometheus tutorials, but for today we're going to use dynatrace and there's three things we need. We need the tenant, the dynametras tenant, that's the url to your dyna trace environment.

A

We need a so-called api token and we need a so-called pass token and I always like to use a cheat sheet. So what I would like you to do is just open up a text: editor notepad or whatever you have visual studio code and let's collect this data, so I will just show you the way I do it. So I will go to my dynatrace tenant.

A

B

A

So the first thing- the first thing I I need is the url. So let's do this, I'm using a notepad here, just as I said- and I paste this in here- so it's just a url without the https in front and what do we call it? What do I have in my slides, dt tenant.

C

A

I already prepare it for later on. I guess you can tell what this is going to be later on. Okay, the next thing is going to be dt api token, and the last thing we need is dt pass token. Okay, these are the three things that I put in my cheat sheet now.

A

Let's do this for like again, don't copy stuff that you see on my screen. You do it yourself. um I know some of you might not be familiar with anything so I'll keep this slow and easy. If you are logged into dynatrace, then you end up on some dashboard it might it's probably not the one that you see here, because this is something I built, but it should be fairly easy to figure out.

A

What's your url for sas planetary, sas customers or size tenants, it's always like a an eight digit like it's three digi three, three alphanumerics five numbers, then dot live.dynajs.com.

A

If you're running damage with managed, which means you run it on premise, you have a url that is longer and it's also it needs to be a url to the environment with a slash e and then your environment id that's what we need now. The second thing we need is an api token. So you click on settings and if you click on settings- and let me just reuse refresh my browser here- there we go- I still have.

A

If you click on settings, then you end up in the settings menu and if you scroll down, there's a section called integration and under integration. The first thing we go into is dyna trace api.

A

This is where you can actually explore all the uh the api options, if you're interested in the swagger ui or read the documentation, but just as with most other tools, what we need in order for an external tool to talk with dynasty is we need to generate a so-called token right, so we click on generate token. I just give it a name. I call it captain workshop meetup, it doesn't can.

D

A

Can be andy can be julio, it can be carlos. The name itself is not relevant. What is relevant here are the things that we have to select here, because captain will not only pull data. It also has the capability to configure certain things in dynatrace.

A

So if you look at the slides, I hope the slides are good enough if you have downloaded them, but the things we need to turn on is the first two then read configuration, write, configuration and so they'll maybe make a mistake: capture, request data and the real user monitoring javascript tag management. So these are the one two three four five six things we turn on all right. Then we click on generate, we've just generated a new token and then you'll see the entry here in your table.

A

You expand it and what we really need is that token, here, okay copy, all right, I'll put it in my cheat sheet, and I will make sure that later on, I will delete that token again, because I know this is recorded and not that somebody is messing around with my tenant. So that's the first token. The second token, it's a so-called pass token.

A

What is a pass token, so the data just api token is to communicate with dynatrace to pull data to send data to configure the pass token is for installing a dynatrace agent on a particular environment. So what captain also automates is, if you install captain on kubernetes on the kubernetes cluster, it can automatically install the dynajs agent on that cluster, so that you get full stack visibility, and we also want to leverage that capability that captain automatically rolls out dyna trace for us on that cluster, and for that we need a so-called pass token.

A

So we generate a token. I again call it captain workshop pass token. The name doesn't have any meaning and.

E

A

Really, what we need is now the token itself copy, and then I put it into my cheat sheet all right. I hope this is all good hope. This was easy to follow so now I will move this a little out of my way, because otherwise it's getting a little too crowded here, perfect good. Let's walk back to my google cloud platform console.

A

So in my case, I have my captain tutorial here. So my kubernetes cluster is up and running, and what I now want to figure out is: I want to connect to it right. This is also, let's just make sure we can connect to it. uh If you have never done this before again, follow the instructions.

A

If you want to connect to it from your local machine but really like you need to then install gcloud locally, but I run it in the cloud shell. So if you click on running cloud shell, I really like this. I get. I get a uh a bastion host, basically provisioned and I can I'm automatically logged in and it automatically adds the gcloud comment into my console and I hit enter, and now I have a machine where I have cube ctl installed. Gcloud is installed.

A

My cubectl is already automatically configured to talk to my to my kubernetes cluster and I have all sorts of other tools installed and if I go quickly back to my slides in my prereq slides in the previous one, I think I said once the cluster is up.

A

Let's try this and just log in and that's it right and what we can do just to validate that everything is good. Is we can go into this shell and say cube ctl, for instance, get notes? Then we should be able to see the one node in my default pool with version 115.9 all right.

A

So I want to get a quick feedback through the chat. Please, if you can do me a favor. Let me know who has reached that point with a quick. Yes,.

A

And if somebody still needs more time, then also, let me know just by saying time: okay.

B

B

B

A

Yes, I'm not sure if I have all the options on the dynamics. Key.

C

A

uh Roll on the the options on the dynatrace key, they are in the slides and the thing is you can always change them as well. So let me show you that just the options again. So if you go back to your dynatrace api- and you can see here, if you go back, you can always switch them. If you switch then make sure you also save these changes, and let me just keep this up.

A

It is number one number two number six number, seven number nine and then whatever that is fourteen, so access problems. First, two read and write, configuration capture, request data and then the real user monitoring.

A

uh Julio is asking in case I have a private cluster. Do we require further steps in the configuration, so with private cluster you mean you have a not a g key, but you have just a just some type of kubernetes cluster on your local environment.

B

A

uh Julia, you can also, if you want to just turn on the audio quickly. We are a small group. You can just ask the question.

F

Sorry, no I mean a private cluster with each node is, has no external public id.

A

uh Well, in our case, we need an external ip. The way my my sample works because we're going to use the the external ip uh to access the api and the bridge um I mean it has to be an ip that you can access. Somehow right I mean, if you are we're going to create a an ingress and.

B

A

And as long as you can access that ip, then it's obviously fine.

F

Okay never mind okay, thank you.

G

The the same is this name is for another cloud provider. Continuing with that with what julio says. If I want to use okey in oracle cloud, the step to follow remain the same right.

A

Yeah sure it should be yeah, I mean again in the end we are. We are going to install pots on a kubernetes cluster and we are going to expose certain services in these pots through an ingress through virtual service, and we don't do any new magic that hasn't been done before.

A

That's the thing, maybe certain things on certain kubernetes cluster are a little special and different, but in general it should it should work what we are going to install later on, with the captain full install it is going to install istio, and that is installed because in the default installation captain also has the capability to apply helm, charts and do blue green deployments and right now we're using easter traffic routing.

A

For that, that's why, by default, if you install captain, we will install istio, we do have some additional installation options where you can, for instance, repurpose your existing istio.

A

You can also there's a new installation option coming up next in the next release, where you can repurpose any any of your ingresses um sort of stuff coming up, but for this workshop today I just assume we just take something. You know a fresh cluster, so we have no problems with any other. Software is already installed.

D

A

Good, so let me quickly go back to my slides. um We got the prerequisites, that's all good, so quick introduction, um because I want to make sure that we all understand why we're doing this whole thing and and and why we believe you know, captain actually brings value. So, first of all what is captain? I know that you have already done your homework because you looked at what captain is. Captain is an event-based control plane for continuous delivery and automate operations, mainly for cloud native apps, but actually the way we've done it.

A

Captain can also not only not only deploy cloud native apps on kubernetes through, let's say helm. We can also use captain to trigger a jenkins pipeline or a gitlab's pipeline, or an azure devops pipeline or xibialabs pipeline. That is then doing anything anywhere. In the end, it doesn't matter right. It's an event-based control plan, but the default use case that we that we see it is for new cloud native projects where people also want to do delivery.

A

But again you can. You can bake in whatever you want, whatever tool to do whatever job in whatever way good. I think the magic thing is here, and you will see this: the application, delivery and operational processes. It's all declarative. It means that you won't see any line of pipeline code. In our case, it's all you declare what the process is and that's true for both delivery and also automated operations for sale, feeding and other remediation.

A

What else do we have? Yes, we are an event-driven system. We are adhering to the cloud events standard. That means it's a standardized communication protocol that we have, which makes it very easy for you to actually extend the product. You can extend captain by writing your own so-called captain services that can either consume these cloud. Events that captain sends out to do certain things like deploy test.

A

Give me the sli metrics promote, or you can also easily integrate with captain by sending events to captain by triggering workflows, and so these are the things it's very easy to be done.

A

Also. The reason why we're building captain is because, when we look at our customers and the people we talk to, we know that a lot of especially large organizations that have a past with let's say the traditional way of software delivery, the monolithic architectures um and now I want to go to the cloud. A lot of people you know are not con and I just you know, they're, not cloud native by nature.

A

Some of them just think we can just uh you, know, wrap an app into a container deployed on any of these platforms, and that makes us cloud native, but that's obviously not the case, a monolithic app that was never built for scalability. There wasn't that. Never that was never built to leverage the the benefits of an elastic infrastructure. That is event driven, will just not be by definition, be cloud native and won't be able to reap all the benefits.

A

And now the problem is not only is this a problem, but we also see that a lot of these organizations have started to adapt continuous delivery, which is great, but over the years there are continuous delivery pipelines that may have started small to do one thing that they build and then they add it deploy to it and then they added test to it and then they added notifications to it and then they added another stage.

A

So these pipelines, all of a sudden, also became monoliths, so they they're, basically just the mirror and the reflection of the monolithic source code counterparts. That's basically what it is, and so, as I said, there's a lot of mixed information in there about the process. The target platform, maybe there's a lot of if statements, a lot of hard coded things in there.

A

The big problem with this is obviously there's no clear separation of concerns, because everything is mixed in one single file that should be separated, because different people in different roles within an organization should have different responsibilities and they should be clearly separated right and that's the challenge. So, unfortunately, the experience what we have is that we have pipelines that have a lot of custom-built integrations with a lot of tools and they're also often copied multiple times in organization.

A

Somebody has a pipeline, you copy it, you paste it and then it kind of gets a snowflake effect where what we really want is not this, but we really want to have a seamless integration. We want to have a platform that is easy to use and that easily allows us to change processes, change, workflows, replace the tool without having to change all the plumbing in the back end. So what we have said to do with captain from a continuous delivery perspective is really uh removing uh these hard dependencies.

A

So if you look at the let's say these monolithic pipelines that that we see out there, what they do is actually individual building blocks, and so we want to. We want to remove those right. We want to kind of break up these monolithic pipelines and and then really focus on the individual pieces of integration. So once we remove them, these building blocks right and connect them, not through a hard-coded spaghetti code control chain, but through eventing, then we can all of a sudden say.

A

Well, we have a clear defined kind of process, build, prepare, deploy, test, notify rollback, but instead of having everything hard coded, we use eventing to then actually trigger the individual events through events, and then whoever can pick up that event and execute that action can actually execute the election. So in captain you will see there will be events like a configuration change event, a deploy event, a test finished event, an evaluation done event, there's different events that are sent along the process as an artifact is moving through that process.

A

The process is declarative and then the events are then there to actually really trigger the right action behind the scenes. Okay, so the the most important thing is that we have really separated the process from from the tooling here.

A

I hope this still makes sense for everyone all right, good last words on the captain architecture. um So if we, what we've done is with captain right, we basically not only came up with that. There's a problem and we have a solution from it in theory, but we actually have a real-life implementation.

A

So if you install captain- and we will do this in a second on your kubernetes cluster- what you really get is a control plane that can be controlled through an api there's also a ui. We call it the bridge, but it's it's down through the api and the control plane is using events, well-defined events to trigger events to to, basically, you know, execute workflows um on on artifacts or on problems. For instance.

A

Now the separation of concern comes in that, if you have the control plane, one group in your organization, maybe the devops team, who is responsible for actually defining which tools should be used. They can customize, captain and say hey. I want to use these particular tools to handle these particular events for these particular types of projects and in case in a year from now, you are switching from one test provider to another testing tool or from one deployment tool to another deployment tool.

A

You don't need to change all of your pipelines where this integration is hard coded. You just change the uniform and they kept from now on. The deployment event is no longer handled, let's say by by by helm, but it's handled by some other deployment tool that you have, or we are switching, let's say from a jmeter testing service to a commercial offering from neoload and things like that. Okay, so you define the tooling in the so-called uniform.

A

On top, we also have the ability, as I said, to define the application plane the actual process for delivering operations. Again. This is typically a different group of people. They define the process like how many stages you have what type of deployment strategy is it direct? Is it blue green? What type of test strategy do you do just functional testing? Do you do performance testing? Do the security testing? Do you do no testing at all in a certain stage, this is defined in the shipyard file.

A

I have clear separation, the control plane with the tools on the bottom and the process on the top, and now, if the developer comes in and says, hey it's awesome. I now have a platform that allows me to move my artifacts through a well-defined process and the right tools are actually minim. Ensuring and executing the process.

A

A developer, then, can just say: hey captain, here's an artifact, here's, a new artifact do something with it and then captain will take it and basically push it through the delivery process that is defined on application plane and is executed by by the execution plane on the bottom by these individual tools.

A

All right, I also added the captain architecture diagram. I think this is you know, hopefully quite up-to-date. um There's you know the latest and greatest are always up on on github, obviously, but just to see what we actually install when we are installing captain later on we're installing a bunch of different components in the captain namespace in kubernetes, uh with the core components like the api, the event broker the distributors.

A

We have the bridge the uis, that's the ui, with a mongodb where we, where we do a store results, and also we have a configuration service that is talking to a git ripple. That also comes with captain because everything uh every configuration element we give captain whether it's an sli, an slo, a test script, a helm chart is, uh is version controlled for us automatically in captain and then the captain services. These are. These are the individual individual tools that are doing things they're.

A

Also, then part of the uniform being installed in that cabin namespace on the bottom. When you use captain for delivery- and you say I have an artifact, I have a service and I want you to deploy it into dev staging and then production. Then captain will also automatically create kubernetes name spaces to separate these um these stages. This will happen automatically when you use captain for delivery or the let's say, the built-in capabilities that captain brings through uh through help deployments all right.

A

So last couple of animations, so you see this kind of in action and then we actually go and install so remember, rethinking pipelines! We, if you have written jenkins pipelines before anything else, they can become really messy.

A

Yes, it I mean jenkins, is a great flexible tool, but in large organizations it can be very messed, become very messy, um and it's going to be hard to maintain it and fix problems and find people who actually wrote this piece of code back then so with captain once it's once, you have installed captain and you have the individual captain services that are listening to different events to do certain things.

A

This is what we'll do later on. So when you start with captain the first thing we do is say: captain create project and we give it a so-called shipyard file. The shipyard is again remember the application process, so in this case two-stage pipeline staging and production, you define a deployment strategy per stage and also test strategy per stage, and what captain does with this?

A

It will automatically create both staging and production, not only git repos, so internally it keeps a git repo, so it creates a branch in in the git repo for staging and prod, where it will later then push and store and persist. All the the stage specific files like a helm chart like your test files, your slis and your slos.

A

Also once we start deploying captain will also create namespaces for staging and for prod, so that we have a clear separation here, all right next step, everything is code, so we've created a project.

A

The next thing is, we need to uh onboard a service, so a project might be your webshop and then a service could be your cart. Your front end your backend. Whatever else you have your recommendation engine. So when you onboard a service, you basically give captain all the different assets that the individual services will need later on the helm charts for deployment. Maybe your chain meter tests, your slis and your slos.

A

They will all be uploaded to git, that's important, especially for helm. For for helm, not only the helm will be updated, but if you decide to do a blue green deployment, then we also automatically configure istio and also create the correct deployments for the blue, green and the deep deployment definitions. So this will all be done completely automatically for you. You just give it an initial helm chart for that service and if you have specified blue green, it will take care of you for that all right. Now.

A

How does the delivery then look like once we actually have set this up, then we can say captain I'm a developer. I have a new artifact and you point it to a container image later on. We will use my simple node service and what will happen is captain will send an event internally for the first step, which is hey. I have this new artifact, so the first thing is: we need to actually update our helm chart with the latest version information, so it will update the the values yaml with the right version.

A

The second thing is once this is done, it will send an event for deployment. So if I've specified direct deployment, then the helm service, in my case, will pick it up and say: okay, I'm just applying my helm, charts that have already been prepared for me for direct deployments.

A

The next thing is once the deployment is done, it will send and next event that should be targeting tests. So the next thing that will come in is the test tool. In my case g meter will then say ah I have a new deployment. Captain told me about it: here's the url and I'm just executing the tests that the user earlier uploaded to the github repo for staging pulls the tests executes them once they are done. The next phase becomes evaluation and scoring.

A

This is now where captain pulls the data from the monitoring tool that we've configured it could be prometheus. It could be dynatrace today we're using dynatrace and pools in all the metrics that we have specified in the uploaded sli and compares them with the slos with your objectives. It then comes up with a score between 0 and 100, and if you have achieved your your own threshold, if you're above that threshold that you can also specify in the slo, it will then either promote it or decline it, but if it's good, it will promote it.

A

If not, it will keep it there if it is promoted. The whole process just goes on for the next stage update the configuration file. That's the blue green deployment, then tests. If tests are available, maybe in the beginning we don't have tests. This is actually what we do later on. We are uploading tests to the later stage, but still the scoring, maybe based on different slis and slos, and then, depending on that we can also decide. Do you want to keep this yes or no right? There's also another team is on is on here.

A

We are right now we're automatically promoting artifacts from one stage to the next. In case you achieve a good grade, but in 0.7 we will introduce also different approval strategies. Where you can say there has to be a manual approval for a particular type of scoring range right. So there's a lot of things that are coming in good, so I think we are finally ready for the installation before we do that is there let's see. Did anybody else join that I haven't seen earlier?

A

I don't think so. Oh john has joined. I think I've not seen john john, I'm not sure if you can't hear me, but it seems you're connected um make sure and I'm reposting this in the chat window. There is a slack invite. There's also a link to the captain tutorial to the tutorial here, the slides and also uh the channel that we've created in that captain's leg. If you wanna, if you have specific questions later on good and john, what we've already done, I'm not sure when you joined, but we have already done the prerequisites.

A

uh So we already have a cheeky cluster and the dynadress tenant for all of us. Okay, so um highlight again or high level. What are we going to do from now on? Remember what you need in order to install captain is a is a kubernetes cluster, where I also added the ubuntu linux option, because we also have a very cool script that can just install micro kubernetes with captain and with everything else you need on on an ubuntu. uh The project is called captain in the box on the um on the captain sandbox project.

A

So this is what we need. We need a place to install captain and for our purpose today, because we're using dynatrace, we need the dynagris tenant, the api and the past token, which we've created earlier. If you run, I want to do the whole tutorial using prometheus. Instead of dyna trace, you can also do this just go to tutorials.dynaj.com tutorials.captain.com and then you will find the prometheus specific tutorials. So these are the prerequisites now in high level.

A

What we're going to do we're going to install captain which will install a lot of cool things in your kubernetes cluster. Then we are going to create a secret where we store the dyna trace credentials so that later on, the inter integration between captain and dynadress can actually talk with each other. These two tools in order so the actual tools that will do it or the actual services there's two services, there's the dyna choice service and the dynamics sli service.

A

So one is there to make configuration uh change, events um or they're, making configuration changes um and then pushing events. The sli service is there to pull data. Then we also say captain oh yeah. Thank you, tutorials.captain.sh. Exactly thanks for posting that uh in the in the chat, then we'll do a captain configure monitoring we need to.

A

After installing these tools, there's an option where we can actually say: hey, dynatrace, configure yourself right, you're installed now now go out and install whatever else. You need that's what we need to do and once that is done, we can actually set up our project. Remember I have a sample app, so we will create a project.

A

We will create a service and then we do again configure monitoring for that particular project so that we really make sure that for this captain project we're using dynatrace, because you can actually have different monitoring tools being used for different projects and we add all the additional resources like slis slos and let's, for instance, say that jmeter script to it and then we can go off and say, captain deploy my stuff good. So now, let's get our hands dirty. um Let's go back to the first step.

A

Is I want to go back to my cloud platform here we go. Hopefully this is not too small for you, but you should have it anyway in your screen. So what all I'm doing now- and this is the way I structured the the slides everything that is in everything I'm going into this mode now, uh all these can be copied and pasted right, don't copy the dollar sign.

A

I guess I should have not if, when I ran through this uh in preparation, I copied the dollar sign so often and then obviously the bash shell tells you what's the dollar sign all about, um but I just you know. Essentially we can go step by step. The first one was just to validate cube, cuddle, get notes. We did this earlier, so we are connected if you're doing this not from from your console, but maybe from your own workstation from your own uh terminal, just make sure you're connected to executed kubernetes cluster.

A

The next thing is this curl comment.

A

Perfect, this query comment is downloading the captain cli and it's actually installing it. So with this um you know we're downloading the latest version and in order to validate, we can also do captain version now and it should say cli version 0.6.1, so downloads it installs it. It's just the cli, the command line. Interface.

A

Now the fun part we install it. So captain install platform gke and I enter now, and then basically it asks you. So if you do cheeky, it basically looks at the um it looks at your uh at the g cloud information and basically um you know which cluster you're connected to and so on and so forth, and this is why it gives you the defaults, and I have to double check that this is really my name. Captain tutorial.

D

A

It also uses some cached values and earlier I used a different name. So this is my cluster. This is my song. This is my g key project. Everything is good. Do you want to really install it? Yes, so now I'm installing captain this will run about five minutes.

A

I'm sure you have already guessed captain install platform. You can add different platforms for gk aks, eks openshift. You can also say platform equals kubernetes. What kubernetes means it just gets. The current cube config a cube, ctl, config and installs it to that um to that kubernetes cluster. If you do cheeky, it uses the connection information from gcloud and so on and so on. So this will run.

A

What did I say in the slides about five minutes right and we'll now install everything that I've highlighted? Where is it installed? Does it generate some kind of home captain folder? No, what it does! That's a good question it. Actually, this the cli is uh downloading an installer pot and basically deploys that installer port into in your kubernetes cluster, and then captain itself is installing itself within kubernetes.

A

Okay. So really what happens? Now? If you look at this slide here from the outside, I say: captain install the cli connects to your kubernetes cluster, deploys the installer. They have a little installer pot and that installer part is then deploying everything from within the cluster and andreas and johannes. Please correct me if I'm wrong, but I think this is the way it works.

H

That's completely right and additionally, of course, we are creating a home folder, so as junior suggested.

A

So they mean the you write, the um the the captain config, because it will not only install it right it will. You can see here what it's doing. It's installing istio, it's installing captain uh tiller, also some good news. The and this is actually worked at andreas. I think you did. uh We are moving from helm2 to helm3 soon.

A

Very cool, but this current version still has uh helm using cell m2. That's why we have tiller.

B

And everything else there perfect.

A

So after it's installing it, it is also and julia. I think that's exactly what you were meaning right. It will uh store the configuration about how to communicate with the captain api later on. How can the cli communicate exactly with that captain installation that will be stored in a captain folder in the home directory.

B

H

In case you are using mac os, we are using the keychain which comes with mac os, so nobody can then access the api token by just opening a file, and this is also the reason why you have to enter your password when you're installing cabinets.

A

So for me now exactly that's what you just said right, so we stored the the the password and the token and the endpoint in the home directory, but essentially I mean I'm done now. What I can do is I can do a captain status and what it actually oh.

B

A

So one thing I do hope this is just an exit. I o issue.

B

Let's try this again.

A

So, as you can see here, the api or the endpoint is api.captain and we're using cpio as a dns resolver to actually resolve you know, api and the bridge, and so on later on. uh This has worked all the time today, but I know that sometimes we have issues with dns probe finished, so we're using this service. This is only good for demo purposes. I mean for a demo, environment, succeed, bio, there's, also nip bio and some others, and we're actually currently thinking of providing a better option.

H

Now it looks good if you um append spec minus ui.

B

Yeah you're right: let's try this again because.

A

Now the it's connection reviews now swagger, there's ui. Thank you. So much.

B

B

B

Swagger, let's see.

B

Let's try this again.

B

A

Maybe one of you can can let me know if you can those that followed, so you should be able to open up your browser, go to api dot, captain and then your ip address and then slash swagger ui. This should be getting you to the api endpoint.

A

uh In my case. It says connection reviews though now already could it could actually resolve it.

H

And can you show us the cabinet pods.

B

For sure cube, ctl get pots.

I

uh Andreas, what do you think about restarting the gateway.

C

I

The program was pointing to this problem today,.

H

Yeah, let's restart the api, please.

A

So I'm not sure if if anybody else has the same issue.

B

Or the api, not the gateway.

B

Let's try this.

A

And this is an issue with uh with cpao right, because I guess they are resolving the name and then, if the dns doesn't work um again, we're using cpio for demo purposes so that we don't have to register.

A

You know dns entries and so on and so forth for all.

H

So I can now access your special ui.

A

Yes, so you copied my appearance instead, sorry there we go and now my captain, uh my captain status works as well, so in case anybody else run into a problem. That's a good uh practice run uh or good troubleshooting.

A

So what happens is um internally? Obviously, captain resolves the same ip using zip bio. In case you don't specify your own custom domain, so there is an option to specify your own custom domain, but for demo purposes and workshops. You always use that and in case that fails, we just restart the api endpoint good. Let's see what else we have it's installed. What else do I have in here captain status? That's what I had oh yeah and and in the slides, as you can see here.

A

I actually point that you know go to the swagger ui and I also explained what are the individual. I am slow, but sure yeah yeah carlos. Where are you? uh How far behind are you? It has captain installed.

A

Perfect, I will take, you know, we'll, take a break in a minute or two anyway, so you can also catch up in case. It's still running just a little explainer.

D

A

uh The way this works right, uh we're using cpio and, as you saw in case, there are problems you need to refresh until cpao cpo, sometimes they've outages, it's a free service. um The way it works, though, is you have your your front? Is api dot captain, so your service dot name space? Where we install everything in the captain namespace, then it's the ip address, your public ip address of your of your ingress and then dot exit, bio and then, in order to access the swagger ui, you need to just add swagger ui.

A

So if I go back to my console here, where did captain status and it could finally successfully authenticate? I get the url to the api endpoint and that's the one I can obviously use if I want to make direct calls to to the api using a postman or some other tool. But if I want to on yes, I don't have. We don't have a certificate here, but if you want to execu access the swagger ui that allows you to explore the api, then just add swagger dash ui.

B

And then it should load in a second.

A

There we go perfect all right, so this is a just a regular swagger ui, where you can play around and then and then test things out good. Also, if you want to know, um you obviously need a token in order to call the api, we can get the token there's a great piece of documentation on captain sh.

A

If you go to docs and if you go to the releases and the references under captain cli, there's a great great documentation here on how you install the cli and actually sorry the aviation cli, and then we also have the api and in order to use the api, we need the token and in order to get the token just check this one out here we go for linux.

A

It's just these two comments right. It's basically, the endpoint is stored in a config map and the api token is stored in a secret, so you can easily get it jose. I was successful as indicator. That's awesome, so I just want to show you this, though right, if you wanna, if you x, if we execute these two.

A

And then do echo captain endpoint.

A

Then we have that and then, if we do a captain api token, then this is my token, and so I could also directly you know, use any type of tool that can send http requests to that endpoint and using that token and uh and work with captain like this or I use the cli, can you paste the doc page where you took those variables from sure it's here.

A

Exactly, and maybe you know what I will do, I will make a note to some additional slides appendix here. So let me.

B

Just add a slide here: edition information working with the captain api.

A

Get your endpoint and token.

B

And then call it yes, vega or u2 of choice.

A

And then there's also documentation here on the api.

B

Perfect good all.

A

Right so, let's see where we are we're here all right, so if we're all authenticated, if you can all go to the swagger ui now the next thing I want you to do. I've prepared some examples. As I say right of my sample app and there's a captain repo, that's called examples. It has some other examples too, um but we're focusing on one still.

A

I want to go into my console. I do a git clone, so let's just clone that github.com captain slash examples and it comes with a couple of examples. As I mentioned the one that we are interested in.

A

Is the one that's called simple node service, so you can look into this. This is where prepared all the files will will later. We will later need, and it also explains a little bit about the app itself that I've based it on an aws tutorial and that have extended the feature set a little bit and I also have four builds, as I said, and every build has a slightly different behavior.

A

uh Some are good. Some are not that good and uh we will we will leverage this application later on. So first thing, though, is we wanna clone that repo?

A

What's the next thing, the next thing is thank you for posting that the next thing is in that once we have it extracted, we go into the examples sample node service, captain folder.

A

All right so cd examples, simple node captain and in that folder I have prepared a script that is called expose bridge because so far the only thing we have seen from captain is only the api end point, but not the beautiful bridge and uh in the next version of captain and all points in 0.7.

A

What's in 0.62 already, where do we get the exchange six too? So in the next version, which will be released next week? I believe we will get this as a built in captaincy live feature uh in in this version. I just provided an s h, shell script, so what this is doing it is. It is creating a virtual service for the bridge uh so that we can also access the bridge through your browser, because the bridge is really where we see what's actually happening.

A

So if you open it up, then we get the beautiful dns error, probe message from cpao and once we refresh a couple of times and then pass the security warning, then we get to the bridge, obviously no projects yet but we're getting there. But that's the that's the idea. Now, if I go to the slides exactly.

B

A

All right so quick check um who has the bridge, who sees the bridge? Who is there.

A

You can either use it audio or just post it in the chat window. I guess nobody's there yet.

A

I'm here perfect roland awesome, that's great, perfect, all right! So then I I have one more slide. I want to walk you through and then I also need a quick bio break and then I give you time to catch up.

A

We working you, that's awesome cool, so the they remember um the little cheat sheet that we created earlier and if you look at the where's my cheat cheat. This is my cheat sheet for the dyna trace credentials, so dt on the square tenant your tenant. This is very important, no http in front no slash in the end, the indeed the underscore api token, the token that we created earlier dt underscore pass token, and then your pass token, and if you take these and we put it over.

A

In my console, that's good now the reason why we need this because we need to install the dynatrace integration and in order for the the two dyna trace services, we have this the dynasty service and later on the sli service to actually talk with diana trace. It needs to have access to these credentials and we store it in a secret. So the first part up here, the first cube ctl captain create secret copy. It will paste it in boom binary secret created. So this creates a secret called dynatrace in the captain.

A

Namespace, that's the first thing. Second thing: is we actually want to install the dynametroid service itself, so the dinatris service is listening later on to different events in captain a deployment event, a test finished event, an evaluation done, and it will send this information to dynatrace. So dynadris knows ah this is when a deployment happened. It was this version. It happened by captain. ah This is when the test started. This is when the test stopped.

A

This is the evaluation result of the quality gate, so this service will make sure that all these events are sent over and so, let's install it cube, ctl apply.

A

This is actually a newer version that thanks thanks to christian who is not on the call today, but he helped me get some of the changes in. So this is a special release. 0.6.9 build so now the service is installed and the last thing we want to do is we want to do captain configure monitoring what this does. It sends a a an event to the dynameter service. We just installed and says: hey, please configure yourself on this captain installation and what the damages service will do. It will install the diameters one agent.

A

It will, as you can see here, it will create a couple of uh tagging rules. This is for the load, testing integration we provide. It will create, come some dashboards and alerting profiles, so it it automates the configuration of dynatrace for this particular kubernetes cluster. For you.

A

um What it does is also, you know, document it, and you can obviously look into dynamite with installation later on to validate what we did, what we did, but this is now kind of enabling dyna trace for that cluster.

B

A

A

Let's do it that way: um let's do it. Let's make a quick three minute break. I just need to refill my water and uh go to the bathroom I'll, be back. Let's continue in five minutes uh if you kind of fell behind a little bit, use this time to walk through the route last to the next steps uh until slide 27, and we will resume in five minutes hope. This is okay for everyone, perfect.

A

A

B

A

So I'm just trying to figure out I john whoever I I saw you you. I think you joined a little later. Can you tell me if you can hear us alright and if you could follow along either speak up or use the chat window? I'm not sure if I saw any of your messages just want to make sure you're there and you could kind of follow along.

A

Well, maybe you're not there anymore and you something something better was available on stream on netflix. Who knows.

A

Good, I hope everybody's back. I think I said five minutes. I believe it is five minutes now. I wanna continue so julio carlos roland you're. All there.

A

What time is it here? It is 7 23 p.m. So it's in the evening 7.

A

30., I think you should see it in my screen- share right on the bottom right: ah yeah yeah, it's a little easier in uh in south america right six hours behind.

A

But we we are with a good pace, so I think we will be. um We will be through this in about an hour, which means I may even get another. You.

H

A

Get some daylight at the end of the day here, all right so jose you are back, obviously julio. I hope you are back in roland. Who else is the chitendra? You are hopefully all here.

B

Perfect we'll see.

A

Good, so then, let's continue next thing: we do. We are creating a project and we are onboarding a service. So just as we did earlier, I have prepared this captain create project, simple node project and then either dash dash shipyard equals the shipyard file that I've prepared for you or just s. For short, I kept the short version because otherwise it would not have fit on the screen.

A

So let's do captain crepe project and what it says hey. I just created a project and it has a staging stage and a production stage. That's it staging and production right. If you look at the shipyard file, it's exactly if we go into the captains folder and if you look at the shipyard file, this is exactly what I specified. I want to have a stage with direct deployment and a performance test strategy and then in production. I want to do blue green.

A

I want to do again performance test, so I'm doing a blue green deployment. Then we run some performance tests and then we also figure out if the monster performance has to done and then the evaluation kicks in whether we want to keep blue or green or whether we want to switch back. I also have here the remediation strategy, so this is something for the remediation workflow. We're not going to go into detail today on that, but there's also tutorials on other remediation and there's also more coming in the next couple of captain versions.

A

So, but this is important: staging and prod direct blue-green performance and performance testing good, so we created a project, and now we need to onboard services we're using onboard service there's. Also, a captain create service on board means you're on boarding a service where you also want captain to handle delivery. Where you actually on board a helm chart create service, would be you're just creating the structure in captain.

A

um Okay, there's a question coming in jose, so you have a problem with creating the dynamics secret.

A

Let's just make sure that, let me just go back what's I can I you just need to make sure that these environment variables, dt tenant, dt, api, token and dt pass token are exactly called like that. Let me bring up my my little cheat sheet again so make sure that they roll up a case that you have the underscore here that you've exported them. First right: this is your cheat sheet. I copied that over or the other option that you have. You can just replace them, but this is just the way I did it.

A

uh Awesome gonna check, yeah.

A

So this is, this could be the only the only thing that, for me uh would explain the problem if something is.

B

Malformed, okay, um I will.

A

I, as I will be deleting the tokens anyway, I will just reply. I sent them over to you. So don't use mine okay, but this is the way it should be. Don't use mine, don't.

A

Let me know if this worked who's, it.

B

Don't worry, okay,.

A

Good, I can follow on perfect, so created a project onboard on board in the service. So what did I say? There's two options: onboarding, where you can add a chart and then we'll do all the magic with we support using helm and we create the different helm deployments. If you do, blue green in a stage create service would be another option.

A

This is typically used if you're using captain only let's say for the the quality gate use case, where you don't really need delivery or maybe where you want to use different tools for delivery right, you can also say create service. You create all the structure in the git ripple, in my case, we're using the full blown captain, including delivery. So I'm copying this in and that's it. It tells me created helm, umbrella, charts and that's it so and now captain has all the information about my service. So what's the next step slide number 29?

A

It says: let's do our first deployment. Well then, let's do it. So how do we do at first deployment? We tell captain captain. I have a new artifact for you right. I might be a developer for this project for this service. Here is my image and here's my tag and I'm pasting this in now the first time this will take a little longer. Why? Because and andreas and johannes again, maybe you can.

A

You can fill fill in the blanks here, but I think for the first time we actually do all the the helm, magic and then also name spaces have to be created and there's more things going on the first time.

I

Yeah correct um we're creating the namespaces and also we are generating a so-called helm umbrella chart which is used by captain internally, um and this is not or it's a duplicate of the helmet that is onboarded but used for doing the blue green routing.

I

And this takes some while perfect.

A

So, however, while we're waiting here, what we should be able to see if we go back to our captain's bridge right, if you have your bridge, still open bracelet captain something and you click on the retry button, because earlier it says you don't have projects. Well, now we do have a project. We have a project, that's called simple node project that has two stages: staging and prod, and we already see our first service and under the service.

A

If you open that up, you also see that the we call it the root event, the root event that is triggering a work flowing captain. In this case, it's a configuration change event. So when I actually said, captain have a new artifact, what the captain cli is actually sending in to captain is hey here's a configuration, change, there's a new, a new tag, a new image and then it starts off. And now the cool thing is you can see here.

A

My my bridge has just refreshed itself and my workflow already continued and on the right side. So if I click on the left side on the root event on the right side, I see the complete chain of events. What is happening in the in that workflow remember. We have, we always have deploy test evaluate, and in our case we said, we have a staging environment and a production environment. So the first thing is staging.

A

We had a deployment finished, so the helm service when it was triggered by captain, because it is listening to the configuration change event it then deployed, and when the helm service was done, it was sending back to captain hey, I'm done.

A

We also have to link here so this little thing here. If you click on it, it will open up a new browser and voila. We have our app deployed. Congratulations build number one is in staging and again if this doesn't show up immediately, because you have a dns error hit refresh a couple of times. This could again be the cpio issue right now.

A

What's also great- and I like this uh in the current version of the bridge- you can always expand these um these tiles and they show you the actual, uh underlying events that kept descending. We will uh we will make this. Let's say we will hide most of this not height, but there's a lot of new iterations of the bridge coming out soon, um where we will uh not make this as prominent.

A

So we want to give you more information and more answers to your questions like what is deployed and where is it deployed, but right now this is really cool and, I think also on johannes in the future. There will be a way to also see the underlying events, because for me this was really valuable to see what has actually been sent back and forth.

I

Yeah, I can confirm cool.

A

So then, it says test finished, even though zero tests zero minutes because we haven't uploaded tests yet so it just bypassed that stage. Also evaluation done, it says the evaluation is done, even though there was no real evaluation because we haven't yet uploaded slides and slos, which is why my pro my workflow by default goes all the way through and it's, as you can see here, the the so-called gatekeeper service has automatically promoted into prod. Now the question here would be: should you really promote something in production?

A

If you haven't done your, if you haven't specified your test files or haven't specified your slis and slos kind of like a test driven development mentality, where you can only pass the test, if actually wrote the tests and if they're successful, but by default now right we just if there's no tests, it just skips that and so on and so forth, and now it's actually made it all the way into production. Again, I have a link here. I can open it up, and this is not my product environment right and you can see.

A

Congratulations build number one and my app is also uh highlighting the the captain. um The captain stage name right in this case it's staging, and this is production right. That's the difference between these two very cool, so first deployment, good, quick uh show of hands or quick comments. uh Please in the slack channel uh you cannot copy slides from the slides. Is anyone facing the same issue? I see them online but view only. Maybe this limits the option to copy paste text. One julio you can you should be able to download the slides locally right.

A

If you open up that link, you should be able to say download local copy.

A

Only download as pdf okay, uh you should be able yeah. You can download them.

C

A

Okay, so role idea to export it to a pdf. uh That's a good thanks for thanks for sharing this information that late an hour later come on folks, but I guess you you thought that everybody had came up with the same idea. So um thanks for sharing this.

A

We'll probably upload the slides somewhere else later on, uh probably to some of the kid reposts. I also we're converting this into a regular tutorial, so under tutorials.kept.sage you will find this as well and then it's all html and then jupisia.

A

Good all right, so this was this run our first deployment and, as you can see here, what I highlighted is exactly what we just did uh navigating through the um through the bridge and then opening up these links now. Another thing I want to quickly validate is if our monitoring is working as expected.

A

So if we go to dynatrace now, let me just go to dynatrace. If you click on hosts, you should see your kubernetes host. In my case, it's a cheeky captain, tutorial default pool all right. So why do we see this host in dynatrace?

A

Because when we were earlier installing the diametry service on captain and then we said, captain configure monitoring dynatrace.

A

This cli comment was sending an internal event saying please configure monitoring for dynatrace the dynamics service picked. It up was reading the damage with secrets and the first thing it does it. It validates on that kubernetes cluster is dynam, trace already installed, meaning the one agent, if not it is installed in the diameter, is one agent operator to roll out dynatrace on that cluster. This is why we have full stack visibility here. As you can see, this is the host.

A

We also have the google cloud tags here and all the different metadata. We see all the processes and containers. Now. What I always like is to look at smartscape, so under the dot dot, you can click on smartscape view, and what's great here is. We should see the complete stack from the bottom up from the data center to the host through all the different processes which are basically you know, parts in containers.

A

The three services here and also on the very top is a is an application. So we actually see end user data. Why do we see end user data already? Because when I opened up my browser window and navigated to that app, it's an app that is running on the kubernetes cluster that is monitored by dynatrace and by default we are turning real user monitoring on. That means everything I click here.

A

Everything I do is also recorded from an end user perspective, which means we can actually see clicks and swipes and uh and user navigation, um and if I go back to dynatrace right, this is really cool oops, not this one. This is the red one um I can go into. You may not have the same nice names. I have a naming rule. I will show you later on how that works, but pick one of those three node.js's and then click on the on the little arrow.

C

A

So you actually get to the the note service that we're monitoring and what's great here is you see, there's tags.

A

Captain deployment, captain project, captain service captain stage, so the helm chart that I've provided for my example, is automatically passing the deployment type, the project, the service and the stage to that node.js process, and when we have set up the dynadress integration with captain that integration automatically created some tagging rules so that it extracts that metadata out of that container and converts it into text, which is pretty sweet, because one thing we can also do if we don't know which service it is that we're interested in, we can go to transactions and services, so you can also get to here in the menu on the left.

A

There is one entry that is called transactions and services, and now what I can say, if we click on the filter by you can say, I only want to see those with the tag. Let's say, captain project equals to simple note project, and then you can say I want to additionally tag it by captain stage staging.

A

So now you have your filters. So that's great right so for those of you that are familiar with dynatrace and I'm sure other monitoring tools have it have it? Similarly, you can use tagging for filtering now. What also happened is, if you remove the filter again and in diameters, you look up here next to where it says, buy now, there's a little filter icon. If you click on that, there's also a long list in my case now, because I've created so many different scenarios.

A

But what captain also does in the dynadres integration? It automatically creates a so-called management zone for all of my different captain projects and stages. So in my case there is a, for instance. One is called captain simple note project staging, so this should actually give me the same result, but now it's on a global level. So I have this management zone and I automatically only get to see those services that are that have the right takes on it.

B

A

Very cool this makes sense. Still. Is everybody uh roland chitendra, jose julio agreem? Are you all there with.

H

A

Good, um perfect and jose, have you figured out and solved your uh your secret problem, a dynamic secret? I am fighting.

B

You're fighting from literal flag, I think it's my console.

A

I, and I think I know what it is, the the dash is actually not a dash. It's a so it's not dash dash. I think you copied it wrong, so this is something that happened to me as well, so you are struggling with this here right. It should be two dashes, but I think the way you copied it is only a single long dash.

B

A

So that's important if you copy things over then make sure that these are two dashes and not kind of converted into one roland. You said you have no metrics in dynatrace, you don't see anything in dynatrace, meaning if you click on hosts, you don't see, you don't see it.

A

You don't see anything here either.

A

If that's the case, then I could create secrets but couldn't configure, monitor, damage race. Okay, um let's go back to this red. You need to create a secret again, make sure that you have. uh If I go to my my example again,.

A

Where is my here, we go man make sure that these secrets.

E

A

Important this is just the url, no slash in the end, the right token and the pass token. Hopefully you haven't mixed the two up.

A

So you create a secret. The next thing is you install the dynameter service and then it's a captain configure monitoring dynatrace, and this will actually then install the one agent on that cluster and uh should do all the magic.

A

Okay, roland, you said not just a project, so you mean the the um yeah but no host. That's um wrong! Do me a favor just do another. Can you just do captain captain configure monitoring damage race, again, yeah, perfect, all right! So, let's go through the rest and then we can also do some some more trouble, some specific troubleshooting uh uh in a sec. A little later.

A

We have this, we have hopefully monitoring in place. Okay next thing is slide number 34.. um We want to also install the so-called dynatrace sli service. That's the component that allows captain to retrieve values from dynatrace. We also have a prometheus sli service. We have a neoload sli service, so whatever data source you have, you will probably end up with an sli service that can then deliver the data when captain needs it. So, in my case, just as before we go over and up, I guess I need to log into my.

B

A

Sure so um I go back to my google console I'm applying that yaml.

A

B

Now, I'm unauthorized: why is this.

B

Unable to recognize, oh, that's interesting as good as a.

B

Let's try this again.

A

Interesting, I've never had that before. So again, I guess there was a short glitch with github, so I just installed the dyna trace sli service next thing is I'm enabling sli dynamics sli support for my project. Remember the reason why we do this is because you can have multiple projects in captain later on, and maybe some of these projects don't need slis at all. Some of them may use prometheus and maybe some other tools right now. We have also the the limit of having.

C

A

Sli provider per project, uh future versions of captain will be able to have multiple sli providers per project right now, it's a single sli provider.

A

It's also one of the reasons why this particular comment here: captain configure, monitor and dynamite project simplenote project will, first of all again make sure that dynamics is properly set up and it will also enable sli support for that project, which means it actually stores a little config map so that later on kept knows to which tool to reach out to when it needs to query data all right now, adding quality gates, that's the fun part, at least for me, the way quality gates work. Remember we have sli files and we have slo files.

A

So in one file we specify what metrics are important and the other file we specify what are the objectives? The one file is tool specific, so we have an sli file that is specific, let's say for dyno trace. This is also the reason. So what I'm doing now, when I copy this, let's just copy this one- it says captain add resource to this project for a particular stage for this service. I'm adding an sli yaml and what I've prepared. In my example, I actually have two sli's one is called sli basic.

A

The other one is called sli proof test, we're using the basic version right now.

A

um That's why the resource field here says: dyna trace, sli basic yaml, we're uploading in it to the captain, repo under dynamite, slash actual iem also, it always has to be sli yaml when we upload it to captain, but in this case we're using the basic version. The basic only has five metrics in there and later on, we go to a different one. If you do this correctly right, you should always see at the end.

A

Resource has been uploaded, which means that's is a good sign that hey you were able to upload that resource to the captain, git repo, and we do the same thing for the slo.

A

We'll do this for the slo perfect. So now, we've uploaded a list of metrics and what objectives we have. We look into this in a little a little later, while we execute the next build and run some tests, because then we have a couple of minutes.

A

So, in order to actually make to leverage these quality gates, not only should we define slis and slos, but we also after deployment need to run some tests. So then we can evaluate that time frame of the test against my slice and slos. So this is why the last step we have to do here, two actually, first, I I wrote a little extension to the default jmeter service, which provides additional capabilities.

A

So I want you to actually use this one versus the one that comes with the default installation. So what we're doing here we're just switching to it a different image so, instead of the jmeter service, we're using the jmeter extended service.

A

So if you copy this over cube, ctl captain set image we're just using a different geometer service from now on, and then the way testing works. There's two things we upload the first one is the testing script itself. In my case, I have prepared the g meter. Script called loads.jmx, that's the file, so we upload that- and the second thing we do is- and this is where the extension comes in of the stuff that I built. It's a so called chainmeter.conf.dml.

A

This allows us to actually parameterize parameterize the the jmeter script, so we can actually define different workloads, so how many users should be executed? How many iterations? How long should the test run, and so I've prepared a couple of different workloads configurations?

A

So I'm uploading this to captain, which means I have a g meter test, uploaded the workflow definition and what's really cool with this particular test. So the way I'm also using a specific dyna trace integration. So when you execute load test, whether it's chain meter or any other tool, you can have the load. Testing tool also add a specific http header to every request, so that dynatrace actually knows when it captures the transactions and does the end-to-end tracing it actually knows. This request came in from the home page test step.

A

This one came in from invoke this one came in from echo. This one came in from this step so later on in dynasty, we'll actually see that we can get that we can do diagnostics and split all the requests up by test step name. So that's also what I've done with this test. Okay, so we've uploaded the tests, we've uploaded the slis and dslos, and now what we want to do is we want to just run the same deployment again.

A

We actually deploy build number one again, even though that I know it's boring, but we'll do it again, but I want to see the difference, because now, quality gates should be enforced, meaning tests should be executed and slis and slo will be forced after the tests are done. So we'll do another, hey captain, here's your new event, so here's a new artifact and now the whole chain goes on, and this runs about two to three minutes.

A

I have decided to use short tests right now, um while these two or three minutes, let's have a look a little bit at these things we just uploaded. So if you remember this captain examples, git repo, where the simple node service subfolder exists. This is where I ended under the captain subfolder. I have prepared all the the files, the charts, the g-meter scripts, but let's have a look at the slis sli that are tool specific. So under the dynatrace folder, you can see here an sli basic gamble file.

A

So this is the the that's the sli file we gave captain. So when we uploaded it, we said: hey captain. This is sli basic on my local directory local drive and you upload it in your repo under dynatrace slide.dml.

A

This is just five metrics and I've chosen this five matrix, because these are out of the box, so so-called built-in metrics that dynatrace provides.

A

So what captain later will do once the tests are executed, it will go to the dynadres, sli server service and say: hey, please pull in all of these metrics and it does it by calling the metrics api every entry here right, an sli specified with a logical name and then with a query and the way the queries work. It's a metric selector. So what type of metric am I interested in in which particular type of aggregation important here is? We always expect a single value to come back from an evaluation time frame.

A

So that's why you need to work with merge and sums and so on, and then there is the entity selector. So from which particular monitored entity in dynatrace to want to have it and remember, we have all the tags on it. So really what we want is, if I go back to.

A

If I go back to transactions and services- and if I go back to my tags, my captain project sample node project take let's say stage staging then I want the metrics exactly from this service, and I can I can uniquely identify it through these tags stage, deployment, project and service, and this is why my sli here has exactly these tags specified in the entity selector. So please give me that metric up front for the entity that has the captain project tag on it.

A

The stage tag the service tag and the deployment, and these placeholders will automatically be replaced as part of that process for the project, the stage the service and the deployment type that is currently kind of pushing mod effect through and at the end, because this is also cool. You can not only query metrics from a service level. You can also query metrics from process level from host level from end user level. So this is where you say from which type of entity do you want to have it? Okay.

A

So these are my slis and then, where are my slos? If I go out to the captain folder here, I have my slo basic and proof test, so we focus on basic. This is where I've specified that I want this particular sli, the response time p95 to be evaluated against my objective. I have a pass criteria and I have a warning criteria for throughput same thing. I have a pass criteria for error rate. For instance, I give this a higher weight, because error rate might be more important for me.

A

If you ever read this by default, every metric has a weight of zero, have one and then you can give it a different weight. Also, I have one metric here. One indicator that doesn't have any objective. This will then just be considered. Informational and I'll still see the values and the criteria you can specify either hard-coded values, so you can do smaller or equals to a certain value, or you can also work with percentage numbers and plus minus.

A

This actually means, if you do a relative comparison, let's say plus 10, it looks at the comparison time frame or the comparison run in this case, I'm comparing with a single result with the last build that past and you can combine it too right. You can say I have a hardcore threshold of 600 and I should not be slower than 10 percent from the previous build.

A

So it's a complete it's very powerful all right. So, let's see we deployed build number one again. If we go to the bridge, I'm going to the bridge- and I can see my configuration change event here- that I executed at 1950. That was five minutes ago and I can already see that it was deployed in staging tests were executed.

A

Now they took two minutes because we've uploaded tests now slis were retrieved because we uploaded slis and- and this is really cool- we have an evaluation done here where, in this version of the bridge that comes with the with the default installation, I can still switch between single evaluation and evaluation comparison. If you click on the evaluation comparison, you actually see the heat map, and this is pretty sweet.

A

Excuse me where you can see that every sli has met its criteria and you can click on it and you see on the bottom, the different values, and I mean right now. We don't have a whole lot of historical data, but you could also click on chart. Yeah, there's not a whole lot to it. So we'll see more once we build, we do more.

E

A

So quick check again into the audience uh into the attendees, hopefully you're still most of you are still with me and not fighting uh either console or xipio or dash dashes or something else.

A

Can I get a quick?

A

Yes, I am there for those that actually made it that far there awesome.

A

And hopefully, you're not just saying this to make me feel good.

G

Andreas, yes,.

C

G

I have a question and a bit confused and excuse me uh for my silly question. But what is the main difference between slice and slows?.

A

Okay, that's not a silly question at all, so it's a separation of concern and I have a blog post and a video. I can show you but the that explains it in more detail. But the difference is.

A

A

So-Called indicators, so you basically specify what type of metrics are important for me, so this would be. You are interested in throughput and error rate rating response time, so you basically just define the source of the data. What is important service level indicators and the slo then says? Okay, but what do I want to compare these metrics with once I have months, I have captured them. What are my objectives? The reason why we separate them there's multiple reasons.

A

One of them is, you can maybe at some point change your query behind the scenes and still so you um so you you, maybe you you change your query and uh you do this in the file where you actually define your your metric queries and you don't do it in in a file that is actually it should be independent on how you get to these values.

A

So this is why we have a separation of concerns. Also, uh in the in future versions of captain, you will be able to have multiple sli providers. So let's say dynatrace provides five metrics and your sonarqube provides metrics and your security testing tool provides metrics.

A

So you define how do you get these metrics from the individual tools and then you have your slo your objectives, where you basically specify hey all right. I have all these input data from different tools, and I want to make sure that I meet my overall objective of this is a secure system, so you can, for instance, say the metric that comes in from this from your security tool has the highest weight and it has a very strict objective and then you put additional objectives on other metrics from other tools.

A

So it's a separation of what type of metrics am I interested in in general, and then what are my objectives.

G

Okay, okay, now I got it thank.

A

G

A

And then, but it's it's a good question right, so they what you should do in the captain youtube channel. We did a webinar last week where I also talked about this and let me just send that link, because I explained the slis and slos and I'll put this into the.

B

uh Into the chat.

B

Here exactly good.

A

All right roland, uh as johannes has asked you: are you still not seeing hosts in.

A

A

The captain looks okay, uh one thing that I noticed earlier uh roland. If you click on hosts, make sure you don't have a click on all in the management zone filter, because the management zone we're creating here only applies to services and not to the underlying hosts. In case you actually, then not accidentally, but maybe on purpose clicked on the management zone.

A

Then you wouldn't see it because it's filtering it out. This is something we need to extend in our management zones, so just double check roland that you just go back to all.

A

That would be great good, all right, so next step um quality gates. Now the next step is, we've just uploaded our tests and slis and slos for pre-prod. Now I also wanna do it for production. So what do we need to do in order to enable it for production? Well, we need to upload our slis and slos for production.

A

Normally, you may have different slis and slos in my case, I'm just using the same file, but I'm uploading it here, as you can see in the captain ad resource to the stage prod earlier we uploaded to staging. Now we upload it to prod in a real life scenario, you would most likely have different slis and slos.

A

Why well, let's say at least different slos different objectives, because in the production environment you have different load scenarios which you might not be able to mimic in pre-products, so you may have different throughput criteria, but the concept is the same right, slice and silo. So now I'm uploading my my sli.

C

A

Production, I'm uploading my slo.

D

A

Production, I'm on slide 41 by the way, and I am uploading my test scripts. So what will happen tests will actually execute also after production deployment. After we do blue green.

I

A

So really just copy and paste just make sure that you're really in the right directory, though, and that you always get resource has been uploaded this. This tells me that this tells us that captain has received that file.

A

Obviously this can once you integrate kept, maybe in your ci process and you you, if you've used jenkins, for instance, to build a new artifact and in all of that source code repo. You have your slis and slos out of that jenkins pipeline. You can say I just uploaded my art defect. Now, I'm uploading my slis, my slos and my tests to kept, and then I tell captain and now I want you to do the deployment, so this can obviously be be fully automated thanks to either using the cli or using the api directly.

A

But I wanna walk you through all of this today, so that you, you see the individual steps involved. Okay, build 42, let's run more builds. I have prepared four builds for you. So, let's start with build number two and as it says here, let's validate the behavior because build number two actually has a bad behavior high failure rate and if everything goes according to plan, build number two should be stopped by captain in stage because it violates our slo, because in my slo I say the error rate should not go beyond.

A

I think two or three percent. So, let's see what's happening.

B

All right, let's go back to the captain's bridge.

A

I can already see that version build number two has been has been kicked off, the deployment has already finished so now build number two is available in its beautiful orange background color. So the way you can differentiate the bills is through the number, but also through the.

B

Really great or bad choice.

A

Of background colors, I'm not that gifted with colors, but it does a good job in telling me what's deployed all right, so build number two is deployed uh next thing that should happen. Tests will execute, remember to take about two minutes, and hopefully now after two minutes we will see a red light, meaning that build is no.

A

A

All right and while we're waiting here again.

B

A

For thanks for being so patient, I know this is not always easy to do it remote, but um I think I learned thanks for the reminder earlier that the link that I gave you was not downloadable as a pptx, so I will definitely make sure to make that available.

A

Also, if you have done other workshops like this in the past, where you saw trainers, uh use certain techniques to figure out how far along people are. Let me know right doing these types of workshops. Remote is also new to us, so always very happy to learn from others. If you have any suggestions, because I'm pretty sure we'll do this more often in the.

B

Future all right, so, let's.

A

See 2005 now it's 2007.

A

It should soon be done there we go. Tests are finished now the sli are retrieved. Oh yeah. Here's also a word.

A

That what what happens internally so we did the deployment captain sent an event to the deployment. Then, once the deployment finished then jmeter kicked in executed the tests once gmita was finished. It basically said: hey tests are finished, then the the lighthouse service kicks in and says. Well, I need to retrieve the data for the time frame between test start and test finish. It's about two minutes.

A

There's a special implementation. We put into place for retrieving metrics from a tool like dynatrace, where, as you know, dynatrace gets data from the one agents and it's it's almost live, but there's still you know a couple of seconds delay. So what actually we do in retrieving the metrics? We retrieve it for the time frame, but then we we retry retrieving the same time frame until we know this data set is actually stable.

A

So the reason why we do this is, if you have a fast deployment, then the tests are finished, and then you say now give me these two minutes and uh the dynadress doesn't have all the data yet fully uh processed. You will potentially get inconsistent results. That's why the dyna tray the captain, sli retrieval, has a logic built in to really make sure it waits on silic until it gets completely stable data.

A

So that means until consecutive calls retrieve the same data set back, and this is now also why it may take a minute or so until we get the final results.

A

Now, if you are using captain to say, enforce a quality gate on a time in the past, that's obviously instant, because that data is stable right away.

A

Good- and here we go for me- I got the red light. Why did I get the red light.

B

Because my error rate is high and.

A

My response time, too, look at that uh error rate is now uh error rate is definitely too high.

H

A

And also response time is extremely high, so all of these metrics are violating.

A

The slo and that's why I have retrieved a failure again. Why is build number two bad remember in my examples: sample node what I've built in here in build number two is a 50 rate, 50 failure rate of a particular endpoint that my testing tool is actually testing that my g meter script is testing. So that's great now I want to do not sure if it, if you're already done, but now I wanted to build number three.

A

And while this runs will do something else, because it's no it's not that exciting to wait for builds all the time, but uh I'm kicking off build number three now, because build number three should be good again. It should go all the way.

A

Through I want to see how this works all the way to the end, with the quality.

A

A

All right so build number three, as you can see, has kicked off now, while again this takes, as we saw in total, four or five minutes, um I wanna quickly jump ahead and and actually update the bridge.

A

This bridge here is awesome, but there's a new version of it already available with some smaller adjustments, and there will be more advances coming in the ui over the next couple of of weeks and months, but one thing that I have prepared for you is, if you go to the slides and if you scroll down to the bonus topics, uh there's one bonus topic on slide 48..

A

This is how we can it's a special builder received today, as you can see it's from the 4th of may actually may the 4th be with you it's a star wars bridge. I wonder if there's an easter egg in there, but if we go to my console, then I can again do my what I copied over. It's really just replacing the image for that container.

A

So, instead of using that bridge that um that image, that came with the default installation of kaplan, just updating that image to that particular build, and if you do so and then, if you refresh the bridge, now see here now, it says staging one service broadband service. I can, if I refresh here, we go now we already. This is a small change.

A

You see the latest version that is currently deployed in this particular stage. So in staging we have version three in production with version one like these are also smart links, so you can click on it and then it automatically scrolls to that particular part of the the process flow, the workflow on where this build was was deployed. If you do the same thing for build number three all right, this is here right now it's in deployment deployment finished. You can click on that link just see.

A

Build number three is deployed right now, yellow background, also what's nice, if I go to, if you see it here, also these tags they're also smart links. So, for my build number two earlier that failed, I can click on it and it automatically scrolls to that deployment. I also get the heat map automatically by the way.

A

Now we can also click between heat mapping charts because we now actually have values at least two for two builds and on the bottom of this chart you can see all these different metrics, so you can all put them in right. You can click on them.

A

uh If you say I want to see how error rate has evolved.

A

And I went up like crazy and I think it's actually probably not the error rate- it's probably uh failed requests. I probably have misnamed that it's not that rate rate for me would be percentage, but I guess I made a mistake in defining that that query, but anyway it worked as expected because it's higher than what I wanted to have.

A

Okay. So then there's some some couple, other cool cool things in here all right. Another thing I want to show you, while the build is still running right, build number three is still running. I want to quickly jump over to dynatrace, hopefully for those of you, where you got it to work.

A

There's a couple of things: if you click on dashboards.

A

There should be dashboards being created for you right, there's a simple node project at captain, that's the one that was created and.

A

It should show uh the number on staging and improved what's been deployed. I think right now, there's no filter on it, so we may want to then filter it on using the management zone. Captain simplenote project.

B

A

Now I don't have hosts. This is something I need we need to fix on our end. The management zones need to also include hosts. That's not okay. uh We should do that. Also. Last seven days doesn't make sense for me, because I've deployed this project multiple times so right now, if you look at the last two hours, I have a lot of these metrics here right. I have staging production, direct links to the service.

A

I see the key metric throughput failure rate response time. I also have links to the actual service itself like where it's deployed.

A

I also have a link to the captain's bridge, that's awesome, but what I want to show you if you are here and if you find your service, let's say the one that we have deployed in staging and if you go to that, then what I'm expecting to see on the bottom right, if you scroll down, are all these events.

A

Remember I told you that one of the things that the the captain dynadress integration also does is send events to the monitoring tool. So when is the deployment happening when the tests are? When do they stop? What's the quality result, so I have the full history also in my monitoring tool in dynatrace in this case. So when was build, number two deployed 2005 and then all the metrics linked back to the bridge, a link to the deployment url again. Where was this deployed to test strategy and so on and so forth?

A

Then the tests which tests were whenever they started when, where they stopped and what was the quality gate result, quality gate was enforced on build number two remember: we only had 30 of 100 points, which is why it was not promoted. It failed number of evaluated. Sli is five score, so we have. We have everything here and especially with the link back also evaluation, time frame, the actual time frame that was used. So we put a lot of uh context data over there cool.

A

So, let's see build number three for me has been gone through the quality gate and staging everything is green again and.

A

It's currently being deployed in prod, and then the tests are also executed in broad and so on and so forth. Okay, so quick, quick check again, um how are we doing? Who can I um roland? I know you've been struggling a little bit with the dynatrace monitoring. Have you uh do you see data in dynatrace, any type of data, or are you still.

B

Blocked good here, okay, jose is good. That's good!.

A

Okay, really some problems with the tokens all right. um I happened to me in the past as well, where I switched. I flipped the tokens. I had the api token flip with the pass token. So maybe you want to try this again and then do the create a secret again, so that's updated and then do that captain configure monitoring again.

A

B

A

Good uh anybody else, as I so jose is good. I assume.

A

Oh yeah, jose is asking and sharing my experience with my friends who's about to practice. Yeah.

B

H

B

Good all it's okay, perfect.

A

Good then um I have, as you know, right there is: uh we've deployed build number three. uh You can probably take an educated guess.

A

That next one is build number four before we deploy build number four, because actually we have build number three at least still running. I wanna walk through one more scenario and then deploy build number four, because I want to extend the quality gates with it, with additional slis and slos.

A

So right now we have the basic sli with five metrics good, but you know boring boring in a way that these are just basic metrics. What I want, I want to extend my slis with metrics that are specific to individual test cases to a particular endpoint of the url.

A

So what I've prepared for you is a second set of slis and slos, so on slide, 45 we're now uploading the sli so-called it's called proof test is the new sli into staging. So let's take this: let's put it over. Let's paste, it.

B

A

We go so that's the new sli, it has more metrics and what these metrics are I'll, tell you in a second, then we do the same with the slo. Why do we also have to update the slo? Well because in the end, the slo really defines what metrics are important. What what do people really? What is part of the objective, if I just add, sli a new metrics definition, but it's never needed an slo.

A

It doesn't do anything, so you have to specify where the metric comes from and you have to specify what the objectives are now the sli that I've just uploaded includes a couple of new metrics that are using a new feature in dynatrace that is called calculated service, metrics, where in dynatrace I can say, I want to get response time number of database calls cpu time wait time, but I want to have it split by a particular piece of the url by a particular context.

A

The piece of context, data and my context, data is my gmeter script is when it executes tests. It is adding context. Data in terms of this is the test step name home page invoke echo version and all that this context, data is captured by diameter. So now dynatrace can give us metrics split by, for instance, the test name or the test step name. Now we need to configure these metrics. We could do it through the ui in data trace, but that's obviously boring. This is not what we want. We want to automate it.

A

So what I have created. I've created a little bash script that you find in the examples simple node service data trace folder, so it means we have to go out one folder and then dynatrace in this folder.

A

You see a couple of scripts that I wrote where played around with automating uh metrics creation in dynatrace, but the one that I want to call is: it's called create test, step calculatedmetric.sh and it takes up to three parameters, and you want to take the one, obviously that I've posted in there so hit enter what it does it creates these metrics hit enter again. When it says, are you sure?

A

Yes, I'm sure it creates these metrics and the these parameters basically specify for which particular services should these metrics be created, so context less and captain project. It basically says only for tags, then only for services that have to take captain project on it.

A

You have to have a filter, so this is the way this works, because otherwise, if you create a metric like this, where you say I wanna have response time split by url and you have ten thousands of microservices in your environment, then this would potentially end up in a lot of metrics that you may not even want in this case you can control it, and you can say I wanna create these metrics only for services, where I have a particular take on it now.

A

My script is very simple: it creates these rules with a single rule with a single condition, but you can obviously create multiple conditions, but this is what I'm done. What I've done, and what I want to do now is: let's see how what else do I need to do here? Next thing is just run the build well that I don't want to run build number one again. I want to run build number four right, um so I'm taking build number four, because I haven't done that.

A

Yet in order to save time, let me first double check. If build number three went all the way through. Yes, three went through, but actually failed in production. That's interesting, yeah, throughput yeah! It didn't hit my throughput mark and.

A

Thanks julio I'll look into this in a second, what do you have the peppa papa bum? Let me just run build number four quickly and hopefully now we get to see mometrix now you have uh message pattern, unknown, request, tsm, parameter, location: uh julia. Did you do a captain configure monitoring earlier?

A

So what what this error tells me is it? uh The rule doesn't work.

A

Yes, you did and you did a deployment. Did you do a deployment of a build where tests were executed.

A

Okay, so here's here's what the error says and actually let me show you this rule definition, because this is important, then you then we can do some troubleshooting. If I go to my dynatrace environment, what this script really did it created these calculated service metrics? You can find them under settings under server side monitoring, calculated service metrics. Now I have a lot of those because I've played around and I'm doing other things as well. But yours that failed was, let's assume, test step response time right.

A

So the way these calculated service metrics work is you have to give them a metric name. You have to give them a metric source, and this is where the conditions come in.

A

So with condition, I basically said, or my script said only create, or only calculate these metrics for services that have the captain take on it or the captain project tech, and also only those where dynatrace was able to capture the tsn. This is the test step name from the testing tool. Now your error mesh such says, dynadress, is not aware of tsn.

A

Tsn is also a rule that was created in the beginning. When we were configuring, the dynadris captain integration, so what you can julio, if you can do one thing, if you in settings right above it says, request attributes, can you double check? If, in here you actually have ltn, tsn and lsn? So these are request attributes.

A

That request, request attribute rules that should have been created now. What these do is when jmeter sends requests, and it sends the test name as part of an http header. It is, it is taking that request attribute or it's parsing, that context data, and this is attaching it to our pubert public data as a so-called request attribute.

A

Now, if this is not there- and you said you have done the captain configure monitoring data trace in the beginning, then the only thing that I could assume what you did wrong is that the token you're right I haven't those attributes, okay, so options uh either captain configure monitoring, dyna trace, for whatever reason failed. There was some issue with, let's say, network connection time. What I don't know whatever or more likely is you have. um You have uh made a mistake with the api tokens.

A

So if you click on integration, if you do dyna trace api and if you look at your token again that you created earlier in my case it was to kept workshop meeting.

A

Then you need to make sure that all of these are turned on and I think it's the right. The right configuration and the capture request, data that has to be turned.

A

A

So if you can double check, if your api token that you used actually has these prop, these uh switches turned on and if one of those was missing, then this is most likely. The.

B

B

Okay, and so, if.

A

This double check, so it's on the integration down address thanks, I'm checking all okay perfect. um It should have also been in the log output, probably somewhere, with captain configure monitoring.

A

If I'm not mistaken, if you do the change, if you find out that you have not turned on a certain permission, you can change it, save it and then do another. uh You can do a. Where is it you can do a captain configure monitoring, dyna trace again, that should do the.

I

I

B

And let me double check whatever do? Project equals simple node project.

B

Yep now it should create it up here somewhere.

A

Good, um let's see, I kicked off, build number four build number four. Is here evolution done here we go now for me. I see now the list of new metrics.

A

So why is this? So? What are these metrics again remember? My test is executing four different test cases and I just said I want to have the response time of my invoke test case, the response time of echo the response time of version, the response time of home page this response time up here is for the whole service combining and aggregating all the requests. But now I want to have it split a spit out.

A

Also the number of service calls so in dynatrace you can say I want to have a metric of a service that is under test and the metric should be the number of back-end service calls it makes. It could also be database calls.

A

So now I get these metrics fully automatically evaluated with every time I run the build. If you want to see it in dynatrace, then two options actually right: you can put them on a chart, create custom chart and we just do all uh test step.

A

For me uh test step uh response time, all right.

B

And here are my individual.

A

Response times and then I want to have it split by test step, so here we go now. I get there's not a whole lot of traffic, obviously on here, but it still lasts 30 minutes, maybe.

A

There we go, but then these are the metrics. The individual ones for echo invoke homepage version, okay, and this is why these metrics are so powerful and not only that we have them available to to dashboard, but we can also configure them as part of our slis and slos. So if I go back to my dynatrace, not this one simple.

B

A

Dynatrace, let's look at this sli the proof test, all right. This is now my new slide. We uploaded and these ones now read rt test invoke so response time. From my test case with the name invoke the metric selector says I want to have this calculated service metric, but I only want to have it for the test step with the name invoke so filter we can do a dimensional filter and then the entity is the same as before.

B

A

And now I also see my mistake with the error rate. Remember earlier when I said this is not a rate because I used errors, total count and I guess we need to figure out the uh not the errors total, but I will error rate.

G

A

That was my mistake. When I prepared this good all right, I think we actually made it through pretty much everything, because we did the extend quality gates with additional metrics. We ran build number four instead of building number one again, but all the sli's are here. The bonus topic was the the bridge we updated the bridge, then tutorials, there's a lot of great tutorials out there uh under w the tutorials dot captain.sh.

A

And I can also very much encourage you to look at helping us extend captain. So, first of all, you can extend captain with existing services that are already out there. There's, for instance, a captain report service, there's a generic executor, a jenkins service. There's a gitlab tutorial.

A

There is an uh there's, a bitbucket tutorial, there's a lot of stuff out there or you can build your own service. So if you want to integrate a certain tool with captain, you can just build your own service, so also guidelines on how to contribute. You can find all this information here.

B

A

So now, uh as we are only let's say, a handful of people, it would be great, you know, feel free to to turn on your uh feel free to turn on the uh the audio and and also video. If you want to that's obviously up to you, but I would like to quickly do a round and see a: how far did you get b, where, where did you potentially get lost, so I can actually bake this into the next iteration of this, so we don't lose people um also.

A

The third thing I would like to know is: have you done these types of remote workshops in the past and what have others done better in the way of preparation go you know, making sure that people are in sync, so.

A

Feel free to just turn on your audio and video.

A

Do you want to say something.

E

Yeah andy, so uh I almost I'm through this the last step for failed me too, the one you know where you create the calculated service, metrics, okay, so for some reason, I'm getting the same error as julio.

A

And do you see the um the request attributes being created.

E

No, no, I checked my api too, and I have all the permissions on the token okay, so I'm not sure I'll I'll try with the maybe with a cleaner setup. I had some. You know tags and a few things from past. I don't know if that messes up this, but.

B

Yeah, let me just do feedback here.

B

So that means uh the.

A

Request attributes were not created.

B

Calculated service metrics failed to create because of missing request attributes. Okay,.

A

Anything else from your site, anything that you've maybe seen in other remote workshops.

E

uh I guess this was. This is really good. I mean I was able to follow your instructions and it's pretty good yeah.

A

Were you able to download the powerpoint? What did you have to do? The pdf export no.

E

I could download it okay, I was able to download here, okay.

A

Good, uh I see julio, you have turned on the.

F

Audio yeah. Well, in my case I double checked. The credentials were okay.

F

I checked also the one the six requirements in the api token, so I have them and I re run the the commands, and besides all of that, I I don't have the request, attributes created and well you just see the the cereal metrics are not created.

F

Due to that, oh well, and besides that. uh Well, in my case, I have a osx, so I had to download the pdf with the slides.

G

F

That's a minor issue when you select text from the pdf, some characters are not rightly or directly copied. You know the white spaces and mine.

H

F

So there is an issue.

A

And you, you think that the reason is because you're running on on mac os.

F

H

F

I'm not fully sure, but there is no, you know osx as a major operating system used by developers, so yeah.

A

I know, but I'm just wondering if you, if you you, you can normally open a powerpoint, though right. It's a powerpoint slide. So.

F

Oh yeah yeah, maybe I could, but in my case I have no powerpoint on installing this macbook.

A

Yeah: okay, okay, that's good to know, stan going with issues with.

B

Command line yeah.

F

Okay, maybe you could have a guest or some page inside your github with.

A

All these comments yeah, so the goal is to do it to put it over here right. So the goal is to put it here.

J

And then that's better.

A

Yeah, that's what we'll do, but I wanted to run this first once to collect the feedback and then convert it over, because I knew there's definitely things we can improve. Yeah.

F

But besides that, I could follow all the tutorials and it works. I'm gladly to have been able to participate on this perfect.

A

Were you able to join the slag channel.

F

Yes, perfect, yeah well I'll. I will check it because I need to uh confirm my email.

A

So it would be great if you can right, because if you the more people that join the slack channel the easier it is for us to get in to stay in touch and.

B

A

Great cool, I also saw that carlos and john had to leave thanks carlos, for the nice words uh all the best in the next meeting. Thank.

G

You very much thank.

B

A

You perfect, um let's see roland.

K

So, first of all, thanks a lot was very good. uh I had some problems with the metrics, but I think there was a token thing uh so.

H

K

Was a bit too fast, then.

A

Do you know, do you know what was the token problem? Did you mix the pass and the api token or.

K

Was it something? Yes, I think yes, yes, I think so, but I didn't.

K

I didn't mention that the api token in the past token are two on two different.

C

A

C

K

Yeah yeah yeah- that was a problem.

A

Yeah I mean there's some good news. I think coming that these tokens, I think in the near future, will be consolidated anyway to just one you just create a token and then one of the tokens maybe has to pass capability, so it right now. It's strange that we even have two sets of configurations.

B

K

Okay was very good and and also the tutorial, maybe a bit slower, because I had uh copy paste with cloud shell problems too.

A

Okay, yeah, that's good yeah! I mean that's something I.

C

A

That that copy paste will go away with once we have it in html. That's that's that will help uh but you're right. Obviously, when you don't have to type all the time, then you are it's impossible to follow. Yes,.

K

Yes, so uh if, if there is yeah me no, it was very good.

A

Cool, thank you. Have you done uh remote webinars like this or hands-on like this.

K

uh Yes, I have done some and I have give some and it's very complicated. Okay, it's it's uh yeah! It's it's a special skill, okay, but it was very good.

A

Okay, thank you, but.

C

If anything, if.

A

Anything comes to mind on how to uh especially understand as an as a trainer where everybody is, I'm not sure. Maybe there's some other tool like raising your hands or I don't know if anything comes to mind.

K

uh Yes, I I think in soon there should be erasing. No, uh no uh wasn't. It was another tool.

A

Yeah, I think you can do a go faster, go slower, yes or no. You have question options. No.

J

So there is a razor and feature.

A

Oh yeah yeah, that's right just raised.

J

A

Yeah yeah, I see it. Oh yeah, that's good! Where do you have this hand? I don't see it in my.

K

I don't have to, but I have the linux client. So maybe it's it's not there yeah.

J

Yeah so I am connected from my phone, so maybe it's on the phone.

B

J

Yeah uh engines, I just wanted to ask that uh I have multiple environments in dynamics, so uh all the environments have multiple tokens so.

E

J

Will we manage that in captain yeah.

E

J

We have to configure again.

A

So the the new dynatris service that was that that we used today um teletrac service.

A

um Oops that was obviously the wrong one. There's a new capability. We extended it where you can actually have multiple tokens. um So let me where was this released into the six nine and let's.

B

See, if that actually has the documentation.

A

Where is it exactly here? It is send events to different dangerous environments, so you can upload a so-called dynatrace.com.mo in the dynadress folder, where you can.

J

Yeah, this is where you can specify.

A

Yeah I'll send the link to, uh can you remind me again, are you? Are you on the slack channel.

J

B

A

B

J

A

Other feedback, anything else that um no.

J

It was great, it was actually very much interactive, so you could follow.

A

Good, that's great yeah! I mean it's it's interesting that this was a supposed to favor stockholm, but then I don't think if anybody actually who actually is from stockholm here it seems we're all over the place.

J

Yeah I'm from india, so I guess there was from.

A

Hi, how are you good? How are you.

D

Good good, thank you for the time and for uh for this well prepared webinar the that's awesome and chose the power of captain with dinah trees, very glad to see that magics happening right now, as was promised in the in the past yeah. That's good!.

B

A

D

You for for your patience.

A

Yeah, of course um thank you for the feedback, and actually the feedback goes to. You know the team that has been doing all the hard work behind the scenes. Johannes is still there, so I think the the kudos goes to them and it's amazing what's possible with automation, right and there's a lot of there's still a lot of way to go and but we'll get in there.

A

Cool all right is there anybody else that wants to say something. I think we have pretty much everyone right.

A

Okay, in that respect, uh thank you so much um have a good evening or a good rest of the day stay healthy, um and uh we will do this. Probably again, we don't, I guess, need a meetup page for that. We can also organize these types of workshops just on a regular basis. I would be very happy to do that and we'll post it on slack and also twitter and whatever else we'll we'll come up with.

F

A

F

I

H