Cloud Native Computing Foundation GitOpsCon NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Our Journey in Integrating GitOps Tools Into Delivery Pipelines - Brad McCoy, Basiq & Thomas Schuetz

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Our Journey in Integrating GitOps Tools Into Delivery Pipelines - Brad McCoy, Basiq & Thomas Schuetz, Dynatrace

As we started our journey into GitOps we soon discovered that for delivery we also needed deployment validation as an addition to the existing GitOps toolsets to find out if our applications are really working as GitOps is mostly about deployment and not about the deployment validation process after that. We started to see that we needed to integrate more capabilities into our GitOps delivery processes, such as observability, and testing. One example was for canary deployments, we started to see that observability was very important to progress to the next step. When we deployed applications at 2 am we realized that if there was no real traffic coming through how could we validate that our canary analysis was correct, generally, this analysis will just ask the observability tool if there were any 500 errors in the canary. So we had to integrate load testing and integration testing into our process to ensure the app was ready. Therefore, we started to build integrations between ArgoCD/Flux in the Keptn Project and faced some challenges while doing this. This Talk will go over the challenges we faced and provide an outlook of what we think the future looks like.

A

um Yes, hello, everyone. um Today, we will talk about our journey about the journey we had when we tried to integrate key tops in delivery pipelines or in workflow or process based um tourings. So my name is Thomas schutz I'm, a principal engineer for diameteries and maintainer of the captain project and I'm here with Brad Victor yeah.

A

So at first, as we will talk a bit about Captain today, I just wanted to describe what Captain is so um Captain is a tour which should help you keeping your deployment stable and keeping your application stable, and it does this by putting data or service level objectives in the middle of your process and help, helps you integrating different tools and does everything by um controls everything in your process?

A

Okay, so which are what what would be the goals we would want to cover when we, when we are using captain at first, we wanted to do some kind of release, validation so, for instance, I want to deploy something and afterwards try to find out if everything works as I expected, and therefore I can use my obserability solution, just like Prometheus or dyno trace or whatever, um then. The second reason why why you or why I want to use Captain would be that you might have a standard for your observability tooling.

A

So, for instance, you define your quality criteria once and can use it um with whatever obserably the solution you like to you like to use, and third third thing is: you can use it for abandon neutral tool Integrations, so you define your workflow once and by use by using Cloud events, you only need a consumer for everything and which helps which helps you executing everything. Last but not least, we can take out the remediation actions So. Based on our observation solution.

A

We can Define some kind of events on which we listen and, if something fails, we can take actions based on that, okay, so um in alphabet Captain, so we said that we want to talk about the github's journey and the first thing we um we dealt with when we, when we had to deal with githubs in Cape, was that Captain was a very imperative tool, so based configuration and you had to to do everything via commands.

A

Therefore, we saw with we thought that it would be a good idea to create some kind of a declarative language for for Captain deployment and build some operators around this.

A

um So there were some um I don't want to get into deep into this into this graph, but it um it should show how how hard it was to implement Captain to implement key tops in a in a in a imperative application.

A

um One thing: so we found out some things so at first I thought this was picky tops um after some time uh spent on key tops and killed with some things. I can say: no, this isn't githubs.

A

This only covered the configuration of Captain, but not the configuration of our deployments. Furthermore, we had to deal with our own artifacts, so we had. We had an own Captain repository where we expected our artifacts. Therefore we also had to deal with them.

A

The third thing we had for deployment we had no real desired state, so we had some kind of deployment configuration. We copied something we modified the configuration, but in fact this was not real and last but not least- and this was the hardest part- was we didn't use argoflux? We implemented our own kitops controller, and this was hard for everyone who wanted to use it so um I think there's a slide missing.

A

So, um yes, Brad will tell you how the how the real key tops implementation looked like.

B

So here we can see we, we first started implementing agar, City and Captain, so we we used rollouts and Flagger as well. But one thing we wanted to make is a solution where you could either bring Argo or flux. So in this scenario, we, the problem that we were seeing is that we, when you were just doing you, know using git Ops and you were promoting through environments.

B

It was hard to do real testing and specifically um so we use Prometheus for metric for metrics and when, when we've done a deployment, we want to verify that everything's, okay and then use that as a quality gate, which I'll show you soon to progress to the next environment. So you can see here that um the the getups controller will see the change and then that will send the cloud event into Captain.

B

So you can use the Argo notifications controller or you can also use flux using a provider, an alert and then we've got a small integration that will turn that into a cloud event and then start the captain sequence. So yeah. You can see here that we're going to run tests evaluate and then we create a PR to progress to the next environment as well, and the pr can either be Auto approved or you can ask for approval as well.

B

If you want to go to production, for example, so this is sort of what we started with before we tried to integrate Captain so everything's in the main branch we would. The first I mean you could change this. It doesn't have to be these environments, but um it's very simply just copying the settings which has all the application configuration pretty much the environment variables and then the version of the container as well. So when it's promoting through environments, it's simply just copying from the folder.

B

So you can see here we have a folder for each environment and it will simply copy over the.

C

B

Settings and the version so for the demo today we're going to just show a quick demo of there's a app that we like to use called Potato Head. So with this app, this is done by the app delivery right yeah and you can change its arms and legs and we like it for demos.

B

So yeah the thing that we wanted to add to this was testing like load testing.

B

We can see here that when using Captain we can have a quality gate between environments, so you can Define your slos and slis to the accepted criteria to progress to the next environment so for the load, testing, we're running, k6 load test and then we're querying the metrics from Prometheus and then, if that passes, then we can do the git promotion to essentially copy the file from the settings file from the load and then copy it into the integration testing and then Argo reflux would pick that up.

B

They would not see the change and then the journey would start again, so it would get the change and then it would trigger Captain with a cloud event and then that workflow would run as well again so.

B

This is what Captain looks like, so you can see here that this is where you define your services, so in this demo we're using around three services so we're using the Prometheus service.

B

You can flip that up for dynatrace data dog, there's quite a few other Integrations as well. So um the thing I like about it is it's good with interoperability, so you can Captain's, not really opinionated.

B

So it's just a declaration saying: okay, I want you to go, do an evaluation, for example, and then that that service will subscribe to that event, so it kept them will send out the event saying: okay, we're on the stage we're gonna um we're going to trigger this event, and then these services that are subscribed to it will then go action. That and- and you can have the event start started- uh triggered, started and finished and failed as well.

B

So this is what it's going to look like, so it I'm just going to pretend that we did a get change so I've just done one before that's ready. So we now come here and we want to say I'm going to start the load testing stage. So if someone's done a feature- and they put it back into the main- we're then going to kick off the the workflow, so this will trigger the sequence and then that will now roll through the environments.

B

So you can see this automation step it's going to go right from the feature into production, so it will go through the load, testing, integration, testing, QA staging and then the last step will have Auto. Sorry, it will have manual proof, so it'll make a PR and then we just click approve, merge and then that will roll up to prod as well.

B

So this is um this is during the load test now, so it'll actually be using k6 to throw a load of it, because one thing that we found is when, when we want to query Prometheus, normally it's for four or five hundred errors to see if we're, if it's our application is working, we found that when there's no traffic it's hard to get a proper outcome of if it's actually working.

B

So therefore, we added more API testing we've just integrated Trace test as well, and that can do open, Telemetry traces and get evaluations from that as well.

B

So yeah, you can see here that this load stage worked so uh k6 load testing was good. We've seen that the traffic went through then you then it sends an event to use the evaluation which Prometheus was subscribed to it, and then it did the evaluation. And then it said yes, we're happy with these results. These met uh requirements and then um it will do the get promotion as well. So for this get promotion, it's just Auto approve and then it will go to the next environment.

B

So you can see that then we went to integration. We did some Trace testing, we evaluated that we promote it again. So in what it's doing at the moment, through each environment, it will just keep doing a PR and then just automatically merging itself as well, so the good Ops tool every time it notices the change for the environment. It will scoop it up and deploy it and then run the test again.

B

So we're in the last step staging now as well and then now it's finishing and then that will create a PR to be able to merge into production.

B

So you can and we're cataloging that microservice data as well, because if we have like 30 QA environments, we want to know which which versions Etc are in the environments.

B

So we're finished and then you can see the pr links here and then that said, promote to Stage production and it will make the pr for you. We want to do things like reporting as well, so we can report back on all of the testing and things we did and then a developer or devops engineer could come in and just say, Okay. This looks good to me: let's put it into product, so you can see the files changed. In this case it was very simple, so I wanted to change the Pod Taylor heads hat.

B

So at the moment here is this hat and that's made it's all the way to a PR and then I can merge that PR and then now that's sitting in main branch in the production file for the settings and then in this case we're using Argo that will then scoop up that change, deploy it and then we'll see that his hat will change soon. Once that's synced.

B

So it's probably it might take a minute or two to to come up.

B

I can maybe just refresh it here.

B

Yeah, so that's sinking now.

B

And then you can see that my change to update his hat has progressed through each environment and we've tested that everything's, okay and it's made its way into prod.

B

So some challenges we have at the moment as well is how do we deal with sorry? You have a question.

B

B

Yes, I'll show you now yeah, that's a good question, so um so these are yeah each. So this is using customize. You can do similar things with home as well, so for this is all on Main Branch. So we're not doing you know some people do on branches, but it's not really scalable. Some folks say it's a bad practice to have on Branch, so this will all sit in Main and then it will just simply copy the one to two and then an example.

B

If we go back here to look at the old PRS, we can see that closed. You actually see that see that did the auto approved, so it actually just merges itself. So it's keeping git is a source of Truth and and trying to stay true to the good Ops principles um yeah. So that's sort of how we've done it at the moment and then Thomas now will talk about our findings from this things like rollbacks, are very difficult and and he'll talk a little bit about what Captain looks in the future as well.

B

So how are we going for time so yeah? Some of the major challenges that we had with this is that obviously, rollbacks are very hard if something's broken, and you know there can be so many different scenarios. So if you keep rolling back and that fails wins enough to to keep going around in the circle, we found it to be quite ping-pong right where you Captain's telling Argo.

A

I think just work um I think it's all at the graph of at the picture of the whole Argo process that we did a lot of ping pong. So we took. We took a look if, in the repository, if something changed, if something changed my notified Captain, then Captain did it did its part notified, Argo and so on. So this was kind of a ping pong game, so it worked pretty well, but it didn't feel really really.

B

A

B

Yeah, especially when you have things like um rollouts and Flagger as well, that that somewhat do that capability, but we just wanted to go the extra step to have. You know bring our own tools for whatever we want to use in context. Yeah. Sorry, you have a question yeah feel free to just answer as we go.

C

I have a question regarding scalability with get UPS like how many deployments do you can Argo or this tool support? Can it like per hour, like you can do a thousand deployments 10 000 deployments? Is that something you've you've.

B

Tested um I guess we haven't tried it at that scale. Have we? um This has been pretty small, uh pretty small scale, but I I think we would run into some challenges as well, because if you have a really big development team- and you just want to keep getting features in and then it's going to kick off these workflows as well.

B

So it if you one thing with the get promotion services that gets quite complex as well, because when you start having editing PRS when you have more than one PR, it can get quite complex, but in general, if the questions is it with just get Ops itself or with get ups and kept in the way we're doing it.

B

Yeah, let's get in get offs, um I, Heard, uh I. Think I was at the flux project meeting the other day and they were talking about someone I think they showed a terraform controller using like a thousand deployments at once, so it very much can scale. So you can just give it more memory. The the good Ops tool- and it generally goes pretty well yeah I've heard of quite a few departments.

A

I can take the other two parts, um two other things we also found out. We need some kind of context.

A

Information to know in which sequence uh service is at the moment, so what you saw in the picture, I, surely prefer or in the um in on the captain UI- was that you um that there are tasks which are connected between each other, and this is propagated via context information and, if you think about githubs, this might be pretty hard um to to get this through a pull request or through a key to people so especially for the whole ping pong game. We did here for the change, detection, kickoff and so on.

A

We had to find a way how to propagate our context. Information through the GitHub store- and this was pretty nice for Argo, because they had some kind of of information we could pass.

A

um We had um this was this was not this easy, and this was also a problem we faced in our g-soft project afterwards,.

B

Yeah, because if it, if it's opinionated payload, then it's hard to get the things that we need so with the Argo notification controller, we can send we for the application sets. We can put some labels and then it will pass the labels and get that as well and then for flux. We did a small integration that will take the payload into our web service and then it will transform it into a cloud event and send it to Captain.

B

So we have 10 minutes left uh Thomas. Would you like to talk about the.

A

Yes, so um yes, so, um as you might have noticed, we faced a lot of challenges with with this workload-based approach and our workflow based approach and pipelines.

A

In the last few months we put our heads together and thought about how such things could be look like in a cloud native, um so without delivering without pipelines, delivering without without workflows and simply rely on githubs and on all the things we we all love, and one of the results was that we create the captain lifecycle controller, which is a small kubernetes, which is a typical kubernetes operator which helps you achieving exactly the things which Captain did before or still does and everything everything starts with your manifest.

A

um One thing we took care about of was that you don't have to change much in your manifest, so you can use the capital lifecycle controller with your default deployment, with your deployments, with your state full sets and so on. So you don't have to change your kubernetes Primitives.

A

The only things which are relevant for us are that you have to add some some annotations, but I think that's obvious. In this case, you can use Argo for that you can use flux for that. You can use your pipeline tool and so on you see everything which applies something on the kubernetes API can be used and what our lifecycle controller does is.

A

It puts observability around all of the things you are doing so, for instance, you see when you're, when your deployment started, how long it took and how long it ended and when it ended and one thing we took care of because this was also a problem we had with with the last approach. Is we try? We are trying to do or we are not trying it. So we are doing this application aware.

A

So, for instance, you can run some kind of pre-deployment tasks and pre-deployment evaluations before the first service of an application gets deployed. So, for instance, you can check check if there is enough CPU I'm left to deploy the application, and only after this tasks and checks are executed, then the workloads themselves get get deployed.

A

um The second thing we are we are doing around this is we are using standardized task execution, so you simply have to annotate your deployment and say which task you want to use and behind this there is a kind of a kind of a definition. This can be enhanced as you like, or whoever who wants to write an integration is able to, and you can exchange this. However, you want at the moment this project is around two or three months old.

A

We are only supporting functions, but this can be um containers or clients in the future, but this can also be external control planes such as directive or such as such as Captain. Now, um where you can send your Cloud events to and a typical workflow of this looks like it looks, you know, looks like this. So at first we apply our manifests to our community control plane. Let's say we are using flux afterwards, if there is an app detected.

A

So if the Manifest is annotated with an app annotation, we are waiting for an app custom resource and are dealing with our pre and post deployment, pre-deployment tasks and and evaluations. After this has been finished, we can do anything on the workload level, so we can check something before the the port gets scattered can evaluate. If this matches are our requirements and after everything is finished, we can, we simply release the Pod um the same for post deployment.

A

So after we detect that the port has been deployed, we can run some checks, some tasks and also evaluate whatever we want, and after we found out that all of the of the workloads in our application are deployed. We can also do a post deployment, tasks and evaluations based on that.

A

So um if you want to get, if you want to find out more about the captain, lifecycle controller um I'll, be at the captain post the next three days um and I'm really happy to talk with all of you about about all of this I think it's a pretty cool tool, um some things we found out so to wrap up the session today. We we think that the ptops and Captain fit perfectly together in terms of use cases.

A

The initial approach was not this GitHub friendly, but I. Think the next I think the approach with the lifecycle controller will make everything better. Then what we found out was that adopting etops in Cape. You know a typical workflow tools might get really hard, because at some point in time you are simply pushing something to keep repository and have to act on it and you always have to find out. When should the tool continue and how can I promote the next stage?

A

Yes, with this Handover too,.

B

So yeah there's a few more talks for Captain. Today we you can come for free as well to the Community Day. um So you can see the schedule here. That's in level three, okay, it.

C

B

Have yeah? Okay there it is um yeah, I, guess we'll take questions now, because we've only got a few minutes. Yeah.

C

So this this tool also support, like blue green deployments,.

A

No way um we decided not to compete with things as Argo uh with flux or Argo rollout, because we thought that they can do it this better than we, um but we can extend them so everything we are doing in terms of pre-deployment tasks and evaluations and evaluations and tasks in general can be, can be used from the capital lifecycle controller. But if you want to do Progressive delivery and want your blue green deployments can release and so on, um I think it's the it's. The battle choice to use the tools which are already there.

B

Because you can still use an agar roller crd with with that, would do that for you as well, so you can do blue green Canary with that.

A

Yeah, so you can also use our tasks and evaluations for algorithms of Lego. So that's also. No, it's also it's no problem, but you have the and you get the benefit from that. You are more or less too agnostic, so you can switch between the GitHub Studio or you like very easy, and we hope to bring some some standardization into the touring landscape.

B

Any other question yeah.

D

So with this new life cycle operator controller approach, does it feedback the health information to the get UPS tool like Argo CD, when it says the health and status of your app? Does that include what Captain knows about or is evaluating I'm.

A

Sorry I didn't hear it too.

D

Good, um the new lifecycle operator approach, so, if you're using it with a CD tool like Argo, CD and normally Argo CD gives you back the health and the status of your application and how it's doing all the components are doing. The the things that kept in is supposed to be evaluating and testing does that reflect back into the health status that you see in Argo CD at.

A

The moment you'll see how your deployment behaves so you'll see that all of your pods came up and that everything is ready.

A

um One thing I found out in the last week was that it would be nice to get back some status information from the application itself, and this will be the thing we will be working on.

C

Thank you, yeah.

B

So we'll be at the thank you.

B

The captain Booth, so if you want to come chat to us as well, I, wouldn't want to have it to.