Continuous Delivery Foundation cdCon 2021, 9 Jul 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Combining Progressive Delivery With GitOps And Continuous Delivery - Viktor Farcic, Shipa

Description

Combining Progressive Delivery With GitOps And Continuous Delivery - Viktor Farcic, Shipa

Three phrases keep popping up when talking about modern workflows and development and deployment techniques. We have continuous delivery to automate the complete lifecycle of applications from a commit to a Git repository, all the way until a release is deployable to production. Then we have GitOps to define the desired states of our environments and let the machines handle the converge the actual into the desired state. Finally, there is a lot of focus on different deployment strategies grouped under progressive delivery. They are all focused on the iterative release of features to make the process safe, prevent downtime, and reduce the blast radius of potential issues. Through a hands-on demo, we will combine Argo CD as a tool of choice for applying GitOps, Argo Rollouts for progressive delivery, and a combination of Argo Events and Argo Workflows.

For more Continuous Delivery Foundation content, check out our blog: https://cd.foundation/blog/

A

What do we get when we combine events with workflows or pipelines and then sprinkling some git tops and progressive delivery? What do we get when we do all when we combine all those things? The short answer is that we get something absolutely awesome slightly longer answer still short is that we can get full automation of everything, or at least full automation, of everything happening after pushing a change to git repository, and we get all that in a way that is appropriate for 2021..

A

Now we could do that before we could automate stuff before, but we don't live in the past. This is how you do it in 2021, full automation of everything. So what do we need? What is all this? What are we going to use today to begin with the obvious one kubernetes right, everybody uses kubernetes or fully partly somehow anyway. Kubernetes is here right, we're going to use git, because that's the only source code repository that matters today, specifically, it will be github, but everything I'm going to show.

A

You applies to github or whichever other git provider you might be using, and then we need the container registry. I'm going to use docker hub again applies to any container registry. I don't care which one you have they're all the same. To me and now comes the interesting part. Now comes the real part. We are going to use argo and argo and argo and arc to be more specific. Argo events for events, argo workflows for breakfast or pipelines, argo cd for github's type of deployments and dargo rollouts for cannery deployments for progressive delivery.

A

It could be blue, green as well, but today, canary. Why not? And finally, the last thing we need is a way how to manage our secrets. I'm going to go with the git tops type of managing secrets, so it will be sealed secrets. We are going to create a platform to create a system, unlike dealing with individual tools like we did before. In other words, I'm going to show you everything you might need to do to automate everything before we jump into hands-on part of all this.

A

There are a couple of things I did in advance to begin with, I created a kubernetes cluster. I already have a cluster. I will not show you how to create a cluster. You know how to create a cluster right. You know it inside of that kubernetes cluster. I installed the nginx english and I installed sealed secrets. That's the only thing that I installed. I need sealed secrets, because without them I cannot create manifest with my secrets and and that's about it. That's all.

A

I have kubernetes cluster ingest, sealed secrets: let's try to figure out whether we can use argo and a few other projects to automate absolutely everything. So I have a empty cluster. Basically only nginx thing just running over there and yeah sealed secrets, and the first step we need to do- is to set up everything we need for production and that everything means everything really, but there is an important rule.

A

I do not want to go to my cluster and do cube cattle install or help this or customize that no, so we are going to apply github's principles from the very start. Actually, we are going to apply it in a minute. We are going to break the rule, but only once I will break the rule only once so.

A

You you'll understand in a second what's going on, so let me go to the directory where I clone the repository, uh and that's this one and let's see what I have prepared here for you for me, and that is the directory production and inside of that directory, I have argo cd applications that define absolutely everything I need today. For you guys, I will need to install argo cd. I will need to install. uh I will deploy an application that will be a demo application.

A

I need argo events, rollouts workflows, istio, prometheus, sealed secrets, everything even those things like sealed secrets that are already installed, they're, defining it as well. So we cannot really apply git without having argo cd installed. So it's a chicken and egg problem, so we do need to install argo cd, but after that argo cd should make sure that all those applications are up and running in our cluster and that the desired state.

A

What is defined in this directory in this git repo is always the same as the the actual state or actually the other way around, but the actual state is the same as the desired state. So I will execute cube capital actually customize, to deploy argo cd and from there on argo. Cd will be managing everything in this cluster, including argo, cd itself, so any future upgrades, but basically I will not be running cube cattle uh commands ever again.

A

This is the last time I will execute any command that interacts directly with the cluster, except in a read-only mode, to show you what's going on so installing outgo cd and making sure that argo cd is going to uh do what needs to be done and that's customize, I'm using customers today, but it could be the same thing with helm, build argo.

A

Cd is the directory and I want I have it defined in overlays production and then I'm going to type that to cube capital, apply, dash dash file, name and then dash meaning standard input, and this will take a few seconds. And after that we should have argo cd applied, not yet up and running but applied there. We are all the resources were created. Now. All that's left from margot cd perspective is to make sure that it is running.

A

So I will do cube cattle cube cattle dash, dash, namespace, argo cd, and I want to wait for the rollout status of the deployment. Argo deployment, argo cd and what is it server?

A

I think now this will take a couple of seconds until it is rolled out fully by the way I will be fast forwarding a few times so that you don't watch me waiting for stuff so uh expect that in your case, things will be slightly slower than what you will see on my screen and there we are now the there are a few things I will just copy and paste commands uh instead of typing them to be faster.

A

Those commands will update the the username and password actually the password only of fargo cd, changing it to be uh admin right, uh like every good password. Admin is always a good choice for a password. Isn't it and let's open argo cd and see whether it is really up and running open http data and argo cd then base host? I think that that's how I defined it- and here we are here- is the argo cd ui admin admin, as you can guess, and there is absolutely nothing inside of no definition, nothing.

A

The targo city is managing. We are going to change that in a second. So far we only installed argo cd and now we're going to do the full blown everything the whole production without. Actually this will be this time. This time will be the last time I will access the cluster ever again in a right mode. Read-Only is okay right, so we need two more things. We are almost done with everything we need from uh deployment perspective. Everything uh one is to create a project.

A

So here is the project cut project yaml, it's a simple project, targo cd project that defines staging uh and production. So here is the production here is staging. I will not bother you with details, so I assume that you watched the video about targo. This is not about target it's about everything right, so uh cube cattle apply and then file name is project, dot, yaml and now that we have those two projects, uh I'm going to create two applications optional and basically the same logic, uh defining staging.

A

That will be pointing to this repository, the repository where I am right now and to the directory staging and another second application that will be production, right, production environment, pointing to the same repository but to the directory production, and this is now now I'm serious. This is the last time I'm going to change anything inside of my cluster myself. From now on, argo cd will manage absolutely everything, so cube cattle uh cuddle apply and then uh file name is uh file, name is upstation and that's about it.

A

Now, let's take a quick look at what's going on in argo cd ui. Here we are two applications created. uh I don't know if you're fast enough, a project, uh production and staging and those two applications figured out that there are other applications and they're all being created right now right, so it figured out everything now. This is my whole product production stage. Everything at least everything for this demo is right here uh and all I did was just tell argo cd hey these are the repositories do whatever is defined over there?

A

Don't bother me much now. This will take a couple of moments, uh so I will fast forward again to the end of this process. There is a lot of going on like uh uh them. Application staging is your egress resources, seal secrets, argo combined approaches, same application, demo, application production, east itself, argo event, sargo, workflow, sargo, rollouts, ingress for istio, argo, cd, uh it's managing itself prometheus and staging and production environments. Whole shenanigans. Everything happening with this in a single command, basically applying uh argo cd application.

A

So, let's fast forward to the end of the process, that's it everything is up and running, except the demo application running in production, that's normal, because I didn't create the first release. I'm actually surprised that the same one in staging is running, that's surprising, probably from previous uh tests of of the script anyways. If you ignore the demo applications, those called argo combined up and I will combined up staging everything, is up and running now. We know how we will deploy stuff just by pushing changes to git.

A

That's what argo city does and we need to now figure out. How do we create a workflow or a pipeline that will do everything that needs to be done uh when we push some changes to a repository of an application. You know it needs to build container images. Maybe some binaries uh create some releases uh deploy to staging run some tests deploy to production.

A

You know the process depends on the exact steps will differ, but the point is that we need to figure out how to do all the automation before argo cd can take over and do the deployment itself. So everything happening until the deployment or after deployment we're going to see. So now we're talking about pipelines workflows. So let me go back to the terminal. The next thing we need are events.

A

We need to tell github to send a web hook request to our cluster and we need to tell some processes in a cluster to trigger some pipelines, and we are going to do that through argo events and the pipelines themselves will be argo workflows and let's see how that works. By the way. I assume that you watch the video about argo events: argo workflow, sargo cd.

A

If you didn't it's in the description right, this is not rehashing the same thing just how to combine everything, uh absolutely everything so argo events and then, if you see to overlays and then production and then even sources right, these are the even sources from argo events and basically, what this is doing is telling hey. This is going to be called github.

A

There are some ports to be opened and I want you to notify a repository argo combined up. Actually, I want you to modify repository arrow combined up to send a web hook request to this specific address. Basically, this is creating a web webhook request that will be firing, uh events uh or notifications to our cluster and english. That will enable that communication right and the second thing so now we know how to trigger events into the cluster from github.

A

uh Now we need to figure out how to pick up those events, let's uh decoupled and do something with them and specifically how to run a pipeline and that's defined in a separate file. So argo events overlays what was it production and, I think, sensors? Yes, this is where I defined it now. This is a long one. I will not go through the details again.

A

Watch argo workflows, argo events, videos what matters is just a quick overview, so uh this will basically monitor those events coming from github uh in the event name is argo combined up, which is uh event from repository the very same name and then uh setting up few secrets, and uh after that we have templates. This is where things are happening: build a container image with kaneko. How do we build container image? I don't know because there is a template that is already defined in a cluster argo cd deployed right.

A

We are using templates heavily here to avoid repeating ourselves here. So this is a reference to template, build kaneka with git that will build container image, and then we have another template that will promote the staging.

A

What that means is that it will modify staging definitions, manifest argo cd, manifest the define staging by changing the toggle of the image that should be deployed there and those are the few arguments and then we have promotion to production, and then we have running some functional tests and integration tests and all those are using dag in argo in argo, workflows, meaning that we have dependencies hey, you can run functional tests after you promote staging, you can run integration tests after you, promote the staging and to promote the production.

A

Dependencies are about functional integration, tests and so on and so forth, so a set of tasks uh and the dependencies. So the tasks are all executed in parallel or certain order or whatever you want basically and then at the bottom. We have parameters.

A

Basically, what we are doing here is we are getting the parameters we are uh parsing uh webhooks from github and then extracting certain values like git url and then passing those values as parameters uh for the workflow above right. So instead of hard coding, what is the name of the application? What's the name of the image? What is the repository? What's or not? We are getting all that information from webhooks right, so this template could be used not for this application.

A

This could be used for any application that uses the same steps right not only for this one, because nothing here is hard coded nothing. All the things are retrieved from github itself, so amazing, very reusable whatsoever. Now what else? Let's confirm that the web hook was created really, and it's just opening the hooks in github- and here we are- you can see here that it created the webcook.

A

Now don't be alarmed that it is red, that's normal, because when argo events created the webco problem, the load balancer that allows capturing those web hooks was not created. At the same time, this is race condition, but trust me it will work. Actually, I can confirm that right away that this is just a temporary problem, because load balancer was not created. If I redeliver look at this, it will be green there. We are right. This is just because load balancer was not created, but web hooks are set up up and running magic right now.

A

Let me go back to the terminal and I'm going to open uh another address, to show you something else: uh http and then argo, workflows and then base host. This is where I'm hosting my this is where I'm hosting argo workflows. Ui now this looks horrible. This looks like a bad joke because it's empty think of it as a bug or a temporary problem. Anyway, we're going to go to workflows, and here you can see the namespace I'm going to change it to see all the namespaces you can see. The first workflow already was executed.

A

We're going to ignore that I don't need it uh for now. So I'm going to delete it. You will see soon what this will do. So we have here workflows. We have brick wall templates, uh cluster workflow templates. Those are the templates that were created by argo cd, because this template builds container image. This one promotes to staging of production or whatever we want all that is already set up, and what do we have else even flows? uh We can see how the events are connecting things.

A

um What is missing here is a workflow, but that will appear when the first workflow successfully is executed. So you will see that that soon and we have even sources. This is the web cooking github we have uh sensors. This is the webhook that is, this is the trigger that is triggering workflows. Everything is here, everything is happening out magically and I'm not accessing my cluster argo cd deployed absolutely everything.

A

So now, um let's go back to the terminal and make a change to the application and see whether the whole pipeline, the whole workflow really really works, and I'm going to do that by clearing the screen. First, clear, I don't know how to type and then I'm going to go to the repository of the application, which is argo combined up, I'm going to change the the address of production, which, basically it's it's based on history. You'll see later, why is you, and only in production or staging you can ignore this command? It's not important!

A

I'm going to also change the the address through which my application production can be accessed. I should have done this in advance uh again, not important. What is important is this guy over here, so I'm going to open conflict tomorrow, config tumble and I'm going to make some silly change to the application so that you see that it's working. So I'm going to change this text here and it says where devops becomes practice and what should they do? I'm simulating it I'm creating a new feature or something like that modifying the application.

A

I will say: hey yeah subscribe now, that's a good thing, so making change to my application and then I'm going to do whatever developer does. Imagine that I already tested it locally that it works as it should be, and all that stuff and I'm going to push it to get so get that get commit.

A

No comment. I don't know how to type today in dashem and what should be the a silly message. I hate writing messages, but they're very useful though, and git push so push the change to git repository and now I should see the new pipeline workflow running in the ui. Let's see whether that's true I'm going to go to the workflows, and here it is, it is running the pipeline is running. It is building my container image and remember not only that I cannot access the cluster, but other tools cannot access the cluster.

A

This workflow will not deploy anything anywhere. What it will do is that it will build a new image, that's happening right now, and then it will uh modify the repository, the defined staging and then argo cd will pull that information deploy something and then I will run functional integration tests and if both are successful, I will modify the repository that defines production and then argo cd will again do this, do whatever it does, and it will be running in production now fast forwarding to the end of the process.

A

I have 20 minutes to go through all this for you guys there we are it's already program promoted to staging, and now it is running integration tests and functional tests. If you're wondering why am I doing that right away instead of waiting for a while, I tend to use functional integration tests uh before deployments during deployments after deployments, but this is mock now, so it doesn't matter. But what does matter is that I will go to argo cd.

A

Where is argo cd and we're going to confirm that the staging changed I'm going to go here and I'm going to go to the deployment and I'm going to look for the desired manifest and the image is still the old one right. So argo cd did not pick it up yet it will pick it up in a second or two up to three minutes: that's the pulling uh frequency!

A

Okay! Now it is progressing. Let's double check the image here here we are right: it's not latest anymore. It updated the image to the one that was built through the workflow using kaneko and all the gs, js gstgs, all the js, uh and now uh now, actually a new version is running in pro in staging and then the tests were executed and uh it automatically promoted to production.

A

I could have created the pull request to do continuous delivery instead of deployment, but I was, I went all the way in complete continuous deployment and we should see uh argo combined up here in production also running in in production, and this is special, I'm using canary deployments and uh because I wanted to allow only fraction of users to use it. So now we're jumping into the fourth the last project of argo cd. We are using uh canary deployments only in production, because canada deployments make sense only in production.

A

So let me show you what it looks like. So how did I define kind of deployments? Here's the rollout yaml. This is the definition only applied to production of this application, and it has some some analysis. Like average number of successful percentage of successful requests, average response time uh all the jazz and it is doing uh 10 and then waiting for two minutes and then reach uh increasing the weight to 30 percent running analysis. All the stuff, argo rollouts, if you don't know what our goal rollouts are.

A

uh The description contains the link and it's probably above my head as well, and let's take a look whether canary deployments were successful.

A

uh I have a strange feeling that they will not be, but let's double check, so I'm going to execute cube cattle, argo, rollouts and then dash dash namespace is production and we want to get rollout called argo combined up and we want to watch for changes and we can see here. Oh, I need to move aside.

A

You can see that it is failing it's failing big time, but that's normal. I can actually explain why it's failing. I expected it to fail simply because it is doing analysis of average request, duration and the error rates and stuff stuff like that, but it has no data and permit use simply because nobody is using this application so expected to fail. If we started using the application, then we would see some results, but in case of country deployments, if nobody's using up that application, that's a bad sign.

A

Absolutely bad sign horrible sign anyway, so it is scaling down. But this is this is the proof we got country deployments as well. We got progressive delivery and that's all combined, so let me go back to the main screen full screen me no more terminal and explain what happened, because a lot of things happened and I tried to squeeze it into 20 minutes. I hope I did. Maybe I didn't I'm not sure anyways, uh let's, let's go through what just happened, so this was essentially a combination of everything.

A

To be honest, I was preparing this video for, like last four or five months. Significant number of videos I made in the past were made as a preparation for this one, in which I wanted to combine everything: progressive delivery, githubs workflows, events, uh secrets management through kit and so on and so forth, and we did that by combining all the tools that I explored before. So, if you're confused, if you don't know what happened, you probably haven't seen those videos so go to the description and check out targo cd, argo events, argo rollouts, argo.

A

What else is missing there? Something is missing, yes, workflows and sealed secrets and conical right. uh I explored those all those in separation. This is how to combine them all together, and that was the sinister reason I had been behind, creating all those videos in the past. I knew that I will get to this point, but I had to get there step by step, and this is the automation of everything in a way that is appropriate for 2021.

A

So this is not the tools which I'm not going to name that already exists for a couple of years, 10 years, 20 years. No, this is kubernetes native cloud native way how to do full automation of everything.

A

Hey, let's see whether you can hear me, I hope you can. Let's see, I I've been answering a lot of questions during the video, uh I'm not sure how much time I have so somebody will kick me out. Let me let me see if I can answer a few more that I haven't from tory. What is your recommendation for github's uh in the monorepo setup? I mean uh the guitar's part is easy right. uh Doesn't matter really whether it's monorepo multi-repo, you just tell argos idiot flags hey. This is the repo. This is directory.

A

uh It can be one rip of one directory or multiple ripples or multiple directories in the same repo. It's all very easy. As long as you tell argo cd, where to look for something uh that part is easier, same thing goes for flux right, so it doesn't really care whether it's multi-rep, repo mono repo multi-rippers, become much more complicated. When we talk about pipelines, you know ci processes, steps and all that stuff. Then it becomes a bit of a nightmare, but for github's parts hey this is where the manifests are.

A

I don't care whether it's monoreport multi-repo, let's see what else.

A

uh Don't see those links to the old videos about the argo tools in the description, could you share a link to them? uh Yes, I I don't have to be honest. I don't have the link to argo videos, but if you go, um this is to emil uh here's the link to my youtube channel and uh within there just search for world argo, and you will see like four or five videos. I have one on argue itself uh what else?

A

What else learned a lot very well done and a great youtube channel thanks thanks thanks, luxor love so hard in the don't know how to type parts, yes and jason says. What's your view of the biggest challenge when using argo.

A

It's more cultural really argue, is extremely simple and easy to use. You say this is where the application this is my git repo uh deploy whatever is there down there? There is no real challenge there. Apart from cultural process-wise, right kind of, like people are not used to the idea that uh hey there are no buttons to click uh there are the approvals are actually approvals uh to for some in approvals or pull requests right. It's not. There are. No managers do not get a button. I want to promote this to production.

A

It's all git based right. If you can, it doesn't really matter how you get there as long as you make a change to the master branch of a git repository and dargo is monitoring that repository. That's all there is to it it's extremely extremely simple.

A

At least the github's part.

A

Yes, let's see whether there is something in q, a uh yeah. I cannot pronounce that name georgie. I think jorge something like that how to handle production deployment with the proofs from different teams. Do we need to move this logic to pair of github, repo or argo cd support this feature, so you don't have to move it to the pr to be approved to appear, but you should, let's start with that right.

A

I I don't see a good reason why anybody should appre approve any change to any part of the system uh without really approving a merge of a pr to the master branch or the mainline right kind of to me. That's the most logical, most natural choice. If you need approval approval, be the person if you need to be the person who approves something be the person who merges a pull request to the main line. That's it's as easy as that. Now, if somebody doesn't like that, you have couple of options.

A

First, you can disable automation of pargo cd and tell argo cd event to synchronize, which is a horrible solution, but you can do that and then that button can execute target sync command or whatever the command is right, instead of, instead of letting argo monitor changes in a git repository.

A

So you can do that, but I really prefer argo pull, request, merge process, uh and then there is a follow-up question from again, I'm not sure about the name georgie or something like that how to handle production deployment with with the proofs from different teams, again pull request.

A

uh If you need multiple people to approve a pull request, then set up your git github, whatever you're using to require multiple approvals uh doesn't really matter. So you figure out what is the best process. You want to approve a pull request: uh either require multiple approvals. That's easy to set up, at least in github, multiple people, multiple teams, you set up the logic and once it's merged, it's approved right now.

A

uh If that's, if git is scary, then uh create a html page that actually uh will make a call to to to github to do the same thing. uh Okay, that's from q, a let's see whether there is uh okay. I think that I'm being kicked out. This is the end of the session. Thank you. So much uh pick me find me on twitter. Linkedin find me anywhere ping me, and I will do my best to help you with every unit cheers everyone.