Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2021, 29 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: GitOps+Jenkins-CI With Declarative Everything - Kingdon Barrett, Weaveworks

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

GitOps+Jenkins-CI With Declarative Everything - Kingdon Barrett, Weaveworks

Flux is the incubating CNCF tool that ushered GitOps on to the scene, and Jenkins is the OG build machinery name-to-know for large enterprises. Learn to deploy Jenkins declaratively with Flux's Helm Controller, how to build an app repo with Jenkins for deployment with Flux, how to test a Helm chart with Helm Test through Flux's Flagger, and how to deploy new releases automatically with Flux’s Image Automation Controller and Helm controller. Declarative delivery with Helm and Kustomize does not require Jenkins to have direct access to the cluster; Flux runs as an agent inside a cluster and uses a pull-based model for delivery of apps, treating the git repository as a single source of truth. Configuration as code runs under the hood of an all-declarative Jenkins infrastructure from end to end, a declarative Jenkinsfile, pipeline for branches and PRs to build and push images. Flux is first configured to pull new image builds into a cluster, then Porter modernizes our image build process!

A

Awesome thanks everyone for coming, um I'm kingdom. This is my first time at kubecon in person uh really glad to be here. uh So welcome everyone to the jenkins and git ops talk. This is my first talk at kubecon, obviously well in person, I've presented a couple of times before I'm kingdom. As I said as taylor introduced me, I'm an open source support engineer at weaveworks, I'm also a flux maintainer and I'm the cow on twitter. If you want to follow me, that would be cool, um so gitops, that's what you're all here for.

A

I think first uh we're going to talk really briefly about what is get ops.

A

Get ops is an operational model for cloud native operations. It's a modern model. I think I meant to say so. Get ops is formally defined at this point. This should give you an idea of its maturity. We've gone through lots of rounds in a vendor-neutral setting aws.

A

I believe red hat microsoft, a number of companies have participated, weave works code fresh to come up with this definition for get ops, and we think it's pretty solid at this point, so it's been made 1.0.0. um That's that's awesome, so the get ops definition is a set of principles, um and I want to preface this by saying that this is not prescriptive or requirements of git ops.

A

Git ops is uh a progressive progress. You have to process. You have to take one step before you can uh get all the way there so step. One is understanding, uh declarative artifacts, you have. You already have declarative artifacts. If you're like kubecon great step, one out of the way step two is, they should be in a versioned store, which is uh holds immutable artifacts, so version store um in this case could be. Git could be something else.

A

uh This is the vendor-neutral definition, so try to be as inclusive as possible, and another guideline of of the gitops principles is that they should be automatically pulled from your source. Oops.

A

And it's a continuous reconciling process, so so continuous delivery, continuous reconciling you have reconcilers that sit inside of the cluster. They pull artifacts and they apply them to the cluster from a version store in a continuous process and the artifacts are declarative.

A

The git repository should completely describe the state of your system so that it can be reproduced.

A

So, okay, back to this talk and why you're here um I I have an idea of who my intended audience is, but I don't know every one of you. So I'm not sure. If you're all current jenkins users can we get a show of hands lots of current jenkins users, okay, great you're, you're, my main audience, uh so the disclaimer is I've shown. I will show in this talk a way to deploy jenkins um and- and this might be better than what you have now, um but I cannot take responsibility for your jenkins infrastructure.

A

It's very big. We do cd. This is I'm here representing flux, I'm here to show you a good way to deploy jenkins and also to give you a good introduction to git ups and how you can use jenkins with flux.

A

So I'm going to start with. uh I don't know how you have jenkins deployed now, but the common use of jenkins is not especially declarative.

A

You have lots of these things, their dialogue boxes, you gotta click on them and that's what we're doing away with that's the point of this talk. So, let's make jenkins install declaratively with flux.

A

So I have this git repository. um I should probably pull the repository up, so you know where to go. uh We'll save that for later. Okay, now we got here, we go.

A

So this is the jenkins infrarepo in the kingdom, ci org, that's my org, and the way that I have this set up is with flux.

A

It's going to take a little while to load, apparently because the network is having issues. So, let's just go right to it.

A

This is running on my pet cluster. It's called the move cluster and I'm looking for the jenkins folder. This is a flux customization.

A

So if you've never used flux or flux for getups, this might be unfamiliar to you. I will draw your attention to the specs section. There's lots of stuff in here we're gonna, go find this patch down here, there's a patch, um so so I've already installed jenkins, because this takes a while. What we're actually gonna do is to scale jenkins up. So um what I just did there sorry that was a little too fast.

A

uh I incremented the replicas count to one from zero and I've had to use a patch here, because the jenkins helm chart does not include a replicas count. It's not meant to be more than one replica, so so, uh let's get commit that.

A

And push- and this is what git ops is I'm going to skip a step two, because this is a fast fast-paced demo. Normally, I would open a pull request. I'm just going to push straight to master.

A

Yeah yeah, yellow, okay! Oh no! Oh! No! That's going to make it not a fun demo! If we can't push to master.

A

A

A

Maybe we'll have better luck outside of the vm.

A

So some things are bound to go wrong as we do this and, like I said this is my first time presenting at kubecon, I'm gonna laugh. I hope you all laugh too.

A

Oh no, okay, so I have to first pull and make sure I have the latest revision because it's a git repository.

A

Here we go we'll put that one in.

A

And push all right, we got to get push.

A

And the next thing I'm going to do is go to slack and, as you see, things are happening already.

A

So, let's see uh if we can see what is happening.

A

We should get some pods. This is an alias here for cube control in the jenkins namespace.

A

Oh yes, of course,.

A

And the reason we can't is because I disconnected from the vpn to try and make that work, uh so I have a vpn here. I have this set up to be a little bit paranoid because, like I said, I can't take responsibility for your jenkins infrastructure and I don't want to put a jenkins server on the internet. uh I don't know about you, but I imagine many of you who have jenkins have it deployed in a private sort of setting.

A

Well, we're going to move on if this doesn't go right away, we're going to need jenkins in a minute, though, so I hope it does work.

A

And I'm going to try to catch up with that guy at the bottom there as we go. This is progress indicator to help us keep the pace. So we're going to outline what we're going to show exactly here. We have we're going to do a helm, update and actually we're going to do it in a mostly automated way and then we're going to do an image update and when we do the image update the first time we're just going to do an image update, it'll be quick and the second time we're going to use flagger.

A

So we can see the image update, go in a progressive delivery style.

A

So first we said we're going to do an automatic upgrade with jenkins and the helm controller. So let's do that ah good. We are getting jenkins to come online. So that's good.

A

So it's in the helm, release file, which is another crd, like I said, there's a lot of new concepts. If you haven't used flux, I'm just going to blast right through all of them and there will be an opportunity for questions later.

A

Okay, so what we have here is our helm release. This describes our jenkins installation declaratively. This is really, I think, what you came here to see, and this is very long once you get to the values section. These are actually things that we're passing into the uh helm install command. So we've got uh persistence so that the jenkins server when it comes online knows whether it's run jobs before or not, but we're not using it for anything else, and we've got a place that we can set the image.

A

If we want jenkins helm chart tells you that you should probably bake the plugins into the image you don't have to there's an option in the helm. Chart the helm chart is great, I'm just going to plug it right now. You should. If you haven't, checked out the helm chart to install jenkins, you should definitely check it out. It's kind of a prerequisite of this demo so.

A

Okay, so before we go into detail about that, we're just gonna upgrade jenkins and that's what it takes that carrot. There says we would like the latest uh release in the same major version.

A

You can also put tilde there if you would like the latest patch release in the same miner version, but we're going to get the latest version here. We'd like it's actually an upgrade to the next minor version, so.

A

So like before.

A

We're just going to do a commit and push.

A

And we're gonna go back to our notifications channel to see if that did anything and it did helm upgrade, has started great.

A

We can watch it progress if we want we've got to move on to something else, so we got lots of stuff to show.

A

So this is the live demo part. As we said, um okay, we've installed jenkins and we took okay. We did that already helm controller, we've seen the semver automation to upgrade jenkins.

A

Hopefully that wasn't too fast, um I'm I'm showing that first, because it's the easy way there's an easy way and a hard way to do automation influx and we establish many of you already have jenkins.

A

Maybe you don't, but now we're going to move on to the app. Hopefully, jenkins is ready, so we can actually do a build and the build will also automatically be deployed and then I'll show you, the artifacts that make that possible.

A

So I imagine that you have some kind of app or apps. Maybe a bundle of micro services.

A

Pod info is our app today.

A

Yes, I'm in the right place.

A

So what we're going to see here in our our deployment patch? So there is a deployment beneath this that has all the details, but this is the easy thing to look at it's the easiest thing that I can show at least, and this over here is called a k. Yaml setter.

A

That is a customized thing, I think, and that tells us which namespace to look in for our image policy. So we have an image policy defined that expresses how to find the latest the latest build for pod info dev.

A

So let's kick it off before we're too late.

A

And I suppose I should make a change here uh rather than just do a build, but um this is quicker and we are actually short on time at this point if you are watching the tracker, so let's do it this way.

A

The build will take a minute we're not going to watch we'll just move on, but when we come back we'll see so there's our build. That fired off jenkins is running on kubernetes and it's using kubernetes agent.

A

So when a build happens, it creates a pod and it pushes instructions to the pod through a side channel, and many of you probably are doing this in your jenkins installs already, but what I I hope that you'll take away from this- is that when you create the new image and the structure of this is a little bit uh arcane.

A

So this is the branch name, and this is a commit hash. Those are not super important. Depending on how you define your policy, I suppose we should look at the policy.

A

So where does it come from?

A

Well, this is a customization overlay. That's one of the confusing things in flux is we have customized customization and we have flux customization. I don't know how we got here but have to have to get over it. It's okay,.

A

I believe I've defined.

A

The policy here- okay in a pod info, folder.

A

And that's what it looks like, so it's got the literal string master in it and it has a space for the commit hash and this part says we're looking for a time stamp. Now, if you're familiar with flux's v1 automation, this is a little bit different, there's a technical reason for it. I can explain later and the opportunity that I'll have later. I hope that you all will join me later at the flux booth at four o'clock.

A

I think 4 30, maybe for um the it's four o'clock, uh the office hours, where I'll be um going into detail about the things that we had to gloss over here. So you will have an opportunity to kneel me with questions um anyway. This part says the group named timestamp is the part that we're extracting and that's the sortable part of the image so we're using sortable image tags and if you, google, flux, sortable image tags you'll find documentation about this. With this very example, so we see our job is finished.

A

We're in the pod info dev namespace now and we see.

A

Two pods that are two minutes old. That's awesome.

A

And for anyone who hasn't seen pod info before that's, this is what it looks like. So this is what we're upgrading. So, let's move on.

A

Okay, so there is a helm chart for pot info. It does work with jenkins we're not going to use it today, like I said, there's a lot of stuff in the abstract for this, and it's all in the demo repo we're not going to have time to look at everything today, but we can at the office hours so we've added jenkins support to pod info pod info does not natively have a jenkins file, and we've done this using the flux guide to jenkins, which I will show you.

A

It is on our lovely docs site.

A

And you can find it in the menu under use cases, jenkins and flux.

A

So there's a lot of text in here go ahead and read it if you like, or just come down here and snag this example- and this is a jenkins file- and you may or may not have a lot of these, but this is part of declarative jenkins, so we've installed jenkins declaratively and now we're configuring, our jobs declaratively great.

A

And how does this work? We have some variables defined at the top. You're meant to change these for your own versions. So you're, not kingdom b, put your own docker hub, username there or wherever your registry is and give it a secret. So it can push. This is a jenkins secret in the helm, release there's a section where you can add credentials uh declaratively also, and then it takes this docker pod definition we're actually using docker and jenkins to build images.

A

So we do build here's how we get the git commit hash in jenkins. This is groovy code and, if you've built any jenkins files or used jenkins for any length of time before you've seen stuff like this, and that's also how we get our timestamp and uh what happens next is probably better explained over here. So here's our pipeline stages. Oh, I didn't install the blue ocean pipe. That would be really nice. We could look at it. How many of you have seen blue ocean before?

A

Okay, all right, so that's not a big loss. Lots of people have seen it anyway. What you would see if you haven't seen it is a fork. So so, when we get to the build, when we build is done, what we'd really like to do is for that image to be deployed immediately in dev and also should be tested in parallel. So that's what's happening here and we have a different definition.

A

um If we look at the actual jenkins file in my pod info fork,.

A

uh That tells it what to do when a tag is pushed.

A

So this is a little bit more elaborated, but we would like for uh when a commit is pushed to a dev branch we like for that to be built and pushed, and then, when we push tag, we would like for that to be built and pushed, but instead of with the crazy three-part tag, now we're going to use the actual git tag as the image tag. Great.

A

So we'll do that in a second.

A

All right so so.

A

I think we've gone backwards.

A

So there are some helm test examples in the docs um that you should all check out we're not going to have time to go through helm test. Today you see we're getting further and further behind here on the clock we're going to catch up, though it's okay, let's go to the linker d demos pod info, and this is going to show us flagger oops, spoiled thing.

A

I've borrowed this example from jason morgan at buoyant they're, the makers of linker d, which is another cncf graduated project.

A

So I've just cloned it inside of my repo here, I'm not using any sub modules or anything.

A

Am I in the right place, I think so?

A

Yes, okay, so there's a canary defined here and the canary is the moving part that does the progressive delivery. So this is going to take longer so we're actually going to start it right away before we explain anything.

A

And uh okay, we've got a pot info, so this is our production pod info. It has a patch to change the background color and do some other things so we're going to change the background. Color.

A

I'm not sure what this color is, but I'm sure it's good and we're going to do something else.

A

I'm going to check our channel real, quick to make sure something happens.

A

It did okay, let's go back to the browser and I think I'll have a minute to explain what we're gonna see exactly so. First.

A

If I can find the canary, we will see that it has a status initialized.

A

If we watch the status, we should see momentarily that it changes to progressing and what that means is.

A

Flagger has detected that there's a new release and it's time to roll it out and what it does to roll it out is first, if you've defined any helm tests or other kind of tests, it'll run them before it routes any traffic at all to it.

A

I hope it is actually moving and then once it started routing traffic to it. It's going to progressively increase the amount of traffic from five percent. Great here we go, it is moving wow.

A

This might actually work.

A

Okay, so in a moment, what we should see is our change, but just a little bit and so what's happening. Pod info is actually refreshing, the page for us, if you run pod info in your progressive delivery demos, it's really nice, it.

A

Kind of pokes, the page over and over again until something happens so that you can see the thing happen and- and you can just keep talking- you don't have to press refresh so.

A

We should probably go back to the slides advance as fast as we can okay, so I hope that so far this is nice. We should see some progress over here soon, um but I want to talk for a minute about if you've seen some flux, demos before or maybe, you've tried it yourself and it didn't turn out like this.

A

Why is that? Why is that? um So you probably should take a look at the web hook guide. There's a guide in flux, docs called the uh notification guide and then there's another one called the receiver's guide and a webhook receiver is um taking notice from github or from an image repository when something happens, to tell flux it's time to reconcile something. Otherwise what flux is doing is it's continuously reconciling on an interval and that's very expensive if you want to do it on a tight interval?

A

Okay, so you have to have some.

A

You'd like to see any part of this here, supposedly serving 50 of traffic from we still haven't seen anything. Am I looking at the right one.

A

I'm afraid I did something wrong in the setup. One of the things that can go wrong at this point is your pods are not meshed. I bet that's what we'll find.

A

When you create uh yeah, that's it so so I've been tearing this down and standing it back up repeatedly- and I have depends on statement which is another feature of flux, to make sure that things come up in the right order, but sometimes when my cluster goes down because it's a home, lab cluster and things go wrong all the time, like literally every 24 hours. At this point, it's practically chaos, engineering.

A

The thing that goes wrong is you: have things come up before linker d and and so they don't get a sidecar injected? That sidecar is a proxy that all traffic goes through, so uh so that linker d can collect metrics and do things like um revert the deployment. If, if you see an elevated rate of errors- or if you see your latency numbers going up your request, latency you, you can take a look in that canary and see how that's defined.

A

So that's all customizable like everything else about flagger and it's progressing every five seconds until it's done and we'll see that it finishes at some point and we'll see a transition all at once. Unfortunately, because I I flubbed that part, but uh so it's still pull-based um and yes, uh the reason that it's actually pull-based, even though we're using a web hook is because flux only pulls it's not receiving instructions through the web book. It's just receiving a notice it's time to pull again, and this is to shorten the loop.

A

So development is an iterative process and the fast inner loop is super important, and this cannot be understated and I'm sure you've heard that a million times before, but this is like one of the important principles of devops and also it makes a better demo.

A

um So five minutes is, I think, the guideline in the devops guide. Five minutes is not too long, but any longer is probably too long and it should be shorter if possible, the shortest possible is the best.

A

So all right, you can use any ci, it does not have to be jenkins and flux gives you this great boundary between ci and cd, now they're, separate jobs, ci's job is to build test and push an image.

A

Flex's job is to deploy it in the cluster. Flagger makes this even safer.

A

You can do this without helm or with helm test or in any combination. There's lots of features to explore in the flux, docs and image. Automation that we saw in the earlier step is probably the feature that made flux. 1 popular in my in my opinion, so get ops is great, but what we really wanted was it should be easy everything should just happen. We don't want to really think about it and cd if it gets out of the way. That's great, that's what we wanted. We don't have to look at it.

A

Super infrastructure is, unfortunately, not easy all the time, and so this is a little harder than it was in the past, because we're building infrastructure we're not just building um cd right. These are composable units of abstraction that you can use for any purpose that you can imagine.

A

So image policy image, repository flux will go out and look at the images that are there and reflect them into the control into the cluster so that it has a record of what image tags exist, filtered through your image policy and image. Update automation takes that information and writes it back into the git repository and that's cool.

A

These are the crds to know.

A

These are the controllers.

A

Git repository is also part of this. It's used for its secret, so you pass a git repository ref to uh image, update automation. It harvests the secret out of it. For this reason they have to be in the same name space and because these secrets are used for writing into your git repositories.

A

They're super important to protect oops. So these are all advanced topics.

A

You can template and scaffold around them so that your team members can use them. If you understand them and they do not.

A

Okay, so we've got our flux and jenkins guide. Please go read it. It took me a long time to write it and get the pr approved. It explains how to use jsonnet together with flux, and if you read the abstract, I mentioned porter. Why is porter in involved in this? So if you're, if you're using docker to build images, porter does that too, but the interface is smaller.

A

And unfortunately, we don't have time to show porter, but what jobs does ci do and should I use jenkins for this? Those are questions that I cannot answer. I am not here to talk you out of it, um but jenkins has made me extremely paranoid. If you go and look in this repo and see all the things that I've done to protect it from random strangers on the internet. While I give this demo, uh you will be impressed.

A

Are you prepared uh for doomsday? I I hope so so um there is a 45 minute session later on. uh Please visit the flux booth. You can do that from wherever you are. You don't have to visit in person. We have a hybrid booth, it's really cool.

A

Also the get ops one stop shop is our milestone event coming up next week and we're having microsoft, aws, vmware, d2iq and speakers from weaveworks all to show us their products that they're building on flux that you can go out and buy.

A

So so, please check it out. We're gonna have some really fantastic speakers.

A

Flux is great for disaster recovery. uh 35 minutes is not enough time to show disaster recovery, but 45 minutes we're going to tear the whole cluster down, and I will show you how it all comes up and it'll actually only take about 10 minutes.

A

um It's my home lab, so uh there's lots of stuff that we don't actually have time to talk about, but it will be easy, uh maybe parts of it at least and uh there's there's a strategy here that I really want to show you and I really want to explain. But um we don't have time, unfortunately right here, but it's super important. So um remember this part at least so.

A

This was the easy thing that I showed I'm going to show a hard thing at the flux booth that actually solves a problem that you're, probably thinking in the back of your mind when you think about how image update automation works. Okay, we have chicken and egg problem, we'd really like to use the git tag so that we can take the manifest that are in the git tag of the application and apply it to the cluster with our new image. But that's not how image update automation works um so so tags.

A

You know that's what you're using for you. You want to mark a release and then things should happen and uh when the image build starts is when you push the tag. So that's not the time for flux to deploy the manifests right, because your build could fail and uh it may fail for surprising reasons.

A

Like you run out of disk, or anything else happens like I don't know, pods came up in the wrong order, so uh the solution that I will allude to is that a new semver image tag actually implies that there is a matching git tag and if you're really thinking right now about how to use this, and if you know about a feature that I haven't shown yet, then the solution might be obvious to you already.

A

You can use that git tag. You can use that image tag as git tag anyway. I think we're completely out of time. No, yes, um please join me at the flux booth.

A

Are we out of time yeah we're actually out of time? Okay, great, I hope you enjoy.