Cloud Native Computing Foundation GitOpsCon EU 2022, 19 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lightning Talk: Taming the Thundering Gitops Herd with Update... Joaquim Rocha & Iago López Galeiras

Description

Lightning Talk: Taming the Thundering Gitops Herd with Update Policies - Joaquim Rocha & Iago López Galeiras, Microsoft

gitops in Kubernetes is a simple but powerful workflow: declare the deployments' desired state in git and an agent (often Flux) should pick it up and reflect the state in the cluster automatically. However, this approach allows for the propagation of issues by "broken" versions of software, which could be avoided with a progressive rollout and enforcement of policies around those. In this talk we propose the use of an update and policy manager – Nebraska – as a complement to gitops. Nebraska allows to set up policies to be met for granting updates, and aggregates the data about update statuses. The integration is accomplished using a new Nebraska Update Agent (NUA), which controls Flux itself, automatically reports statuses, and has a minimal impact to the gitops users’ workflows. With NUA and flux, users can manage new deployment rollouts in a more controlled way, by defining policies for updates, for example: update just one cluster at a time and halt all updates if one cluster fails to update; update cluster only during certain hours; see a global view of the updates’ statuses and drill down to any error reports. Hence, tying gitops and policy-based updates in Kubernetes.

A

uh Up next is jochem and iago talking about uh taming the thundering get ops herd with update policies. This is particularly interesting for us large cloud providers who have problems when configuration management goes wrong. So I welcome this. Talk have addict. Thank.

B

You so yeah hi everyone, um I'm jago, and this is joaquim and we're part of the kinfold team. At microsoft, azure and yeah we're gonna talk about uh update policies so yeah. I guess I don't have to sell you git ups here, but yeah github is great. So you describe the system declared declaratively.

B

You have your state and gate. There's another trail. You know who did each change and yeah. You have a really nice development experience in general, so uh the problem is that github supplies all changes at once.

B

So you know you do your change, you commit it and then the operators or the controllers in all your clusters will start pulling the changes so yeah what happens when something goes wrong um and while changes are tracked on it, uh the rollout is not so you cannot just say and get just roll out 20 of the cluster uh yeah, that's something you have to do somewhere else so and the other issue is that you don't have a global view.

B

So if you have several clusters uh yeah, it's usually like one controller per cluster and yeah having a global view of where your update is going. It's great, so we explore existing solutions, of course, like flux and flagger yeah. So flux is nice because it's simple, it's very flexible, but we haven't seen rate limiting and you don't have a global view of how updates are going and it's usually for a single cluster and flagger is a bit more complicated web app oriented.

B

It does have you know a progressive release and canary deployments and things like that, but yeah again, there's no global view of updates. It's a single cluster tool and it's somewhat complex. So our proposal here is instead of having flux, just pull the git changes directly.

B

You put an update manager in the middle, so these update managers should determine uh not only what commit to to apply to the cluster, but when to do it- and it should be this- the single source of truth- for what version change changes is returned to the to the clusters so uh yeah. It should also have a global understanding of each application's update and it doesn't matter you know when it where it's rolling, so where cluster is rolling, you should have you know a global view, so yeah.

C

Hey so let me first tell you about nebraska, so we have several projects involved here. The first one is nebraska. This is an update, manager and I'll be very fast, just describing that it's uh in this case passive, so it doesn't have to connect your uh instances or applications or whatever it's just the opposite. This is the applications that connect to it usually to get updates information.

C

uh Besides giving the update information, it also gives you uh the ability to monitor and to and to release uh stuff based on policies like was mentioned before, policies means, like maybe rate, limiting, uh restricting the updates to happen only during certain hours and all that. So, of course, like I said this is this is passive. So when you have an application uh using githubs on a kubernetes cluster, then you know both don't know about each other. So we're missing a part, and this is the part.

C

So uh this is called the update agent or nua for short and uh yeah. This runs in cluster, and this is the the part that is in charge of checking for updates from nebraska, applying the the configuration for flux, as you will see, and then uh checking the flux state reporting it back to nebraska. So you have this sort of control.

C

So essentially uh you know instead of of letting flux, pick everything from git directly applying it and if, if something goes wrong and you have several clusters connected to it or looking at the same repo, then you're going to have the same issue in every one of them. um Instead of doing that, we're putting nebraska before that we're putting nebraska in charge in this case um yeah so no way in there, as you can see it pulls. The data from nebraska says: okay, there's an update, so the update is going to have a git graph.

C

This is going to configure flux with that ref and then, from that moment flux operates as usual. So all nua has to do is to pick up the the status and communicate it back to nebraska.

C

This is the reporting part, so I'm going to show a demo very fast. It's going to be under the limit time. We have thanks to santosh and suraj. There are colleagues that did the heavy lifting on this one and uh couldn't be here, couldn't be here so uh yeah, let's go so in this case we have, um you know we have a a cluster and uh this cluster is running a web. Well, an application in this case is nginx, as you will see.

C

uh So, as you see, there is a version 117 running and uh now we're going to show nebraska, so nebraska is also running. This is still like has nothing to do with the with kubernetes. In this case this is just running elsewhere, if you will, and uh but this is just to show you that there is- you know this nice ui.

C

If you will, uh it could be nicer, but but you know and uh yeah, so you have the policies part that you can control this way as well, and then there is also uh you know. When you go to the instances uh or to the the update status, you can see how many instances there are usually it's much more interesting than this.

C

If you have several instances- and you can see that you know there is one instance- and there is the version now- this version is different from the nginx version and I'm going to explain later why but moving on very fast, you know you can also check the the story of uh you know the updates, what happened throughout so far. uh You know you had one update that happened, but we'll see the process of updating. So this is wait.

C

This is the uh you know the status of your application or what you wanted to uh you know the status to be so. This will be uh tracked on your regular githubs repo right. This is the stuff that flux would pull from and actually does pull from. So when we want to update the version, we just go, we say: okay, now I want version 1.21 in this case and then the regular regular spiel you just update, you know, commit and push until now.

C

Everything is is normal right, so you you push it, but now, instead of having flux, just pull it and it's like okay, it's broken chaos whatever.

C

What we do is that we instead we pick the the commit ref and then we go to our you know you could go to the ui of nebraska and set it there, but since some people don't like uis, we also have this terraform provider, where you can just change stuff there. So very quickly. We go there. We say okay now, and this is why we had a different version in the ui. So this is like a notion of a package in nebraska.

C

It has lots of information because it can mean more than just one app right. So you up the version there like that, and then you, you replace the commit that you want flux to look into sorry, maybe this yeah, so you replace it and yeah yeah. There's all the other configurations that you can provide to it. This is all flux stuff, as you as you can see yeah. So this is the diff.

C

We create a pr in this case, going just to go very fast. Hopefully you can still track. What's going on, uh you know open a pr. Then we have some actions to perform the terraform uh magic. As you can see here, let's see yeah. So this is like uh what you get from the pr when the the action runs. Everything looks fine if you're familiar with terraform. You know how to read this, then we press, merge and, and then it you know it will eventually change the nebraska configuration.

C

So there you go, you got some feedback, that's it now, nebraska got configured, it says. Okay, I got a new update whenever you know instances come and ask me for an update. I got I got new stuff to deliver so uh and that's what happens uh because noah the agent in this case will will have communicated. Let's see if it happened.

C

uh Let's see oh surprise, it happened and uh and uh yeah and as you can see, you know in the back, like I said, newa checked for the new update information got this new update, saying: okay, there's a new ref that flux needs to pull. Here's the. Let me change the cr uh that describes it and then flux from that takes it uh on. So it already reconciled and everything and if we go- and you know I talked about the reporting part.

C

So if we go back to the nebraska ui, we should see the new uh information. If things get refreshed, you can see that it's tracking already the new version. Now again, this is not spectacular because it's just one instance, but if you had like 100 clusters doing this, you could apply policy like okay. If there is one that reports, an error, abort all the updates, so the first one that reports an error, you only fail until that time.

C

Hopefully the first one uh yeah, and you can see that you know you also have the history here and uh and that's it so.

B

Well, we have 15 seconds left.

C

I don't know I don't know if we have but yeah that's it thanks.