Cloud Native Computing Foundation GitOpsCon NA 2022, 2 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Complete DR of Stateful Workloads, PVs and CSI Snapshots via Flux and Vault OSS - Kingdon Barrett

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Complete DR of Stateful Workloads, PVs and CSI Snapshots via Flux and Vault OSS - Kingdon Barrett, Weaveworks

Stateful workloads present a heavy challenge for platform operators to secure, backup, and provide for their recovery in the event of "black swan" or cluster-ending catastrophes. Developing a strategy for CSI volumes and managing snapshots should be a priority for cluster operators who otherwise may leave their important data vulnerable to ransomware attacks or other threats; what mitigations can be applied in the event of a full compromise? With GitOps, managing recovery of STATELESS workloads is made easy, almost effortless. It hasn't been shown how that level of convenience is replicated for STATEFUL workloads and DR on persistent volumes. Cluster operators may already know enough about CSI to understand that PVCs are backed by PVs and have a ReclaimPolicy that may be set to Retain. How does one manage this persistence with GitOps? How to manage snapshots? What exactly does snapshot restore executed via GitOps approach look like? Let’s endeavour now to offer clear prescriptive guidance about effective GitOps strategies for managing disaster recovery. What are some strategic approaches to apply to bring our risks back within tolerance? We will find out more in this talk.

A

Welcome and thanks for coming, this is complete disaster recovery of stateful workloads, persistent volumes and CSI snapshots via flux and with Vault. um My name is Kingdon I'm, a flux, maintainer I work at weaveworks and the developer experience Department as an open source support. Engineer.

A

Take a picture of this if you want to visit us later or keep abreast of all the git Ops things that are happening. We have a nice website where you can see the flex Centric view of get UPS days and everything else at kubecon.

A

Okay, so here's our agenda, uh we're going to learn about kubernetes CSI and we're going to use a lovely little tool called Helm, which you may have heard of that makes working with CSI volumes a little bit more approachable.

A

Okay so CSI over by and large, not a beginner topic. So why is this a beginner talk? This actually is in the schedule as a beginner talk, if you didn't notice that, so how is this a beginner talk? Well, we're going to start small and we're going to build upon what we know or what we've learned.

A

uh So here's our scenario for today this is what we're going to start with. We are going to install some chart using Helm. um I have picked bookstack completely random.

B

A

Enough persistent volumes to be a useful example for this demo. Okay, and how does this work well? A persistent volume claim and the template is instantiated and then kubernetes fulfills the claim with a persistent volume okay. So why is persistent data different than other declarative resources?

A

Well, they're stateful persistent volumes are stateful, they may contain information, um for they may contain information. Persistent volumes contain a state, they have an identity, they cannot be wiped and recreated without spoiling the statefulness. So um I bet you can guess what happens when the persistent volume goes away when the claim goes away. I bet you can guess, but we're not here for guessing we're going to see. So we are going to install bookstack. That's the example and I actually had to search for a while before I could find the maintained Helm chart for bookstack.

A

So does it have to be book stack again? This is just, for example: we've chosen this chart. No, it does not. What are the requirements to make a useful demo here?

A

Well, we want to see something in the values.yaml file of our chart. We want to see the words existing claim, because that will allow us to use an existing volume an existing volume later so, and would you look at this? Actually there's no such field in the values.yaml, so we're going to have to we, we are going to have to pick a different card.

A

uh So why is this not a good example? Because if you look here's the persistent volume claim um it's not wrapped in any conditionals, so this one creates the PVC.

A

It's not moving. Okay, there! That one um creates everybody.

B

A

Okay, it can creates a persistent volume unconditionally, so we've chosen poorly. We need that existing claim field for a reason I'll be illustrating shortly. So, let's it's not moving forward.

A

Okay, here we are, this is the slide. We we need, that existing claim field, and sometimes a bad example, is a good example. Okay, so I was actually aiming for a good example. So here's the good example uh in the helm charts repo from the stable disk, which is yes, uh it's a bit long in the tooth. But this is the example that I was actually looking for when I thought of bookstack. Can we all see? No, it's not moving forward, I'm, not sure what to do about this.

A

Okay, so here's our good example that we're looking for um it has uh storage section for configuring, the storage in the helm chart and you can set an existing claim.

A

So we're going to do a Helm install now the old-fashioned way actually.

A

Yes, this is git Ops con, but uh I promise. Before too long we will get to get Ops, so we are intentionally.

A

We don't need git Ops to have this problem. Let's put it that way, we're trying to get ourselves into a jam and we don't need git Ops to illustrate the problem, but it's part of our ideal solution. Later it looks like I have to flip back and forth a few times, I'm just going to try to do that and not talk about it. Okay, so.

A

We started with this question: what happens when the PVC goes away? We're going to answer it.

A

A

Okay, so here's the screenshot- uh hopefully this is visible, or at least semi, visible, I'm, doing Helm install and just to note at this point as a consumer of the chart, we actually may have no idea that it's doing this, it's creating persistence on our behalf um and that's good and bad.

A

uh The Keen Observer at this point May notice something else interesting. We have persistent stores for storage and uploads, which the distinction's not important here, but what you should notice is that we have no persistence for Maria DB, that's not good! So that's, okay, we're just asking questions right now.

A

We want to know if the persistent volume can be deleted and whether that's going to become a disaster for us and so let's uninstall and find out.

A

um That's right: it wasn't deleted, wait, they're, perfectly safe. Both volumes wait. R1 volume is perfectly safe. What I can't hear you and it's gone.

A

uh So all right, what do we learn? It's not a disaster. Yet it's a policy decision we have chosen or kubernetes has chosen for us to make the default policy delete, retain recycle delete are all valid. Reclaim policy settings but recycle is not very interesting. So let's say our choices are, retain and delete.

A

So we told kubernetes to delete it. uh Yes, it's the default.

A

And where is that default from uh here? It is in the storage class. You see that reclaim policy of delete in the storage class. Did we not mention before that? Each persistent volume inherits its settings from a parent storage class.

A

We did not okay, so um we can edit it in place. No, actually we can't edit it in place. Updates are forbidden, so we have to Define this when the storage class is created, so we'll just have to patch the PV after it's created. So that's what we'll do all right so back around to where we started, but we're adding one thing we learned: we have to enable persistence for mariadb. It doesn't come out of the box that way and another problem. All these resource have reclaimed policy of delete.

A

So there's a piece of kubernetes Docs we're not advancing the slides.

B

A

And let's talk about what problem did we solve.

A

We should not be unceremoniously deleting our important data so what we prevent from being deleted through the regular Helm life cycle. We can do that by setting the reclaim policy to retain so um and also we've added another persistent store. So we have really only dug the hole deeper at this point, but one thing at a time we're building up to it all right. Let's uninstall the chart.

A

That's interesting: they did not all get the same status. I wonder why Ah that's interesting I bet that has something to do with it. This one still has its claim. uh Maybe it's a stateful set thing: let's have a look at that PVC, oh boy! That is a lot anything we can do to clean it up. uh Yes, there is Cube cuddle meat.

A

Cute petal neat we'll trim it down a bit. This does what it does here is worth pause to talk about for a minute.

A

So this is Cube cuddle, neat, a crew plug-in. This just goes through that resource definition. It takes out anything, that's been added by kubernetes as an expanded default, and it does a few other things to make this resource safe, to apply and reapply on a continuous loop.

A

um It doesn't quite do everything we need for persistent volumes, so we're going to take a few of these annotations away from the persistent volume claim uh there, where there's one for um bind completed and one for Bound by controller we're just going to wipe those out, then we are going to. We have another detail. We can't really have known in advance um that kubernetes is about to fill out which node the persistent volume should attach to that.

A

Has this weight for weight for first consumer setting back in the storage class, so when the Pod is created on a node and scheduled? That's when the persistent volume is created according to the configuration in the storage class okay, so we now have solved another problem. We know how to prepare a PV and a PVC definition for rebinding, but we've just understood the relationship a little bit better.

A

Well, this isn't actually a problem we've encountered yet right, but Helm creates a persistent volume claim which spawns a PV and we need them both to complete the puzzle. So now we can recall that our PVCs were deleted, except for the Maria DB, so we can reconstruct. We could we Could reconstruct them, but we're still just poking around. So let's put the delete policy back and just wipe them out. Okay and then, when we delete that claim, the volume is really gone again.

A

Okay, so we haven't really uh solved the problem yet, but so, okay, there is a video here of we're going to try this again, and uh so we pipe each PV and PVC through Cube cuddle neat and save them on disk. Make the changes, as we Illustrated here, I'm not going to play all these videos, because we don't have time, but hopefully you have an imagination and there will be more help later. For in case you don't have an imagination so.

A

Now my screen is not updating, I'm still on Slide 56.

A

All right, I'm, not sure why it's doing that here we go okay, so once more, we're going to do a Helm install with the same options as before we're going to patch the resource of the retain reclaim policy we're going to pipe each PV and PVC through Cube cuddle neat we're going to save them on disk. Is this going to work this time?

A

There's only one way to find out, let's reapply those definitions and finally, um well the.

B

A

Is it did not work? There is more behind this video to see, but we uh we missed. One thing we needed to erase the node Affinity. uh It's not pictured in the previous video um and.

A

I'm sorry, my slide is just not updating Okay. So why did this not work? Here's another important thing. um No, it did not work because we we forgot the database password no longer matches what we have in the database itself, that's pretty complicated, but how much generating us a random password for each install and uh first time, Marine DB runs it actually records that information in the database and if it doesn't match again later, when mariadb won't start.

A

So, if you're following along at home now it starts starting to heat up. So we've got a nice collection of values here, we're actually going to put them in a file values.yaml and if you'd like to give this a try here, uh yeah. So we know we need to set the password actually I'm going to skip One Step here and say: there's two passwords: we need to set because there's a root user in the database and there's also a user user.

A

So we're going to do that. We are going to uninstall and then apply these definitions to the cluster with helm.

A

So here's our amended list for now is this going to work. We've added passwords to the list this time and if you follow the video this time you know it's working, uh you you better believe me. It took two tries to get that right.

A

Yeah, the two password thing that I tripped over that at least twice so now all we're missing is get so if you're new to git Ops I hope that this whole beginner's Journey hasn't scared you away, but remember that we set out to do just one thing for now and we've done it right. So we can uh we're gonna. Do this Speed Run actually of how do we get this into flux?

A

How do we get it into git and and use that as part of our Disaster Recovery Solution? So there is another video that should go here um and I apologize I've been working up to the last minute. um So but if you've read the flex stocks from cover to cover this, one might not be familiar because it's a bit new, it's about encrypting, values.yaml and storing it for use in a secret.

A

Then how to use that secret with a flux, Helm controller Helm, release to drive the operation of your Helm chart.

A

So here's the helm repository we're going to create it using the flux, CLI and then we're going to use flux, export Source, git.

A

And here is that Helm release repository a little bit zoomed in with some details, highlighted we've actually because there's a reason that the stable repo was deprecated, it's large and unwieldy, so we actually have to increase the timeout to make this work and we're going to do the same for the helm, release convert the helm, install command to a declarative form.

A

A

So there are some details from that last page that you ought to read the flux docs to understand if they're not immediately obvious, but there's one that I'll talk about here. The value is from is part of the guide I link to a few slides ago, and this is straight out of that guide. This is customized config.yaml.

A

um This points to a field in the helm, release it doesn't need any changes. This one does change just a little bit. Okay prior to this, I was using pod info, and uh so let's change the book stack and then, let's make sure that all the file names match what we've created we're going to need that values.yaml file that we created earlier. Also and oh, this reminds us we're going to have to declare the namespace if we want this to work with Git Ops now, I think that is everything except one more thing all right.

A

We said we're going to have secrets in a values.yaml file, so that means we need to encrypt it, um and that means we need a stops configuration. So if you do follow that link that I posted before this is the same link- and it has information connected to the Sops guide and flux, to tell you how to set up stops from scratch with flux.

A

So all right, so we're going to do that just going to follow the guide and not talk about it and commit everything to get and flux's. Helm controller is going to take over this release and if we've lined everything up correctly, then you should see Helm controller taking over the helm release.

A

You should have flux bootstrapped already at this point, um thanks Pinky and some tochi throughout the day, you'll see Flex bootstrap we're not going to do it here, but hopefully we've already seen enough of that or we'll see enough of that and um hey it worked cool.

A

So there's our our book stack, Wiki, uh We've restored it and it actually has all the data still on it. So what's missing at this point, actually uh we still don't have a backup. That was the first thing we wanted to do so that's that could be a problem, um but we're well prepared now for what comes next Okay so check out this talk from last year's kubecon that already covered CSI snapshots at large.

A

um We can use them in githubs I'm sure, although unfortunately I can't prove that right here as I'm about to tell you more about why.

A

um And there there are still some problems for us, so uh we cannot manage our Azure snapshots. I. Don't think I mentioned that I am on Azure here, but with Git Ops we cannot measure. We cannot manage our Azure snapshots without getting another third-party tool involved. Kubernetes cannot do it directly. Azure does not support it, and this is actually not too uncommon.

A

um I did not start on Azure When I Was preparing this I wanted to make it accessible. So I tried not to make it about the Synology file server that I have at home. That has an open source CSI driver that does support the common snapshot interface. So the demo is basically all the same there, and if you don't have one at home, you can try it on these clouds and if anybody shouts an am right now that also supports external snapshots. I will happily add it to this list.

A

No questions asked so: okay, let's move on so there's some documentation that I've been working on to go with this talk.

A

Disaster recovery runbook: this is the beginning of a run book that will eventually cover everything, we've discussed here and more, but it needs some work yet so I don't think I can do this by myself, but you can help.

A

I'm, sorry, the slides are not in better sync with what I'm saying, but hopefully it's still possible to follow so um get used to saying that you need help. I need help future. You needs help from present you or maybe present. You needs help right now.

A

um If you need help, you know where to find us I hope we're in the cncf slack or I am uh Kingdom B and I'm, usually in the flux, Channel.

A

So this run book has a section that is focused on what to do in a disaster.

A

It assumes you will have already followed all of the architecture guidance, so the architecture section is much longer. The runbook is short I thought for a while about how to publish this I decided not to use bookstack.

A

um Okay, so lesson number one: actually, let's back way up, don't get yourself into jams. If you can simply avoid it.

A

In this case, it would have been absolutely avoidable. Maria DB can be run externally. You can refer to it as an attached resource in a secret, just like we've been doing since 2011 or earlier as described by a 12 Factor.

A

If your organization is large and fairly confident, it is highly likely that they have backups worked out already to a science. So, if figuring, that stuff out is your job, then I hope this talk has helped.

A

If you don't have such guidance, you may want to start with a better resource than my run book that I started um three months ago, so there is prior art without a focus on git Ops Valero actually solves most of these problems and others too now uh about snapshots. Another thing to consider is: if your snapshots are housed in a resource Group or another analog like we have on azure, they can be Cascade deleted. So if you're creating snapshots is your plan for a backup?

A

uh Well, that's not the only limitation right. uh We talked about mixing secrets in in our values.yaml file with the rest of the configuration. Isn't that really the best way to do it? I, don't think so.

A

I think you should put your secrets in a purpose-built Secret store like Vault or any KMS solution, that you can access with an external Secrets operator, but it does have limitations when you use that with flux, flux, customized controller needs those things together, so that example that I showed and the other examples that are in the family, the config map, and um there there are a couple of examples there.

A

If you follow that link and just scroll up a little bit, there are examples without encryption for things that aren't secrets on how to use a config map generator or a secret generator and customize, and you need the secret right there in order to do that, it needs to be decrypted in the context of customize, so um there there is some stuff to think about here.

A

So let's talk about the risks again, you might have a single cloud account um say: the mode of failure is a total account takeover.

A

This is actually a kind of similar scenario in that we may have lost access to the account and we're unable to recover it or we've lost access to the snapshots. Like we would have if we deleted the resource Group uh case in point, you need a backup outside.

A

So what else can we do for backups? Well, if your architecture for backups and Disaster Recovery planning includes these possibilities, you probably are going to want to check out rustic. It has many scenarios for copying in ways that weren't supported or desired to be supported by Upstream apis like the Azure API.

A

Now we can't show them all here today, unfortunately, due to time constraints, but we will add these things to the runbook soon and we will continue adding them, as people are interested. If that should come to pass. If we're not prepared to go outside of what our cloud provider supports. Natively there are other Solutions. We can approach too.

A

So you can lock the resources again using the Azure API or the Azure portal to prevent their deletion. I'm sure other Cloud providers likely have a similar structure. That's available and terraform also has the perfect destroy life cycle, setting which, if you're using terraform but again one, can just comment that out and run it again. So that's really not a complete solution.

A

So what else can we do? We set off-site backups, something we want to consider? They should not live in the same cloud account if we want to protect against a total account takeover or really anywhere, that's accessible from that account.

A

So we've reached the end of our content and you can take a picture of this again if you missed it earlier. um Thank you very much for your attention. I think we might have time for a question or two. We do yeah yeah one over there.

A

We've got a microphone on the way here.

B

So all the steps you were going through to create a backup manifest of the persistent volume claim persistent volumes. So ultimately, what do you do with that manifest? That was a little bit. I wasn't sure about that. It was was that, to put it back into your own version of the helm, chart or put it somewhere else.

A

What we did with that, maybe I did skip over that a little bit. So what we did we uninstalled the helm chart or we uh yeah. We did uninstall it. So we we have taken an initial definition. The values.yaml with here. This is the slide. This is what we did. We took on the left where we have no existing claim defined in our initial installation, and we have storage enabled so that the volumes will be created by Helm and then we took them over.

A

We captured their definitions, we stored them in git um and we changed a few things. We changed existing claim to refer to the existing claim and at this point, um I uninstalled the chart and reinstalled it I had a problem when I tried just straight upgrading it there, but the actually the problem uh somewhere in here, you'll notice. If you didn't believe me when I said it all worked at the very end here here is a mistake: it's not called my book stack release.

A

It's called my book stack Wiki, so I did have an error at the very end and I had to fix that. I wound up uninstalling and reinstalling, but after that now your volumes are part of your definition. They exist and as long as they exist or they can be restored to where they were or another place. Now you can do a full restore on a new cluster, so you tear it on the cluster. You bootstrap a new cluster and it should restore all of the data from all of your persistent volumes.

A

This works basically on every CSI driver with minor variations.

B

A

Yeah all right great come visit me at the flux Booth thanks for your attention. Thanks.

B