Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Sponsored Session: Securing S3 Backups Against Ransomware - Tom Manville & Michael Cade, Kasten

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Sponsored Session: Securing S3 Backups Against Ransomware - Tom Manville & Michael Cade, Kasten by Veeam

Sharing of Personal Information with Sponsors In order to facilitate networking and business relationships at the event, you may choose to visit a third partyʼs virtual booth or to access sponsored content. You are never required to visit third party booths or to access sponsored content. When visiting a booth (e.g. by clicking on a third partyʼs logo in the Solutions Showcase or exhibitor directory, and any actions within the booth thereafter including viewing resources), when accessing sponsored sessions in the Sponsor Theater, or by participating in sponsored activities, the third party will receive some of your registration data. This data includes your first name, last name, title, company, address, email, standard demographics questions (i.e. job function, industry), and details about the sponsored content or resources you interacted with. If you choose to interact with a virtual booth or access sponsored content, you are explicitly consenting to receipt and use of such data by the third-party recipients, which will be subject to their own privacy policies.

When backing up cloud-native data, there is no better place than object storage. From databases to data protection applications, the cloud-native landscape is littered with projects that backup data to object storage. With all this mission-critical data being stored in object storage it’s no wonder that it’s become a high-value target for cyber criminals. This, in combination with the rising popularity of cryptocurrency, means that ransomware protection is now a requirement for IT departments of any size.

The S3 protocol has become the de-facto standard API for interacting with Object Storage. It is implemented by nearly all object storage providers and consumed by modern cloud native applications. Thankfully it provides all the primitives needed to develop a robust strategy to protect against ransomware attacks.

In this session, we’ll walk through how backups saved in an S3-compatible can be made ransomware resistant. We’ll show which APIs are used and how to configure the buckets. If you’re considering backing up data in object storage and your data is too valuable to lose, then this talk is for you!

A

Thank you for joining our session on securing s3 backups against ransomware.

A

So as we get into this just a bit of an introduction, so I'm michael cade, I'm a senior technologist here at cast and by veeam and I'm joined by tom hi. Everyone.

B

My name is tom manville, I'm director of engineering here at cast and by veeam, and I was on the founding team at caston.

A

Awesome tom's going to be giving the good news later on before that I have to put in some of the doom and gloom around well just the reality of. Why do we need to protect our data against things like ransomware, so I'll get into the the bad bits? First and then tom can tom can bring it home later on. So the first thing that we want to touch on is well what is the need for a mutual backups and as much as we're talking around specifically around kubernetes clusters and kubernetes data?

A

The actual premise here actually resides in any any other platform, so whether it's virtualization, whether it's cloud-based ios workloads, whether it's on-premises physical machines, they all of these player, play a a reason why we we need- and we should be considering that mutability within our within our backup chains or off-site copies as well, so just to quickly run through what these look like accidental deletion and I'm going to come through and put some of these into specific use cases later on.

A

Like we know, people make mistakes, and if we look at, if we were going to look at um the analysts who are constantly speaking to to you guys and understanding what happens from a failure scenario, point of view, accidental deletion is always in the top three things that are causing for data loss.

A

Then you've got policy gaps in terms of well. I thought you were backing this up. I thought we were backing that up and there's that confusion gap around what is being what is actually being protected and what actually is being um backed up to a different location for to prevent those failure, scenarios or not being able to recover from those failure scenarios and then probably more and definitely over the last 12 months is around security. So both internal security threats, there's a growing number of insider malicious activity going on.

A

I'm going to touch on this a little bit later, as well as well as that everyday news wire coming down around external security threats and namely around ransomware, and that's really the point where a ransomware attack is going to try and get into your system. It's going to try and encrypt encrypt your data at a very 101 level, uh but there's so much more to ransomway.

A

Ransomware is getting so much more advanced in terms of being able to understand a lot more about your environment, like I said before, you've probably got different platforms within your infrastructure within your environment, so ransomware could wreak havoc understanding. What that looks like and it's getting even more intelligent that way.

A

So the the whole prevention is one thing, but the fact that ransomware and cyber cyber attacks are getting more um advanced in how they're attacking their their victims just really resonates that we need to be thinking about how we store that backup data in a in that immutable fashion, so that it cannot be modified both from an internal point of view, so your backup admins they can't even they shouldn't be able to touch that that backup file, but also, obviously, we don't want the external threats doing that either and then also legal and compliance, especially in certain sectors.

A

Verticals, they have the requirement about well, this cannot be touched. This cannot be modified and, whilst the other, the other, four that I've mentioned before are really important.

A

The legal in compliance, obviously in some some verticals are, are also business, threatening as well from that point of view, so just to really hit home and really touch in on some of those targeted, kubernetes use cases here. So I first mentioned around security or external security threats, and this in particular, this hildegard happened at the very beginning of this year, so we're talking january 2021, so very new, very topical, and it really.

A

The premise of this is that it's going to target cloud and container infrastructure, so your kubernetes or cloud native environment, and it wants to inject itself in there and start mining for cryptocurrency.

A

It's going to leverage the the underlying hardware that you've you're paying money for, especially in the public cloud, and it's going to leverage that to to to gain that cryptocurrency to to mine that how it does that is, is via deploying malicious code that targets exposed docker demon apis.

A

It's been active since that very early 2021 that I mentioned, and the potential to exfiltrate sensitive data. So, whereas I mentioned around ransomware in the beginning years and years ago, was all about encrypting and holding you from ransom. Well and and the ransom was a monetary value that you would pay, probably in some sort of bitcoin.

A

That's not always the case anymore, and actually the data is somewhat. Some could be more uh valuable if they were able to exfiltrate that out of your your environment, then we touch on, and hopefully this is the worst it gets from a from a ransomware point of view.

A

But then we move on to more of that insider threat, which is actually maybe a little bit more scary in that all of us work for companies that are obviously looking or have adopted cloud native or at least some sort of infrastructure platform over the years, and we probably have some responsibility when it comes to that and just to touch on the ransomware side of that is that well, ransomware is quite a hard process in the it's always good, being able to write the ransomware or the the threat.

A

But then you have to find an entry point into the business or into the vertical that you're looking to to attack. You've then got to find some way of compromising the user accounts, finding misconfigurations or vulnerabilities to be able to gain more of a footprint within that environment, whereas this is where it gets. Really. Quite scary is over the last again over the last 12 months.

A

Maybe due to the pandemic or not, is that there's been the ability to buy network access over the internet and that could be anywhere between three hundred dollars to ten thousand dollars, and this kind of gives you a pick and mix option to be able to well. I want to target this particular vertical, and I want this particular access, because this is where my ransomware could really work, and you might have also heard the term ransomware as a service.

A

Well, this kind of coins and goes into that that same same bucket- and this is the scary bit a disgruntled employee, uh an employee that is is leaving, and this is really playing into that- that internal security threat and that we should be thinking about that least privileged model, making sure that the people only who absolutely need access has a have access, leveraging things like role-based access control, leveraging things like policy based um access as well all key things that we should be considering, and I think, from a kubernetes point of view, that's very much front and front and center of mind, but it still obviously has to be implemented.

A

It doesn't just happen, um and then we get back on to that accidental deletion. Now people do make mistakes, it doesn't matter what platform you're on human errors are absolutely bound to happen and you can prevent as much as possible around how that doesn't happen.

A

Kubernetes is obviously largely um made up of code, so it takes away that human impact or that human error, but still there is that chance that the accidental deletion process could could take um take hold and then in particular I found this on stack overflow, so I accidentally deleted my kubernetes namespace and funnily enough. There's not that many answers in there um because there's not much.

A

You can do about that if there's, no, no backup or or uh or I'll the ability to bring that back and something that I've I've coined here or many people have coined. Is this code 18 and again this is this is applicable for any platform across any any uh environment? Is the code 18 being 18 inches from the screen user error, type type, um type error?

A

So, just a couple of points before I pass it over to tom to to bring in the the good news of well how the how? How can it? How can immutability help in these instances and some of the prevent preventative tasks that we can put together or help with around making sure that our operating systems, our software, is all up to date, it's using the the correct um patches, etc that are being shipped out on a daily basis, regardless of what platform you're on.

A

Also then routinely audit those access lists that you have making sure that still, the people that need to have access have access and probably a broader, because not everyone.

A

I consider the I.t, or at least the the operators. The developers have a good understanding of what ransomware is what phishing attacks look like from a wider scale, but maybe the hr, maybe the finance, maybe other business sectors, maybe even sales. If you're not in a in a tech orientated um company, then maybe we need to educate as well and continually educate, because ransomware has changed dramatically over the last just over the last three years.

A

It's been become more mainstream and more vicious in how it attacks so just and and also much more intelligent in how it how it um how it attacks as well and then.

A

Finally, the backup backup is super important and that's really what's going to lead us into what tom is about to speak about is making sure that we've got a copy, and the final point that I'll I'll raise is around mastering the three two one rule and again this methodology works regardless of platform, and it basically means three copies of your data on two different media types, one of those being off-site air-gapped and immutable away from any malicious activity, whether it's insider, whether it's external and with that I'll hand it over to tom.

B

Thank you michael. Given the number of entry points for attacks, the possibility of accidental data loss and the complexity around governance, compliance requirements, immutable backups are an important part of any data protection strategy. Object. Storage has become a popular destination for backups in the cloud native ecosystem and so we'd like to demonstrate how object storage can be used to create immutable backups.

B

The scalability simplicity and the robustness of object storage has made it the perfect target for backups in the cloud-native space objects. Also called blobs are accessed via simple quests, which have been wrapped by a number of tools and libraries. Many databases have object. Storage support built in their backups are automatically pushed to object stores.

B

In addition for data protection, vendors supporting object, storage as a backup target has become table stakes. You can also find several open source projects that make it easy to back up to optic storage copy is my personal favorite, since my team is actively contributing.

B

Many projects treat blobs as immutable, never updating a blob one certain this has become a best practice to help deal with the consistency provided by object stores and their caching layers in the simplest scheme. One one blob can map to a single backup. However, more advanced projects will split back up split, a backup across multiple blobs enabling better performance.

B

Treating blocks as immutable is an important requirement when implementing immutable backups. However, this is not sufficient, since your code may not be the only one attempting to write to the bucket.

B

The only way to ensure immutability is to use the primitives provided by the underlying storage. Let's take a look at what the s3 api provides, although not it's not the only api for optic storage, the s3 api has become the most prolific and includes all the primitives needed for creating immutable backups with a feature called optic, locking in order to use object blocking you need to make sure that your bucket is configured with object.

B

Versioning based on your requirements, you'll need to set object, locking parameters in your api requests for the most basic schemes using s3 object. Locks will be similar to using any f3 bucket for backups I'll go through the changes needed in the api requests to configure immutability.

B

In order to use object locks, your bucket must be configured with versioning. Any request that modifies a blob in a virgin bucket will create a new version of that blob. The previous versions may still be accessed deletes, will delete, will create a delete marker that indicates a delete which was issued. The availability of previous versions of an object depend on the relation on the retention settings for the blob, the behavior of object, locking is determined by the retention mode. There are two retention modes. Governance mode is more permissive to bypass the retention.

B

Setting, users must have permission. In addition, the user must exclusively set a special header. Compliance mode is more restrictive and once set a retention period cannot be reduced in aws s3's case. The only way that an object may get deleted within a retention period is by completely deleting the aws account.

B

There are two ways an object can be retained. The first is by a mechanism called the legal hold. A legal hold may be set or released by any user. With the put object, legal hold permission, an object cannot be modified or deleted when a legal hold is set. The second mechanism is a retention period.

B

An object cannot be deleted or modified until after the retention period expires, a retention period may be configured by default at the bucket level at the blob creation time or may be extended via a separate api call, let's dig into some of the apis when creating a blob as part of a backup you'll want to ensure that the retention period is set long enough, so that the backup cannot be deleted or modified for an amount of time determined by your retention requirements.

B

Since the bucket requires versioning enabled the results will have a version id for the blob. It's important to save the version id since you'll need to query for this version of the object.

B

If you don't specify a version id when querying a blob, the latest version of the blob will be returned. This works well for the simpler backup schemes, but is not sufficient for the immutable backup use cases. Blobs may be overwritten accidentally or by attackers, and therefore blindly using the latest version is unsafe.

B

For this reason, you'll need to track all the versions of all the blobs for your backup.

B

In addition to being cost effective, deleting backups may be required for legal reasons, deleting blobs associated with a backup in a version bucket requires deleting a specific version of the object. If the blob is currently object blocked, then the delete will fail for immutable backups, it's best to always specify the version. When deleting the blobs, if not specified, then the latest version will be will be set to deletion marker, but older versions of the blog will still continue to exist.

B

You may not know up front how long you'd like to retain a blob and a blob's original retention window may not cover the lifetime of the backup. This is especially common occurrence with the duplicated products where blob may be referred to by multiple backups.

B

To handle these situations, you'll need to extend the retention period of blogs using the put object retention api to reset an object's retention settings, since this will create a new version of the object. You will now need to track the latest version id. The previous version will still exist.

B

Hopefully, you'll now understand what immutable backups are and why you may need them. The apis, provided by s3, make it easy to implement simple backup schemes and make it possible to build more advanced schemes as well, whether you're interested in building mobile, backups yourself or would like to take something from off the shelf. We'd love to hear from you and understand how immutable backups can fit into your data correction strategy and with that we'd love to take your questions.

B

B