Apache Cassandra Apache Cassandra World Party | July 2022, 2 Aug 2022

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Safe OS/system upgrade by rebooting | Apache Cassandra World Party 2022

Description

Presentation from the second Cassandra World Party on July 20, 2022. Learn more about Apache Cassandra at https://cassandra.apache.org/

Serban Teodorescu of Adobe presents “Safe OS/system upgrade by rebooting”
This talk will cover how we implemented an idea presented at an ApacheCon by Netflix. In order to safely upgrade the OS and perform various system upgrades without having to move data around on 900 instances in one week, we had to implement a simple state machine with some puppet and Python scripting. We used DynamoDB for a “cluster lock” and Prometheus for metrics to alert when the process was stuck.

About the Presenter
Serban Teodorescu is an SRE at Adobe, where he’s part of a small team that manages 30+ Cassandra clusters for Adobe Audience Manager. Previously, he was a Python programmer, and he’s still trying to find out how a developer who preferred SQL databases ended up as an SRE for a Cassandra team. Apart from Cassandra and Python, he’s interested in automating his tasks with terraform, AWX, and StackStorm.

Apache Cassandra is an open source NoSQL distributed database trusted by thousands of companies for scalability and high availability without compromising performance. Linear scalability and proven fault tolerance on commodity hardware or cloud infrastructure make it the perfect platform for mission-critical data. Get started: https://cassandra.apache.org/quickstart/

The Apache Cassandra World Party 2022 was a virtual, one-day celebration on Wednesday, July 20, 2022 held in anticipation of the launch of 4.1

Join the Cassandra community: https://cassandra.apache.org/community/

#ApacheCassandra #database #NoSQL

A

Hello, I'm sherban del dorescu, I'm working in adobe for the last five and a half years and in the last four years, or so I've been working with cassandra in an adobe project, audience manager this time I'll, do a quick story of how we've done two major system upgrades of cassandra instances using the method that I've stolen from netflix at the presentation- and this is how we'll probably do the upgrade to cassandra for one when we'll do it, which I hope will be soon and also.

A

I should apologize that I didn't have time to rehearse this. As a couple of days ago, I was, I had an unfortunate emergency room visit, but uh so we can go to the next slide.

A

So, as I said, this started at apache 2019, there was a very interesting netflix presentation hosted by joey lynch named how netflix manages petabyte scale apache cassandra in the cloud. The presentation was not recorded. This was before the pandemic times, so things were still happening on the offline, but the slides are available at the link that I included.

A

There were many interesting ideas there uh among them, um that's a concern about the imperative versus declarative control, plane and also software migration by using immutable emi- and this is described in that in a medium netflix, blog post- and this was from their presentation. The idea is that you have in aws instances with local data stored on instant store. This is data that is not being persisted when the instance is stopped, so you cannot just replace the root image.

A

What they were doing instead was determining that an image needs to be updated, installing a special ubuntu bootloader on top of the image booting into that ubuntu, and then in turn, that one is not doing a full boot but doing some booting tricks. Some all cysts have been stuffed to get the image from s3 and overwrite the disk before the root disk is actually mounted. As I said, all the see something stuff uh back to our setup. Why why we have this problem?

A

It's where we have 34 clusters in eight lws regions with replication factor two, which is a bit strange, uh usually to do a big upgrade. In this case we had the os upgrade assembly. We would have done. We have used the simply replace each instance, but doing that on 1000 instances. It's you are just asking for problems. It's anything that might happen will happen when you do it 1000 times and even worse, is that we have to do it on 34 different clusters.

A

So I already knew that. That's that wasn't an option. um The uh applying the uh netflix solution implied uh having um emi already baked with uh amazon linux, two new s and cassandra pre-installed.

A

That means no terraform to be played to plan, apply using the same hardware and just replace the route without moving any kind of data around, and here I have a nice story that at some point I was discussing this with aws support and they had really big problems in understanding how we can move the data around and uh saying that there is no way to move 1.9 terabytes of data in 10 or 20 minutes as I as, and they were failing to understand that we are not. We are not moving.

A

The data were actually switching the root ebs, so that was a fun moment to explain them how things are working in theirs in their systems. One of several nice things what this was um very consistent. um It was always going between 10 or 20 minutes when it was working so how we implemented it. We used because netflix is open, open source, only the uh bootloader stuff and ubuntu image.

A

We implemented the dyno db cluster. Lock it's easy to find this on google.

A

uh We are already using puppet and user using that for control, which instances are going to be upgraded, for instance, per availability zone or the entire cluster and a bunch of python scripts called from chrome tabs installed by puppet and on the first as a node and the first upgrade we had to move some folders to different dbs volumes so that between replacements, some some data is being kept, such as logs or instance, names and so on.

A

um On the first uh on the first try we had the. If I remember correctly, about five ten percent. The instance is stopped to uh due to various problems, mostly due to hardware, because uh enterprise has a hardware integration for for the um and that was blocking s3 access.

A

No, we are on a time on limited time, so we had no metrics. You can see. On the second execution, I managed to install some metrics, and this is an example of running in parallel on five different clusters and each of those nice lines is a lock being held and then released and then moving to the next instance in a distributed manner. But this was only on the second execution when we had to do. uh We had to install the security agent that required.

A

And we'll probably might maybe we'll share this in a blog post as well with some more videos.

A

That's great good job. You got through it in five minutes, well done and yeah a blog post. I think we know people. Yes, that would be great on the cassandra website for sure um or on your own medium. We would love to talk about it.