Ceph Conferences, 31 Jul 2012

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: OSCON 2012: Ross Turk, "Ceph: The Future of Storage"

Description

The quickest way to get a proprietary storage vendor to change the subject is to ask them what they think of Ceph. What makes them so uncomfortable? Ceph provides object storage, block storage, and a distributed file system -- in a single storage platform -- for free. Ceph runs on commodity hardware, allowing you to power your storage with the best technology available.

Ceph's powerful distributed block device allows you to scale cloud platforms without limitation, and its object store is compatible with applications written for S3 or Swift. Ceph has been designed with no single point of failure and intelligent nodes that are self-managing and self-healing. It's time to throw away all the old rules and start a new era of free storage solutions.

You can pay for expensive and limited proprietary products. Or you can use Ceph and invest that money in your business instead.

Ross Turk

Inktank

Ross Turk is responsible for building a strategic relationship with users, contributors, and the open source community. Ross brings more than 15 years of experience creating software, managing complex IT systems, and helping companies understand and serve developers. Before joining Inktank, Ross managed developer communities for Talend, Alcatel-Lucent and SourceForge.net, the world's largest open source community. In the more distant past, Ross ran the engineering team for SourceForge and provided architectural leadership.

A

So I have to say first when I read Pierce's description, I thought: how can I survive being wrapped in plastic and cooked alive, because it was about looking good laminated anyway, I'm gonna talk today about storage and scalability. As long as there have been humans, humans have been creating information and we keep building technology to store information, and we keep outgrowing this technology that we use to store information and I.

A

Don't think this is a trend that we can stop or reverse and I'm, not actually sure that we would want to stop a reverse because it's become part of who we are. We just have to find a way to scale it. Okay, who traveled to be here today, I, actually don't have time to wait for you to raise your hands. I'm. Sorry, if you traveled here today, you're most likely holding one of these in your pocket, it's a room key when you go to your hotel, when you check in they say Oh mr.

A

Tuerck you're in room 806 or whatever they give you your key. It opens the door usually and it's a simple thing in a small process. It's a little too complicated for me, though, because I can never remember my room number. So this is my trick. I take a photo of my room number when I'm leaving the room, so I don't have to later on, go down to the front desk and go through, but I can't remember a number, because I can't remember a number.

A

It doesn't work so well when you're in a place like Las Vegas, though, because the room number 85, 3, 2, 6, 4 8 tower whatever is not very helpful. Las Vegas is an interesting study, though, in how the hotel industry has taken the lessons from running a small hotel and they've applied them to running a big hotel by just scaling each individual task more check-in people more valets. But what if you had to build a hotel that had a million rooms in it?

A

Could you hire enough people to process check-in for a million people at a time? Maybe maybe, but probably not what? If you had to have a hotel with a billion rooms? Now you can forget having a front desk entirely. You can probably forget even knowing who's in your hotel. At that point, the way you would run a hotel will be completely different. I know this doesn't make sense. It's going to my trick also would not work in a hotel with a billion rooms because I mean so. Let's say I know.

A

My room is eighteen million four hundred sixty two thousand three hundred twenty-two. So so what you know I have the picture. It doesn't really help so running. A hotel with a billion rooms is problematic because also the hotel changes every single day you're there. If you have a billion rooms in your hotel at any given moment, a hundred thousand of them are on fire right.

A

You know a million of them are probably being renovated. You know toilets, don't work in half of them whatever, and it's less like a building and more like an organism. It's changing all the time. So if you run this hotel with a billion rooms every day, you come into work, you're working for a different hotel, pretty much so I explained all this to talk about scalability. What is the answer to this problem? More checking clerks, more valets, wider driveways, no because the tricks you learn to make things big are not the same tricks.

A

You can apply to making things infinite. It's just different. It's a very different way of looking at things. So if I wanted to have a hotel with a billion rooms, maybe what you do? Is you figure out how to make the hotel itself assign people to rooms right instead of having a centralized place? You procedurally do it and the hotel grows itself through process like a fractal? And so maybe this is you come in and you put your thumbprint in an elevator and it goes. Oh, hey, you're, Ross, Turk I recognize your shape and I.

A

Don't have a lookup table anywhere. I, don't have a centralized registry, but I know you're in this room because I calculated that and then, when I come back on Tuesday. Maybe it's a different room because there's been like I, don't know alien warfare and half the hotel's gone or whatever, and so it gives me a different room on Tuesday, but that's good, because the robots have moved all my stuff from the old room to the new room right.

A

So as long as you have robots, you're, good, so I guess what I'm saying is in order to have a hotel with a billion rooms. You need two things: you need a deterministic pseudo-random repeatable placement algorithm that can put a billion people in a billion rooms and not mess up even once, and you need intelligent hotel rooms that can move people around I, don't know liquefy them put them in a tube. You know whatever, and this exists today. It doesn't have anything to do hotel rooms, but it's called crush is the algorithm.

A

It is a controlled replication under shared hashing is what it stands for and it's the way that Saif the open source distributed storage platform figures out how to put data inside its cluster. So it's not hotel rooms, but it's the same problem. If you have an infinitely scalable storage system with an infinite number of nodes- and you can't write down where stuff is, you have to calculate where to put it and where to get it. So this is how crush works.

A

You call crush you pass it, a cluster map and a series of placement rules, and it tells you here's your data, it's in this node and when you want to go put data somewhere, it tells you where to put it and then the nodes are intelligent, that if the cluster changes it moves the data to where it should be, so that when people calculate the next time, it's where they expect it to be.

A

This is one of the things that you have to think about when you're building something that truly scales to infinity and it's one of the things that makes F a really interesting project to work on, because it's solving problems in a different way. I haven't taken a breath in like five minutes.

A

Okay, so the set project is young and it's vibrant and it's growing and if you believe as I do that this is an interesting problem to solve, and if you think that open source is the only right way to solve this problem, then we want to talk to you and we're having a buff in about an hour, and we invite you to join us.

A