Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Rook: Intro and Ceph Deep Dive - Blaine Gardner, Red Hat & Satoru Takeuchi, Cybozu, Inc.

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Rook: Intro and Ceph Deep Dive - Blaine Gardner, Red Hat & Satoru Takeuchi, Cybozu, Inc.

The Rook project will be introduced to attendees of all levels and experience. Rook is an open source cloud-native storage operator for Kubernetes, providing the platform, framework, and support for a diverse set of storage solutions to natively integrate with Kubernetes. Rook turns storage software into self-managing and self-healing storage services. The benefits and use cases of Rook will be explored along with an overview of each of the Rook storage providers: Ceph, NFS, and Cassandra. A deep-dive will be presented for the Ceph storage provider on how Rook provides stable block, shared file system, and object storage for your production data. The recent features in the v1.6 release will be covered to further illustrate how Rook continues to be enhanced for production environments. Rook was accepted as a graduated project by the Cloud Native Computing Foundation in October 2020.

A

Hello and welcome to the rook, intro and ceph deep dive today, I'll be joined by my colleague, satoru also travis and sebastian, who will help out with the q a section before we get started. I do also want to mention. Travis gave a lightning talk a little earlier in this cubecon. Please take a look and watch that if you want just a quick, broad introduction to rook, but for now let's, uh let's start with some of kubernetes storage challenges. This is where I like to start when talking about uh when.

B

A

About rook kubernetes is a platform used to manage distributed applications and in the ideal, these applications are stateless, but this isn't always the case. This isn't something that can be always achieved. There is very often some reliance on external storage.

A

This external storage is non-portable, it can be a deployment burden and it also needs someone to manage the storage, especially for day two operations which composes 99.9 percent of uh the the bulk of what storage is.

A

There also may be a cases where you have storage available, but you are sort of vendor locked into your cloud provider or to something else you have set up in your organization and rook aims to overcome these challenges. So what is rook in a little more of a deep level?

A

Rook makes storage available inside your kubernetes cluster and rook supports multiple different backends. Each is a different kubernetes operator and has its own custom resource definitions for users to create storage from it.

A

Rook provides automated management for any of these storage operators. This includes deployment configuration as well as updates, and once the storage has been set up, it can be consumed like any other. Kubernetes storage, with storage classes and persistent volume claims for applications.

A

Rook is also fully open source using the apache 2.0 license.

A

The storage providers I mentioned before ceph is the most stable, but we also have cassandra network file system and yucabyte db available. Also, some quick stats that we like to highlight for rook uh we're a cncf graduated project. We have just released our version 1.6 of rook, which we're very excited about, and we've recently hit 215 million downloads of the rook application.

A

And as promised, I I'm getting into a cef deep dive here with rook, so let's hop into that. Starting with what is what is seth um stuff is also open source and it is a tried and true distributed. Software defined storage solution.

A

Ceph is pushing 10 years old at this point and provides block shared file system as well as s3 compliant object, storage. All in one package, it's been enterprise tested and it has a lot of performance features as well. If you'd like more information on ceph, you can look at the link below.

A

Let's talk about the architectural layers of rook with seth, starting with the rook operator itself, which owns the management of ceph, there's also ceph csi, which is the csi driver that dynamically provisions and then mounts storage to use user applications.

A

Ceph itself is the data layer which rook encapsulates and manages.

A

This first layer, rook management, is shown by the blue boxes. It is primarily the rook operator in the middle top and also consists of some additional daemons, depending on the configuration something I can also note here about. Ceph is that these ceph osds, that you see several of those are object, storage, daemons and they connect to underlying storage and then with ceph. They aggregate that pool it together in a software defined cluster and then expose that to the user at layer 2 we have the csi provisioning of rook.

A

This is where user application requests block or file, and then those are provisioned based on a storage class and presented to the user generally block is a read write once volume for only a single application and file system can be read, write once or read. Write many there's also a similar system in rook setup for object, storage where a storage class exists and the user can make a bucket claim for that object, storage and rook then implements a bucket provisioner to create the bucket for the user.

A

While I'm already talking about buckets in purple, if the user requests object, storage, they can then get information about. The storage rook has created via a kubernetes secret and then connect via s3 protocol using those connection, details, they're provided in the secret, and you can see that in purple for block and file storage, the user application is directly given access to via a kernel driver to the cef cluster. Underneath so rook does not sit anywhere in the data path. It is straight from the application to ceph.

A

I'd like to highlight some of the key features of rook now that I've talked about the architecture.

A

First of all, simplicity, rook is really focused on simplicity, starting with installation, which really is installing some common components, including our custom resource definitions, followed by the permissions that rook needs to operate. Following that the user can generally just directly install the ceph operator, though there is configuration if they want and following that just configure the ceph cluster resource as desired and then deploy that there is an example of a very minimal self-cluster resource off to the right, so it really can be quite simply expressed.

A

We also want to focus on as many environments as possible in rook, it's pretty natural to think of bare metal where you can bring your own hardware or use shared hardware and provide storage to your ceph. Cluster rook also operates and is focused on cloud providers, and this is largely to expand upon cloud provider storage with rook's capabilities, digging a little deeper into that last bit in a cloud environment.

A

Ceph will use persistent volume claims as the underlying storage there's no need for direct access to local devices, and this can give a consistent storage platform wherever kubernetes is deployed, regardless of where kubernetes is deployed. This is great for multi-cloud installations.

A

This also can help overcome some of the shortcomings of cloud providers storage, whether this is allowing storage across availability zones or a lot of providers actually take many minutes to fail over storage. That ceph and by extension, rook can fail over in seconds.

A

Rook also allows for a greater number of pvs per node, effectively limited only by whatever performance is there than strictly saying. Only 30, like some cloud providers, and some users also really want to get better performance to cost ratio by picking the storage from their cloud provider. That gives them what they want for the best cost and then using that storage uh within rook rook is customizable for basically any cluster topology.

A

It is customizable across and within topologies, and in order to provide high availability and durability, we can spread ceph daemons across failure domains.

A

We can also deploy on specific nodes if you desire to totally separate storage nodes from application nodes and have some more support that way or even just have a single storage node. If you have a very small environment similar to ease of installation, rook updates are mostly fully automated ceph updates and evensef major upgrades are handled totally by rook.

A

Similarly, rook patch updates, uh for example, going from version 1.6.0 to version 1.6.2. That update is fully automated. You just need to tell kubernetes that you want to update rook rook minor upgrades do sometimes require some manual work.

A

This allows for taking advantage of the latest features, and occasionally a kubernetes, cef, csi or rook feature is deprecated and users may need to take some manual steps to migrate. But this is pretty pretty rare. In more recent releases, everything will be documented in rook's upgrade guide for upgrades. We do still try to make everything as automated as possible and as simple and streamlined as possible.

A

The csi driver, which I've talked about a lot, provides a lot of the functionality that rick provides. It allows dynamic provisioning of read, write once or rewrite many volumes for both block and file system. It also allows volume, expansion and it allows snapshots and clones, though that's still in beta. There is an older flex. Volume driver that's still available, but support is limited and we really, we really recommend using the csi driver whenever possible.

A

Another feature that rook provides is the ability to connect to a ceph cluster, that's outside of the current kubernetes cluster, for example. If your organization already runs a ceph cluster, you don't need to create a new one inside of kubernetes with rook. You can connect your kubernetes cluster with rook to the existing external ceph cluster, as we call it, and then you can still dynamically create your block file or object storage for your kubernetes applications.

A

The ceph cluster is just managed outside.

A

I also talked a little bit about the object, storage, provisioning. This is something that is pretty unique to rook. This allows administrators to define a custom storage class for object, storage and then a user can create what is called an object, bucket claim or obc and following a pattern, that's similar to persistent volume claims.

A

The rook operator will create an object bucket when requested, and then access to that bucket is given via kubernetes secret there's. Currently, a kubernetes enhancement proposal called cozy, which is short for container object, storage interface, which aims to supersede this functionality. Rook has a what I've often called a proof of concept of the themes of cozy, and you can look forward to hearing more about cozy and rook in the next. I would say year or so.

A

I'd also like to specifically talk about the new features. We've added in rook version, 1.6 really quickly. Firstly, starting with support for a new ceph version. That's ceph, pacific in rook version, 1.6 support three step versions.

A

We generally aim to support two ceph versions, so support will be dropped for version 14 in version 1.7, but we do want to give this time for users to migrate as they desire.

A

Broadly ceph file system has a couple new features which are exciting. We have support for multiple file systems, perseph cluster, which is in uh general availability. It's no longer in a preview state. We also support mirroring a file system from oneself cluster to another.

A

We also support high availability for ceph manager, daemons and pod disruption. Budgets are enabled by default, which helps us maintain a stable ceph cluster during maintenance and updates.

A

Osds have a couple enhancements that we can call out specifically osds can be bulk, updated in parallel. Rook will do this automatically and it respects failure domains to retain that high availability and also we are continuing to try to use lvm the logical volume management less and less for new osds.

A

This has allowed us to restore support for creating osds on partitions and just puts less stuff in the way of what we're trying to create for the user.

A

I'd like to pass it off now to my colleague, satoru who's, going to give a demo.

B

Next I'll demonstrate how to create a log safe cluster.

B

There are two types of clusters host based cluster and bbc base cluster in host based cluster. You specify hardware configurations directly in safe cluster cluster resource, then push standard data is on hostpaths in ppg-based cluster. You specify volume claim templates for ost, then push center data is on ppc.

B

Host-Based cluster is suitable for a simple cluster, especially if you use all nodes and all devices, but cluster resources get complicated in host-based cluster. If not all nodes are used, or there are various hardware configurations for each node.

B

In such cases you might have to list all nodes and all devices like this.

B

In ppg based cluster, you are free from describing hardware configurations.

B

Instead, you should specify volume, claim templates inside a glossary source, and you should specify the number of ost and the storage class names and the oysters size since pbc based cluster is not so intuitive I'll. Explain its detail by creating a simple ppg based cluster step by step.

B

I'll use this environment to in this demo, there is a kubernetes cluster consists of one node. This node has two local purchases and volumes: local 0 and local 1, and this kubernetes cluster has rook operator.

B

So this demo contains two steps. First, one is create a simple cluster. The second one is expand this cluster. So let's set the first step, create a simple cluster to create a simple cluster. Let's apply support sample cluster.yaml on the toolbox. Yaml progress.yaml contains a simple, uh safe cluster cluster resource, and the toolbox yaml contains a toolbox port. The definition of toolbox port toolbox spot is to use executing to execute a safe command from kubernetes.

B

So, let's see the sample.

B

Cluster.Dml sample cluster okay, so it's safe cluster cluster results and the most important part is the number of counts and volume. Gram templates inside storage class device sets number of ost is one and this the osd are consumed from local volumes and its size is uh at least five gigabytes.

B

Okay. So let's apply this yamu.

B

B

Okay, let's up also apply toolbox port.

B

On the right watch, the list of all save look save ports, and these are csi drivers and next this is a monitor. Ports.

B

Okay, so now monitor port initializing and now running next uh manager, port and ost prepare ports are created.

B

So please wait for a while: okay manager, port is creating and oc prepare port is now creating. Ocd prepare port is to initialize the new osd. So now it's running so it means configuring the new osd. So it takes a bit long time. So please wait for a while, but probably okay, yes, it's completed so rock safe cluster is launching and now initializing.

B

Okay, it's running so the get bot. Okay, all ports are created. These are csi cc drivers and it's a manager, ports monitor ports, operator, ports and usb ports, and we also created. I also created toolbox port, okay,.

B

So kubercontrol get baby. Okay, local zero is bound to ppc, that is created by rook and its size. Is the pvc is bound used by this looksafe ost00 port? Okay? So we also, we can confirm the status of the safe cluster by executing save commands from toolbox port.

B

Take I save, choose, save okay, the safe cluster is successfully created and its row capacity is six gigabytes and it's the same as the capacity of local local, zero percent volume. Okay,.

B

So all ports uh created toolbox, monitor manager and ost and local 0 percent volume consumed by ost with t0.

B

So the next step is expand this cluster. It's very easy. You just need to increase count field in cluster resource.

B

Control rule save edit save cluster.

B

Okay, look safe storage grass device set okay, so currently, of course, the number of ost is one, so let's increase it to so and save so, let's see, let's watch the.

B

Port and look save namespace now, look the operator port is under reconsideration.

B

Okay, the new oc prepare port to initializing. The new osd is running. It's initializing, the new sd.

B

So it takes some times: okay, it's completed and the new oc1 port created.

B

So it's running okay, so the osd one usd one is added to my safe look, safe cluster control get b; okay, both local volumes are bound to oyster ports and, let's see k status.

B

So now the row capacity increase is increased from 6 gigabytes to 12 gigabytes the it succeeded to expand my kubernetes by safe cluster okay.

B

So if you need to increase increase the number of oysters more and more you just prepare, you just need to prepare people, persistent volumes and increase the account period. It's very easy.

B

And there are some advanced configurations of pbg-based cluster. The first one is: if uh you use csi drivers with dynamic volume provisioning, you can create purchase and volumes for ost on demand. In other words, if you don't need to prepare passes and volumes before increase count, you increase account then purchase. The volume are created.

A

B

Demand and the second one is: if you need a spread ost evenly among all storage nodes, uh I guess it can be applied to. In most cases you can use topology spread constraints feature in kubernetes.

B

So for more information about these topics, please read this blog post.

A

Thanks satoru and thank you all for tuning in I'd like to leave you all with some information about how you can get involved or just get more information about rook I'll leave this up. While we now start our q a session. Thank you again.