Ceph Ceph Virtual 2022, 15 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: RGW Zipper

Description

Presented by: Daniel Gryniewicz

RGW Zipper

RGW was developed to provide object access (S3/Swift) to a Ceph cluster. The Zipper abstraction API divides the RGW into an upper half containing the Operations (Ops) for S3 and Swift, and a lower half, called a Store, containing the details of how to store data and metadata. This allows the same Ops code to provide correct S3 and Swift semantics via a variety of storage platforms. The primary Store is the current RadosStore, which provides access to a Ceph cluster via RADOS. However, new Stores are possible that store the data in any desired platform. One such Store, called DBStore, has been developed that stores data in SQL, and specifically in a local SQLite database. Additional Stores, such as S3, are planned to provide additional flexibility. Zipper also allows intermediate Filter layers that can transform Ops, perform policy (such as directing different objects to different Stores), or perform caching for data and metadata. The first planned Filter is a LuaFilter, which will allow rapid prototyping and testing of other filters. An individual instance of RGW will consist of a stack of Filters, along with one or more Stores providing actual data. This presentation will cover information about Zipper, about the existing DBStore, and plans for the future.

A

Project zipper.

A

um For a while now, the S3 implementation in rgw has been considered a high quality, S3 implementation.

A

It provides sort of Best in Class capabilities and testing, and so there has been interest in allowing us to provide these features for other storage systems besides just stuff, so, for example, ISP storage sources like the gate or Intel or cloud storage.

A

In addition, we want to provide a flexible, API based framework that allows us to stack filters that can make changes to operations based on interesting scenarios, for example, tiering or caching, or something like that.

A

um We've actually had requests over the years from people to allow us to to allow them to integrate with RS3 protocol stack to provide a better S3 for their systems. In addition, we would like to provide the ability for developers to have easy to install local S3 compliant systems for tested for development and local testing of apps.

A

The way we decided to do this was we decided to decouple the protocol layer of rgw from the storage layer of rgw, which we called unzipping it and then develop an API that cuts the entire way across rgw, so that, above it is protocol and below it is storage, so that we can add in other plugins, that provide storage for the existing protocol systems such as S3, but here's more or less what it looks like before.

A

We had a single large system that had all of the protocols and the integration with liberatos API all the way down through ratos to ceph.

A

What we've done is split it apart, so that now you can have several different loadable drivers that provide to the protocol layer, the correct amount, type of storage, and then each of those can talk to their own storage system. And the obvious example is the rattles store which talks to the existing ceph clusters.

A

They architecture looks like this at the top. You have your protocols below that. We have a core layer that provides operations that are common to all object protocols below that we have our zipper API, and this provides an abstraction of the various things that object. Storage would use such as buckets and objects.

A

The fact that this API Cuts entirely across rgw and there's no side channels allows us to actually provide intermediate layers and they implement the zipper API both on top and below, so that they can perform transformative operations such as caching and then at the bottom. We have storage layers which implement the full zipper API and provide canonical data storage for actual operations.

A

Zipper has been, to a large extent, embraced Upstream. The first project was cortex motor from Seagate.

A

They had an S3 implementation that was not up to their standards, and so they looked around and found zipper and decided that they would attempt to implement a driver for the zipper API which they did in a couple of months, and it is now in their Upstream and it is in our Upstream and it's been pretty successful. The second one was deos from Intel. They have a memory based storage system and needed an S3 implementation to live in front of that and again they implemented a zipper driver in Fairly short order.

A

The third big project is aquarium from susei. They were providing a hybrid file, slash database Standalone driver. The goal is to provide a high performance, single node, S3 implementation. It can be used in um in situations that require single nodes, such as extreme Edge,.

A

So the zipper API at its base level, we provide a set of pure virtual based classes, and these cover the various Concepts needed to implement operations on an object store.

A

The most important of these base classes is the store. It is an entry point for a storage, abstraction or a filter.

A

There is a single instance of it for each running rgw that provides access to the other classes and allows the proper stacking there's a set of primary abstractions that we use for getting stuff done, such as users and buckets and objects and multi-part upload.

A

There's a set of helper classes that let them help the main classes provide provide their work, such as a serializer or a writer which is an uh um an asynchronous writing class.

A

We provide lifecycle, implementations and we provide cluster layouts such as zones and Zone groups, and then rgw has the ability to embed Lua scripts, and so we provide a Lua manager that allows you to store and retrieve Lewis scripts in your store.

A

A

Implementation doors sit at the bottom of a stack and they implement the entire API and they provide canonical storage for reading and writing data and metadata the base. Implementation is in rgw cell store and it provides the keys and the key data for each type so that all stores will provide will have the same key data.

A

A filter is an implementation of the API that lives in the middle and performs a transform a transformation on an operation as it passes down the stack.

A

A filter does not need to implement the entire API because we provide a base implementation that is passed through so that any portions of the API you do not override just get passed through for you, a filter will pass, will optionally make a transformation and then pass that down the stack and then the results from lower and the stat get passed back up and again. Optionally perform a transformation.

A

um At the top level we have The Zipper implementation is in style.hncc, it's the pure virtual API, and it also contains provides a manager that allows the loading and stacking of doors and filters, and also rgw cell.h, contains the canonical documentation in the form of oxygen for the API.

A

We also that have the implementations of the base classes for stores and filters and those live in Sal's store and style, filter, respectively, implementations of stores or filters live in Sal implementation, so Sal ratos is the Rado store, implementation and Sal d4n is the d4n filter, implementation and there's a store directory that contains an implementation directory for each store or filter that has the detailed implementations of that door or filter.

A

Each store or filter is a loadable module.

A

um The config for it lives in a Json document which lives in a config store that is cluster-wide so that every rgw in the cluster has access to the same information about stacking filters and stores. So at config load time the config store, the config store manager determines which stores and filters need to be loaded and loads. The associated plug-in and sets up the stack of filters and stores correctly plugins can be created in tree or out of tree, and only the loadable library needs to be available at runtime.

A

The loadable plug-in work is not fully complete, but it is almost complete and uh that should be finished within the next month or so.

A

um I would like to go a little bit through code, so this is the uh as an example. This is the pure virtual base class for an object um you can see it contains some subclasses that help, for example, reading and deleting it contains operations such as copying. It contains accessors, for example, to tell if something is compressed or changed compression.

A

um There are methods for accessing attributes, transitions to and from the cloud placement rules and so on ways of interacting with the bucket which owns this particular uh contains this particular object and then um omap and such things as that.

A

um The store object, provides the base class for a store implementation of an object, and it contains the the key data for this particular object, which in this case is the an rgw object state. It also contains a pointer to the bucket containing this object and it has a cache of the attributes for this object.

A

This allows every store to have the same key data and to not have to implement the key data on their own, uh which allows for consistency across stores other than that. This only provides access to the key data that it stores and does not override any other methods.

A

So a store would inherit a a given stores object. For example, a ratos object will inherit from this class.

A

Filter the filter object contains only a pointer to the bucket at that contains the object at this level. So the same filter object, the same filter bucket at the same filter level, and it contains the next layer in a point, the object in the next layer in the stack.

A

So if there is another filter below this object, then next will point to that filter. If there is a store below this object, then next will point to that store and all the implementations of all of the methods, as you can see here, just pass through to the next layer down. This means that any filter which We'll Inherit from this, such as d4n filter, can override only the methods and and objects that they can methods and classes that they care about. All the rest of them will fall back on this default.

A

Implementation which passes through down to the next layer.

A

That's the basics of zipper. Are there any questions.

A

Question is: is there any lands on having some form of grpc back end? Where zipper would call services, uh it can deal with the operations and they would decouple it from C plus plus. uh Currently, there is no plans to decouple from C plus plus.

A

um There is a plan to write a a Lua filter which will be an implementation of the filter layer in Lua, which will allow scripting of filters for Rapid development, um but there is currently no plans to decouple this from C plus, which doesn't mean it won't happen. It just means that we haven't, we haven't planned to do it yet.

A

Other questions all right. Well, if there's no other questions, then thank you for coming and if you do have offline questions, feel free to email me or hit me up on Google, Chat and I will be happy to answer them. Thank you.