Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2019 (Shanghai), 5 Jul 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive: Rook - Jared Watts, Upbound & Dmitry Yusupov, Nexenta Systems

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Deep Dive: Rook - Jared Watts, Upbound & Dmitry Yusupov, Nexenta Systems

In this talk, we will be taking a deep-dive through both the architecture and some of the more recent developments of the Rook project. Rook is an open source cloud-native storage orchestrator for Kubernetes, providing the platform, framework, and support for a diverse set of storage solutions to natively integrate with cloud-native environments. We will learn about using multi-homed network configurations for popular Rook storage operators such as EdgeFS and Ceph. Backend and client networking isolation will be explored in detail, focusing on performance analysis and security benefits of such configurations. Rook was accepted as the first storage project hosted by the Cloud Native Computing Foundation in January 2018.

https://sched.co/NrrE

A

So we can I be talking about deep dive and particularly ruk, which is currently a project, isn't sincere and edgy Affairs as a operator. It's a newer curator which is very similar to what CF, but.

B

A

Little bit different use cases, so you're gonna be talking about that a little bit about myself, so I'm founder of Dixie, anta and CTO. Recently, we just saw that with EDM and before.

B

A

A

Lin Xiao Jie was a Linux hacker for very long time didn't mess with leanness but, for instance, if you using AI Scaasi within a Linux kernel. This is what I've done back in 2000 3000.

A

A

So what is ruk ruk is essentially is a Orchestrator which is used for in cloud native storage,.

A

Particularly in kubernetes and extends kubernetes with custom types and controllers which are tailored for storage and soft storage issues, it had put deployment.

A

Does the bootstrapping automates configuration provisioning, scaling, upgrading migration disaster, recovery, monitoring and resource management, they're all very complex tasks, and if you do this by any other means, you have to think about essentially implementing something which you take damages. So that's why we cannot leveraging kubernetes for these purposes. It is also a framework for many storage providers. It used to be just cf a few years ago. Now, these days, it's modern cf. It's obviously add your first, my presenting at your face. There are other providers and talk a little bit more about this later.

A

It is hosted by cloud native computing foundation and it is totally open source, so storage on kubernetes. What is this really okay? So this is the example of classic kubernetes, with three nodes, one master or maybe more than one master for high availability. And then what do we see? We see processor machine networking with XI disks.

A

We saw ok, so why don't? We just consume those workers in the kubernetes and build up because this all we need, we need jazz orchestration and we want to run our software as essentially our pods or stateful sets, or something like that and consume the storage devices and provide virtualized software-defined storage on top.

A

So we essentially deploy storage into the cluster itself, so it runs and managed by the kubernetes, the harness of power of kubernetes. For that we automated management by smart software, it's a portable abstraction for all our storage needs.

A

Ruk is a framework in a sense that route by itself is not a storage data plane. It's just in Li an Orchestrator framework for different providers. For instance, it does this storage resource normalization, it discovers the disks and.

B

A

Those disks to the provider, so they can utilize them. Non-Acidic can be just this week with the other persistent volumes which operators can consume the operator pattern plumbing, so we're leveraging this between the different operators and storage of apparatus, common policies, specs logic. This is all the leveraging testing, and these are the current providers which we have. We just add your face. You have cockroach DB Mineo and a phase Cassandra and more to come this year, so watch out.

A

Now, what's in 1.0, and we just recently released 1.0, we announced this at previous cube. Con 1.0 has too new yorky readers, which is Apache Cassandra and extend the edge of s, and we can be talking a little bit more about edgy fest. Today,.

A

And we will talk about multi-home network, as example of usage, and what you've done for each of us. Okay, what is edge affairs really. First of all, this is gate, like architecture I just wanted to give you this snippet from the get documentation. So you can read about this and you can kind of you better understand how a GFS is implementing the storage within within the itself, to like kind of three important points highlighted in red. One is: if two objects identical, they will have the same.

A

Cha-Cha is a fingerprint signature of the data block, engage how many of you know how get operates and using it today raise your hands. How many of you knows the get using.

B

A

Today, yeah I'm surprised that any people who didn't raise the hands here because get is ultimately is a widespread source control management system. Wide so widespread is because it's a fault, tolerant and highly scalable system, and we saw that if you build storage system is the same. Similar architecture and.

B

A

We can achieve enormous fault, tolerance and availability if two objects different. They have different shara, that's how we essentially detect all sort of errors and corruptions, and then we do automatic self-healing by fetching the data from different sites and regions and refers objects always keep a graphically self validated and therefore globally, unique. So what that means is it doesn't matter where your data or metadata is located? If you always have exactly the same signature, therefore we can match the signature, and I can do a global verification of that signature.

A

You can think about benefits of global cache, for instance right. It's totally, mutable storage. You can cache it. You don't need to invalidate the cache it becomes really scale about that moment and where you fall tolerant, so same as indeed, any modification of three modification is fully version. Essentially edge of is a system which manages virgins whatever you do with the system, whether this blog device object file or noise kill. Yes, we support noise scale as well.

A

Any modification to that, in instance, essentially, will be the urgent, and those of you urgent is what the managing and those version is. What we distributing across different locations- it is scale out system and protocol unified, object, storage solution. Ultimately, it is object, storage solution in this protocol when you fight in a sense that it supports different protocols such as NFS I skies II. It supports s3 variation of s3 for particularly for HIIT use cases and its most importantly, designed for gear transparency and hOAT use cases and we're going to talk about this more.

A

All right, so look at your face essentially deployed as a kubernetes, a curator. So when you see your staple set, you'll see: oh ok. This is your set all set representing the targets of the storage system. It's a full service lifecycle management. You can install update a rolling upgrade uninstall reinstall, so the whole thing is managed by essentially so-called custom research definition.

A

If you want to upgrade you just open your editor, you edit the version and goes to the next version. It knows how to upgrade correctly. It will take care of your inter dependencies, which are complex in the storage system, and we will do the right thing.

A

It is also tightly integrated this permutation graph onon. So you can monitor the behavior of the particular data segment. It is easy of use built in GUI. That's what we have. Sometimes it is kind of tedious job to go ahead and edit the CD to change something so we're sort of providing a GUI visa which allows us to modify C or D. So you don't need the actual editor to do that.

A

But the editor is good because that's your default style right. So you can edit this er D and essentially commit this to the repository and then have kind of control of what's been changed. The history of the change of your infrastructure.

A

It is runs in embedded environments as well as well as can be deployed in very large-scale environments as well in when I transcend embedded. It can run operators in just one gigabyte of memory and to two cpu cores.

A

What it does is really consumed locally connected row, disks or directories, and we actually working on adding a mechanism of collecting their system volumes as well like in a cloud and then present a virtualized SDS on top of it. If the exposes different data.

B

A

One of them most important one for today is s3, which represents the object access protocol. Additionally, we provide s3x, which is no scale abstraction and POSIX. Like extension, on top of s3, you can do very nice things with just a X, for instance, it outperform s3 for scenarios like time series, databases, BigTable databases by almost hundred 200 times, scale out high-performance on the face by heart by scale out and high-performance. This is what really it is it's going into.

A

Potentially hundreds of thousands I hopes you can reach million of a herbs if necessary, scale out the sky, see blog devices what it means you can have many Borg devices in the same segment scattered across, whereas persistent volumes or just applications, and when you write to one device, it's really going to be leveraging all the disks and servers and zones connected to it.

A

Engi first connect up data, so the important aspect of a GFS that it can connect multiple segments together in it's not just one and can be many segments- can be hundreds really and again. If you remember, I talked about git and how big it is architected very. Similarly, they do reconciliation of the versions and eventually we provide fully consistent view of whatever you want to synchronize cloud connectivity. What it means is that we can actually transparently present different object formats as internally just metadata within the age of s, global namespace, block level.

A

Gog duplication is critically important because it allows us to save quite a bit of the bandwidth when we transferring data between the links metadata only transfers. So we because we kind of know the fully consistent representation of all the changes in making. We can make a assumption that you don't need to transfer this or that particular data block and all we need to do is just transfer metadata. Only. This is a tremendous savings when you synchronizing multiple clouds, together, local caching.

A

Obviously, when you enable in just metadata there is no data yet so we basically have local caching mechanism, which allows us to locally cache them in on, pin and clone the data sets you know, intelligent prefetching is also part of it.

A

We kind of do not wait for the your to complete we just reading ahead, and that is all built in and transparent to the application, because all that we get into the geo transparency and it's real geo transparency, where you do not need to think about the data management complexities, do not need to think about natural, synchronization and stuff, like that, it is transparent to the application and it builds global namespace. Really.

A

The prodigal transparency is like, for instance, you can see the NFS files in s3 and you can see. S3 file are objects in NFS, transparently and multi-protocol. That means you can actually from one segment right here: have different protocols, whatever your application, a.

A

Bit more about the GFS cloud connectors and what we do so, if you want most importantly once you when you're talking about cloud you're talking about objects, storage right, so we're talking about SD, everywhere, transparently syncing regions, maybe later only synchronization- is critically important. These local caching, because otherwise you're gonna be paying doubles and maybe triples, depends on how many connected sites you have.

A

So we currently support AWS cloud, Google cloud, Microsoft, Azure and Alibaba also recently been added, and most importantly to mention is that it operates with unmodified native objects. So what it means, as the object in a double layers, can be seen as OSS object in Alibaba, and vice versa, and you can set up. Synchronization is just metadata, so you don't need to move the data actually.

A

So ng face is really a multi cloud layer with gear scalability, which spans network of geographically distributed sites connected as one global namespace. It's a gate like architecture, you stop fault, tolerance and immutable version, metadata design. It scales equally well for object, file, built-in, no scale or block devices. It is Gayle transparency. It is always on. It is big direction, access to the same as a bucket or NFS export. Only in a different regions, it's automatic last writer wins updates for s3 object, so what it means.

A

If you there are two users or applications updating the same as the object. The last writer will have a preference if versioning is not enabled for that particular bucket there's now a few groups is what we have as a concept for the NFS and I Scaasi synchronization and a gear. Consistency is what we have the snap view: groups which are literally floating this connected gear namespace.

A

So what does it mean floating by floating I meant to say that when I create a snapshot, let's say in one segment, that snapshot is transparently will be available in the other segment. You don't need to send it, you don't need to kind of transform it manually. It's just appears on a different segment, any green variety protection, so we protect files directories back its logical unions, no scale, databases, it's all sort of built in into the paradigm of get consistency, give locality an active every occasion.

A

So what it means is that these add your files, you don't need to worry about latency of access to your data. Your latency is always local. If you have local SSD, your local writes going to be the latency of what SSD provides. Your local reads will give you the letters in your local SSD provides. This is critically important for a 4h IT connected devices, because the latency is essentially killing the whole idea. An idea is great. Yes, you need to move your computing to the edge on. How would you do that without HFS?

A

So we need that your first occasions synchronized asynchronously, and that means geographically eventual. Obviously we cannot break the law of physics. Ultimately, there is a latency between different locations. Yes, there is a 5g on horizon. That's going to be another five years when the 5g gonna be really available for everybody, but even when is gonna be available, you still gonna have problems of sort of like okay link is disconnected what do so, how I can IRA connecting Canada, synchronize and so on?

A

So that's why we designed and developed and brought in the edge of it solve that issue.

A

Use cases to mention. So there are different use cases I. Think most important ones are here: the multi-cloud CDM workflow. So obviously it's a classic content delivery network. You avoid full replication. You can do pin on pink of lock cache content in second session of primary source from a SS, as you are able by and others cloud travel a bit by T, so cloud can fail, your site can fail, so you need automatic follower. You need some failover over the cloud links, so redundancy is critically important, I create in offline mode.

A

So yes, so link can be disconnected, but the locally. We can catch the data up to seven days and then we eventually will be synchronizing and after synchronization is complete.

A

Your data will be exactly consistent as it used to be before nget to and from cloud so capturing the edge data in a local cache and provide clouds for any IML processing is something which STX protocol was specifically designed and, most importantly, we also do not need to send data if it is deduplicated and if you send, we can essentially guarantee that what we send is necessary to the same access global namespace transparently, while avoiding the need for full replication is very important for this use case and because we kind of routed into their book and we started with kubernetes, we natively supporting persistent volumes across clouds and it's actually very directional.

A

So what it means by leveraging a few groups and snapshots you can provide, get on spot synchronization hirsute can use just metadata only and it's years of support, CSI for file and block TVs and obviously consistency groups. Data segmentation, region, awareness is also supported, means that, if you have let's say large, kubernetes cluster and you want most efficient access to the data, our logic will select the closest segment and execute your application on the closest segment.

A

Alright, so one of the topic which we want to kind of talk today about is networking between the room. Quick, currently leveraging. Just the port network, which, as you know in kubernetes, is the one big flood network which.

B

A

Great right very easy, so you have one G pen, G 25, G, 4G, hungry G, whatever this is great but is sharing same port network with the converged application, and this is a problem when we running applications on the same queue: branches crisis. Alongside this storage, you probably want some sort of isolation, so not anymore.

A

What we kind of come up with is a multi home network, and this concept is when we essentially create a different network for different user specific purposes and the network which essentially just responsible for kubernetes servers, API couplets and so on. Limas readiness, probes and I kind of lives. Here we have, you can't simply attach per application, which is specific user traffic to children. Networking ports you'll be orchestrated within the same port right here.

A

If you want to take a look on a definition, an.

B

Example, how it.

A

Looks like like here, you see: okay, this is the flannel SDS and you see there is a rule kg first name space for which this network was allocated.

A

We selected maltose Eni for very practical reasons, really, and it's just essentially just first scene. Our castration, which we selected we're gonna, be adding CNI Gini ammeter later, but out, while a multi CNI, because it's backed by in toe- and we know that as far as networking in toe is probably leading the space, so it makes sense to kind of select Malthus. You know it's a flexible selection of its the ends. You can do not just flannel, you can do calico. You can do other second OD PDK.

A

You can do SRA, VD BTK, all sort of things you can do with motifs. It enables namespace isolation. This is important feature because whatever you define in this namespace will be a network just for this namespace and other ports will now I actually be able to get the same IP address from this network. So it's totally isolated to the totally different networking devices.

A

Okay, what does it mean for edge your face for edge your face from the edge of first point of view? It means improve performance characteristics and they actually have done some work on. Measuring this. It means improve data security. That means, if your application and increasing in grass running on a different port switch network. There is no way they can see, what's happening on a back end and the back end is often shared between various tenants and users.

A

So it's critically important to isolate the application network from the back end network, and this is how it used to be so. It is just one port, Network and now we've kind of transitioning in this particular example to this from pet isolation and back and isolation, you can have more than two networks: it's configurable, it's high, it's flexible and so on does important mechanism. Here is what we gain is improved, QoS and SLA.

A

When there is no packets kind of flow floating from application side, we can better guarantee QA it's going to be better response time. Therefore, it's better SLA agreement when it's isolated here.

A

Okay, so the demo setup. This is very typical demo setup, though in production running larger setups than this here you see for node cluster. So it's one two three four hosts now after the deployment he will see target port in my example, was for terabyte each. So it's like the Ephesus d4 sort of metadata. They offload. We have HDD switch one terabyte each, so we got one port. Four terabytes now support four terabytes now support water about sincerely stateful said we talking about. So this is a stateful SAP.

A

Additionally, we have s3x deployment, it's a kubernetes deployment style, so you'll get Asterix pod, which we'll be serving this protocol and connected to the backend and front-end. We also have cause bench is essentially it's a workload generator which runs on a separate hosts just to simulate the behavior and kind of highlight the fact that we mount a whole network you'll get significantly better parameters.

A

Now, if you take a look on look at your first crisis, any typical ERG looks like that. So what do you see here? Okay, so you see the name of the cluster you'll see the namespace and from the previous slide, you remember that for that namespace we've done different network. You will see the image version, so we're selecting 1.20 you'll, see the service account for our back and other primitives you'll see the the localhost directory where a Java is going to be storing kind of configuration of files of each target in them.

A

You'll see some parameters which are coming from the rogue framework like okay, we want to just use all the nodes, all the devices we dynamically detect all this configuration and building this totally automatically and some configuration like being able to either have we enabled some optimization for performance, metadata, flowed and some other stuff and some resource limitations.

A

So when we execute this, CRD ruk operator will notice okay, so this is edge of s and their food will execute a particular operator implementation which build up all that for you, so you don't have to edit configuration it automatically detects s DS the match, with the edge d DS and build most optimal pairs for you. This is essentially all you need to write to get the cluster up and running. You can play with this if you want, after.

A

So here we go, we have some you've done some performance analysis. Obviously, as I've been expecting, the results were pretty much. I was expecting actually better numbers than this, but it's still quite significant number see about this. So what do we see here? So better response, so response time is in case of multi-homed is significantly better by almost 30 percent, and this is when you're just leveraging one pod Network. This is multi home network.

A

So you might wonder why there's so many milliseconds well, because we have 128 right threads and 256 threads and obviously be bound by that. I have only 16 disks in that system, but I've done this on purpose, because I actually want to highlight how the system gonna be essentially behaving this iterative application and the example is just 2 megabyte s3 object, transfers these three stinking replicas and 6 terabyte of random I/o data set. So it's it's a real test case. Real use case of how system can benefit from multihomed an I/o bandhas.

A

You see actually even better numbers you'll see 40%, better, better bandwidth with multi home network versus they just bought network. So we can conclude that this multi home, if we enable multi home network, we will get significantly better response time and better bandwidth. In addition to other benefits, which I've highlighted like better isolation, data security and other benefits.

A

To summarize this brief introduction to walk a geofence, so first of all root community is growing, we adding more providers. So there are a few providers from with hub which are coming out. There are few other independent providers be improving existing providers, we building kind of new practices how to contribute, and the top contributors I would like to like is about a year red, hot mix, antibody D and Susie. These are the companies cloud ability who essentially putting the efforts into making this better, and your first is emerging toward areas multi-cloud and AAT needs.

A

We edited affairs for the reason, because we believe this outage of s or something like that: the edge computing problems, the problems which computing will not be solved sufficiently and efficiently. So that's why we kind of wanted to focus community to pay attention for edge affairs and how this can be used for those sort of use cases. The multi home network is going to be available in upcoming one one release, but you also learn that multi home network can be seeking significant in your isolation for the data security purposes.

A

Also improving performance in it's actually possible to do within the kubernetes itself, without introducing any additional additional layers or custom modifications. So all kind of clean built in into the roof, and we will be soon available not just for its affairs but also for chef.

A

And thank you very much so I can accept a few questions. We have three minutes left. If you guys have questions, architecture go ahead,.

B

Basically, ers provide the s3 and s3 acts like API, and the data is starting. No local disk or blob storage in public cloud, I think. What's the difference between a GFS and came compared at GFS, and them me mean I, oh I, think so.

A

Mina yo, yes, great question. Thank you Mina. Your is just altered. Mina yo is simplistic object. It runs on top of directories of repurposing. Existing infrastructure. Hfs is more kind of enterprise object, so to speak. It runs at the high performance. It can run on top row disks, for instance, and it consumes raw disk without file system.

A

It provides HFS, also provides good compatibility versus three and has additional options such as sta X, which is more designed for a sort of BigTable and connect cloud, scalability and transparency.

A

Additionally, obviously GFS is a unified protocol, so if you have bloke and in the first wish, Mina yo doesn't yeah.

B

Another question is about metadata storage of HFS III I noticed that you have the IMDB configuration in.

A

Your country, but.

B

I say own DB is not scalable, so so.

A

We've done a lot of work on when DB, but as far as metadata metadata is globally scale out so each essentially it's there is no like centralized metadata server. This. This architecture, metadata, is exists on all the poles, so it's gradually distributed and because we're using the get architecture right. So we can address this very easily because it's always immutable. So essentially it's a sharding across multiple locations, multiple devices, but.

B

I'm not sure, if you you, you just mean the method is automatically shared across every node, but how.

A

B

The the, for example, one copy of the metadata is destroyed or how to do the H, a or so.

A

A GFS provides replicas counts and every encoding which essentially gives you have all ability. Okay,.

B

A

It is in addition to the segmented storage paradigm, where you can synchronize between the clouds. Thank.

B

A

Thank you great questions, any other questions.

B

A

A question is the edge FS open-sourced, so I couldn't I couldn't have found it in the kid hub or anywhere else yeah. So we we have great news. We open source in geo edge of s within next month or two, so it's coming out. So this is. This is very good news for linux foundation and community. We also will be working with LF h community to make it part of LF h at some point, so it's getting to the full open source, essentially, okay,.

A

Any other questions guys.

A

Thank you, so is that I guess we can conclude this session. Thank you very much for coming.