Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Nezha: A Kubernetes Native Big Data Accelerator For Machine Learning - Huamin Chen & Yuan Zhou

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Nezha: A Kubernetes Native Big Data Accelerator For Machine Learning - Huamin Chen, Red Hat & Yuan Zhou, Intel

Large training datasets used by machine learning frameworks, such as Kubeflow, are usually stored in low cost and high capacity S3 or Google Cloud Storage. However, S3’s rating limiting and slow data downloading significantly challenges training performance and limits compute scalability. We introduce NeZha and explain how it improves Kubeflow’s training. Nezha is an open source, community driven, and highly collaborative project, contributed by storage and big data engineers. Nezha is based on Kubernetes Initializer: it rewrites Pod spec, adds a sidecar S3 cache, and redirects Pod to use local cache to accelerate. Nezha is self contained and easy to use. It does not require modification to existing applications or user visible Pod changes. Nezha improves big data application performance. Our initial Kubeflow benchmark results using MNIST dataset shows NeZha achieves ~2x speedup.

To learn more: https://sched.co/GrWB

A

Advancements in the machine learning and as well as how we can have you, use the Cooper native native building blocks to do high-performance machine on you, accelerations, so the product of you, which I haven't, got a chance to miss. My name is Carmen Chan I work for were hat and this you, and so he works for Intel. So we, you know, as you see, we come from different background and difference are the companies. But the goal is the same.

A

We can use technologies to help people, especially those who are using kubernetes, to do machine learning and so the generalities target, so we're going to go through some of the challenging issues in machine learning and if you are using cube flow or anything else, there's a shared concerns that how some of the bottlenecks exist today that we have no resolutions. Yes, how competitive you in these areas and then we are going through different examples. How this framework can be used in both deep learning, as well as data analytics.

A

So for those who have used the machine learning and has been deeply involved, so the acceleration in machine learning has been growing rapidly. So you know hardware plans. We have the hardware acceleration C in the CPU and CPU slides, as well as a GPU and I've PDA in the software stack, and you can tune your algorithms to such extent that you, for example, you can change your arm. Pre-Processed or the other says you can have speed ups like 20% or 30%. If you can cast your, for example, the image cropped in a right way.

A

You can get reasonable accelerations and you can also use some of the latest developments in the in memory stacks like, for example, apache spark or medial rapids. These are all great form. Works can give you accelerations, but they haven't solved one more issues. Yet, let's take a big take a step back looking at what has happened in the past ten years. As we see the the cpu- and you know in terms of computes, are the computers growing exponentially over the past ten years and I have inhabit data for 2017?

A

So, as you see the CPU, your almost doubles performance and when you talk about GPU that the progress the grocery is even higher, so the endpoint us we are confused, is no longer or just become in no longer the bottleneck for while you do it or machine learning, and if you are come to 2018 and beyond picture in the chip industry is getting. You know more interesting because they're in competition driven technology, advancements have never seen an end. You have seen companies from like AMD the epic CP used as well.

A

Have generational improvements like four times and the processing I was going to some nanometers or ten nanometers depends on which company are talking about and the microarchitecture improvements we are in a bumpy analysis. All these are good things in CPU and you are seeing the GPU side, Nvidia releases, new architectures and the also coming up with a mile 60. So all these good things have it bump up the performance and driving down the cost.

A

Yet if you are looking taking land and look at the machine learning, if especially, why your random workload, the picture, becomes kind of difference, I'm, just taking a small sample of the public tail cells and how much time you need to download SS and how much time you just weigh in the model.

A

So if you are looking at the magnitudes of each other in assess they're, pretty much on par with each other, so which means you spend a big chunk of your time, just paranoid analysis and Mastriani in your GPUs, expensive, GPUs or CPUs, and which leave them underutilized.

A

Even though this just a public dataset is the story is the same, especially when you are hosting your data, for example, on s3, and you run your computer somewhere else on premise on different data, centers or just know. If you have a hydrogen Estella says, run your own host along different cows. That's it's going to be a problem for you, so the goal of this project.

A

As you see it's going to address the last mile issue, so we are trying to accelerates how Denis has can be done out it fed into the your machine, learning pipeline and accelerates the overall performance, so the solutions can be anywhere in my opinion, in this in these spaces, either you build a customized file system which takes years to succeed and by the time you have a file system. Maybe the constraints you had before no longer exists or you can just catch your own. You know the data in the computer knows everyone in your laptop.

A

You Runyan habit. If you use in your data center, but a trouble is once we come to the kubernetes and, as you have perfect sublimity policies, the base assessment must reside on same computer. That's your parties running, so another is actually another is actually the third proach somewhere in between. So we are not attempts to build up proprietary or different file system was trying to use whatever we have existing in our quebradas.

A

The building blocks with no basic primitives in kubernetes and to address the latency issues and scaling issues for those who have not seen that before another is a Chinese authority, figure equivalents to peel of Han in Western literature. The node is fast, you just like the pit of hell and another never grow old. The idea is, once you have your data in the cache. You now spend more time down order data and with us I hand over to the end.

B

Okay, so I'm going to introduce more details for this project. First is our project goals, it's quite simple and straightforward. First, we want to improve the data download speed by bypassing the network latency and also we try to put more data close or a tool to the compute node so that we can have a faster IO speed. Second, one is: we want to build a kubernetes negative project, which means it's very simple: there's no other dependencies. The the core ideas we used is host, alias and also web cook.

B

Our introduces in later slides the server is we want to make this transparent to end-users. This means your existing applications to have to change anything. It can run transparently and can benefit from this project country. We we still have some issue when handling HTTP traffic, but for other protocols is already working.

B

Okay, so here's our general diagram for deep learning training. Basically, you just download the source file and then you do some ETL and then do some retraining get some checkpoints and then you just try to adjust as a way or bias some parameters to kill the training and then, after several iterations, you get a well trained a model that can be applied to your real application.

B

So, as I mean imagine, people are trying to fix the compute band latency, but there's also actual there's also actual latency in the up layer, the network and I all layer. So this project is going to try to fix that latency.

B

Okay, so here's our general architect for for our project. We actually only added the middle layer in the middle here. It's very simple: there's a caching port which can do caching to a local device.

B

You may old, you may need to expose a faster disk may be like SSD or even decent, nvme SSDs to those caching ports and also there's a injector which can do host ls4 for your application. Prods arrange this detail steps for a detail workflow for this project in later slides.

B

So, with this simple architecture, all of those data will be fetched from a local port. That's that that means you can have a very high performance and also availability will be much higher.

B

Okay, here are some deployment models we have designed the first one is a centralized model and it means you have to start a caching part first in which part in which you may need to expose a SSD device as the caching device, and also you need to create configure map which can do can map those data source costs to to the service IP of the caching caching part, and then by by specifying some labels for the application part. Then all of those reads from those applicants were with direct to our caching port.

B

Thus it will be much faster than fetching from a public network. Here's a centralized immortal and we also have another deployed more local sadqa', which means your application port. Are your cache in Cordoba, along with your application port, so your application can fetch the data from a local Container Network, which is even even more faster than your internal network.

B

Here we listed the comparisons for different deployments. If there's no caching, then your performance were being limited to add to your external Network performance. Also, your availability were also being limited by the network availability. If you, if you use a centralized deployment, then the performance will be limited by your internal network, usually maybe a 10 gig on network. But if you deploy with a sidecar model, then it will be. Data will be fetched from a local local container network. Here is a loop device. Then is it's going to be faster.

B

Okay, here's some very details for the deployment.

B

The first one is actually there's our instruction on where we get hub loophole and I just tried to brief the steps here. The first is first warnings. We need to know labels the application where yours, for example, usually for a coop for a job. It will be labeled as Kissel net and we just use realsies label as the signature.

B

Then your need to create configure map. There are three three important components here. The first one is first, one is a label it just we use a kiss on edge here as an example. The second one is the caching port IP, which, which is actually can you can fetch this IP from sheer script. It's quite easy, and last part, is the upstream servers for follows. Source data here are some examples for MINIX and also say RFA and also cold data set.

B

Usually it's a tunnel speed will be like maybe several mirga's a second, but if you, okay, so I'm doing introduce the the real application job. Here's an example job which we just did a simple dublicate. You just try to download the data source from an external website and with with our caching, so all the first tunneled you can see. The performance is like maybe several mega per second, but if you're for the second and following downloads, the performance will be much much faster.

B

Okay- this is a very very simple example here.

B

Okay for next I'm, going to introduce a more complicate or interesting use case, so we have seen bhagadatta are also analytics. Workloads are also moving to the cloud. The key technology here is the use of HDFS.

B

It's called Hadoop compatible file system which can translate those hadoop api to an object, store API like s3 or Google Cloud object. So, instead of using traditional HDFS setup, people are trying to move their big data were close to the cloud to use s3 as their data store or Google Cloud that as their their store. The benefit of this of such setup is quite obvious. You can actually scale in your compute cluster and your storage faster independently.

B

If you, if you need more computer source, then you can add your computer notes and if you for your storage nodes, it can also be acting as a common data. Lake. The other departments may also put data into this storage cluster and also since spark 2.3. It is able to run on kubernetes natively. So we also tried to see our project if our project can benefit Sparkle kubernetes.

B

So here is several important object store connectors for Hadoop world and among these most popular one is Rea because people there are many private implementation for us. 3 like safe or Swift or menial people can set up their own asteroid storage and do some work with that part. So astray is actually picked here and usually, if you're, if you are using a public three as a storage, then the data reading is very, very small. Your public networking is the bottleneck.

B

Even if you use a private s3, then you still have to fetch the data from the remote cluster and then do some computing work in your computer, no computer cluster. So there's always a latency between the computer cluster and the storage cluster.

B

So here here's some example for SPARC / kubernetes with x-ray. Caching, as you can see, we we actually use. We have a computer cluster in a player. There are several compute nodes and some compare notes are on I deployed with caching and for for the first I'm sorry for for the storage layer we use, you may use a public s3 or you know private, as Ray as the storage cluster, but for both solutions. There's a storage networking, isolation between with your compute compute nodes.

B

So there's a latency when you fetch fetch the data at the first time. But if you have a caching on your cash import running on your kubernetes cluster, then for the following, weeds will be fetched from from the local caching. Port I actually have done some basic tests with modified PVCs.

B

It's a common, it's a quite popular quarry workload and we found four for those IO intensive quarries like the chorus will read many many data and the performance improvement can be like 30 percent or even 40, 40 percent improvement. So that's a very big improvement.

B

Okay, as I mentioned before, so you may usually want to expose an SSD device, SATA SSD device or even decent via me, device for the caching, so that you can have a very good local, just local reading speed. But can we can we do better? Yes, we can so here's a some.

B

New device from Inter, it's called optim data center, persistent memory, it's exactly the same form form factor of DDR memory, but it is build with native resistance.

B

Also, this speed speed frawls for this device. It's going to be faster than SSD device a bit lower than the RAM device were more interesting. Seeing here is you can see the size for this pedestal memory it's going to be like it can have 512 Giga atom. That means, if your sink think about your servers today, most of the set up may be like sorry to gigas padding, that's a very luxury expect a luxury set up right now or with this team you can build a won't error.

B

One terabyte memory cluster will single, always two tips. That's a very, very attractive point.

B

So here's some performance comparison with normal SSD device. So in that in the right side, is a vm e SSD. As you can see from the from the picture for frock a week in p.m. est, it's going to take like nineteen microsecond.

B

But for sorry, so in the middle there's a bar for Optimus, T, optimizer sixties, they use the same meteor but different from factor. Optimizes thing is using a PCIe factor: it's not a game Tim device. So, as you can see from the picture in Optimus St the little that the performance is going to improve a lot, the latency is like 15, 15 microsecond, but you're still suffer from the driver file system latency and in the right in the right bar. That's a performance for for the new game device.

B

Without those file systems with with all those to our agency, then the performance can be improved. Like 10 nearly 10 microseconds.

B

That's that's very, very, very fast, actually.

B

Okay, so just a heads up, so in Chow, kubernetes teams are developing CSI plug-ins for this team and they, the the I, think they have a general working code right now. The idea is to expose a notable call game to Conan is as a probe device. This is a starting point in the future. They will provide more features like different from fact different memories, so they have a real poor public repo here and if you want to know more details, just go to inter blues, there are some technical guides there.

B

B

So just a summary, so no dry is kubernetes negative, big data, accelerator project that can improve your input, data performance we use hostile eyes and also webhook, which is transparent for applications. So you don't have to change any code in your application. This can be applied to machine learning, deep learning, workloads.

B

Sorry, as well as big data, analytic based on object, storage in our basic tests. We found the performance. Improvement can be like 30 percent or 40 percent of depends on the input data size. As for next steps, we're going to check on the HTTP support, as well as the sidecar container solution, so.

A

A

The question is: can this could be applied to continue? I think that the because the container image is oftentimes, their hostility in the s3 or object store? Yes, so the idea is yes, so we can definitely registers. So what you nearly do, because this is generic, a configuration. You know config map we have for the injection. You just need two points to the endpoints of your registry, but the downside is, we haven't figured out and we haven't completely solved. Hdtv I see huge yes, so because that whole yeah, so the mic remember.

A

The idea is to inject the certificate. You know this cruel world, your self signed certificate, your self signed certificate, and then we inject those certificates as the secret volume. This could work, but you know that's just my technology point of view. There's a compliance audit in yes, yes, so that's going to be a different thing. So can it be helping? Yes, this can help to what extent we need to validate.

A

A

Do you have a secret connector like just for our data, analytics data warehouse? So if you are doing adjustable downloading and I think this we are only doing the the truth is as long as there's a way you can catch the data. Yes, we can definitely do it, whether that's educated protocol or some other protocol us to investigate. But the truth is the host areas. Injection can work for any hosts. As long as the host name is already in a part, then the networking routes can be changed, so I mean the next step.

A

You know more or less we'll have to take Lucas, as how am voice is doing this proxy right. So we are doing this independence of van voice, I'm a boy and in steel, because we don't think that is going to be used in a crib flow. But there are some sort of mythologies in angular and steel we can borrow and just to use this project.

A

A

All right, so that's two questions about circuit, basically, cycle I. Think it's sidecar! It's going to be a promising one, because even you have the centralized the cash or you can. Even you can replicate the cash there's still chance to all failures. It was your deduction fails or the partners response with responding and then you're in trouble, and you are also subject to your VPC network performance side highest interesting because it's always available if the is not so available and that no, that means that application part is not running as well.

A

So you can safely discard, as note so I, believe, that's going to be an automatical for projects and also some other things we can explore. As you know, the kubernetes 1.13 suppose the snatchers volume right. So, if you have your volumes, does have the raw data cached. You can snatchers Acronis volume and just rips them on each of the cycle volume. So the idea is that you can have the same source of data. You just need to take a quick clone with snapshots, and then you can use that volume on each of the sidecar.

A

So you said you have the data available on each of the sidecar and you don't need to download or data again. So that's going to be a even higher scaleable solution.

A

Well, you can still do the caching, but chances are the data you can watch. You do snapshots that is already there, so you don't need to download it again, but as it's going to be a very interesting expression, yes,.

A

A

All right, so it was if the cash is already changing. I think that's a I have an art Hawk coming for kinetic. Basically, we detect something changes. If something changes I mean that's it's wrong home story, but the idea is that, if anything changes we can invalidate that cash and the problem can just in various testing objects we don't slow in Ho cash. So you see there's a possible solutions. We had need to take some some.

A

You expand relations, I think if you are doing the, if the proxy is working the right way, it should be able to detect upstream change as well. But if not, we can go to the service way. We can detect objects, change and then various percussion.

A

Oh I'm, sorry I can probably sir.

A

A

So maybe you are not funny into same issues, so, in my opinion, so the idea of a using a cache instead of town Orion directed from topic will be poll. Data status. Repo is a the time is always consistent. You don't have to worry about when part. Where starts I mean if you, if you are talking about the suede empower the that is whether there is available, you can guarantee that the trade embargo starts immediately, so there should be no latency are in between.

A

So, if that's the difference issue, maybe we can chat offline, all right restart. We are happy call for lunch and they enjoy your meal.