Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Intro: Kubernetes SIG - Da Ma, Huawei & Zefeng Wang, Huawei

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Intro: Kubernetes SIG - Da Ma, Huawei & Zefeng Wang, Huawei

Kubernetes started as a general purpose orchestration framework with a focus on serving jobs. But as it gains popularity, users want to run high performance workloads on Kubernetes, such as Spark, TensorFlow, etc. When running these workloads in Kubernetes, several advanced capability are required, e.g. fair-share sharing, queue, job management (suspend/resume), data management. This Intro will present the work in community to bring "batch" capability.

https://sched.co/MPi7

A

B

A

Everyone I would like to get word. This is a cattleman dinner trick, talk about what we have done in the kubernetes, and this topic is about what we have done about the back workload in the kubernetes, and this is the Klaus and this co-leader of six cadelec and the other one.

B

Hello, hi I'm Kevin, one I'm, a net contributor and currently also working on the volcano project, we'll be introducing this session. Okay,.

A

Okay, before we talk about the batch workload, I think away near to I'd like to say why we want to have such kind of a topic, and you know when the crew brain has so we already talked about several things about the sed. That was the cruiserweights sourcing, the desires we try to run, release tomcat and mexico in the kubernetes. But all of that workload is a kind of service model. It's a can of or long-running long-running work level.

A

Right and recently, we will hear that most of the people try to run some computing workload, high performance or cloud in the kubernetes, such as a test flow. We have spark- and maybe some blink yeah, and maybe some some others not ephemeral here, but when they tried to run such kind of workload include crew Bleus they. Finally, we found several gaps here now. The first one is oh, nothing, for example to the queen. The queen one is a one drop and the blue is another drop and the each box is a mean uphold.

A

So, for example, in the in the computing workloads such as the MPI- and it is very training of AI workload- they need to get in get all results together to run at all. Otherwise they don't need. The job should be pen here, because the each volunteer to communicate or exchange data bridges each other. Otherwise they cannot finish the talk, so you know so there's two worst case. Another.

A

This one is the some people. Can't you get the two job. We only get a part of results they are going to raise to another. The other part work to get enough result. So to start so this can of deadlock. So that's me the blue table when they get only part of it, blue top star and a training to with another tuple to start and as the result is occupied by the green jobs, so the they are waiting between each with each other forever. This is a deadlock, and the two job cannot be finished forever.

A

They were gonna be pending here, except you. You kill them all disabled to something many to handle that another one is that if the blue job get enough result to start but unity, Falls gather with continued to these two schedule, the other two poll here so the green product. We will start here, but the green job were not running because the other poll on all star they need to exchange the database. Rather, the the wrong input are waiting for the others. They consume the results and possibly didn't do anything.

A

Yes, they called as a busy with you, keep just a hound. Some loop consumer results and waiting for the others. I think the this one is what we want. We want the job start, one by one. We don't waste the results and the trouble can't be make sure start one by one yeah. So the solution is rights. We are going to do the scheduling, scheduling the poor group of the pot. We were to scheduling in by the group of power so either of them care enough results, or we don't put.

A

We don't care any resources on yeah, and this is our test. Our title result of based on this is on this case now. The first of all the first case is the travel with two PS worker and how for work, sorry, two PS and for workers they say the or job with the six port to start together- and this is the first one case, so everyone I will job can start. I can finish in 24 minutes. The blue. You can see that is the blue one is we have a project?

A

We have some solution for that another one is a cool flow based on the default scheduler. This is the the second. The second case is the way how to charge here now the the two job. Aha, there are gonna, be two peers and for workers for each job. This is the this is that has a result of peace, a wage in case, because you know some. Some polar wrong in here is awaiting the results and consume the results and the first.

A

The third case is we submit five job here and because you can see that I used a different color here, because only two or five are finished and the other one didn't finish. We can make sure that we do not finish because when which have in the in the Knights code, a more than 220 hours. That's the opening here, because the each of the top should be finished in about half an hour, but it's did the other story.

A

Job didn't finish in 20 hours, so you know the result is here, but the job cannot be finish. There's going to be a really bad cast for for the users assume. This is the all or nothing case we Israeli. We resolve the problem. The job finished one by one, but there is a issues as we still have. Some results are idle here right way, at least how some results for the other two poll yesterday.

A

If we have a smaller code, a smaller job, what we can do- and we can a better solution- is issued backfield the two smaller jobs added to the idle result, to make to improve the resource utilization here yeah, but.

A

In the way, this feature is ongoing. We have requests about the overall implementation here, but when we we also provide some some pals profile, some interface to the euro, to the user, to decide which small bank photoshoot with small jobs will be backfill details. As you know, if the small job take a really long time to finish, you should not bail through this one, because the you will make sure the other, the blue one. If the blue one finish, they need time or you can really short time you will.

A

It will make sure some more idle result here. Yeah.

A

Another case is that if we try to backfill- and this can be worse for the green job, because we have smaller normal job one by one here and we try to backfill one by one and the green job didn't get a chance to start right. This can't be your channel for starvation. The the green job cannot finish in time, yeah, so for us, so we tried to avoid such kind of case now.

A

The first one is waiting when, in the our in the simple to simple implementation in size up to one backfill, we will not go through another smaller job for them. How I'm sure we, both you or the the first up law or bron-bron talk? We were not thankful and others a small one, this one we will give it idle here, just tried to make try to improve improve the results. In addition by once that, but we still provide the bank call by here. So you can.

A

You can decide whether we should do another round, Iran or backfill, or something like that. This sir, this feature is going on going. We have some also have some discussion here. Yeah, yeah and I think that this is the for the gap. One and another is we have gap to to capture. Is that some people? You know in the current default scatter or in the current implementation?

A

If we, how to namespace wake on offshore, which make namespace get how many results right, if we one names based on meet 100 polled and the other names based only only submit one or two poll, this the first names based get more chance to get all results right. It's gonna, be you know, the scheduler cannot make sure that, but this is unfair to the post namespace, because they they have enough. The how similar priority have similar weight, or something like that.

A

So the target to target is that we would like to share resources between the two names face of mornings based in proportion. Right, you can share, share, have a result of Makassar and you should get another half of class, sir. Yes, together, we have to do that in depends of how many results.

A

How many pulled your Samir to the caster yeah, so which there is a proposal that we were going to support fair share by the namespace, let's cross, the namespace for them all in the NIP in the the name of a system in the Pluto.

A

Even he have lots of to car result requirement is there, but we only give here, for example, three CPU here yeah and we also give a fossil to the other other to the other namespace anyway, for the photo, if all the green polar finished, the blue one can get all the result of castor, because there are no other people to share the sausage here. Yes, the first year will you know the fair share policy will support the Dominator resource fairness to share results between the different images. The funnel will always support.

A

You know, cross names way cross names list, because we have. We try to do some simple on the talents, but for now the tenants we can group with identifying what is the tenants so for now we will support by the namespace. For this part, and- and this is the second, the second gap- and we also found another gap- is that different in the upstream, the polled for them of the drawable deployment team aside something all this kind of workload only is important.

A

One pull the template, but when you try to run batch workload, they also require multiple template for de novo. For the disability training we are, we need to have parameter server pod and we need to have worker pole. The pole. The template may be different. The total image may be different. The the resource requirement may be different for worker. They try to use the GPU, but for permanent server. They try to use the CPU.

A

If this is enough, another case is, as for the, if we try to run the MK I walk alone when you to hire unique because I'm here to talk with each other but the better ID for them, one two, three: four: they need to know how many workers are there, so they can talk with each other. This work an awful hour reduce, which is used by the better AI workload. Another requirement is dice for distributed training of MPI workload.

A

If one polar would fill the whole job should be restart because you know the cognition, you know the pollen you to communicate each other and the. If one for the world restart the the data would allow the community history will last so the how to restart the hotel if we, if they didn't, did any checkpoint in the cluster. So all the all of this features are required to run bash workers in the in the criminal haze yeah and there.

A

You know there are lots of other requirement or there are other gaps there without job amendment I think there also have something to estimate. We also need to support queue. This is a similar to the batch system. Yes, we also need to support promotion. Trouble. I will prompt ssin for this one yeah and we also support Hardware top logic, such as network and accelerator, something like we are going to do that yeah this song, this one. We are going to how our but I will have a portal that we include to all cheer.

A

So all this cannibal gavel for this power, yeah.

B

Okay, so this is the what we recently working on.

B

We have a create a new project to leverage them, so in the above, you can see that we called domain framework, so it's like the first one is: is the kind of example of long-running workload right. It's it's a wordpress and also like cube flow, it's kind of running workload and framework and also spark and the last one is actually a cube gene.

B

It's for genome sequencing, a framework, so basically this kind of free framework they also they have actually similar requirement that they all ought to like deployment installation for the order frameworks and also they have their own concepts or CRTs to define the job or queue, and they they also have some requirement for that to the underlying layer, especially for electron scheduling, but that's as close just introduced that that's not enough so so to fill the gap between the frameworks and vanilla kubernetes.

B

We use volcano to to offer the advanced functionality. So we have a common job at abstraction to serve all these frameworks, so they can manage map or the their definition to the volcano CRD. So then, the user could be much easier to define how your framework can be deployed and also we are.

B

We also improved introduced, more advanced functionality, scheduling functionalities like fair share and account scheduling, and also the the germ management here, is to reflect the common job, job CRD and also for the sales band we we will later on, try to to have a an R and a better integration on GPU and FPGA, and also in the end, the user can still just use cubic ro to do all the operations. So it's still kind of you know analyst native experience with with this project.

B

This looking into more a little bit details. So so currently we have like the green box are the compote components of volcano, so we have a scheduler. So currently it's a cube batch, but we also have more advanced features, will be added to this volcano scheduler and for the controllers is currently, we have two one is the the the common job destruction and another one is the queue because for especially the the high-performance were batch workloads there you may have tons of pod created.

B

You know in the cluster there's only limited resources you want to them just to have the poor the first there and when some of the port's complete you can spin up the the other processing in the queue okay and the for the for the for the volcano control, the it's a command line to to manage the volcano components lifecycle. So the policy in the volcano scheduler will still be pluggable, so you can either depends on or which kind of workload you are using.

B

You can enable DRF or priority between jobs or schedule, may be more common among all the frameworks and also working volcano controllers, as I said. Currently, we basically have the job, controller and queue controller and the volcanoes.

B

So in a nutshell, we want to make volcano the best solution to handle order, high-performance workload, so a little bit about the road map. Currently, we have just released our first release, so basically it's we offered the job management and in the following in the following releases.

B

We want to deliver queue, management and hierarchical queue and also the priority priority, pray and the preemption and a reclaim between the jobs and also we want to do some entry and enhancement and the performance optimization, including the run time, and also we want to take the the Saltman resource like celebrator and a network resourcing and the topology into consideration. So we are still a very young project. We definitely welcome everyone to contribute or share your your idea.

B

So here are some of the useful resources we have the website, and currently we also have already the first version of our code there, and also we have. We can be found on the Twitter with volcano Sh and also we have the Select channel and Marylyn list, so so definitely welcome everyone to to share your idea or your requirements or contribution yeah.

A

Okay, okay, so any question: oh.

A

A

Hugh, do you mean the state's current editor of the kill right? You know, wait right, reintroduce the queue because of Q is important for batch system. Yes, there we didn't. We try to introduce a new CRT in kubernetes, because namespace or something didn't include enough. Information such as the party priming.

A

Scaling policy, something like that for now we already how will kill each? How wait to we also do the proportion to share this also between the different, the killing cool batch. But you know this is not enough. Yes, you know, I, believe that there several feature you know, kill now the first one is we try to support a higher accuse queue? Yes, so the queue of the structure of queue can map to your department. Yeah. You can share this house and there are also some other requirement. Is that different, a queue?

A

The job we in different kill me record a different policy to do the scheduling right. Furthermore, some queue is try to schedule drop by the party. Another queue maybe try to share results by the at the TRF or maybe some other policy. We try to also suppose that can do for feature yeah another one is you know we have a. We have a first year feature right to cross namespace and we also need to support sure sure for the namespace cross the queue for the move.

A

If one names based on one user tried to submit a lots of job in different queue, we need to share results to make sure this to namespaces share is also cross secure. This is also in the progress we are trying to that anyway, and another policy is that you know for folder state for the photo queue we only support weight doesn't mean we doing this. We shown results by proportion right, but in some case user would like to support priority.

A

For example, high priority mean all the job in a queue how our hair party to have more results. Here we are going to support more and more feature with sonic you, but anyway, right now we have really basic feature of queue in volcano. We are going to continue to enhance this power yeah.

A

Any other occasion.

A

A

Compared to the new code base.

A

Crew patch is a new code base from scratch. Yes, so there's a total different with the defaults gather details the unity photo scanner, didn't do the scheduling pulled by pod yesterday it lost us a concept about the job. Some you know for, for the moods were wrong or to run or NPI job. We need to run them start to gather such a drastic as we need to have certain information in the scheduler. Yes, so before before the scheduling framework crew patch is already there. So we started from scratch.

A

So Indian ask in the nest AB we can fine them together, based on the scheduler framework, but you know the we already already have something there. So there's really long journey for us to combine the two scheduler. Yes, but I singular saving our plan. For this part.

A

Any other question: yeah! No! Okay! Thank you. Thank you. Okay,.