Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 4 May 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Building a Kubernetes Scheduler using Custom Metrics - Mateo Burillo, Sysdig

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Building a Kubernetes Scheduler using Custom Metrics - Mateo Burillo, Sysdig (Intermediate Skill Level)

The default Kubernetes scheduler does a fantastic job for typical workloads, but when you have specific requirements (like higher level application metrics) you might need other scheduling methods. You probably know how to use node affinity, inter-pod affinity, taints, and tolerations. But what if you could use any of the metrics already present in your monitoring infrastructure to dynamically configure the Kubernetes pod scheduler? In this session we will demonstrate how to create a custom Kubernetes scheduler using any monitoring source such as statsd, JMX or Prometheus. We will walk through the code required for a custom scheduler, and end with a short demo. We will provide attendees the source code and placeholder containers required to run the examples in their clusters during this practical session.

About Mateo
Integrations engineer and tech writer @sysdig Mateo greatly enjoys tinkering with container technologies, creating homebrew lab environments, and the perpetual newbie feeling of this fast moving world of tech. He spends most of the day getting in the shoes of the user at many different levels: user interface, task automation, proper documentation and error handling, code examples, and dissecting deep technical concepts for the newcomers. All his former job positions revolve around the confluence of humans and technology, having worked as Evangelist, Project manager and pre-sales engineer. Relevant experience: - Currently working for Sysdig playing with Kubernetes and writing about it: https://sysdig.com/?s=Mateo+Burillo - I’ve been featured in the the Kubeweekly newsletter a few times: https://kubeweekly.com/?s=mateo+burillo&submit=Search - I’ve spoken at a few Meetups (video of this one @ Docker Meetup Madrid: https://www.youtube.com/watch?v=eg7aiQn14ag)
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Good evening, everyone I'm Matt Emery here I work at CSD, so I'm. This talks about obviously kubernetes schedulers okay, so we are going to start with the index. For this talk first I'm going to talk a little bit about the default cornetist scheduler, which has a lot of features. Maybe you don't know about some of these alpha-beta features and yeah it. You can use the default kubernetes scheduler, don't go and Raymond the wheel. Okay use the default one. Then we are going to look at creating your own scheduler, hey I. Have some code?

A

I have some examples: I'm going to do a live demo after that using cloud cluster. So hopefully everything goes well and yeah I'm going to show you a little bit. You know the main loop are the the main functions that you will need to implement. If you want to create your own okay, this first implementation is going to be very naive, very basic, but the at the end of the presentation I have some improvements that you will need to implement to move towards a more production, ready, implementation.

A

Okay, some of them are actually implemented in this code. Some of them are proposed. Some of them will come from the public at the end of the talk okay, so we will end with the Q question and answers.

A

And yeah looking forward to hear your ideas so yeah, okay, this talk is not about 60, but allow me 30 seconds so cystic is the container intelligence platform. If you are using kubernetes, if you are using micro services, it's just one unified platform that will do monitoring, alerting ranting security, troubleshooting, forensics, okay, we have open source versions. We have the commercial versions. Try this at home. Take that's all I'm going to say about that's it!

A

Okay, you know that the kubernetes terminology and concepts are growing every week every month, it's crazy. So what this talk is also not about it's not about the coordinates additional policy, recentl Paulo to scalar or HPA. Actually, the HPA is a little more difficult to implement. I have another running implementation. If you go to that URL, okay, so I suppose everyone knows whole how to implement. I mean how to run an HPA and what an HPA does scale is the number of poles in a deployment.

A

It's not about the vertical scalars, which is a quite new feature that will actually size dynamically every individual path according to current stress and resource consumption. Okay, it's a different thing and it's not about the no, the scalars, no, the scalars are mostly club and on the dependent, so every every you know every Amazon, Engine or Google will will provide you with a nappy to scale the notes.

A

This talk is about the kubernetes scheduler. Okay. So, as you probably know, this component is in charge of watching the newly created pots and then deciding in which node this pot is going to live. Okay, it's going to be created, so the scheduler is well, it's not a controller, but is a control in the sense that it watches the API, looks at the decide of the state. The size of final States looks at the current state and move things towards the desired State. Okay, yeah I actually stole this from Joe swear, that's law!

A

This is what I really like this diagram, because it's very important to know cycling, at which point the schedule is going to interact with your with your API okay. So first you write the pod F initial into at CD.

A

Okay, then, the scheduler retrieves, the list of pots that are pending, but the scheduler is not going to act upon the pot directly. It's going to write a rescission back to @cd, okay and it's EDD I mean the API and the cubelet, the other components that are going to actually create and run the the pot. So it it's important to note that this happens after authorization by the AP. There are mission, controllers and persistence in EDD.

A

Okay, we will talk at the end why this is important, but happens before the kulit actually pulls the image and runs the image yeah. This is just going a little deeper.

A

You get the list of posts that are published. You apply, your scheduling, filter create a binding. You write back the binding in the observer, and then this body is updated with a new node name: okay, yeah. You need to watch for posts that are impending, have no no, no, no name assign and are explicitly requesting your scheduler okay. We will go into that later.

A

So yeah I want to talk a little bit about the default scheduler because it's a good starting point. Okay, so you, if you know what the default scheduler is doing, is you will get a lot of ideas for your own scheduler? So basically it has two phases: one is filtering and the other is ranking filtering is we can call it like this hard face and the soft requirements phase so filtering is based on the on the pod resource requests.

A

So if this node cannot accept this port is out of the list, filtering depends on the data volumes that you need locally or knows that I are explicitly discarded by selectors. Okay, once you have the heart requirements out, you will you will rank the node depending on the of course, the least requested priority. Basically, the the note that is: has more free resources: okay, the affinities, an aunty affinities, I'm going to talk about that a little bit later and the I like this, the last one, the calculator spread, a split priority.

A

I didn't know about this one. It tries to separate the pots that belong to the same service. Okay, so by default is going to do a nice balance and spreading over all your notes.

A

Those are the hard core strings that you can. You can use with the with the default. The scheduler, you probably you are probably very familiar with them. I mean you can have tines. You can have tines tolerance. You can have a note selector that explicitly matches a set of labels etc. Not going to talk much about this because it's common stuff, soft constraints are maybe a little more advanced, so you have, for example, the preferred no schedule effect, which is soft version of no schedule.

A

For example, in your master knows, you may prefer, if you don't put too much effort in the master notes, but if, if that is an emergency and the other nodes are not responding, you can use prefers, prefer no schedule. Okay, for example, know the affinity is the same thing, but for the note selector and the pod affinity, so Paula affinity have you used both affinity before and pull some the affinity and this kind of things? Okay, so you can. You can do a split it optimization.

A

For example, if you have two components that talk a lot: okay interchange a lot of bandwidth, so you want them together, because it's more efficient or the way the other way around. You have a pot that has a lot of privilege and a pod that has a lot of sensitive data, so you prefer, if those two are not together. Okay, because you know, I, you are gaining privilege into one of the containers will be able to access the other, and you also have. This is an alpha feature. You have up waits for the pots.

A

Okay, one thing: it's very cool to think about the relations between your poles and okay and start to imagine that you want these two poles together. But you know this is our relation graph and this is not linear. Okay, so don't don't become very happy with these kind of things, because then the scheduler will be more complex and will take more time. Okay, so keep keep things to the minimum if possible.

A

Yeah this priority is an alpha feature in 1.10 and you can establish a wait for the for the pots and it will have three different effects preemption, so a pod that wants to be scheduled will kick out less important pots. If, if there is no room for for everyone, okay, the scheduling queue, you will have different priorities, so the more the most important pots will get a schedule at first and also out of resource eviction, which is like the first point but the opposite situation.

A

All the pots are in the in the in the null, so they will start a bit in the last important ones: okay and yeah. Then you have this paint base eviction, which is also another alpha feature: okay, where you can express problems that you are having in the node as stains and tolerance.

A

So, for example, if you don't have network connections, you can you can dynamically tack this, this node with a taint- and you can have also these Toleration seconds, which means that the pot is going to be a victor, but it will wait, for example, five minutes. So if the network problem goes away in two minutes, it will stay in the node. This can be very useful if, for example, you have a lot of local state and it's heavy for you to destroy and recreate okay.

A

So you want to leave that note, but not not so fast. Okay, not not inmediately,.

A

Okay, so that's basically what you can do with the default: we're not a scheduler, let's go and create around kubernetes custom scheduler. Why the first? The first question is: why would you want to do that right? Okay, so, if you use the default, the scheduler, you are mostly limited to a set of hardware hardware constraints like CPU this pressure memory.

A

There are four or five things or you need to come up with a manual, an explicit set of labels, and you know affinities and the affinity selectors, note selectors, which is basically a manual and a static and a static thing. Okay, what if you want to schedule based on different metrics, like application metrics, for example, you want for HTTP, latency or HD HTTP bandwidth. You want to schedule paths closer to the consumer or GPU performance. This is I mean this is very hot.

A

Now you know for artificial intelligence, crypto coins or well or yeah, artificial intelligence, graphics or these kind of things. So you want to know exactly how many flops are available at this time. So this is a variable metric, it's like CPU, but for the for the graphical unit. Okay, you can do it with with a custom scheduler or you have. You are having a bottleneck.

A

So most most of the nodes I mean the service in general is behaving well, but one one of the nodes is having problems, needs more workers there, okay to to carry away all the work. Okay, so you can. You can send more workers to that particular node.

A

You have three examples, and so you can come up with more so from governmentís 1.6. You have multiple schedulers at the same time, so you don't have to go and replace the default one. You just cover additional schedulers in the same cluster. Okay, this is very important. You will see why this customer scheduler can run outside. You can directly contact the API and run it outside the cluster or run it inside uh suppose, I mean what I will recommend is you will run outside the cluster for testing developing when you don't want to?

A

You know recreate the pod and deploy and do all these kind of things, but eventually, when you have a proper version, is much better. If you run it inside, this outer container is a pod inside the cluster, it's not external. So what you have to contact I know, usually, once you create your own scheduler, if you know sky's the limit any API any parameter, you want to use for a scheduler. It's good yeah, a couple of considerations at this point.

A

You will need to create a service account for this pot, but you, you already have a cluster role, that is the Cuba scheduler okay, so this pot is going to contact the API and is going to modify and send objects and write objects to the to a coordinates API. So it needs this set of priorities way not important for this tour, but just in case you actually want to do it, and also there are two two types of bindings. This is the.

A

There is a bending object, which is was the default, but is deprecated in 1.7, and now you are supposed to use the bending super resource of the path.

A

Remember that the scheduler doesn't actually do anything, it just post, a decision which is basically a bending object. It doesn't take the action itself yeah, and this common line is basically showing that, with this air back role, you have both. You have access to the buildings and the both meanings: okay, okay, so let's go to the cold.

A

Suspense wheel, okay, so basically I have a vanilla, pretty simple nginx and the only thing I need to do is to claim that I need that the scheduler in particular for this set of pots. Okay, that's all you need to do on the poor definition on the implementation. You need to do a little more okay, so the implementation. Basically, what you do is you subscribe to this event? Queue you subscribe to the new IP pots, and you check that this pot is, is one of the one of the interesting to you.

A

Okay, so you spend in the scheduler name, is the one that you are looking for and it was just added okay and then you have the decision algorithm. Obviously, we have implemented this using Cystic matrix okay, but you can go and replace this method with any other.

A

Api that you want and that's basically the main loop, so let's go and try it out.

A

Okay, so this these pots are going to be pending forever like they are requesting, are a scheduler that doesn't exist, so they are never going to leave the pending estate.

A

Basically I'm running my scheduler I will I will post them. They did have a address at the end. This is my scheduler. Obviously, I need the coop config for the certificates and so on, I specify one of the metrics you can put plus or minus depending. If you want to order for I mean if more is better or less is better okay in terms of ranking, and then this is the Cystic token for for the api.

B

A

So it has detected that these nodes are pending conducted, the API and, according to this net HTTP request time schedule at all all the nodes. Okay, so hopefully.

A

Okay, simple simple: as that.

A

Okay, now, if we go back here.

B

A

That's what I told you too I mean first look at the default: the scheduler because creating you desk I mean this. This example was very simple, but creating your scheduler is a brave thing to do. Okay, I will show you some improvements that I have coded for this one, but there is a lot of things that can grow grow. Okay, first improvement. This one is implemented, I will go back to the terminal. Matrix Kiche is a very common case that you have a new deployment and suddenly you have 20 new pots.

A

Do you really need to worry the upbeat 20 times? Probably not okay, so the first obvious optimization that you can do is create a matrix catch, a every time that you retrieve a metric from the API. You put a time stamp and you decide if it's fresh or not. If you want to go even further, you can do adaptive may adapt ah to catch a so you know metrics that are changing more often you retrieve more often from the API lots of things you can do okay there.

A

This is very important. This is very important. The failsafe mechanism, you need a lot of different. You know you need to take into consideration every everything that can go wrong. For example, the matrix API is not responsive, it happens twice a month. Okay happens twice a month. Matrix API is just not responding.

A

Okay, let the gut check and mitigate this, but for a few seconds when told you have to give up, you are too strict with your restriction and there is no good candidate. Maybe you need to relax you your requisites time out. Okay, don't forget that we are writing a scheduler and the scheduler is an optimization mechanism. If it's taking too long it, it's not worth it anymore, okay, so it's better to timeout and go back to the default. One and multiple exception code has box. Okay, you need to catch any any exception.

A

That may happen in your call. You always need a a plan. B, that's the main idea. I mean if your scheduler fails. Pots are pending forever. That's not good I mean any decision is better than no decision at all.

A

So one of the most obvious strategies in this case is to delegate to the default scheduler.

A

There is a small problem here, and that goes back to the first diagram and the small problem is that you can. Actually, you cannot actually change the scheduler on a running path. It's not a permission. Thing is forbidden by the sign. Okay, so what you need to do is you need to actually go back to the parent. Let's say the deployment changed, the parameter in the deployment. The plumbing will observe that the decider state is not compliant, it will kill the post and it will recreate new ones.

A

A

Another thing that you can do is that you can Ricky, you can do the queuing. So if your main method returns a certain cold air or the API will understand that it needs to put the pots back into the pending queue. Okay, you can do that. I never tried myself, but you can do that so yeah in this case, just.

A

Okay, another first simple, stupid demo.

A

Okay, we have the pending pots and now I want to schedule, but my token is I made a typo. In my token, so the API is not going to respond so I'm alone, okay, okay, I, have the fallback that will delegate to the default. The scheduler in this case.

A

All this part of the call, basically what it does is it ask for the parent object. The parent object is a replica said: ask for the parent object of the Perito object, which is a deployment retrieves. The deployment modifies this parameter and then the deployment.

A

Will actually, if you realize the names of the pots, okay,.

A

Those are new ones before what I've said, I mean it's definitely before there is no way of changing changing a running container.

A

Okay and another improvement, number four number five I, don't know. If you know everything I was explaining at the beginning of the talk about all these kind of cool things that you can do with the default scheduler. They are not implemented in your scheduler by free, okay, you need to you need to actually go and code them, so another obvious improvement is adding more variables to the mix like respecting the the CPU limits, honoring, the affinities and the affinities, all these kind of things, which is probably quite hard to implement. All this.

A

All this stack of priorities and relations.

A

Yeah, another thing that you can do is writing a mutation and mission webhook. Okay, in case you, for example, you have 300 channels with 300 definitions of deployment samples and you don't want to go into every one of them and change the scheduler. You can always create an a mission controller. This also goes back to the first diagram, because there are missions. Controllers happen before the object is persisted in @cd. So at this point you can actually go and change the scheduler name at this point is it is possible later on.

A

It is not yeah and then, if, for some reason,.

A

You want to have I mean one of the posts that is pending can be selected by one more than one scheduler I. Don't know why? Maybe for high availability reasons, then you know you have to be careful with who gets the lock of debt who changes the state of the poll leader election, but you know that was a an entire subject at college, so I'm not going into into that at this moment. Okay, so yeah, that's basically what I wanted to show. If you have any more questions at this point, yeah.

A

Yes, I mean the default. One is I mean you have the source available, so yeah. He was asking that instead of writing your own scheduler, if you can go and write.

A

Yeah yeah like taking taking the default one and to use the affinity.

B

And so on, but also extend on these features with your own yeah.

A

Then you will assume you will get the code of the default, the Cuban a scheduler, and you will. You will commit on that. You will complete over that. That's actually that's a good I mean if you want to do thing, is that if you are doing a scheduler, you probably want a very specific task, very, very specific task. So you don't care that much how all the ante affinities held and these kind of things.

A

But if you want to do more complete more complete one, probably you you are better off, starting with the with the default. The scheduler can.

C

You actually like evict some in our pots, based on your scheduler or just shadow, a new one. No.

A

You cannot evict scheduler once you have the decision and you have taken the decision and rewritten the decision to the API scale. Work is done.

B

B

If you happy to scare, if you happen to have your custom scheduler as simple as your rotator and tries to put a pod in, you know that has no memory left or not enough resources. What happens then, let's kubernetes does anything about that or it doesn't care. You just come in here. Basically,.

A

Okay, so you put this okay, so you are not taking into consideration. There are the limits and you put they not I.

A

Think at that point I mean the cubelet will start misbehaving. I have not tried that, but it's like any other process that that goes over the harbor limits in your in your operative system, not.

B

Try to perform any sort of.

A

If you don't call them performance, that's that's. What I was saying that if you you know, if you can't use the default one, you say for one, because otherwise you will have to call a to call a lot of things. A lot of restriction.

B

D

When your scheduler is failing.

C

D

The deployment and change the scheduler.

C

D

Can't edit a stateful set so you're saying it can only use this for deployments. No.

A

No I was using deployments. What I say is that you cannot go and change the pod directly. You need to contact the entity that created the body. Yes,.

D

But stateful sets you can't edit them unless it's a replica and something else, so some I'm just asking if there is a way of doing it without editing the resource.

A

I, don't think so, I mean I know. If I suppose you need to go to a state full set a check and change the parameter. Okay, okay,.

D

E

Hey just a quick one if you use the mutating admission hook,.

C

E

You got to equip it to your custom scheduler, but it fails and you want to use the default scheduler. Aha, we didn't the hook. We hook it back to your custom, scheduler.

A

Okay, good question: you change it, so you used you change the mutation and then no no, because because you are directly writing the object, that is in HCV, you know- and that happens after if I'm correct, but you do authorization, you do the mutation we hook, it gets modified, it gets written to the HCV and you retrieve the object from its CD and write in to add CD again. So if I not grown it, it won't happen. It will still work this mechanism.

B

D

D

A

You can do presentation. Yes, yes, I will put my presentation actually read.

E

Your Twitter handle oh there.

A

It is right is my Twitter handle and yeah.

B

A

You know open-source pr's welcome all that kind of stuff, so this implementation is naive, but it's not so naive. I mean it has catchy, and it has. A fallback mechanism is starting to do some more interesting things, but still needs more improvement because I can consider it. You know something that you will put in production.