Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Scaling K8s Nodes Without Breaking the Bank or Your Sanity - Brandon Wagner & Nick Tran, Amazon

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Scaling K8s Nodes Without Breaking the Bank or Your Sanity - Brandon Wagner & Nick Tran, Amazon

Kubernetes (k8s) has enabled applications to be mostly agnostic to the underlying VM infrastructure it is running on. Many clusters can benefit from the cost savings of utilizing spare VM capacity offerings commonly called Spot. In this session, we will discuss some of the best practices for utilizing spot capacity within a k8s cluster and some of the tools that will make your life easier managing the underlying VM infrastructure.

A

So this is uh scaling kubernetes nodes without breaking the bankery sanity. I'm nick tran- and this.

B

Is brandon over here.

A

um So we'll talk a little bit about what spot is exactly and some of the best practices you can use for spot and then we'll go into kubernetes and spot how they work well together and then how to auto scale your nodes with these spot nodes and then we'll, hopefully go into a demo after that.

A

So what is ec2 spot um so ec2 spot is spare virtual machine capacity. This is available at a steep discount from on-demand instances, and these instances are the same underlying instance. The only difference is that these are interruptable, and so what that means is that ec2 can claim these instances when they need to for an on-demand customer and they'll. Give you a two-minute notice from which you can reclaim this instance.

A

Sometimes ec2 can also give you a rebalance recommendation to give you some more time, and now we'll talk a little bit about that later. So some common workloads you can use with spot are quick, continuous integration.

A

This could be something like tekton or you could also do batch processing status, apis or dev workloads, basically, something that's tolerant to the interruptions and you can clean up quickly and so some of these spot best practices, we'll talk about, are spot max price and not how not to set it flexible instance, type requests and rebalance recommendations, and these are all going to maximize your instance run time uh minimize the cost you have and also maximize the chances that you can get an instance when you ask ec2 for one so prior to ec2, sorry, prior to 2017, the spot pricing model was a bidding system, and so what this meant was when you set your max price.

A

um This is what you would pay for the instance and it would. It would maximize the instance run time based on how high you set your max price, and so sometimes you would see with this bidding system is that it would go over the on-demand price, and so after 2017 ec2 decided to change this and make it based off long-term supply and demand, and max price is where uh what you're willing to pay.

A

And so, if you don't set max price, um it would default to the on-demand price and when you have the max price set, that opens you up to another condition from which ec2 can reclaim the instance for you. So most of the time, what you want to do is just to not set max price so that you can maximize your instance runtime another one you can do are flexible instance types, so ec2 has a lot of vm capacity and it's important to know how these are segmented across all regions.

A

um And so there are these the concept of capacity pools, which is a tuple of availability zones and instance types, and so you can see each of these orange squares right here uh would be a capacity pool, and if you have some capacity requirements or resource requirements for your instances, let's say you need at least a size 2xl, the on-demand price is, would say, 44 cents here per hour. You can actually request in 2xl and also 4xl and 8xl, and anything above because the 8xl is still cheaper than the 2xl because of the spot discount.

A

So capacity pools are the amount of unused instances in ec2, and so here we'll have these three capacity pools, which is the m6i.large and then the us east, 2a 2b and 2c availability zones, and so there's something you can do with ec2 when you ask for an instance and that's using the capacity optimized allocation strategy and when you ask ec2, given these capacity pools, it'll pick from the capacity pool that has the deepest amount or the most unused instances, and when you use this capacity, optimized allocation strategy that will really maximize your instance run time because it'll pick from the ins, the pool that has the most instances, meaning that ec2 will be the will be more less likely to reclaim your instance because an on-demand customer needs it.

A

You can see here that here are some instance types, and these are the frequency of interruptions. So it's important to note that these interruptions, if they do happen, are not going to happen all that often as well.

A

So the one of the best practices you can also do is rebalance recommendations, um and so this is just a warning that the interruption is coming and at the worst. This can also come at the same time. At the interruption notice, it will be vended through the same service. The instance metadata service that you'll see for the interruption notice, and so, if you have workloads that can't be cleaned up in two minutes, this rebalance recommendation might be useful for you and so something else. So sorry so going to the next thing.

A

Kubernetes and spot work super well together um and within spot and your cluster. You might want to have these programmatically configured to watch for these termination.

A

So these interruption, notifications, so within aws brand, has actually worked on the aws, no termination handler, and you can configure this within your clusters to work and watch for interruption notifications in rebalance recommendations.

A

So when you do this, there's also an important kubernetes concept called the pod disruption budgets, and these are used mostly to ensure that a certain amount of replicas on your pods are running at one time and sometimes, if you use pod disruption budgets, you might run into problems when you have spa instances and they need to be cleaned up within two minutes.

A

So we'll go into a little bit of an example here. So the kubernetes eviction api is what is used when you have when you drain your node and the cubelet will basically be sending sig terms and sig kills based on this kubernetes eviction api if it respects all the pod disruption budgets within your cluster.

A

So let's say you have no termination handler installed in your cluster. What this will do? You can configure it to watch for interruption notices and when it sees an interruption notice, you can configure it so that it also drains the node and when it drains the node. What the keyblade will first do is it'll send some sig terms to the pods once these have like say: let's say that the pod disruption budgets won't violate, be violated and that you can evict these pods.

A

um For instance, this pod could be saving data or shutting down um and that from that qubit will send a sig term and once it's done it'll be deleted, and then you can also do another one where this could pod this pod could be draining connections. This could be something like batch processing.

A

Sometimes when you are cleaning up your pods, sometimes they take too long, maybe too long way too long and that point when it reaches the termination grace period seconds, the cubelet will send a sig kill to the pod and at that point, it'll be forcefully deleted and then it'll be cleaned up.

A

So that's how you can use kubernetes in spot to clean up your workloads programmatically, but sometimes we need to also provision our workloads to get those instances in the first place. So we'll talk about how to auto scale your cluster, so you can use hpa and gpa to auto scale. Your pods and so hpa is the horizontal pod autoscaler. And so, when you horizontally scale your pods, this will be in the form of scaling the amount of desired replicas on your deployment for your pods, and so you see on the top one.

A

You can see the pods going from one replica to four replicas and the same would be for vertically scaling. You could go from one smaller pod that maybe has one v cpu and five gigs to a larger pod that is 2v, cpu and 10 gigs, and so within this.

A

Sometimes you might not have the nodes or the instances to hold these pods, and so you need to also auto scale your nodes, and so a very common node. Auto scanning solution used in kubernetes is the cluster autoscaler and for aws. This is a simple interface between the ec2, auto scaling groups and it would look at the existing pods within the cluster and if they can't schedule to the existing pods, sorry, it's the existing nodes.

A

It will increment desired capacity for the associated asgs that you have configured.

A

And so cluster autoscaler in this sense uses externally managed infrastructure.

A

If you have a lot of different capacity, type requests, availabilities and requests, and instance shapes you could have this set up where you have six different asgs, where they're each of the yellow ones, and that leads us to something else that we'd like to talk about, which is carpenter.

C

Yeah, so oh clicker cool, um so yeah, so that's why we decided. Can you okay there we go. That's why we decided to build uh carpenter. So carpenter is a new node, auto scaler for kubernetes, it's a groupless, node autoscaler, so cluster autoscaler kind of takes the approach of using this externally managed infrastructure like autoscaling groups or virtual machine groups that you'd increment this desired capacity on like nick was talking about.

C

Carpenter is completely groupless and kubernetes native, so we use provisioner crd, which is our custom resource in in carpenter, to define this configuration for scaling out. Your nodes carpenter is familiar to use if you're coming from cluster autoscaler, because we do we do just in time provisioning. So we look at pending pods that are unscheduleable by cube scheduler and that's how we know when to scale up your cluster carpenter is designed to be completely vendor neutral.

C

Currently we only have the aws cloud provider, but we're we're hoping to add on more cloud providers in the future.

C

So let's look at an example here, so you have a cluster. You have some pinning pods coming in. uh Just like cluster autoscaler coupe scheduler gets the first pass. So if you have existing capacity in your nodes that are running in your cluster, the cube scheduler will just schedule that those pods onto those nodes, if you have unscheduled pods. So this you know pods that keep scheduler, is not able to place on nodes because of scheduling constraints or just not enough capacity.

C

This is where carpenter comes in, so carpenter will look at these pods it'll. Look at your provisioner, spec and it'll kind of do an intersection of the requirements in both the pod spec and the provisioner spec, and then carpenter will look at the nodes available in the cloud that you're launching capacity in and figure out the set of nodes that it should launch to fulfill the capacity.

C

So, like I was saying, carpenter is completely kubernetes native, so there's no external infrastructure that you have to set up to use carpenter other than a few permissions, and so we use a custom resource definition that we call the provisioner, and so this is an example of a provision respect.

C

uh So we have some like main sections of the spec uh main one being the requirements, uh so the requirements uses the same. You know kubernetes requirements apis that you're probably familiar with if you're, using like affinities in your pod specs, and we use this to kind of layer on these scheduling constraints.

C

So in this example like the kubernetes, I o arc. That's a well-known kubernetes label to set the cpu architecture.

C

This provisioner is flexible to both x86 cpu architectures, as well as arm64 architectures, so this provisioner would be allowed to launch pods that require either of those or, if you build, your containers for both architectures uh carpenter will union together all the nodes or all the instance, types in the cloud that support those two architectures and pass it to ec2 for ec2 to make the best decision like nick, was talking about with the capacity optimized allocation strategy with spot.

C

This works really well because you can pass it a huge array of instance type options and then let ec2 select the one that's going to run the longest with the best discount.

C

So, similarly, to uh you know the cpu architecture, you can also be flexible across capacity types, so your pods, as always, can override kind of what the provisioner's flexibility is it'll. Do this layering intersection of of constraints, but if you're flexible to both spot and on demand, and your provisioner carpenter assumes that you want the cheapest node option, if you're flexible to both so we'll default you to spot.

C

uh But you know, of course you can override to on demand. If your pods require that uh kind of or aren't interruptable, so another section of the provision, respect is the provider spec. So this is a generic spec, but in this example this is the aws cloud provider. This would be where other cloud providers are also implemented, and this is just you know, uh cloud specific parameters for launching your your nodes.

C

So nick was talking about this spot best practices of like not setting a max price, because that was kind of the bidding error of spot, at least on easy tube. And then you know the big one with spot best practices is flexibility, and that was one of the main reasons we built carpenter is because we needed to be really flexible with the compute request. With all the instance types that ec2 is releasing and then rebalance recommendations gives you a way.

C

If you can't tolerate a two minute interruption window rebalance recommendations can uh potentially give you more time to to tolerate or to gracefully shut down your application. So we're going to look at kind of how carpenter helps with this approach, and so these are kind of the things I already mentioned, like the architecture. Flexibility carpenter builds this right in uh here, where you can be flexible across different cpu architectures and then, like. I was saying your your pod spec here.

C

If you require specific architecture, you can override it in the pod, spec capacity type works the exact same way with spot and on demand. It will default to spot if you're flexible to both and you can override in the pod spec uh and the requirements section isn't uh limited to just the ones I'm showing here these. These are kind of well-known labels in kubernetes, uh there's not a capacity type one on label and kubernetes. So we've used the carpenter namespace for it, but you can use your own labels here.

C

So you can kind of make your own scheduling constraints if you need that so another one that is a well-known label, is the node kubernetes instance type label. So if you do require, if your pods can't run on specific nodes, you can do an exclusion list here and again it uses the requirements, api and kubernetes. So you just do a nod in operator here.

C

So another interesting scheduling, constraint you can do is with gpus or extended resources. More generally. So in this pod, spec you're you're doing a resource request for nvidia, gpu and carpenter knows how to look at your gpu resource requests.

C

Query your cloud provider for nodes that support an nvidia gpu and then provision that instance for this for this pod.

C

So here's a little bit more details on the aws cloud provider side and how we're doing this flexibility that we've kind of talked about a little bit. But this is more concrete on the apis that we're using.

C

So we use the ec2 fleet api in what's called instant mode, so carpenter will get these unscheduled pods and then it will compute this constraint, intersection between the pod specs and the provisioner spec, and once it does that it'll query the cloud provider for all the instance, types that match the scheduling constraints and the bin packing that carpenter is doing for the pods onto those nodes once it computes this list of instance, types it'll pass that to the ec2 fleet api with an allocation strategy.

C

So if you're using spot it'll use capacity optimize and if you're using on demand it'll be a lowest price allocation strategy, but you can potentially get you know a huge list of these instance type options which makes you know. Obviously your request very flexible. uh You you have more guarantees on getting capacity, and hopefully your spot capacity will also live a lot longer.

C

So and all of this is taken care of with one provision respect you don't need to make a bunch of these provisioners to get you know hugely flexible, a set of instance, types that are launched and so like this provisioner will support on-demand spot different cpu architectures gpus.

C

All of that, we also support topology constraints for zonal constraints, host names. You know all the regular stuff you're doing with your pod topology today.

C

So we talked a lot about scaling up so now, let's talk a little bit about scaling down. So carpenter has two main fields in the provisional spec that controls scaling down. So we have a ttl seconds after empty, and so this is when you have a node up and all of your pods are evicted or you know, exit if they're jobs, and so you need to spin that node down so carpenter will look for empty nodes, excluding daemon sets, and then you know turn those nodes off after this time to live has expired.

C

The next one is the ttl seconds until expired, and so this fulfills a requirement.

C

If you need to roll your nodes over a specific, you know, time period, you don't want them to live longer than 30 days or a week or something carpenter will basically fulfill this expiration period for you and roll your nodes over and whenever carpenter rolls a node over or turns it off, due to it being empty it'll gracefully drain, these pods from the nodes it'll use the eviction apis, it'll respect your your pod disruption budgets and we even go a little bit further, where we install a node finalizer on all the node resources in your kubernetes cluster, so you can actually cube cut or delete a node.

C

That's provisioned by carpenter and carpenter will call the eviction api to drain the pods, respecting your pod disruption budgets and then we'll terminate the backing ec2 capacity behind it.

C

So carpenter today doesn't support natively spot like being able to watch for interruptions, but you can always install the aws node termination handler in your cluster, which is pretty easy to install, and it will pass that signal back uh so that carpenter can kind of handle your your node evictions for you, and so this is a similar um kind of walkthrough of like what nick did, but with a rebalance recommendation. So this would be used if you need more time than that two minutes window, that an interruption would give you.

C

So no termination handler is installed in this cluster. It watches for rebalance recommendations, and then it sees one and decides to drain the node. Due to this, this rebalance another way you can handle a rebalance recommendation is to simply cordon the node and that's an option in node termination handler. If you just don't, want further pods to schedule onto it, and so then the eviction process happens where the cube sends it a sick term. Hopefully the pod shuts down in a reasonable amount of time.

C

You know within your termination grace period and then it just shuts down and everything's good.

C

So what do we talk about today? So we talked a little bit about spot and how spot can drastically reduce costs of your cluster. You know discounts up to 90 on your nodes, where it's the same backing capacity. The only caveat is you have to handle that interruption notice but handling the interruption notice? Isn't that different from an on-demand right, because you still have to handle pods shutting down for on demand hardware failures and on demand, so you kind of have to be able to shut down fast anyways in a lot of applications.

C

So why not? Why not try out spot? We looked at spot best practices, mainly on the provisioning side, on being super flexible to different instance, types which will allow your spot instances to run for longer periods of time and also hopefully get better discounts if you're being super flexible.

C

Then we talked about how kubernetes and spot are really. You know a match made in heaven. Really the kubernetes node lifecycle abstractions, make it super easy to handle these spot interruptions and components like the node termination handler, standardize everything for you.

C

Then we talked a little bit about how scaling up your pods requires you scaling up nodes and how you can be flexible with cluster autoscaler or you can try out carpenter and use our our crd approach with a provisionary spec to do groupless, node autoscaling and we did have a demo, but we had severe technical difficulty before so, I'm not sure is it gone? Okay, yeah. I think my laptop died like right when I walked up here and wouldn't charge anywhere. So no demo.

C

um But if you catch me around and I get my laptop uh working again, we can definitely show you a demo of of carpenter in action. So yeah, that's all we have so are there any questions? I think we have like four minutes left. Maybe.

C

Sure I I think we have mics up here. If you wanna.

C

Justin garrison emcee.

D

So you will, would you recommend only if I use carpenter um to scale my notes, would you recommend to only use spot instances in production.

C

But I mean it really depends, so there are certainly applications that can't tolerate interruptions like you just can't do it. So in those cases, if you can handle some interruption, but you need to guarantee some amount of capacity, then you can do a topology spread across on demand and spot nodes.

C

So you get like a 50 50 spread and hopefully one day we'll be able to do weight based topology spreads where you could do like a 10 on demand and ninety percent spot spread, but there's definitely workloads that require spot and carpenter will fulfill that use case as well. But uh but it's really easy to use spot with with carpenter.

D

Okay, and also, is it possible for copper carpenter to get rebalance recommendations to use to.

C

Yeah, so as long as you install the node termination handler in your kubernetes cluster, a carpenter knows how to drain the empty nodes or knows how to shut down the empty nodes, and so no termination handler will actually do the graceful draining part of the process when it sees an interruption or a rebalance. And then carpenter will shut down the node for you.

E

Hello, you said that carpenter can shut down the nodes that seems empty. Excluding demon sets what about moving pods that can be scheduled on remaining capacity to defragment the cluster.

C

Yeah exactly so yeah defragment is definitely on like the the top of our priority list. Right now, we've done a lot of work on it. So far, it's not released yet, but um we're thinking about that problem. A lot there's a work around today, where you can actually use the expiration feature where you can set an expiration and if you have a good pod disruptions set up, you can, you know, shut down instances that have expired and then the pods will go back into pinning and they'll be packed more efficiently on that round.

C

So that's kind of our workaround currently but hopefully, defrag will be out soon.

F

Hi, do you see spot and carpenter actually helping with carbon footprint for companies? Part of the green? You know revolution so to speak.

C

Right, I I don't have any data on it, but I I definitely see carpenter as uh more efficiently utilizing nodes and capacity, and I you know hopefully moving forward capacity will be more utilized and therefore more efficient, and you know therefore, more carbon efficient as well.

G

Hi, have you noticed any changes in the spot markets over the last like two years? I would imagine. Companies are using ec2 a lot differently after pandemic than before.

C

Yeah, I I have not. I don't have access to that data, so I wouldn't know to be honest but yeah, hopefully more people are using spot.

C

I don't I don't know if the pandemic had anything to do with it or just you know it's a it's. A good deal right. People like discounts.

F

B

Hi- and this is a great slide to ask the question for because I see that you have r5 instance specified here- does carpenter support the attribute, based notes, selection,.

C

So that's an interesting question, so we're working on a addition to this provisioner spec, where you could specify attributes to constrain the list of instance types, but by default carpenter uses all instance types and it'll look at the pods to kind of figure out what constraints would prevent it from launching certain instance types but yeah we're working on the attribute based like filtering down in the provision. Respect. Thank you hopefully, that'll be out soon too.

H

Hi I I also have a question about carpenter. I mean the spot. Instances is really interesting. I think pretty well understood carpenter's interesting for a lot of reasons. You've talked about the bin packing on at least the metrics of cpu and memory. uh Have you also considered the constraint of volume attachments.

C

Yes, yeah, we definitely have uh so. We support some uh like uh empty directory packing today we're looking at supporting more dynamic, like ephemeral, storage attachment, uh like scheduling and bin backing awareness. We are aware of, like persistent volumes, so we'll schedule in the correct zone, if you're trying to attach like an ebs volume to it, but yeah. That's definitely something that we're working on currently.

H

Just because of the volume attachment limit right right.

C

C

Make justin run.

I

Hi, um you said, carpenter doesn't work with uh auto-scaling groups yeah, but how does it work with managed nodes.

C

Yeah so today, carpenter doesn't work with manager now so they're separate, like solutions uh so with managed nodes. You'd use uh customer autoscaler today for managed nodes because it is in an auto scaling, group and carpenter would just be completely without without any groups.

C

I

G

I think that's time so. Thank you very much. Thanks.

C

Everyone thanks.