Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2022, 1 Jun 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Autoscaling Elasticsearch for Logs on Kubernetes - Radu Gheorghe & Ciprian Hacman

Description

Don’t miss out! Join us at our upcoming hybrid event: KubeCon + CloudNativeCon North America 2022 from October 24-28 in Detroit (and online!). Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Autoscaling Elasticsearch for Logs on Kubernetes - Radu Gheorghe, Sematext Group & Ciprian Hacman, polypoly

Elasticsearch (and its fork, OpenSearch) is the go-to storage for logs. As with any storage, the cluster likely needs to scale to keep up with the change of load. But autoscaling Elasticsearch isn't trivial: indices and shards need to be well sized and well balanced across nodes. Otherwise the cluster will have hotspots and scaling it further will be less and less efficient. This talk focuses on two aspects: - best practices around scaling Elasticsearch for logs and other time-series data - how to apply them when deploying Elasticsearch on Kubernetes. In the process, a new (open-source) operator will be introduced (yes, there will be a demo!). This operator will autoscale Elasticsearch while keeping a good balance of load. It does so by changing the number of shards in the index template and rotating indices when the number of nodes changes.

A

Hello and welcome in this session we're going to talk about elasticsearch or open search, we're going to use the terms interchangeably for the most part we're going to talk about them in the context of logs and other time series data and we'll focus on using a kubernetes operator to manage elasticsearch clusters in this context.

A

But before we start, let us introduce ourselves, I'm radu, I'm a search person at sematext and most of my time goes into consulting production, support and training for elasticsearch, open search and solar, and sometimes I contribute to our observability platform, which is called semitex cloud.

B

Okay hi, my name is chiprian, I'm a consultant for kubernetes and automation, and also work as a software engineer for poly poly, mostly on privacy related projects, also in my spare time, I'm an open source maintainer contributing to many of the projects in the kubernetes ecosystem.

B

So, let's start with the agenda we'll talk about why we want to have such such an operator, so the use cases we will talk about, how how it should work when should it scale up and down, and what should we do when it does that and available options and last a quick demo of our proof of concept?

B

Let's start with the! Why? If you have a small cluster, then you don't want to think about elastic search to learn all about it, the ins and outs, and you would like to have as less maintenance as possible, so ideally zero maintenance.

B

This kind of operator would help you get started with your logging, without caring about learning elasticsearch too much for bigger clusters.

B

Usually these clusters are multi-tenant and you you could split them easier because it's not such a big maintenance to have multiple clusters. If you have the operator.

A

All right moving on to how so, in other words, what does the operator need to do in order to auto scale um in order to get there? I want to talk about three things, so one is using time-based indices. So, if you use, for example, logstash before you might have noticed that it creates one index per day, um we're going to talk about. Why that's a good idea.

A

Next, I'm going to argue that for most use cases rotating by size instead of by time, will work better and the third thing would be. How would those indices behave when you scale your cluster up and down?

A

So, let's start with with time-based indices, you know they don't have to be daily. You can have like one index per month or one index per hour, but the idea is the same and the advantages are pretty big. So when it comes to indexing typically, the bottleneck is down to the lucene segment merges which happen in the background and if you're indexing in a smaller index, it's going to be much much faster, so we're basically comparing the time based indices with just indexing.

A

Everything in in one index when it comes to searches, most of the searches tend to be in the latest data. So again, if we have it broken down by time, we can hit just a slice of our data and that's going to be faster, but even if we search in all the data, even because their indices are not written anymore, they're much more easily cacheable by both elasticsearch and the operating system underneath it.

A

So, um in my experience, both indexing and searches will be orders of magnitude faster with this design compared to having just one index.

A

And finally, when you delete when you have to expire data um with time businesses, you can simply delete whole indices, which, by and large, implies deleting some files on disk, as opposed to deleting documents from within an index which are only soft deletes and will trigger additional lucine segment merges.

A

In practice, you would have like multiple series of such time-based indices. um um Normally, um we would break them down by uh how you'd search them. So, for example, if we search nginx logs uh separately, then syslog in general, we don't have to like. We can always search through everything. Then it makes sense to keep them in separate index series, but this design is not perfect and we may run into what we call the black friday problem.

A

So let's say we are an e-commerce website and we're logging access logs, and so hopefully, during black friday we will have a lot more traffic and so that index will grow larger but much larger than the indices from the following saturday or sunday, and then you have cyber monday and then again it's a spike of traffic and then again it goes down and so on and so forth, and the problem is that um the big indices that will get generated on friday and monday will behave.

A

Much like that big index that we talked about before so indexing will be slower. Searches will be slower exactly when we need them the most.

A

So to fix this problem, we can rotate indices by size instead of by time.

A

So the way this works is at least typically, we would have a an alias that would point to an index and logstash will write to that alias or whatever puts data into elasticsearch, and then, when that index gets to the target size uh typically 10 gigabytes per shard is, is like a good rule of thumb in in our tests, um we're going to create a new index, we're going to flip the alias to the new empty index, and then we can continue writing there, and we just keep on doing that.

A

Thankfully there's some automation in recent versions of both elasticsearch and open search in open search. This is called index state management and in elasticsearch it's called index lifecycle management, and these can be used to automate this process of okay, let's create a new index, let's flip the alias, even let's remove old indices, which is going to be a bit more challenging in this context, because you don't have indices strictly divided by time.

A

um Of course, this design isn't perfect either. So, for example, if you need to backfill data right, we're just onboarding some new project that already has a lot lots of logs. Those will all go into the latest index and that's problematic um and also when we're searching, let's say: you're searching the last 24 hours, which indices contain uh those 24 hours.

A

This is a bit harder to figure out, though elasticsearch again will have this kind of stuff out of the box, so shards that don't have data matching your time frame will quickly kind of dismiss your query saying: zero.

A

Okay. So how does this work in the context of auto scaling? So, let's start with the simplest or one of the simplest examples. Let's say we have uh two elastic search nodes and we have one index uh with two shards, so we have a spike in load. Let's say we want to auto scale, we add a new node. Obviously we cannot take advantage of it. Even if we had previous indices before some shards will migrate, but the third node will not be able to contribute to indexing, which is our main load.

A

um So our suggestion is to say: okay, even if we're not at 10 gigabytes per share or whatever our threshold is, let's force rotate the index and like create a new one that will be evenly spread out throughout our cluster, and this way all three nodes can can contribute to indexing.

A

This will typically imply three steps, so one we would change the index template so normally all your settings and mappings and all that stuff will leave an index template so that when you create a new index, it will inherit everything.

A

So we'll need to change that template to say three shards instead of two and um we're going to do this, uh forcing first rollover like create the index and and flip the alias. And finally, we may need to adjust the life cycle policy.

A

So if, if the life cycle policy said uh 20 gigabytes before, because we have 10 gigabytes per shot times, 2 shards now, it has to say 30 gigabytes in order to um keep consistency and we're going to do much of the same when we need to scale back down, um the only difference would be that we need to make sure that nodes are properly drained before we shut them down right, so we would exclude them from allocation and then elasticsearch would move the shards off of them before we take them down, but once we take them down, we get into a similar problem as before, in the sense that the cluster is not balanced right.

A

So in order to make it balanced again, we can do the same thing. Change the template back to two shards force roll over and adjust the um the policy again. Is that make sense cool? So that's what we're trying to automate, but before we get there, I want to talk about three more sort of best practices. We can't be comprehensive here in terms of best practice, but I think these three are important, especially in this context.

A

So one of them is that it's often tempting to judge the size of a cluster based on indexing throughput, because that's our main workload, we do much more indexing than searches. Typically, so let's say we index one million documents per second.

A

What kind of cluster do we need, but typically the unit of scale would be either search, latency or just outright disk usage, because if we want to if we index 1 million documents per second- but we want to keep this data for like one month- we'll need a big cluster and that big cluster would be big enough to index um one million documents per second right. So this is typically what we will look for, and since searches are our bottleneck for scaling, then I think it's worth mentioning that searches do lots of random.

A

I o, so I o latency tends to be more important than throughput or even iops. So, with most cloud providers we would typically go for the local ssds, the ephemeral storage, rather than the managed over the network disks, because we have much much better latency, and so we can uh put a lot more data on the same node now. Obviously, this has a downside in the sense that if the node goes down it takes the storage with it. um So we'll have to account for that either.

A

By having more replicas obviously depends on how important the data is and regular backups to sort of compensate and the last one is, you may have heard of hot warm cold colder kind of architectures. The idea is, um you have like your let's say, best hardware in the hot tier: that's where indexing happens. That's where most of your searches, uh recent searches will happen and as data becomes less relevant, we move it down to lesser hardware.

A

um Frankly, our proof of concept does not support this, but I'm not sure if it's needed, because um I think there's a limp, there's limited use of this design because you cannot put like trash hardware on the call tier, because elasticsearch will still keep need to keep the data open. It still needs to be able to monitor itself and stuff like that, and if the call tier is too slow, it will just become unstable.

A

It's not the problem that you wait like 30 seconds more for a query: it's just it's gonna. The nodes will drop out and stuff like that, um but if the so, if the cold tier is fast enough, um at least this is what we notice like hey these are these nodes are idling, we might as well get them back to work and do indexing, and then you end up back to like a flat kind of design.

A

Okay, so um this idea of an elasticsearch operator is not new at all, so we thought. Okay, let's see what existing operators can help us with and unfortunately a lot of them are either unmaintained or they don't do auto scaling at all, and but there is one bioelastic that obviously supports elasticsearch very well. It's called elastic cloud in kubernetes trouble. Is it's not exactly open search? We don't want to get into that flameware, but not open, search, not open source, um because it's on the elastic license, but particularly for auto scaling.

A

You need to pay you. You need an enterprise license. um There's I think it's worth mentioning uh opster, which is an operator um used by open search, it's under active development, but right now it does not support auto scaling. It's on the roadmap, but it's not there yet so the one we went for is called es operator it's from zalando. It was presented at kubecon cloud native con before and it is open source.

A

It does support auto scaling. It does support draining uh nodes like um what I mentioned. We need to move shards off before um uh shutting the node down when we scale down and it works really well for, let's say e-commerce type of use cases.

A

So when you have one or more indices- and you just want to increase your cluster capacity to to handle them, even if that means adding additional replicas, so that works really well, but we wanted to change it so that it works for logs, so that it does this template management and lifecycle policy management and the force rotate. Everything that that I talked about before- uh and this is what uh cpan will show you uh in a second okay.

B

So uh we're going to do the demo right now. uh I will start with uh a little uh about the zalandu operator, inner workings.

B

It requires you, first of all, to have your own master deployment before running it. So it's not fully managed the cluster. But after that you have some simple options: that control scaling and other various aspects like excluding system indices from any calculation that the cluster that the operator does.

B

In our case, the most important options are the minimum replicas maximum replicas that we allow for it to to create, and we will be using this usage percent for scaling up and down the cluster okay. um So at the moment we have already prepared the the demo in docker in kubernetes docker. So it's pretty easy to to do without networking.

B

um As you can see, there is only the master we already applied this configuration and once we will start the operator, it will create the data nodes and then we will show how it scales up, how it applies, the templates, um the ism configuration and what it does at each scaling step. So starting the operator now, okay, immediately, it notices that it needs two replicas and it has none um already started the pods and looking here it should take very little to get the cluster up and running until then.

B

B

Let's look at the cluster health, so it's a green cluster.

B

There's no index yet.

B

There is no template, there is pretty much nothing as you can see. It already found that the nodes are healthy. It created the component template for logstash and also a separate component template for scaling, and then it applied it to the index. You can see that it applied both logstash and the scaling components, and it also created the ism policy with the minimum size, 10 gigs.

B

Okay, let's see here indices, we have one index with one primary shard and one replica. um We have template with logsdash and scaling and the cluster should be healthy. So it's still green.

B

If we go back to the configuration we can change this scale up percent boundary to 21. This is something that works on my setup. We don't have the time to ingest that many logs or data to to do a real demo, so we just fake it with.

B

Decreasing the thresholds, so um let's see.

B

B

Should be applied and very soon, it should detect that um the bound the disk usage percent boundary has been reached and it should start adding more replicas. While it add more, it adds more replicas. It will reconfigure the scaling template the ism.

B

And roll the logstash index to to match the new number of nodes with bet best practices.

B

So, let's see oh okay still waiting for the third note to be ready, I guess.

B

So, let's see here how it does.

B

Okay, it should be up and running and we should see how it sets the scaling template right now to to match the number of nodes and it increases the ism policy to minimum size to 30 gigs to match the three primary shards, because all the nodes that we have share the same storage. It will not stop at three replicas right now. It will go down.

B

It will go up until the maximum allowed by the configuration so for replicas, and once it reaches that threshold we will change back the uh scale up disk percent boundary and we will make it go back to to the way it was to two nodes, and at that time it will start recreating each index, um no, not recreating sorry rolling the indices and applying uh the correct uh scaling template for them. So, right now, if we will be looking here in this series, you can see that we no longer have one.

B

We already have three uh created at each step, so it started with one uh primary and one replica. Then uh it went to three primaries and one replica so that the allocation can be uniform on that amount of nodes. And lastly, it went to two primaries and one replica, so I guess right now we should have the cluster with four nodes. It should be healthy, let's see so it's healthy.

B

We can try to see that the ism policy was applied correctly, so it's 20 gigs the minimum size for all over, as we expected.

B

And um scaling and the scaling template matches what we see in the in the logs.

B

Okay, uh reducing this uh changing this back to 75 percent and increasing this to 20 25 percent, the scale down disc usage percent boundary so that it starts uh decreasing in size applying so right now it should quickly scale down also drain the nodes at each step and reapply. The scaling templates.

B

To two indices, while rolling them- okay- let's see in this, is you can see already have the fourth index. That's matching the second one created like three primaries and one replica and pretty soon it will. You see draining the pods ensuring the cluster is green and pretty soon it will get to uh two ready nodes instead of four, as it was in the past. So in this is it's already at five indices and we have again two data nodes in elasticsearch.

B

At the moment, this is pretty much hard-coded, many of the things that we did here. It was most it's mostly a proof of concept, but we want to improve it and publish it as a usable solution for people that want to do logs, um and I think this concludes our demo. So thanks everyone for coming here and watching us. Thank you.

A

C

D

For me, should I keep it? I don't know.

D

So uh basically, at the end, you have a log stash o5 index with strip three primary shard and one repeater for each, but you have only two nodes in your cluster. So is the cluster beginning to be yellow at that point,.

A

No, no, it wasn't. It wasn't three, it was one at the very end.

D

You have an index with three screw primary at some point and you keep it so. The cluster should be yellow right.

A

Yellow because we, when we oh, do you want to take this sure.

B

So we modified the shards per node setting while scaling down, so it allows the cluster to scale down properly. We don't want, we don't want that part to scale to hold back the scaling. Okay, thank you. No problem any.

A

Other questions.

E

Yeah, just uh more of a high level question on the the operator, so um the reason I'm interested in this is because I have some customer facing real data, not necessarily for the observability aspect. It's I mean. How confident would you guys be with this operator for genuinely customer facing critical uh usage as opposed to a little bit less worried if I lose it.

A

Well, this particular one that we showed it's not production ready. I can tell you that um the one that is sort of open source already that we linked, if you don't have logs, if you have like- let's say e-commerce, it should work. I mean as far as we work with it it. I think it's obviously up for sort of testing for your use case, but I uh they use it for quite a long time now, okay at their shop, so I guess it works.

A

A

Thanks any other questions.

F

What matrix are your monitoring in order to know when to add more nodes or scale more less nodes and stuff.

A

Yeah yeah, where we initially the the the metrics, were cpu usage um and also um number of shards per node. So you can say I want at most five shares per or something like that. That's what the original operator does. We added the disk thresholds so if the one that cheaper modified like if you go over a specific disk and we scale up if we go under, we scale down um and there's another one that has to do with disk, which is like a safety net.

A

So when you scale back down, you don't want to get over a specific um disk usage because, like you, don't want to run out of disk when you scale down or run into some sort of endless loop. So those are the metrics. The plan was to add something with that has to do with search latency as well. We can pretty much get any metric that we want from elasticsearch, because these this this metrics um are from elasticsearch itself, like from cat nodes or something like that.

B

We we'd want to use metrics from inside elasticsearch, because elasticsearch does quite a lot of stuff in the background and if we use something from external sources like some monitoring tool, it may differ from the same the data that elasticsearch uses for internal uh thresholds. So we cannot do for disk. uh We cannot use for this external tools.

B

um I think the zalando people are trying to extend their operator to use horizontal podatto scalar, so that would make it pretty easy for us to gather more metrics and add the logs related ones there so improve on that.

B

But at the moment it's uh from what we read in their comments. They are heavily developing this feature, so we cannot do a merge of our code there or even discuss the possibility, because we don't know how it will look, it's kind of frozen at the moment.

A

Any other questions yeah. Thank you.

C

uh Hello uh in in in a big cluster that uh has uh big charts like 50, gig charts and I don't know 60 notes or something like that. uh How does the scaling up and scaling down performs.

A

We did not test it with that size.

A

I think my recommendation would be if you can not get there and but rather split it in multiple smaller clusters.

A

I think for a larger cluster. What we would want is to scale in larger increments, like you, don't want to scale up all the time because we force rotate and then it would be kind of sub-optimal if we go 61, 62, 63 and then go back and so on and so forth. So I would add some sort of increments. We, the operator, already has stuff like cool down both when you scale up when you scale down.

A

So it's not that jittery and you can obviously configure that, but I would add some sort of steps, so you don't so we kind of reduce the noise and for large shards. This is again something that you can have control of and if you use it for time series data, I think 50 60 gigabytes is kind of large, but of course you cannot have like a million shards. So there's a trade-off there.

A

You may be aware of that, um because if you have a lot of shards, then your cluster state becomes large and then it's going to be difficult to coordinate that across the nodes and that's in my experience at least ultimately the the scaling limit of how much data you can put in a in a single elasticsearch cluster, because at some point it becomes unstable because of the large cluster state that has to be coordinated.

A

I hope I answered your question yeah. Okay, any other questions.

A

Right, if not thank you very much for attending um hope. You.

D

Have thank you.

A

For great rest of the conference.