Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 22 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Es-operator: Building an Elasticsearch Operator From the Bottom Up - Mikkel Larsen, Zalando SE

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Es-operator: Building an Elasticsearch Operator From the Bottom Up - Mikkel Larsen, Zalando SE

Operating stateful applications is a challenge and Elasticsearch clusters are no exception. At Zalando, the search infrastructure heavily rely on large scale Elasticsearch clusters consisting of 100s of nodes. In this talk Mikkel present how Zalando utilize Kubernetes and the operator pattern as introduced by CoreOS, to reliably and cost efficiently operate large scale Elasticsearch clusters. The talk will walk through how the Elasticsearch operator was designed, what problems it solves and how building it from the bottom up allowed getting it in production fast, gather more learnings and later extending the featureset to make it less manual to operate and reducing the cost of the overall infrastructure. While the talk revolves around running Elasticsearch, the ideas presented also apply to running other types of stateful applications in Kubernetes.

https://sched.co/MPcP

A

Yeah all right we're gonna go in and get started a little early since they won't let anybody else in anyway, so I'm Doug, Davis, track host, but obviously you're here they're here you're here to hear Mikkel talk about operators, I was actually reviewing the proposals for the CFPs. For this thing, operators was one of the hottest topics out there. So obviously, as you guys are testament, this is obviously very very exciting topic. So this is. This would be really good.

A

One quick reminder, please rate the top the talk when you're done through the app that way. The auger dieters know where there was a popular topic. Speaker did well or not, that kind of stuff they leave. It make comments as you possibly can and with that yeah. Thank.

B

You so welcome everyone who could get in to my talk it's about ears operator, which is an open source project for running elasticsearch like running big scale elasticsearch and how we run it and kubernetes at solando adam. This I want to go like to a story of how we built this operator and I think the patterns here also apply to other applications, not only lesyk, search but also sample. These kind of applications can also benefit from this pattern. I think so. Who am I.

B

I'm Meg, Larson I am a software engineer at solando I work at the cloud infrastructure team which deals with kubernetes AWS infrastructure, so we management clusters and we were like in a joint effort to develop this elasticsearch operator between the search department and our team. So we came with the cominius knowledge and they came with with the elasticsearch knowledge. So I'm, not a expert on a likely search. I know some things, but if you really want to know this, then I will forward you to some some colleagues who are unfortunately out here at the conference.

B

If you want to tweet at me, saying that is stupid to run or most critical stateful applications on kubernetes, then you're welcome to do that. But maybe you have a different opinion. After that's all good see what is the Lando? We are Europe's leading online fashion platform. We sell socks and shoes and all kinds of fashion articles all over Europe online and you go there. You order your things, get them, and then you can send them back.

B

If you don't like it all three and we are in seventeen markets, we have a revenue of 5.4 billion, so quite a big company, 26 million extra customers. We had 15,000 employees in Europe and roughly a little under 10 percent of these working with intake, so software engineers and so on, and these are kind of the customers for our team internally.

B

This means that we have a kubernetes set up at solando. That looks like this. We declare it the default deployment target this year, meaning that everything that is the develop new software has to go to kubernetes. Ideally this means we have 125 kubernetes classes right now that we manage in our team. Roughly 50% fat 50% test clusters, this amounts to around 1400 easy to instances. We run an AWS. This will cost scales in and out during the day. So it's just a rough number because we have auto scaling with the cluster autoscaler.

B

We started this interfere with kubernetes in October 2016 and since then we have updated our clusters from 1.4 to their oldest clusters. To now 1.13 last week we rolled out and really like continuously upgrade the cluster. So we don't create new classes, move workload. We just upgrade them in place, and this is of course an elastic search of a radar would also have to to take care of this.

B

So what the search looked like at solando, we have 300,000 products per country, 17 countries. We have 2000 brands, roughly 700 categories, and all this data is in the elastic search. So you can find it when you go to the website and search for whatever shoes or t-shirts or whatever you like, and this is all powered by elastic search. It has 12,000 appears on our QPS on like the normal peak during the week, and we can also handle like a que.

B

Eight thousand requests per second, the updates per second. From from like ingesting the data from the different streams we have internally into the cluster at Black Friday, which is our biggest commercial event of the year end of November. We have roughly doubled of what we have in the normal week as, and you can see here, we scale to 512 notes last Black Friday.

B

This is not in company this is we just moved it this year, so we will see how it goes, but just to see what the scale is, and normally the workload is not 500 but more like 200 each two instances for this elasticsearch pasta. We have different instance types depending on which country we are serving. They are all part of one big elasticsearch pasta, but it was running on ec2 AWS, with a little bit of tooling around some bash scripts here and there.

B

Some Jenkins pipelines running quite well managed by the team, but there was a lot of manual things involved and considering that we had made kubernetes the default deployment target, it made sense to move it not only to kubernetes but maybe also make some improvements along the way.

B

So goal was moving to kubernetes and by moving it to kubernetes. We we had some goals that we wanted to achieve, so we wanted to have Siri, save automatic updates of the elasticsearch cluster. This means both that elasticsearch can update like updating a version of elasticsearch or changing a configuration of the of the nodes in the cluster, but it should also handle updates of the underlying kubernetes cluster.

B

So the way we roll to veneers processes, we scale out the nodes and then move workloads to the new nodes, and this is done all automatically and the tooling for elasticsearch has to be aware of this. All the application has to somehow handle this. Another thing was that now we are moving anyway. The they're all set up in impure AWS was very static, like we could add manually nodes and for for, like Black Friday, but we couldn't like auto ski and scale down during the week where we didn't have so much traffic.

B

So actually the cluster was usually scaled to the peak to handle the peak load of the week and for the whole week. We maybe the double amount of notes that was needed in the night and so on. So we wanted to have advanced, auto scaling to like reduce costs, to be able to scale down and when the classes are due so much. This was the criteria we looked into first like how can how do people run elasticsearch and kubernetes?

B

We found the kind of official help chart for a running elastic search, which is a stateful set. I hope you all somewhat familiar with staple sets in companies. I will not go so much into it, but the idea is that you, you have a way to say: I want to have X number of pots, and they have some volumes attached potentially and they have a certain criteria for what is the naming of the pots, and so on that you can kind of reliably and the way it works.

B

If you just take the helm chart and you, for instance, think of a situation where you need to update the cluster, then the cluster would like want to drain node. So let's say we have three elasticsearch pots or nodes in the lecture class. The gray are nodes of the company's cluster and we want to drain one node, so the tooling will drain a node.

B

This means that we need to delete this pot on the on the node that's being drained, and what will happen with the home chart setup or the stateful set is that it will yeah trigger draining from our infrastructure, basically terminating their pots, as kubernetes will handle termination. Normally then, in communities you can set a pre stop hook on the on the pot definition, which can be like a batch script.

B

You can run or whatever, and what it does in the home chart is that it excludes the node from the elasticsearch class time, meaning that data will not be moved there. So electric search will stop moving data there and will actually like drain the drain, the data away from this node this we want to do so. We don't just drop there, all the data we have there. If you have volumes, of course, you could have it still attached to a new part.

B

It comes up, but we also want to make sure that we, we are not letting any traffic go to this node. Why we're training it? So this is kind of why why you want to do this? The data is moved as shown here and then the part becomes empty, doesn't have any data and then it can be be deleted. With this pre stop hook. You have this thing that the part is already going into a termination phase you cannot like on.

B

You cannot stop this in company, this is already terminating, and you said that the chart sets like a one hour, delay of the training. So if your train takes longer than one hour, then you have a problem here. You cannot really handle this. Also. If the batch script doesn't handle network issues or something during the phase, it will just fail. Maybe, and then you yeah doesn't really execute the the moving update dates are correctly. This is one problem with this approach.

B

If we go further, what in the ideal case when it works, what would happen is a new part would be replaced by the stateful set on a new node, and this has a post at hook, which basically removes all these excluse in elasticsearch, meaning that days can then be moved to the new part and elasticsearch will like rebalance and so on.

B

The problem here is that if you try to alter scale at the same time- and you have this priest at hook- which just unblocks all of them, then you could have one part that is training and have excluded itself, and then you have another one that just like on excludes everything, and then it starts to get data again. Of course, you can implement more sophisticated scripts or maybe writing something in the more advanced language and handle these cases, but it begins to sound like you actually need something like an operator.

B

So the operator pattern is it comes in here. I took this from the chorus they kind of coined this phrase operator in 2016, I. Think I, don't think they like invented the idea, but they kind of called it operator. I think I, like this term, and they stated an operator represent human operational knowledge and software through we manage an application. So the way I understand this you're in code like how an offer what you would normally do as a human cube detail. Do this call Dre, nonetheless, research?

B

All these things, you incorporate into code that can perform this. For you, this is one way I see. Operators I also see when I look at what other people are building as operators, operators can also has have the functionality of like providing an abstraction so, instead of users defining a stateful set, a service volume claim defining all the people talking behind.

B

Sorry, instead of the users defining all these resources and what environment variables to give and so on, you can create an abstraction that, like hides all of this, for the users, this is not what we focused on. We focused on the purely operational part and to start with, because this was really the problem we had, the defining making user friendly and so on was not so important because we're very like the team that operates is a very technical knows a lot about elasticsearch, so they want to tune it to to their liking.

B

But the idea we first had, we call it like be zero. What to just manage a stateful set in a bit smarter way, so what we could have? What we said was: okay, we cannot put. We cannot modify the replicas directly that you normally have a stateful set, because if you drain a part, it has this problem that I described before it just stops terminating and and then you cannot really stop this operation and if you take longer than one hour, then you cannot handle this case.

B

So we wanted to have it split in like the replicas on the stateful set and then the desired replicas, which is what the operator tries to achieve. So you might be scaling. Obviously easy. You just add another part, but if you're scaling down you could have like replicas smaller the desired replicas, which we put in an annotation as you can see, it could be small and then the other one could be high, and then it would eventually like drain and achieve this.

B

We also used update strategy on the lesion, which meant that instead of the staple set controller like modifying and updating the pots, when we make an update, the operator can control this person and kill parts once at entering them. So this didn't really work for us. It was too low level and, like it, was complicated to just update the stateful set without because you have this, you have a certain number of replicas in your jungle file and if you update this, then you override, what is it about? The operator is doing.

B

This was one thing and state must be stored in annotation, which is just not nice. It's just a string that you have to then pass and it just becomes a little bit ugly. It's doable about it yeah. So what we did instead was we thought. Okay, we just need some layer of abstraction. On top of the stateful set, the staple said idea is kind of good. It just needs the extra knowledge, so we created something we call elasticsearch data set, which is a very thin wrapper. It has the scaling thing for configuring.

B

Your auto scaling I will talk about this later and then it just have replicas like a stateful set the template for the part template, so you specify the full thing and the volume claim template so like a stateful set. So this way anyone can just take a stable set, modify the bit and then they would have it work with their with the elasticsearch setup.

B

And this way we have replicas on this state on the Lessing's data set letting search data set, and we have also a state on a stateful set which is managed by the operator underneath and then we have these two different values that it can reconcile towards.

B

So this is idea and from like a higher-level elasticsearch data sets can either be like a full cluster if you want, but you can also compose them into into like one big class that consists of different elasticsearch datasets, and this was what I mentioned in the beginning- that we have different type of instances for a different type of countries, for instance. So Germany is like a lot of traffic. We get there, so we have big instances and unless it's search data set for that these this country and then for other countries.

B

We have smaller or instance, titles. Smaller parts requests in company this case, but they all combine into one big elasticsearch lasta, which is how the tooling in our search infrastructure works. But you could also have like one set of masters for each of your elasticsearch data set. You can kind of compose it like you want, and the operator then manages all these things.

B

So if we look at the example again like I gave with the stateful set now with the operator involved, we now have a service created by the operator that that, like goes into the elasticsearch API, we have the same parts and, let's think of the same scenario where we have a pot node kubernetes node, that is being drained.

B

What the operator does it detects this because it looks at the label of the node or the status of the node okay, it's being drained, then it scales out the stateful set underneath. So we get an extra pot. So now we have somewhere to move the data from this one we want to drain. Then it calls into elasticsearch similar to this priest. Up script would do and says exclude this particular node from the cluster.

B

So data is moved out of it and this will then move to the to the new to new elasticsearch node that is available, and once the data is moved, the ears operator can safely delete the pot, because now that it's completely empty and it's fine to delete it and then this case is handled and also like if it takes more than one hour they operate, I can just continue to try to drain. If the operator dies, it will just start back up and it will just continue from where it stopped.

B

So it's like very resilient in that way, and and lastly, the participant okay. So this is like the upgrading of a cluster cup of kubernetes and elasticsearch. We have this operator logic for doing this, and this means that now we have this training logic, it can scale out. It can move the data. Then it's also easy to like add auto scaling, because we have this functionality of training and so on already.

B

So what we implemented was like custom, auto scaling in two dimensions, so we can scale the number of No and we can scale the number of replicas inside the elasticsearch cluster. So the replicas of the shots of the indices we loop into using the horizontal part autoscaler, but it's very, very simple in kubernetes, and it basically just takes an average of like if you're scaling a CPU takes an average over all the parts and then calculate how much would be needed to actually satisfy that.

B

Whatever constraint, you said, this is too simple for us, because both it's too reactive, like it, can be scaling up and down too fast. So we added something like a duration that the CPU has to be above a certain threshold for a certain duration that you configure and we also have a cool-down. So if you just scaled up, the cooldown has to exceed before you can scale to another scaling edge and either up or down.

B

So you don't have this waiver scale up and scale down and scale up and move days all around, and in this example, we can scale up. So you see the graph here. It shows like the that the CPU is over a certain threshold. For some time we haven't.

B

We have a very simple case with just one node, which is maybe not so so safe, but it's very cost efficient where we have six shots, and if you want to scale this up, then we can add to one new node and then we can split the shots over these two nodes. If you want to scale up further, then we can split it again and have three nodes and split to charge two charts on each node, and this way we are reducing the CPU per node.

B

Obviously, and if we need to scale further, we can do so, and here we are like limited by you can say that number of max Potts, the max part you want to have, but we also limited by the minimum number of shots per node. So if you have one as minimum, you obviously cannot go below because then you would get notes with no shots on it, which is just a waste of money. Also, if you have like a limit of two, then you would kind of stop here and then you could could not scale further.

B

But this you can all configure in the in the ideas the electrics search data setup similar. If you go the other way scaling down the CPU is below a certain threshold for a duration. Then we start to scale down scale down to the three nodes. Two shots is and so on. All the way down- and here like the minimum replicas- are the boundaries. So if we have a limit of two replicas, then it would not scale further than this and also the max shots per node.

B

So if you have a max of three, then we could also not scale, even though we the minimum replicas, might allow it. And then we have another safety which is max disk users, and this is to prevent that we scale down, but the less.

B

We have so much the less research data that we actually will fill up the disk and cause problems that way, so we also checking into elasticsearch what are this disk, the users of the different notes, and then we check before we do an auto scaling action if it can actually happen without reaching this boundary. So it's just a safety measure.

B

Another thing is that if something goes wrong- and this is mostly when scaling down and the cost is not in green state- the operator backs off and it doesn't do anything so it just waits for the cluster to be green either. I can do it by itself or if an operator like a human operator is needed, then they will not do anything until the clusters green again and someone has like it might need that you need to yeah whatever.

B

So this is like an all safety to prevent that it yeah misses of the cluster further than it was already. It could be that you're scaling down. Then you lose a node because of the cloud provide or whatever at the same time, and then you didn't replicate enough and and and and you are in a bad situation, but it tries to like protect you as much as possible the same when doing an upgrade. It will not continue an upgrade if the clusters is not green, so I mentioned there are two dimensions for scaling.

B

The other one is like scaling the replicas of the indices in elasticsearch. So here we have an example where we just have one shot and like adding another node wouldn't do.

C

B

Because when we just have a node with no shots on it, so what we want to do is we want to scale the the number of replicas for the for the index and then we just add another shot, and then we can scale up and actually utilize the nodes. And this we can, of course, until we hit the boundary, which would be minimum number of shots per note and the maximum Potts replicas that we can scale to.

B

Just to show that this actually works in production, so this is over seven days. The green and orange is like the CPU you choose, so we can scale based on CPU. This fits into our use case and the blue line is the number of replicas or number of cuts in this particular.

B

So this is only showing de Germany, so only one of these elastic search data sets so only a subset of the whole cluster, but it scales for like from like 22 or so to 44 or 45, and the reason that it that it here is so steep. You can see it when I zoom in here. You see the scale up. It looks for some time. Okay, the CPU is over a certain threshold and then it scales up by a lot.

B

This is because we basically have a case where we had why we are duplicating the replicas, because this is how we can how it configure to scale in our case, and then later on after people have looked at all the things and shop during the evening, then it starts to scale down slowly and you can actually see it scaled down within one hour.

B

So back to the to the lower limit that we have configured and this way we are saving instead of running with the high line of 40 for every time all the week we are running most of the time with only 22 or so.

B

So so what are some of the lessons learned and and takeaways I? Think that, like if you have all these batch scripts lying around for operating all your staple things, turn them into operators. I think there's a cost up front of building an operator, but in the long run it's really deserved because you yeah you, you move a lot of manual effort away from yourself and can focus on other things.

B

Of course, if you have an elastic search first of three nodes, only storing logs and okay, if it's unavailable for a little time, then maybe you don't need this. But if you have a bigger, bigger setup, then I think it's definitely the way to go and kubernetes is one way to do it. You could also implement all these things in AWS or Google cloud without kubernetes, I think but I think ranee.

B

This just makes it much easier because you have this API you have like if you use the stateful set like we do you get all this volume claim support and all these things out of the box. So you get a lot of things for free that you don't actually have to implement.

B

Another thing that we also like learn or that I kept like saying to the other people in the team is like assume the operator can die at any point when you're writing the code. It's like easy to say: okay, we just need to put something in elasticsearch via the API. Then we store this in a local variable in our go program, and then we do the next thing. But if you, if the operator dies because the cluster is scaling down and there has to move the operator part, then you are losing this state.

B

So always do things where you store the state on the custom resource or wherever but store the state somewhere outside of the operator. Every time you do something, so you can always come back and continue where I left off. This is very important and they have this in mind when you're, when you're writing the code. I also think like. Maybe this is a personal opinion but start simple and add abstractions only when you need it.

B

So we started with the stateful set, be 0 where we try to just use that this was too simple, and then we identified why we needed this abstraction. We needed to have the replica stole some whales and we needed to be able to update the resource without affecting the existing replicas and like starting the thing to scale up and down. Unexpectedly, we might add a like an abstraction later on.

B

That makes it simpler to just define and elasticsearch cluster and get things out of the box right now you have to deploy the master yourself just a deployment. The stateful set, it's not that hard, but it it is like an effort from the user point of view, and this we could maybe abstract in the future. But this was not our goal. Our goal was to like make all these batch scripts and so on encoded into an operator, so we don't have to do so much about scaling and and updating and so on.

B

Lastly, I just want to mention some of our own Souls projects, so realistic search operator is obviously one kubernetes and native. This is where we run our configuration for kubernetes production clusters, so this is how we can figure all our clusters you can get inspired by this. We also have post press operator and I think like elasticsearch over radio, it's much simpler, because you don't have this like single master that you have in Postgres.

B

So it's a much simpler problem to solve and I think it also applies to like, like I, said, Cassandra and other things, and then we recently have one of my colleagues released price on kubernetes operator framework, so normally the most of the things I go. They offer our elastic search operators also and go, but the his team was using Python, so they developed their like pythonic framework called cup, and this, like a very nice little framework that abstracts a lot of things away. So you can like just a few functions in Python code.

B

Then you can write an operator and it takes care of managing in bins and handling the customer resources and all this for you. So it's quite interesting, also I think you should check it out and that's it. Thank you very much for attending.

A

We have 10 minutes for questions.

D

I have a question regarding the scaling and scale out. The scaling and scale out are relevant only to the pods right. You are not scaling down and up the kubernetes cluster as well I mean if I will run a kubernetes cluster based on ec2, then it might, it will scale only the pods, but not the cluster. The ec2 will still be running yeah.

B

So we use a cluster autoscaler in the clusters. So if the, if the parts demand for parts are less than what the cluster provides of resources, then it will scale down, but obviously it will not force the elasticsearch thruster to scale down. So it's only when, if you, if you scale down the elasticsearch class, that in the notes might also be scaled down underneath the ec2 instances, okay,.

E

F

Hi, did you have any solution to safely shut down and startup elasticsearch, so.

B

The question is: if I solution for say please no.

F

Not an operator to implement any logic, you've heard a distributed, elasticsearch cluster, to stop and start it properly, because if you kill one of the nodes usually starts replicating to the others. Yeah.

B

What do you mean like turning down the whole cluster and then.

F

Bring you back up you starting stopping the obvious. We don't this.

B

We don't have we haven't, had we don't have this use case, but I mean maybe it's feel free to open issues on our open source repository. If you need something that we didn't think about.

C

Yeah, so thanks for the talk, it was really interesting. You explained how you did it. I was actually wondering very much the reason why you did it I mean, as you said earlier, you could do the same. Using some ec2 instances, you decided to implement an operator to manage it on kubernetes. It's a lot of effort, so I mean I'm sure you, you had very good reason to take these this choice. What are these good reasons and do you validate them so.

B

Like I said that we declared kubernetes the default deployment target, and this is for the whole organization for several reasons. One thing is that we have like a CH DD system that is integrated with kubernetes and it's much easier to use, and it's much like nice of our developers to deploy against kubernetes and what we had before was like one of the reasons that we are moving to kubernetes.

B

This also like automatically means that it makes sense to actually run these try to run, reduce workloads. You could say lecture to staple set our stateful application it's more difficult, but they were kind of needing this anyway, they they had like they were managing it, but it wasn't like sufficient. They used a lot of time on managing it manually, so they needed to implement something like this anyway and, like I, said I.

B

Think kubernetes just provides a lot of the features out of the box, makes it easier so yeah and and by having this, we can also like offer it to the open source community. Other people can benefit from it. We can improve. It. I think also like that. If the code is so abstracted in the code that it can be used also to implement and an operator for Cassandra or something we haven't ventured there yet, but it's I think it's possible.

B

So these are some of the side effects. I mean in the end it's kind of the organization going this way and I think it's good reasons for it, but yeah it could also be done otherwise, but yeah.

E

Yeah and I realized. This was only announced yesterday, but I wondered whether you'd had an opportunity to compare elastics own operator again.

B

We have been a little bit talk, I have not been so much involved, but I haven't I haven't seen it at all. Yet so I cannot comment at all on this. Thank.

E

G

Hi I would be interested in how large are these elasticsearch data sets when you scale them up and down how much data is moved around yeah.

B

This is actually good questions that'd that I, don't know the answer to so I said in the beginning that I'm not so much in the elasticsearch side and I also don't manage this now so I actually don't know. Oliver ocean from my from from solando is, is the one to ask I can I can tweet about him and then you can can ask this way. He would he would know it.

A

Any other questions.

H

How many clusters are managed by one operator and how do you handle the relationship between the two.

B

Essentially, it can manage any kind of any number of clusters. It will it. You have to deploy the masters yourself so, depending on how you configure the elasticsearch datasets to connect to which master be part of which cluster, but the operator doesn't really care about this. It cares about the data sets and they each have a service that is pointing into the cluster day apart off, so it will, it can operate any number.

B

You can also limit it to a certain namespace and you can also limit it through just a single elasticsearch data set that has a certain annotation. That means only control this one, so you can also run multiple controllers and they won't operate us and it will not like interfere with each other. But there's no like rules for this. You can do it. How you want how it see fit. I cannot here.

H

Have you run into any scalability issues, sort of that the operator didn't manage to sort of handle all the classes it had to manage? So you.

B

Could have two around the question is if we had any scalability issues that we were running it in production for around three weeks, and we had one issue, which was that we reached the limit of our network all in network size in the kubernetes cluster. So it wasn't really because of the operator, but that meant that we basically had to reduce the number of IPs reallocate to his node. So we could get more IP space. This was kind of interesting to see.

B

It didn't really have an impact because it just meant new nodes didn't come up, but we haven't. We haven't seen anything like. Obviously we have fixed issues along the way, but since production we haven't seen issues with this auto scaling of the elasticsearch itself.

I

Hi here we heard probably about the coop DB operator on the markets. They do opera. They operate actually a lot of different storages elasticsearch as well. If so, do you have any like plan to merge feature sets between these two operators to like to leverage both operators experience into one I.

B

Think and correct me if I'm wrong, I think the approach is a little bit different in the way that I see that they take it more from the abstraction point of view.

B

That's how I see it that they like try to make it nice for the users, and you just have a simple definition and you just get a full cluster out of this and we kind of approach it in my mind, from the other side, where we try to first fix all the like operating the hard parts of operating it, especially for our use case and we don't haven't fixed, we haven't focused on like making it super easy to just go to our page and just deploy be a one simple Yama file.

B

So this would be like the things where we could maybe converge, but yeah we haven't looked so much into it, but when we looked around for something before develop our own, that wasn't really something that that did what we wanted it to do on the operation sides on the heavy heavy lifting there.

J

So any question I mean how is operator one per namespace or you have namespace for operator and the different clusters in different namespaces. You.

B

Can run it, however, you, like you, can say either one operator can only charge this namespace where it's running or you can control with a bug rules what it gets access to. We run it just like in one in the cluster and then it can like manage any number of elasticsearch clusters within this kubernetes cluster, but.

J

Do you how you want, but you can run one pair of names, yeah.

B

We just we have one just for the whole cluster and then it just manages no matter the namespace, because we don't like respect the namespace in our setup. Yeah.

J

But it is quite dangerous because if you want to have one Operator for all, namespaces update of operator can broke me database in all namespaces yeah.

B

A

We might have time for one more short. Question may have one all right in that case. Thank you very much Mikkel and don't forget to please rate it on the app. Thank you thank.

B