Cloud Native Computing Foundation Kubernetes Forum Delhi 2020, 27 Feb 2020

Previous Meeting

⏯

youtube image

►

From YouTube: Story of a Kubernetes Go Operator - Aravind Putrevu, Elastic

Description

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects - Learn more at https://kubecon.io

Story of a Kubernetes Go Operator - Aravind Putrevu, Elastic

Kubernetes is the undisputed leader in container orchestration world. But simply running all sorts of applications on Kubernetes are not so easy. The Operator Framework is an open source project that provides developer and runtime Kubernetes tools, enabling you to accelerate the development of an Operator. Operators are purpose-built to run a Kubernetes application, with operational knowledge baked in. They will be smarter and more tailored than generic tools. The cloud-like capabilities that are encoded into the Operator code can provide an advanced user experience, automating such features as updates, backups and scaling. All of this is accomplished using standard K8s tools, CLI and API. Elasticsearch is one of the most downloaded and used container on docker hub. This talk explains our journey and thoughts in writing a GoLang based operator for Elasticsearch.

https://sched.co/YWJT

A

So welcome everyone, after the small break, III, think I'm audible to the last row. Like am I audible, clear, great, so I, Marvin and I work at a company called elastic which the creator of elastic search, locks, Devon and all these open-source projects, and today we are going to talk about the story of how we have developed operator on, go to be like to run elastic, search on kubernetes. So before I get started. This is a usual practice that I do and a lot of conferences. Where and like a lot.

A

Many people a lot many times like people take photos of the speaker, but they don't take photos to the attendees. So this is like a trend that I do and before we get started. If you call can stand up, I'll take a photo selfie with you and then like I'll post on Twitter like, and we all can like you can. You all can be part of the session as well. So if you can stand up and stretch for a while and say what what a power or whatever yeah.

A

A

Thank you. Thank you. That's good, so yeah. How many of you know elasticsearch or heard elasticsearch, how many of you use in production so for the people who didn't raise hands are probably like. Oh I have heard about it, but I really don't know what it is. So I'd give you a brief, very brief intro and then we will talk about more more interesting stuff.

A

So if you have used uber or if you have given a point, A and point B in any of these ride-sharing apps, and you try to do the try to map the who the rider get a cab actually or the dried or dry wall painting happens on elasticsearch so basically like trying to search for a right, you're, literally doing, search for rides or even when you are trying to order your favorite food like you, are searching on elasticsearch and in a lot of these places.

A

Similarly seamless experience a lot of these digital experience that are powered today are powered by a distributed system called elastic search. But today we are not really like talking more about that. We are talking about something different. Like you know, there are, there are multiple ways that you could run elastic search I mean. Are the entire stack we're going to talk about? How do you run this? What are the various ways or how we have developed a specific way to run on to run it on kubernetes?

A

So a lot, many communities is like the the big container orchestration platform and a lot. Many organizations have kubernetes. First strategy like develop, deploy, run, do anything and everything, but on kubernetes right, so we even have kubernetes operated and then like it's called elastic cloud and kubernetes. You could deploy run or like develop or like even do. A lot of these things are on, on the end, run on the entire stack on the kubernetes platform. So all sorts of versions, vanilla, open chef like even GAE a kay.

A

He cares in any any platform the type of kubernetes you can use that. Similarly, like integrated and also like, you, have different sort of architectural templates, but all of these are powered by a common cold framework or the API structure that rims are even uneven, like the other. Other speakers have spoken about. It's called custom resource definitions or or otherwise these are called as operators. How many of you use operators in communities today operator heard about operator?

A

No okay cool, like I, look more about operators, but yeah like we will, will talk more in detail about the problems as well. So operators are the clients of this API like they act as controllers. They have a custom resource definition through which you can deploy any general purpose software. So there are multiple you know, base of like multiple general purpose. Software like radius Kafka, my sequel and all of these are data bases, distributed systems stateful systems. More importantly, this share data between them between inside the same system.

A

So all of this is really complicated in a way to run in regular infrastructure. How do you run in a scalable, much scalable or much optimized platform like kubernetes? So that's what operators help you solve? That's how you can run all of these complex systems, much easier by automating, the regular processes so like no I I didn't really get it. If you feel in that manner, I have I, have a UI, so I could explain it much better.

A

So kubernetes has API server everyone between write, API server and then the API server will be the one. Just like a state machine. Wherein you ask talk to it to get more details, you do cube. Ctl will get parts, it literally hits a API server or more in between cache and like try to get the parts. Try to get the resources that are deployed in communities so here custom resource depletion is something like a template that is there on.

A

That is gate given to the API server, which enforces a specific sort of state with the application that is running on top of it so elasticsearch, even though it's a system, it is application in the kubernetes place, and then we are running as just like an application. So, but what does it? What more? It does not just doing this suppose it spins off it watches the resource spins off a part, especially in a distributed system like elasticsearch.

A

You have multiple nodes, so nodes are probably like you could think of like parts or containers which runs this software and ordinates with everything. So you see and watch all the parts it does create, update and delete. Get the resource pack also like creates the expected resources. You know that the way my expected is highlighted will will I'll explain more in that in the deeper terms like why and what are the various problems that that are there.

A

So there are, there are multiple things when we say expected in a specifically state-based machine like you have you have a specific sort of spec that you want to deploy, for example, say that you have I want to deploy a three master node and two data node based elastic search cluster right. So then you are having a five node cluster right. Then you might also want to do all the wiring it between them all the ApS that are exposed and a lot of these details.

A

So this basic template, through which the operator, when you ask the API server it, will get deployed, but there are many things more than that, not just deploying these things, but there are many things like how does the elastic search works more? Where are the backups stored or probably, what are the naming conventions whether the nodes are TLS, are having a TLS encryption in between them or are there like mayor the secret stored? Then many many things like many many things that are happens in a distributed system? This is not per se with elastic search.

A

It can also happen with, like other systems which need which are stateful in nature. That's where, like the stateful sets even come into play so like much more more and more details, not just that. You also have a HTTP service through which, like connects all of them like exposes the API, so majority of the stuff. Also, everything gets reconcile like every time when you for something to be done. There are all of these operations that needs to happen.

A

Suppose you add one more node, then it needs to join the cluster get into all of these share. All the data share, all the TLS and everything certificates, everything and be part of the cluster start working for the problem that you want to solve. So these are many many things like stateful sets enable us or any stateful system to get all of this pretty easily.

A

Now, that is how you end up interacting with the software like elasticsearch like this is the end goal wherein you you deploy, run and, like you finally start looking at something like elasticsearch, but again more than that, not just deploying this configuration for one time. You also might want to upgrade the cluster over a period of time like because there are newer versions, newer features and a lot of these things.

A

You might also look for migrating the data in some cases where you have better Hardware into your kubernetes platform, and you might want to like just roll off and like put put in something right. You could also do no drain the node and like replace the part. So how do you do all of this? All of these updates, so operators are the custom resource. Definitions are controllers which helps you to do all of this.

A

You automate this logic into a place and makes elasticsearch like a resource to be managed on the kubernetes, so you could literally do it for your application as well, if it is if it requires, but this is a story that I am trying to tell you like how do you do it, for even you are application, so there are specific tools and lips that we followed under the kubernetes special interest group, six, there's something called Q builder, which is really popular in building this.

A

You know the custom dispose definitions which in turn, users like the controller tools and controller runtime. You could also go and like look at these and like try to develop this stuff, but these are some some little details now all this is like kind of power or the verbally. If I tell you like, maybe you might not believe that whether this is possible? How do I get and do it so there two ways to do kubernetes.

A

Operators, one way is to like use the go-go Lang and deploy or create and write controller and put it into there like put it into your kubernetes, using using still using animal. But if you want to see the code and how this reconciliation, how this entire process of logic works. This is how it is like there is. There is a specific method or a function called reconcile where in each and everything like services, certificates, nodes or how the client and other secrets are configured.

A

Let us take a look at one specific service which helps you do like get a deeper understanding of what happens now, at the same point of time, I'm trying to tell a story here not just like what we have done, the problems that we also faced when we are run trying to run a massive system like elasticsearch in kubernetes, another massive system like kubernetes, which is also having a lot of moving modules. So, if you see here, we we follow a specific naming structure around pods.

A

We also like do specific, like particular ports, that we enable do a lot of checks. Whether this is already created, or not, most importantly, sometimes when you kind of like commit or give an issue like I, want to I want to delete this part and the part still exists by the time you try to create another part. The other part still get is existing, so there are many multiple conflict and other areas that that does again and again so.

A

So all of these, like there are multiple methods or multiple functions which are sequential in nature and also like returns early when there is a issue, so you keep doing it until you get into a desired state. Actually, so until you get to the state that you want to do like you have set a state in your amble file, it keeps happening. The controller ensures that this hunt entire thing, but this is a logic that is written by us. So if you are writing your application, it is pretty abstract in nature.

A

You probably might want to manage this through your Tube Edmund I mean the the administrator of kubernetes cluster, so the two basic or primary themes that I want to like like explore here, is like the operator lives in the past, where in whenever we try to create a specific part and see if there are ok, this part isn't exist and I just want to like create another pod like you might end up creating more or you might not get part that you want. We have already created.

A

So this is week because sometimes that the the client uses a cash reader where in the cache is stored and like if you would not be able to like get the latest resources or the latest updates, but it's also very tended because in some cases you can't go and call API server all the times because there's a performance limitations.

A

So that is one another one problem that we faced when we are trying to build this thing, but this creates multiple downward spirals and like a lot of other problems that we face in a regular distributed system. So, for example, we could continue, keep creating the pods forever pod missing, create one.

A

So we because it's a logic that we write and that could happen as well or we could also come into a situation like if there is a split brain which is very specific to elasticsearch a problem it could and it doesn't reach the quorum because elasticsearch is a distributed system and you have consensus or several things that helps you to maintain high availability.

A

If we don't reach that quorum, you kind of end up having this problem as well, then double rolling upgrade suppose you, you kind of like come on to upgrade and you try deleted apart recreated it and you get it, it doesn't work. So again you try to recreate it. So these are some of the problems that we be understood and, like you, keep doing this reconciliation forever. Sometimes when you are trading this oke operator.

A

So what we can do is something that we already like we thought of. For example, there is no way that we can't live without running operator or running on kubernetes, so we there are two major themes like optimistic concurrency that that could specifically help optimistic. Concurrency is a strategy that is used in transaction or a relational databases structure wherein, like it's a model of concurrency, a way wherein, like you, you kind of read the data, but you before you commit your changes.

A

You check whether the data that we have shred by you is not changed so by somebody else. So this is this should happen mostly in less data retention or less data contention places.

A

So what we dan is like, we be kind of like named name the particular parts, with a specific name that we know so so that, like it's always topics are prefixed with a specific name. There is also something called the source version which you could take help of.

A

Similarly, you also have similar sort of things when you want try to delete this parts like we also get something like UID for the pod UID that we have seen, and you need to keep track of it to ensure that you are not deleting it if the UID is not there. So these are some some processes that we we found. So the quick takeaways here is like used deterministic, naming that you can track and also like always assume a stale Cash like because the client uses a cached reader.

A

Then you could also look at like the cons. Reconciliation entirely should be idempotent, like the state should reach in a certain manner. So this is the learning when we are trying to do operator. These are some best practices. There might be more as well as we as we develop more and more into different platforms. I would like try to use it for a large scale. It might also be like imperative and like more and more and more things might come so, but also like.

A

We feel that it's important, that you empower users and let people decide what they want to do with with the kubernetes. So a lot many people might have a specific sort of pod template or guaranteed or run by their security teams, etc. So we offered them a specific, not just defaults.

A

We offer them a structure so that, like they could go and deploy in a say in a specific policy, like no affinity rules are, I mean, maybe like even their desired ram consumption rate all of this, so so that is one another thing that you could also do not like.

A

So when we talk about scale, I mean I'm a stateful set, and when we talk about disability systems, one thing that, like one thing that comes out open, is like the data. So how do you ensure that the data data is always there when you kill a node? How do you ensure that the same data comes up or or the data rebalancing doesn't happen, and that is what stateful sets meant to do right. It binds you with the pod and the persistent volumes get binded to a specific.

A

It gets minded again and again so that, like you, don't need to create this entire volume again so yeah. So we have a specific structure or we have implemented that we also have some problems around that as well through which, like we have implemented. So, for example, if you have two stateful sets, one is like master notes and, like you also have like data nodes, then imagine that you have binded them with the volume claim templates made like say you have share in a different route provider. It might be different sort of stories class.

A

But if you are using your own file system, it might be again different, but if there are that sort of truck storage classes- and you bind it with the thing so whenever you you see a specific node or a pod, both of your data still be the same, and it will try to like reconnect to the same volume that you have attached to that part. So this is one one such thing that that helps like really large scales, large scale. But again all of this will come with ifs and buts.

A

So so that way like you can take control of what sort of state that you want to maintain, and all of this logic resides in the operator that you don't need to manage. So this is what is managed by us and helps you to develop solve your own business problems. Not working only on elastic starts running elastic search because not you might not want to become an elastic search expert. You might want to solve your own business problem.

A

So at the same time you there are other things like. We make sure that the the until the cluster is green, the the the the consolation or everything should be stable and should be present, but otherwise, like you, will risk losing the data. So this also puts you in a pressure like wherein, if you are, if you are having less resources, probably there might be a chance of latency, but we will also disable the shards like the the shards or a data rebalancing it between.

A

So all of this is automatically happening, but it is helpful if you also know what, like what sort of things that are achieved using this yeah so end of the day like when you delete a part it automatically gets created. It gets attached to the thing so should I build operator, should I should I be using an operator. So that is a question that everyone has when you are running.

A

Applications on kubernetes, so I just want to like give us some huge final summary wherein, if you have complex business logic, and you have systems which interact with data, are stateful systems like say, trade systems, business-related trade systems, and you want to preserve data or in-memory stuff, then, probably like you might want to take a look at operators. We really tried. We have automated a lot of stuff, but that is one another thing, but also like.

A

If there is complex business logic itself like we're in in dealing in computing stuff in ensuring that things are stable, which are internal to your application, then also, you probably might want to look at an abstract if I that logic, so that, if whoever is deploying and scaling like your support people, they would definitely enjoy doing running your application at scale when there is a demand so also like you could think of using something like help charts.

A

We also have a health chart, but if not, you are a regular app wherein you want to see ACD functionality and, like you, want to do a continuous deployment, you could create a healthy art and like do versioning and start deploying version based applications that still works out to be well. But this is much on a layer above the whole chart and like helps you to do more more stuff, so more or less like when we also test. There are multiple things that we do testing the operator is also not so simple.

A

Like you see here, it is. It is also like really really complex, and you need to take a call in like whether I want to do it or whether, whether whether I don't want to like, do it and like go and go into something like help.

A

So if you, if you are like interested like please take this away and like I, would like to hear the feedback about the project and learn more and also like you might stand a chance to be moving home. So so yeah. That's it from my side and thanks for coming.