Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2019 (Barcelona), 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Managing CRDs and Operators in Practice - Zhen Zhang, Alibaba & Wei Guo, Ant Financial

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Managing CRDs and Operators in Practice - Zhen Zhang, Alibaba & Wei Guo, Ant Financial

Manage one Operator is easy, but manage tons of Operators is hard, especially, when those Operators come from various teams throughout the country. In this talk, we will share experience & practices of developing and managing CRD + Operators across teams, with most of these developers are new to K8S API. We will talk about CRD standard we created to restrict common labels and fields for different kinds of CRD and ensure all CRD and Operators be formed to K8S conventions. We will share our CRD review process with code and config checking tools as well. We will then walk through a real use case to demo how these technics apply to refactoring a traditional middleware app into Operator. We will share challenges of using Helm to package these Operators, including Service Catalog integration and dependency mgmt. We believe this experience applies to any public cloud as well.

https://sched.co/MPXH

A

Hello: everyone, I'm, Risha and I work on the link, Rd surface mesh and I'm gonna, introduce you to Jen will be talking to us about how to manage CR, DS and operators in practice. He is a software engineer at Alibaba and he works on cluster management. So let's hear what he has to say.

B

Hello guys, so how many? How many operator talks have you heard in these days? What do we have? Not another competitor? Ok, our personal have three three operator talks, so I think I believe I'm the winner. Then, in this talk we will talk about the general ways to manage ceoddi customer resource definition and the operators, so I will not focus on one particular operators, so Who am I.

B

I'm Johnson from Alibaba cloud for record Alibaba cloud is also maybe this largest cow providing the world, and you may contact me through slack or just just a meaning, email.

B

I begin this talk, because in Alibaba we are undergo a journey to cognitive.

B

Actually Alibaba begins journey to cognitive. In very early days in in 2011, we had already moved our virtual machine application from virtual machine to linear clinics containers and in 2015 we introduced tok, tok image and tocopherols, and that we also built our first orchestra continuous orchestration platform using docker swamp, but in two thousand of in eighteen.

B

It is very clear to us that kubernetes standard, declarative, API of kubernetes is much superior and he has great it customizability. So we began to our began to migrate our container platform from swan, dr. Swan, to kubernetes and during the process we are building our many enhancements and are also helping our developers to build their operators.

B

So currently we have around like one more than 100 customized resource definition. Those are developed by more than 10 teams across the country. Actually, most of them are not very good at kubernetes and out of these customers was one third of them is built in-house. So in other words, there is no like 70% of the customer resource is in just using the our opens over source software's, so why we are building customer resources and our operators.

B

First of all, we during our journey for cells to migrate to communities we find Sun. We need to give some enhancement to communities. You know now into several ways, so we we kind of build us our own staples as replacements, so that we can update the port in place instead of week report. Every time also. We also builds custom I'll control SOS so that we integrate our company's support with our own infrastructures, such as config manner, mansion management database.

B

You the other hand. We also have many stables. They will stay for applications. Most of them come from meadow where team, for example, message kills. So we are helping them to build their own operators and also we have many teams, take teachers more. They just used some open source open source software as such as could be flow. They will install the them so from those software in our clusters. So most of the t, software now shipped with some kind of customary resource resource, and there are some operators within.

B

So when we manage some stuff, we we have to know who the user and our partners, so the first user of course, is just ourself and I believe most of most of the people here come to this category, who I named it captain and it's actually just a beautiful name for the administrator of their crafters.

B

These people have green knowledge about the kubernetes and also the goal languages, but because come with the grid power, because you have the administrator privilege come with the power. Is your responsibility. You have to make your clusters in order to have to make them running all the time. So sometimes we get a kind of kind of obsessive-compulsive disorders.

B

So nowadays we have to also be psychologists just in communities instead out of engineer together, water turns so another kind of person. We call we name it cowboy is just a name for other teams.

B

They may be some expert for some at home and they have limited- maybe maybe not also great knowledge about kubernetes but very still, very limited compared for us, but they do have some great knowledge about about some domain and but they always mess something up in your crafters and always outsmart you in surprising ways the sort kind of piece person we animated busy man, but actually they are also engineers, but they are experts in some particular applications.

B

They have in-depth knowledge about the application as there and there's business projects, but they basically have no or little knowledge about the kubernetes or : google language itself.

B

They are very curious about kubernetes, and but they don't have. Many cannot invest many time in learning them. Of course, they also come with a lot of legacy in those non chrome native days.

B

Also, some of those engineer are just forced to write some operators because we are driving them so as a captain, our crafters, how we organize those those produce, three kind or persons so that we can build better and good operators.

B

So here are the first and ironically, the first problem is how to make them right. Operators actually right operator, is not very easy, so the solution, I believe is, is to make sure that, whether whether or not you just actually need to write operators, I will talk about that later. But in many time you just need. Maybe you just need some transistors that I will surprise many cases. So if you do have some business application logic, you need the right rhythm in the interpreters, so you have to learn how to write operators.

B

For starters, they have to come overcome many difficulties they have to overcome. The concept overload our communities and they are so how to break language barriers, at least that the you you can build some operators.

B

But if you fail to come overcome the language barrier, many many hours developers come from Java world, so they just get color feeds and and maybe they're just using Java client library, and they call kubernetes in your external way.

B

So they do not never get a chance to write a native kubernetes operators, but in the end, if they do come across the language barrier and the company's concept today, they also have to learn the controller pattern there I believe there is no shortcut here, but here obviously we do think that if you can use copy control or operator to case you can quit reading speed up the learning process. So we come out with some some some tips to speed speed. A lot of the learning process.

B

I will just skip to the last two two tips is we have to give them easy access to a good sample operators.

B

For example, you have to set up some kind of your own version of Awesome operators, your own version of operator hubs, so that you can just follow the good examples another you have to make some standards and offer some best practice, because those developers do they just they can write controllers in their own in their own understandings.

B

So here is what what standards? Basically in the profit program, we have to solving standards to come up with some naming conventions, the developer.

B

They can when, when they do start to write or filter or customer resources, they tend to make mistakes in naming those resources of, for example, they can name exactly exactly the same as the poutine kubernetes resources that will make gave many confusion when, when use them, they may also give the resources a very unclear group names which is also very variable and because they also don't have access to some kubernetes suggest recommended labels. We will come up with some their own names, so we just make some kind of C audio standards in first row.

B

They have gave them some naming conventions in Alibaba. We made some simple rules. For example, you have to give a clear name such as a diamond, it's a vacuous names and the you. You can also have your API groups, which may contain your your business units in your companies. We also define some common resource resource category. Category is Krishna introduced a field in customary source definition.

B

For example, we intervene apps for workloads, customized sources for the those apps. Because of my resources. We we enforce that they must provide a scalable resources and also a rural control definition. We will recover that later and we also make some commonly across company.

B

When we went a developer have followed our rules to write the operators, they overcome a deal to often make some common mistakes. One of the mistakes here in the first item, I believe, is a wrong committed. Many of our operators, developing open source project, also made that mistake is that they tend to use operators as a replacement for package management management.

B

They simply wrap the wraptor underneath resources in our operation and that doesn't include any operational logics in the operation. So we believe that it's all over queue and it doesn't waste our time for our developers, so we will command them. If you don't have any specific application, operational objects, just use, helm or other package management management solution. Another common mistake they make is that they tend to reinvent the wheel because they don't the developers tend to. They don't have what broad access to our kubernetes resources.

B

For example, they don't have a great understanding about stateful set or a config map. They tend to.

B

Add many fields in the customer resource customer resource, so we recommend them until you reuse as many as possible those those resources building in kubernetes also they have tried to. They also try to. We try to return the controller cancellation. They always try to retry encounter an errors or they tend to always wait for some time consuming operation, so we will always recommend them and to test for all the all.

B

These cases just append the work item into work, accuse and last last mistakes, just they tend to use old imperative wave our design, design controllers designed this design software. So it's a have to converge to two new declarative design.

B

Another problem is when they have finished their the operators. They want to get the operator and as customer resource installed under in the new clusters, but unfortunately we asked administrator of the caster's we we are the only person to have administrators provide, so they have to come up comfort to us for help.

B

Unfortunately, in in in previous days, we use a man with the hole request and request and process process is done manually. In fact, that in here is an example that we use chat application as a channel to serve those CRD or access requests. I will not go through the detail of these conversations, but it's clear that it's to use free human, pre-tax or human language to describe the target. We saw a peck to describe the source definition.

B

Resource definition is very inefficient and a time consuming and when we will try to make try to make the bag manifest manually, you tend to. It is very, very prone to errors and always come comes with bad user experience. And, what's more, is that you, if you have a more casters or to use a won't just edit or modify some of the APEC's rules, the whole process musta to it or all over again? So it's just not sustainable and not repeatable.

B

So we may ask why not just using helm, you can use it as a sardine store just put customers for definition, manifesting your charts. We think that if you're using home because helmet is just as a giant as you do studio servers, you just give up all your responsibility to manage the crafters. So we don't take this approach, so the solutions is that we introduce. Another repository is for resource metadata for our partners.

B

So in this repository host meta configuration such as customary sauce definition up a truth, services count and maybe some web hook. Web kook related configurations also served as a awesome. Awesome operate Lister for internally. So yet you have a list of the operator CR the CRD transformed corresponding to the operators. You also have a developer name so that other developer can contact them and communicate.

B

They also have a link to the Seattle operator repository so that when we never add other developer want to reference or call those API they can access to the type definition and the client said it's also a place where we we make review review process.

B

So here come to review the CI CD based access to reveal process. So when a developer wants to commit some to make change to some customers or definition or can access to a new cluster, they just folks take the folks to resource meta meta data and made some change and sent sent a murderer's to across an administrator plasma. Treater will try to review whether they of those change a proper or whether they confirm to our standards.

B

So once they have accept those fed those who know to request the Jenkins or others continued integration, assistance can can take over and the tudors syntax check. For example, you could do a dry run against the kubernetes api server. You can also write a script to make sure that all that your custom original definition or also the count of labels just compliant to our our c rd standards and the junkies also makes the last jab says that is to apply those resources one by one to one of those two thrusters.

B

So here there's some missing pieces here is that we know, and so when, when we to the run Jenkins to the test against those resources, there's some common common problem. We found that, for example, one when you want to a can access to a clusters, you have to provide a request to a kuba config or just provide a name for the service counter you want to you want to get, but if it doesn't provide the necessary few necessary resources, it's definitely definitely not gonna works.

B

Also you can you can test here. Is that you can check whether or not your provider cut was named is has conflicts with other building kubernetes names. Also, you can check whether does those name has already being used by other by other team, so that it needs a very good chance to late just rename. Those resources for the human for the Catherine Streeters, with the common review problem is that many of them developer to just want to get access permissions, for example, they're just one.

B

Basically, we want all those permission to pour to note every staff so that we can. We have to question them in a guild limb. Let them make to make the reasoning so also. We have to make sure that the developer doesn't try to try to write a web hook, for example, phone for some internal internal type like ports or nose, because if you you write web fuga for some such core types, core resources, we have to make sure that they have made every made effort to ensure the reliability of the web servers.

B

So here is the last pieces to apply the resources into the kubernetes. You have to have the new care to give the ability to request for could be could be configured so here we make our little little sure that we we define our own customer resource called, could be configure request. You can fill in your username for the public, config and corresponding groups, and and it's a feeling, a email address or some chat. Some trade chat accounts so that we can send those could be config files to write directly to you.

B

Another little utility is that we've made another an efficient service counts so that whenever web hook web hooks server web full server startup, you can mount extra serving credential credential from the service from the service account.

B

So when our developer has set up two crafters correctly, they also want was that that operators can get good throughout. You know you can roll out and integrated well with other CI OCD pipelines. I mean here. The resource here at me is that for operators they usually is his come is actually at one type of workload. So you want to deploy or deploy workload online. You have to go through all this order pipeline, for example, why not?

B

When a developer 4matic you manicure operators, they change their medical implementation, they just commit the code in the report code repository and may let Jenkins take over and build a code appears image and push to the doc registry here come. The first problem is that Jenkins? Don't they don't have no idea that, where to put those generators doc image into because customer in cut my resources, they have no clear standards where to put those images.

B

So in typical Peter of set house there were another row here mean other operator. The human operator from our operator will committed those communities, resource configuration into the configurable repository and the continued evolution will do some templating template rendering or do some customizing and in the energy will apply those resources into kubernetes into kubernetes api. If you have server, but even you want very reliable, reliable we're out, you have to come up with some controls. For example, when something comes wrong, you have to get have the ability to pause the rollout process.

B

You have to also have the ability to to know the progress comma to progress of the rollout, whether he have finished, whether he has just get stuck in the middle.

B

So our first problems is that we also come up with some. The local standard standards is that we ensure that each operators come each each operator muster pact with with ham chart and they have some common helm values in the data that is, for example, they have some values called image and in the in the lab or in the right side in the customer resource camels template you have, you can just render, render those those values into the correct place in the in your customary source. It manifest.

B

Another problem is that how to control the customer, Greece or customized resources? Currently, there is no no available solution in nobis in existing open source world. So we we come up with some. This kind of also customer our own customized customized resource called rollout control. It contains some common knobs to control the rules, for example, which you can have you can specify. The name of you know your customer resources, your kind, the kind of your customer resources and the possible to to control whether you can post a resume the rollouts.

B

You also can help control your your. You are the partitions it's just like in staple set. You can also control how much on the available parts is tolerant. It tolerant it in in deterring the rollout such a, which is such a lot like in Japan interpolant in the status. You can also gain a will to come as common common information, such as how many replicas are there, how many of those replicas got updated?

B

The problem is that not very.

B

Crd for customary resources contained all those knobs, so we we get, we define another customers were called the rollout control definition. It just basically defines that for for each common, real all control knobs, but that exact Jason field of the Jason pass of the of the field in the in your operators. Cr CR, DS cuz, my resources and if you do not have such capability, so you just can leave it blank divider plants so that the continued delivery system can know that it cannot support pause. It cannot support the partition, abilities.

B

Accurately that most of our operators, we found seen in kill hubs, many of them just don't provide any role of rule out your other ability. So this.

B

So we have, we have to make this such kind of roll out control rules our internally.

B

So that's never our and conclude our lessons, and so we make some standards for CRD and operators we also in for those operators with our contingency is AD pipeline reveal revealing process.

B

Lastly, we make ours: see: is ad pipeline works friendly with those operators.

B

So that's all since.