Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2018 (Shanghai), 21 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Rage Against the API-Machinery: Writing an Operator for Production - Tom Manville

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Rage Against the API-Machinery: Writing an Operator for Production - Tom Manville, Kasten & Ilya Kislenko, Kasten

An operator is a set of CustomResourceDefinitions (CRDs) that extends the Kubernetes API and a controller that handles the new API objects. Not only have the number of projects following the operator pattern exploded, but so have the number of ways to bootstrap an operator. The operator pattern is the basis for Kubernetes’ extensibility, but it is difficult to achieve the same robustness as in-tree APIs/controllers. In this talk, the speakers will present what it takes to write a production-ready Operator based on their experience developing and running Kanister in production. They will compare popular operator kits, SDKs, and guides, presenting their trade-offs. Best practices for building, testing, and API versioning will also be covered. After the talk, the audience will feel comfortable developing a production-ready operator. Familiarity with CRDs is a suggested prerequisite.

To learn more click here: https://sched.co/FuJQ

A

Hi everyone get started, I think we're on time. My name is Tom Manville, hello,.

B

My name is Elia Kazan and.

A

We work for a company called Caston whose goal is to help simplify the management of stateful applications, and today we're gonna be talking about operators in production, so Ellie and I have a lot of experience. Writing operators we've open-sourced canister, which is an operator whose domain is stateful applications and also our proprietary product k10 has a few api is some of which are based on the operator pattern.

A

So we've decided to use the operator pattern because it helped us automate and centralize our operational experience in one place in a way that is well integrated with the Korea's eco system. What we found when writing these operators was, that is very easy to get started.

A

There's many projects within the community that will help bootstrap your project, but, like anything in production, the devil is always in details, and so there's a lot of kind of things that we'll go through, that we experienced the rule of thumb that we developed try to figure out if our operators were production-ready was to compare the operators to Corcoran's, api's and controllers. By doing this, we were able to implement API. Is that felt familiar to developers and also integrate with existing tools from the community?

A

So today we're going to talk about how to extend the kubernetes api for your operator, we'll talk about the requirements and features that we think are needed for an operator in production and finally, we'll talk about the pain points that we hit when developing these operators and we go through expanding the Canaries API will define some terms and then go through some of the bootstrapping projects and for that I hand off to Ilya.

B

So what is a printer? You have an application which happily runs in kubernetes, and you know about everything about that operation about this application. Your DevOps knows everything or your ops. People knows everything. Let's call it the main knowledge of your application. It works fine, but this is the time when you want to do upgrade or you want it's time to scale down or change something um pretty much what you want to do. You want to do an action on your application and you submit custom resource.

B

What it is, I would tell you about a little bit later, so the controller the code which lives inside the kubernetes sees the custom resource, does analysis and then performs needed. Changes for your application sounds like magic, not really it's just an operator so operator domain knowledge was custom. Resources was controller. Let's talk about operator design pattern. When you creating operator, you literally moving human ops knowledge into software.

B

Each operator follow is three simple functions, its observe or watch just waiting for something to happen, then analyze what needs to be done and then act on it gets back to the first step and watch. Wait following these three things: you can support really complex operations. Also active reconciliation helps you to change application on really different ways, and do it safely or not safely depends on you also. Operators are super extensive.

B

Let's talk about what is custom resources or to be specific, what is custom resource definitions, what the difference? What the similarity in kubernetes, if you wanna, create an object. First, it needs to be defined how it's going to look like it's a little bit easy form core objects or entry objects, because the definitions already there so just creating port or deployment or replica set for them custom resources. First, we need to create customers of definition, which literally defines new API. This is kubernetes object, all of them cluster wide scope, which is important.

B

We will see a little bit later why this is in tree API and it adds new endpoint to the API server. The custom resource actually instance of the CRD. Also kubernetes api object. It can be scoped as the cluster white or namespace, and it's out of tree api is two examples. Our CRD is defining new, stable, example.com API group and the new kind of that group called crontab and our custom resources actually instance of that custom resource definition, which is also kind from top, and it has some specs they're arbitrary, JSON or Yama's.

B

It's a cron spec and the image I'm, getting back our presentation to tom who's, going to talk about the kind of API conventions so.

A

There are many types of API in communities and using them all feels very familiar. The reason for that is because there's a set of conventions that all these api is follow and so to write a really robust production operator. We recommend you do the same and follow these conventions for your api. So this document is actually the document that lists the conventions, but I'll go through some of them right now.

A

Communities API is our restful, meaning that these API objects are controlled through HTTP verbs like get and post. If these objects are named and named and potentially named spaced together these fields, name and namespace form a unique identifier for every object in your system. There are other fields that constitute the metadata for each object. There are things like labels and versioning resource versioning enable the synchronization and labels enable ways to query your object. The commune's api is declarative first, but can also be accessed through imperative requests. Most objects have a spec in a status field.

A

The spec field defines the intent your desired state of the world, and then the status field is updated by the controller. To say what the state of the world is.

A

So there's a lot of conventions, and it can be very difficult to implement these. Luckily, there's some help. So there's these these tools called operator kits, which will help you bootstrap your project a little over a year ago, when we started working on the canister project, there's two out there so they're, both called operator kit, one was released by rook and one by a giant swarm.

A

Both these projects are built around the kubernetes official go sdk called klein, go, we ended up using rook because it had more helpers to help. You initialize your api, but it turns out that these two projects are not mutually exclusive. Giant. Swarm has a bunch of libraries which help wrap client go to make it more robust and easy to use.

A

Had we started working on these operators today, they're, actually many more projects that have come out in the past year. So for those I'll hand off to Ilya, okay,.

B

Let's talk about new school frameworks, there's three like the most popular I would say this many of them, but we'll talk about only this three number one is operators decay from core OAS, no sorry from Red Hat. Actually, no it's from IBM and I check five minutes ago. It's still IBM wicked and it's a really good operators, decay. It has a really good community, a lot of talks, a lot of guides, a lot of support and they also introduce really cool concept. Called operator lifecycle manager, a pretty much operator for your operator.

B

Let's do motor tools because it's cool and pretty much the Giants answer the question who's watching the watcher, a cool concept, probably your phone production, another one is the computer. This is the child of I, see gap, API machine rim super opinionated framework. But if you love go struts go types. This one is, for you also a lot of code generation and it's super close to actual kubernetes api and how it feels like and looks like, and the last one is the meta controller.

B

If you don't like go on for some reason, and you want to write your operator into JavaScript, you can do it and also you can deploy your operator in the really fancy ways there function is the service you can do it. I will not recommend you doing it. I would encourage you to pretty much pick one of the goal based operator frameworks. They work better, they close to the kubernetes api and it will feels natural.

B

Some, as you can see, or as we described previously. Bootstrapping is relatively easy. Let's see what it takes you to get into production. You just follow a common sense in some best practices, which we will share a little bit, I mean in a few seconds. Let's talk about clients, clients is the way how you communicate with your operator. I would recommend to use the native ones. I will describe each one, our back. If you in production, you need our back period.

B

Safety, see if it's not about safety. It more like your operator should not harm your up. If it's not sure what to do, it should bail out, do not break it. Events I assume you run in kubernetes in production because you're listening to the how to write operators in production, so kubernetes in production, you probably have something like login monitoring some events listening, so the way how you integrate your operator with those beautiful tools you using default events from the kubernetes, that's the easiest way, testing I mean production code needs to be tested, I.

B

Believe it's obvious. Let's talk about clients if you're using frameworks go frameworks, especially this one. Her percent guarantee that you're getting integration with the default clients like coop Caro or some go SDKs is the example. How we can create. One of them exists, object in the kubernetes namespace, the coop cutaway, and the goal way your custom resources will be the same way if you're using proper frameworks. If you doing it from the scratch by yourself, you might get in trouble.

B

So please use frameworks, auerbach everyone afraid of our back I have no idea why it's not that complicated. But it's a double-edged sword. You can prevent something happening or you can allow something happening so be cautious. Don't forget, users are not equals server account, which means you, as the user, can do things with the coop cutter, which probably the operator which works as the service account, might not see, learn from our mistakes. We for some reason, a small, tiny mistakes and one of the rules.

B

We decided that CR these years are the same, which prevented our operator to listening. Anything we spend two days of debugging that one remember your operator at least should be watching CDs and listening CDs and getting them and for the shares you can give all access or non access. I mean you probably need something like edit, because you're going to update events, please follow principle. The lease privileges, if your operator doesn't need a need, something to be caught, do not use it use the least privileges as possible.

B

It helps in production, and this note back to Tom. Oh sorry, my bad grace no changes and when I was talking about do not harm it. You know when you're transferring knowledge into the operator. You should trust it because you're trusting your ops people and they created operator, so it should work, but from the operator perspective, you should be really careful what you're doing and when you're changing something or you're applying changes do not break your up. Please, if you're not sure, do not apply awesome.

B

We notice recently that some operators think it's a really good idea to managing or to themselves. I would say it's not a good idea. Please use some workloads like deployments or the stateful sets. They have really cool features like stay for set, for example, always guarantees that your pods will be named in their predictable fashion. Also, if you're doing any changes, they will be in the order which kinda hard to implement yourself. Also, all those workloads and objects from the doors were closed already tested by community runs in production.

B

Please use those so back to some.

A

Thankfully yeah so these the bootstrapping projects that we mentioned earlier will help set up your cogeneration. Cogeneration is really useful in for any go developer. It will help you eliminate a lot of boilerplate code and will prevent adding new bugs from that code, so these bootstrapping projects will create will help you set up the core community's API generators.

A

Four of them are listed here, so deep copy Jen will help you generate functions and then the other three help generate clients like the one that we showed earlier from client go so using cogeneration is very easy. You using annotations. These are comments above your ghost trucks. I have an example here which we pull from the canister code base, but you can find examples like this all over the place. In this example, Jenn Kline just means create the clients you can pass parameters to the generators.

A

In this case, we pass a no status mean you do not generate libraries around the status fields and finally, the last annotation here says generate the deep copy functions to implement this specific interface would called runtime object, so venting eventing is a really useful way to figure out. What's going on with your application.

A

Events are created on API objects on other API objects, and so, when you want to query them, you can describe that API object and you get a list of some information about the object, but also the events that were created for that object.

A

In this example, here the top snippet this is from canister, so we've we have an API of it called an action set, and we, if we describe that action set, we can see all the events that were created on that events are also API objects themselves and they can be queried directly using coop, cuddle or favorite. Other clients in this case could cuddle get events will list all the events in your namespace.

A

Unlike other API objects, however, you cannot they're not created in the same way so because they're created on an API object on a different API object. You have to use slightly different code to create them. So this the snippet, is similar to what we do in cannister, where we create an event recorder in the first part and then every time you want to create an event. You use a second snippet with that recorder on some object, so testing any production code should definitely have a lot of testing around it.

A

If you're using the go library, the go native test framework is pretty powerful, but it is missing a few features so some features it's missing include test Suites and matchers to check the output of your code. Quran DS uses a library called ginkgo to solve this problem. Ginkgo is a behavior driven test framework that allows annotations to describe the behavior testing and if your test fails, you can determine exactly the behavior that is broken, so this is really powerful.

A

If you're trying to make robust tests one trick that we we use for testing, our operators is to actually run the controller outside clusters. So, typically, when you run a controller, you instantiate the client configuration using in cluster configuration, so the first snippet here is what you'll see all over the place, which is in cluster config, which grabs the surface account token from that's mounted and every pod and then we'll use that to instantiate a client. What we found is that we'd like to use we'd like to use clients outside of the cluster as well.

A

So if you want to run, for example, controller in your laptop, you have to use your local context to instantiate that controller. So the second snippet is how we do that this snippet will actually, if you're running inside the cluster, it will use your in cluster config and, if you're running outside the cluster, it will use your local coop cuddle context.

A

If you want to have very fine fine grain testing, you may want to mock out certain api's one cool thing about the generation code that we showed earlier is that will generate a bunch of fake lines for you, and so in these snippets we create a reactor. We can sorry we created a reaction. This reaction will always create a new object when we instantiate the fake client. We add this reaction and will always return that that fake object. Every time we do a get on. We issue would get to that.

A

That client now back to Ilya for our paper cuts originally.

B

This slide was called rage. What Tom said is not appropriate. Now it's over boom, so pretty much. It's not rage against API machinery. If a machine is great, I actually love it. It saves a lot of times it works and it's great, but there's a small, tiny thing with it. When you work in with the court objects, they were just fine, but when the CRD is this actually gaps and this gaps are sometimes annoying sometimes can be dangerous, and sometimes we had the piper cards for that.

B

So here's what we find out and that's what we can share validation. You have to do things yourself. I will describe later what it is because I mean CRD lifecycle, um custom resources, as you remember once you creating that before you creating vent one CRD supposed to be created, it's not the case for the entry objects, but it's the case for them out of treat object. Soft is controlling them, we'll discuss it a little bit all day later object, versioning entry or the core objects are really good with the versioning versioning clients.

B

It's not the case for the sergey's OCR's getting better, but there's some gaps and my favorite code generation code generation is amazing. It's super cool when you just run something and it creates things for you and it works great. Unfortunately, when we started using coop builder, we spend a lot of time to set it up and it was throwing some really interesting, weird errors, but once it start working, it's super useful. It's super great, highly recommended so validation you're responsible to bully dating whatever you getting into your operator.

B

What I means by that, so yes, open, API validation will validate schema. So all the objects will be X with the expected schema or predictable schema. But let's pretend and let's, let's think about it- you using something like s3 bucket and your operator is getting some name and before actually work on it. You probably wanna validate if they even s3 bucket or here L is violent, because kubernetes has no idea. If it's there or not.

B

Yes in tree objects or core objects, they have a really extensible, really good way of validating, what's happening inside not the case for the shares. Another way of validating objects, your custom resources is validating a mission webhook. This is a quote from them official the commutation, as you can see, they're quite dangerous, some use responsibly, but the end what's happening, you're trying to meet new custom resource and you getting called back to your operator to validate, what's inside, so we're getting back to the first point, you're responsible, so validating what you're getting ERD life cycle.

B

As we know she are these cluster white objects, which means everyone can access them and there's only like one active version of them.

B

So what the problem with that someone has to manage them, because if you want to change schema, you wanna bump version what's happening, there's two ways how you control in it, and this right now, outgoing debate in the community, how it should be done, there's a few ways and pros and cons if you're, using only one Operator in your cluster, you probably want to control them from the operator, because it's a little bit easier, less instrumentation operator wakes up in the cluster trying to get custom resources, don't find any create new one or update new one, and it just works.

B

But if you have more than one, if you have three four or five, it's a little bit cheeky. If you update only one of them this year, D version cluster Wyatt will be changed for all of them, which might lead to some errors. When you try to act on the resources on the different version, they will just not work. Probably in this case, if you have many operators running, you want to control them outside of the operator by deployment or any other mechanism, for example.

B

So when you're updating all your operators, you bumping version changing version, so that's probably will a little bit easy. If you have many operators again, it's same outgoing debate. We don't know the answer, but we'll find soon I hope. Another small, tiny honorable mention, because on how many customers or definitions you are creating this really great service mesh application called Eastern. They are creating 55, zero custom reserve definitions when you're doing coop cut or get CR DS on the cluster it takes.

B

It might take really long time and I highly encourage you try the thing deploy. Eastern do coop capital gets here, d- v9. That thing is amazing. Back to Tom thanks.

A

Ilya, it's another big issue we hit when developing our operator was custom, resource versioning, so custom resources have different versioning scheme than core API objects. One thing we did initially was we'd have constant schema changes to our our API objects. This cause problems. If there was existing objects in the cholesterol ready that had different schemas, things would break. In kind of you know random waves.

A

Luckily, in Korea T is 111. They added support for multiple versions for custom resources, which is really good. However, the support is still relatively limited. The conversion between different versions of these objects is very simplistic. They call them the Noah or basically passed through conversion. Luckily, there's discussions around kind of adding more advanced conversion code, as well as adding specific validation for each version of your object. So we have to look forward to those.

A

So in closing you know we have a few suggestions, for you definitely follow. The API conventions will make it'll, make it much easier to integrate with the tools in the community. The best way to do this, we think, is use one of the bootstrapping projects and probably a go one so that you can integrate with the rest of the go tools. We recommend you generate your codes, so you can avoid rewriting the same code over and over again for production features, definitely use our back for security and events to understand the state of your system.

A

We suggest you invest in your testing, you know, and we also suggest to use the the techniques we we mentioned around. Operators in this talk. Beware of the gaps between the core API objects and the the API extensions, there's, definitely a few of them, and they can cause some issues. It's important, remember: you're, not alone right. This can be hard, but the community is very helpful. We're certainly willing to help.

A

You know reach out to us on the canvas for slack or tweet at us, and so with that we'll take questions, we actually have some swag. So if you ask a question, you get some of our you get a nice bag which you can hold other swag in so ask away. Thank.

B

B

B

I'll be the muscle sure.

C

Thank you. My question is: what do means a gap between CRT and coal? Ap is the kappa what mean what so.

A

There's some features that are that you that exist for core API objects that don't exist for custom resources, so I if I write a CR. For example, I can't have theirs conversion code for core API objects. You can't have that converted code for your CRS.

C

Thank you, yeah.

A

My fault bathro thank.

D

You I have a question in sir OpenShift 3.11 and you say it is the operator and the Prometheus operator manages the magnetized operator life cycle manga and to have a plan to aim too many to yo-yo traitorous by the operator life circle. Men drop prey, deform developer at the PI's already had the.

A

Question the actual kit that we use for the newer operators was coop builder, but we're definitely willing to try other bootstrapping kits I. Think the one you're referring to kind of integrates well with the operator SDK, is that accurate yeah? So we actually, you haven't used that for any of our operators, but it's definitely a great option if you, if you use out yourself. Thank you.

A

In the back, I see.

E

You see him here, I.

F

Can share some experience on how to can redeploy your operators or to some beta release operator because operator we can only have one running operators. Even we have maybe three replicas, so deploy or operator may be dangerous. Definitely.

A

It's a great question, so unfortunately we have to the way we do it is we use different clusters, especially because these CRTs are cluster scoped right and that is kind of an issue and I. Think I did hear some chatter about name spacing: CR DS, but I can't I actually don't know the details on that. So, but that is a great point. I think there's, maybe not a great solution that I know of right now.

G

Thank you. My question is, you talked, could you mentioned? There are two options for the CRT left circle right. Firstly, it's in the controller and the second is during the deployment as so, and there are also a computer so that so the logic in the computer is the opportune. One is created, the CRT controller right.

A

Soku builder actually creates CRD amell's for you, and so it's up to you to deploy those in your cluster somehow before running the controller, and so that's really the option where we you kind of do that during deployment. Really before your controller starts running the alternative, is you could actually instantiate and create give your controller permissions to create the CRT when it starts up, and you know ensure that it's the correct version, but the KU build approach is actually to do it before running your controller. Thank you.

B

C

E

A quick question, so it's like a best practice when the controller gets deleted. Is it is it? How should I say like? Is there like normal to also delete the customer resource definition or just let it leave it? Leave it there. So.

A

We we definitely choose not to delete the customer resource definition. If you delete the CRD, then all the CRS in the cluster go away, and you can imagine you know rescheduling of your controller. Like you notice the coop schedule, you can move your controller around to different nodes, for example, or doing upgrades without wanting to delete all the CRS in your cluster. So we choose, we choose not to you, could imagine an API that would require that. But for us, our code, we don't delete the CR DS when a controller gets deleted. Okay,.

E

A

Thank you and feel free to come grab a bag. If, yes, question.

B

Cool. Thank you. That's it. Thank you so much yeah. It says.