Cloud Native Computing Foundation KCD Berlin 2022, 14 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Build your own Level 5 operator on K8s - Frank Kloeker, Deutsche Bahn

Description

Kubernetes is all around. Also on your refrigerator? Learn how to extend the Kubernetes eco system for your own purpose with things whatever you want.

Website: https://www.bahn.de/

Organized by @Microsoft @kubermatic7173 @SysEleven

Thanks to our sponsors @CapgeminiGlobal, @gardenio, @sysdig, @SUSE, @anynines, @redhat, nginx, serve-u

A

A

Operator on kubernetes I have three topics today on the agenda. The first thing is kubernetes. Okay, we are on the kubernetes community days here first time in Berlin, then that's the main topic, the second topic of Telecom clouds. We know so datagram cloud. Yes, many and the second thing something about hamster or some ideas about. This must cut the tree. The third thing: yes right, a very new program language or what is new, uh how it is going.

A

What do you mean.

A

One years: five: ten, yes Ted, White, happy birthday this year, 10 years of growling, so maybe follow us. Some girls are part of Telecom Cloud. It's a openstack based public Cloud for business located in Germany in brm.

A

You can see you can find us on Google Maps. There are the two data centers data center, one and data center. One two three, the second is lost no way. The second is in Market, it's in the near just in the South, so you can see in the middle of nowhere. The next big city is uh Borderland, I think there's also one shop there, but I think it's closed and some other side seeing step you prepare and Sue of Schoenberg.

A

So as a very good waiting on Google, Maps, 4.4 stars and the best thing is open around the clock. So some pictures inside from the data center, so some fancy marketing stuff there behind the doors or the computers.

A

There are also some welcome wall and entrance. If you take the tour there, you can buy the tour after the talk and yeah. That's me: I'm from clerical technology manager, Cloud applications at Deutsche Datacom um broke since 2019 in a team named manage cloud services platform services, so there are lots of services in the name must be good, so at the time before 2015 devops use of the orientator Chrome Cloud, so I'm not involved in the operation of the potato gun Cloud, but I'm a user term.

A

The time before 2012 um was the openstack cloud operator or sometimes support this term and yeah the time before. 1995 someone on the internet and the time before, 1986 I was doing something with Railways operations. This kind of locomotive there and I think this comes also the name for my Twitter account or the quick GitHub account, because the locomotive has a kind of nickname on the radio so e and yeah. This comes the name from there. So the topic today. What is uh kubernetes operator?

A

You know something about operator. You know some comment operator. Yes, yes, okay, some many, okay, maybe a short definition uh with kubernetes operator. You can extend the kubernetes API with a function that it doesn't have before. So that's very abstract. For example, uh configurator you can deploy as this kind of thing in your kubernetes cluster. You can say, for example, I have a kind of stuff regulator. There are some specs like a color or the size.

A

Wait shut up, so many shelves into a three shelves. So just can you apply to your kubernetes cluster at the end, there will be not a refrigerator there. Then you need a kind of controller or machine, so you need something in the middle, a controller you have the real things are happen in a traffic or atom manufactured. So um there's a connection between your computer, your kubernetes cluster and machine there. Where these things are happening.

A

So can you mention the kind of industry, 4.0, maybe another example um kind RDS you can deploy RDS database also with some specs there, data store type data store version, flavor of volume size for your type and so on, and to make this happen, you need a kind of crd there's this API extension in kubernetes cluster, where you can extend this API.

A

We just think about with a customer resource definition and it is custom, Mark resource definition. You can choose your own domain name or group name. Typically, it's a project name like k8s IO or your company name telecom.de or your product name. What you want I choose the mcspe, it's my own domain, but it's not required and then you have a resource name. The resourcing name is for the RDS instead. So if you have S resources, name IDs and at the bottom, we have. We also run ifr1.

A

Custom nodes now what the RDS is and which specs store, are expected. Also your controller and meets the kind of definitions. You need at least a package with this version, one Alpha One, where you can explain again the specs, you can see their type RDS type of specs, so the same things, data store, type, data store, reversion and so on.

A

So in the middle is uh the controller. The controller is very stupid to you. The controller is a knowledge only about three commands at update and delete. That's it all other things what you have in the cluster like on, get or keep City I describe or whatever that's inside the kubernetes cluster. But if you want to really things happen, then you need this controller at this controller is a very common part of the kubernetes software, so you'll find this very often there.

A

For example the example one is the deployment controller to make deployments happen in your kubernetes cluster and also the main functions. Are there at function? Update function, delete function to make a deployment happened in your cluster that you can deploy them, puts a containers and have other specs there. Like peptical set and so on because of tests, this is very common in the uh in the kubernetes software function.

A

There are also lots of additional features or inclusive. For example, queuing you can imagine there are you? One user deploys one RDS instance the next user, another RDS instance. Then you make some order there. Then you need a queue. You build a queue and behind the queue or a one broker or more worker and the worker get the next job on the Queue it, and you have some also caching, if you have again again the same resources, then you can use a kind of caching that you don't need the same results.

A

At the same time, a rate limiting to prevent the controller for overload is also inclusive and the leader election. You want, of course, one not only one, but with one uh your for with one controller, you want to look at one two or three and 42 or three. They need uh elect uh trust. The leader, who is the master of this controller and a lot of us ask or ask the master if you're alive, again or not and then elect again.

A

So that's um kind of high availability, also Auto generated code, is a plus feature, so you have very often the.

A

The software running you have the same functions again. What we saw in the deployment controller, the same function, that functions, update, function, heat function and this code can be Auto generated because it's very common part from this. It does also an example controller. You can look on GitHub into kubernetes organization paper.

A

There are lots of things, they are in December control and only we need to copy the central control or we place the names. And then you have a running control at this point.

A

Another part also with um article Reddit code, is this operator SDK. They are also a tutorial for Googling.

A

um There are lots of code is out again or rated if you lose the CLI, if the project name and the API version and so on, and then lots of code will be Auto calibrated.

A

So apart from this um now comes I'm really unique.

A

Now we have around a thousand lines of code, for example here where we connect to the potato gun cloud with some service clients on the left side to make real things happen and also there we have followed the three mantras create in RDS instance, updates and RDS instance, and delete this instance.

A

So if you make this uh there, then maybe some too many things happen. If you don't check the return code, so at the end you will have deploy one audience instance, but on the Telecom Cloud people have many many RDS databases running, because when you not check, for example, if the database is really there, so uh you have every time to check the status from the cloud with the status in your in your kubernetes.

A

So you have to look up the resource IDs. You have look up. If the name is there and then comes the SDK, you have SDK, for example, for openstack uh the go for cloud SDK, but we have also the own Fork a potato cloud with an SDK, and then you can look into SDK that which, which features which features are provided. For example, look up in OBS ID or look for the RDS name, so sometimes of features missing.

A

Likewise, what we see here a point in time, recovery from the database, so that's open source also to find on GitHub. So you can contribute to the SDK at this function, learn some new things about unit tests and yeah. When you are part of the community.

A

Our first success with the version 001, which is able to create an RDS instance and OTC, and then we release very fast and describe the work. The version 2 has also to check the status. What I said or.

A

Delete the database also and resize the flavor, but to get this done, we have this status on the RBS spec. Without the spec, we have what you had with the data store, type and data store version, and we have also this status fields where we have the RDS, ID or also the IP address, or the status like here to statuses is so. At this point we can take a look on this operator or level approach. It's also a slide from the Urban stack SDK.

A

So there are five levels, but you can reach with the operator starting from the level one. What we have now, where we have a basic installation of our RDS instance. We can create this. We can delete this and that's it, and there are also some tools or program languages in this SDK is hand mentioned or ansible or go, but there are also other languages or tools that are also in bash operator.

A

If you like this, it is and with hand you can only reach a level two, so there's no more functionality to reach the level five. So you can see um how important this is to choose. The right tool or language, so in the level two we have um seamless upgrades there are only not so much functional in the open trader, Chrome Cloud you can only choose, for example, in my SQL version, five seven 21, and then you can upgrade to five seven, twenty nine or something like this.

A

So it's not so important and yeah other things is this full life cycle where we have this application lifestyle opening job? You have also a kind of backup and restore for the RDS instance implemented.

A

So and then we are back on our restore point in time. Recovery function, which we can use now in our control on and to make. This function happen in the operator that we choose at the first and backup restore timestamp where we choose the time when we want to restore our RDS instance.

A

Put this to this backup function from the SDK and then the open telegram Cloud to restore based on the API call our RDS instance to this timestamp.

A

So it has some more level four deep insights, so you have to interact with some metrics with some monitoring stuff which locking so most of these things are done with orientated from cloud there's. A service named Cloud I have lots of Matrix for the RDS instance on measured, but we can also have an lock API. It doesn't look API where we can get the log files from the RDS instance, so we have Arrow locks, slope, very locks and another look and yeah to provide this now to the operator.

A

How can we interact with this, so they asked an event recorder in the kubernetes framework.

A

If you describe your resources like puts or other things than you have at the end, an event log where you can see what happens? What is this awesome yeah, but for the RDS instance for the Locking there are around 100, plus block entries and then yeah you have to think about. All these entries are going to your adct, so you can really nice mess up your STD with lots of lock entries with some RDS errors.

A

So the idea here was to spawn a kubernetes job connect to the operator, Chrome Cloud API and get the API call for the locks and then put out a push out the locks to the standard out of the bot from the drop, but there's also a problem because to connect with the Nintendo Chrome Cloud. You need the cloud potentials of course, and if you spawn this job, then this job needs this card, but then just hit. This job connects to the open telegram API.

A

So the job runs in the username space, where the RDS instances also running. So this means that you, each user, would have this master Cloud capacity. So to prevent this, we um create your dysfunction token, based on our credentials. Give this token do the job and the token it is token, can chop connect to the open, Telecom Cloud, get this lock entries and then returns and Benz returns. Then we invalid the total with our credentials.

A

The important thing is to invalid a token each case. So if the, if the pot breaks or you're not running or there's no return in each case, we have to invalid. The token so another case, the token will be valid and it's a kind security page I, don't know so that's one of the import properties. So then we are very fast on the level five RDP load.

A

These are the five all these things are now together. What we have before The Matrix, this monitoring alarms, so the operator can interact to all of them things to um each list. There is a kind of flown for this RDP load feature so at first, we need to create alarm rules in the OTC Cloud I servers.

A

This alarm rules interact on Netflix like CPU or memory or disk usage, for example, and this alarm, who will send messages to a simple messaging service, also a service from reputation card. This simple message: service, SMS, um uh create an https webhook, and this web hook needs a Target and for this target we have for our operator and increase control over where we can, if it increased, that the web group can be perceived by the controller and then operator interacts on this event like this CPU and memory.

A

So this is a screenshot of the of the sequencer, so you can see that there are three alarm rules for one or the S instance. So one alarm is here happen for example, say view: alarm is Switched, then how we can we scale up? Of course we can search for the next flavor, so we need to function to uh search for available flavors, which is not so easy, an open telegram card because not all flavors are available in each availability zone, so we have to look up and which so in our RDS instance, is running.

A

For example, we have a ha instead, so we have two zones and then we have to look up in each Zone with the next available flavor or the next with the bigger CPU is available. And if that's the is this the the case, then we go back with the next flavor, send an update command to our controller and the controller updates to the next thing. So there's also some crafts inside of the open data from cloud. So we have here some traffic CPU some queries and to switch over 90.

A

So if this for longer than 50 minutes, then yeah, then this event occurred, that um a scale up is required and uh are the aspects scale up and then the load is going down, and you can see this also in here in the event recorder, we set this RDS instance to RDP load. We have an endpoint there with audience operator and yeah. If it's scaled up, then this event is also recorded to um the event lock.

A

So there are some there's a course. If you remember me to the next available flavor, the disk usage will be plus 10 gigabyte and we checked the situation after 30 minutes. If again has a high load and we scaled up again. If not, then it's cool, so some more additional features um very fast SQL user management is into where you can create schemas or create users.

A

With some user permissions to test the user can set up at the client can set up the user database user with their rights permissions and also an extra feature um if the user deletes the RDS with Cube TDL delete RDS. Maybe you don't think about this. This also deletes the database in the open, Telecom Cloud, so to prevent this failure we make and pick up before, so you have a pickup in the backup in the back end. So if this was a failure, then you can restore the backup very very fast.

A

So that's the level 1.5 approach of the operator level. There's also one Vapor it's written by archive, but it's also contains many many operators, this GitHub .com operator framework, Orson operators, the, although some helpful tools when you develop on GitHub, for example, the dependency port. It's also a tool which wraps all the code on all repos, and if there are some security issues on your code, then you will a lot here on the website or also an email. That's um really helpful.

A

Another thing, totally underrated, GitHub action where you can build your grow application or also build your container image and push the container image to the GitHub repository.

A

So to have your all things done to get on so some people time, maybe.

A

Work check with this.

A

Yes, okay, yeah I have here a test cluster of my test environments. Then we have your some examples into vapor I'm here on this OTC OTC RDS operator of April. There is one example which will be a little bit bigger where we have all this feature into we ports, users and so on, and there's also a small example which we can see here, but I already need a password for the database and VPC from the network.

A

Subnet and the security group and the rest are from the parameters are default, so the control, because the defaults so we'll see if this work. The first thing we installed operator WP and hand chart with some values. The values are the cloud credentials. Other things are very default, so we'll see if this working.

A

So it's now deployed you can see if the operators running there's one instance running it's a test environment. So not the full feature set is the, but, as you can see, it's fresh installed, it's very very fast, because the GoPro cam is only 40 45 megabytes, so it's very fast to start and yeah. Then we can apply. This obvious single instance, for example, is created, get obvious instance on this address one namespace you can see here. The obvious instance is applied, which are my with our specs are into to Stacks uh default.

A

Mysql database pick up keep. This is also default, availability so and just default so and after a while should go to this build state, and you can check this also with this um purpose. Tech RDS instance list where we can see I'll, probably see maybe some serials issues.

A

Yeah uh two testing standards in the dot instance screw it up now, which my secure version: it's zero yeah. This takes around 10 minutes, so I can show other use cases, but this would take too many time, but you can see it's running. So how do you end some.

A

Key takeaways from my side of the development this operator, we did at first division what you want to achieve so, for example, of obvious instance, so you need a division. First, it must be, must be enough rocket science. You know cheap limitations, so there are lots of operators around on GitHub, but most of the operators are only reach the level one. So there are no more levels which there are no more features set.

A

So if you want to use the operator, you have to check the features that first, so it's a common tasks or if it's brutal or bad easy to use for housewives I know a little bit complicated. But if you think in the past uh the the the ice box there's only one switch one switch, you can choose a temperature zero to five or seven and take it this direction. If you build your software or build a tool set for your customer or clients start small, so quite small programs to us.

A

It's too small for gum Snippets, like this flavor search program. The first MVP is okay, but you can. Then you have to continue very fast and get familiar with the environment, get familiar with the SDK. What you used get familiar with the program language, if not code and another tools.

A

That is also a very nice yeah, then look always for sdks, which provided by the community or from the vendor what we have Automation and pipeline what we what we saw on GitHub- and you know, then after that's also the life cycle after the first releases, so they could use those and maintain- or maybe here and yeah, both documentation, testings all these tasks, and if not, then you can also go to a GitHub Mark your project as retired, and that's it so one minute. It's also my last slide.

A

So if this was to pass there's always a blog post on GitHub, you know all this information and links are into yeah and that's it. Thank you very much.