Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Go? Bash! Meet the Shell-operator - Andrey Klimentyev & Dmitry Stolyarov, Flant

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Go? Bash! Meet the Shell-operator - Andrey Klimentyev & Dmitry Stolyarov, Flant

While using Kubernetes, the need to automate our work is everpresent. Usually, the task boils to a simple pattern: subscribe to Kubernetes resource changes and acting upon them. In the simplest case, a ConfigMap creation will suffice. In the complex one, interaction with a Custom Resource might be required. Most of the operations engineers know a scripting language (e.g., bash) and are familiar with the jq tool, which employs an indigenous paradigm of traversing JSON structures inspired by the functional programming paradigm. A general-purpose programming language, without a doubt, provides tons of benefits, but do they not create a steep learning curve on the path to automating and extending Kubernetes? This talk focuses on writing simple operators with the shell-operator, using a scripting language (bash) and jq, and justifies such an approach.

https://sched.co/Zeo1

A

Hello, I'm andre clementiev and I'm a solutions engineer at fland.

B

Hi I'm dmitry stolleroff, I'm cto and co-founder of land yeah. We both work at fland. We don't want to waste your time, telling you about what our company does, but, of course we will appreciate it if you check our site.

A

And we don't want to waste your time, because today we're gonna talk about.

B

Yes, we are gonna talk about cloud native.

A

Shell scripting, so let's talk yeah. Let's do it.

A

Before expanding on this notion of cloud native shell, scripting we'd like to talk about kubernetes, yes, this platform that we all know and love from its wonderful api to well-thought-out programming patterns upon which the kubernetes is built. Instead of working with low-level system stuff, such as configuration files, systemd units or even containers, we operate on a higher level of abstraction.

A

Such high level concepts as deployment stateful set secret or even node are available to kubernetes users.

B

When I think about kubernetes api, I imagine the file server, having folders for each kind of the object and objects or resources being gmo files on that server, and it has simple http api, which allows us to do three things with this server. First, we can get a resource knowing its kind and its name.

B

Second, we can change the resource, and, what's important is that we can only store the object if it's valid. So if we try to put the object into the wrong directory or if it's just not formed object, the operation will be denied.

B

Third, we can watch resources to do that. We ask the server and we immediately receive the same result as if we would use the get method. But if the resource changes we instantly receive a new version and if it changes again, we receive the update again and so on.

B

So basically kubernetes in some way is a yaml file server with three methods. Of course, there are many others, but we skip them. We just need those three, but the server, except for storing information does completely nothing and to put life into the kubernetes. We need controllers the second most important, fundamental thing in kubernetes in my list generally, we have two types of controllers: controllers of the first type, read from kubernetes process information using some logic and write back to the kubernetes and controllers of the second type.

B

They also read from the kubernetes, but they change in some way the state of external world. Let's just quickly look into what happens when a user creates a deployment in kubernetes the deployment controller. It's part of cube controller manager gets this information and create the replica set. Then the replica set controller gets the information and it creates two replicas two pods, but these spots are not scheduled, so the scheduler get them schedule them and write their yammels back but update it having not filled in them.

B

And then we have cubelets running on different nodes and that and they get their parts and update information in external world. But this time it's docker and then what happens? Then then, basically, almost all the chain repeats, but in reverse order, so cubelet checks, status of containers calculate status of the pod and report it back then replicasetcontroller receives it updates status of the replica set. The deployment controller does the same and then user.

B

If he watch receive final status or he can get it later if he doesn't watch so kubernetes is all about controllers acting together, and if you are going to automate anything, you need to write controllers.

A

But we all are cis, admins, aren't we and we all love unix philosophy and we all are used to piping and shell scripting. So when we want to count files in a directory, what do we all do? Well, of course we use find and wc and when we want to find the process by name we use ps and grep and maybe an extra grep when we want to find top 10 ip addresses in the log we cut.

A

The log extract the second field, because we know that it has an ap address, then sort count, unique values, sort again and take the first 10, and that is our result.

A

That's how we used to do things and on one side, kubernetes adapted to it very well. If you want to find most used images in our cluster, we can use kubectl and tell it to get all pods from all namespaces and extract the image field from them and then use the same approach as with logs sort count unique sort again and take the first 10..

A

But what should we do if we want to act on a change.

B

Controllers, as I've told you, and I think, as we all knew, kubernetes is all about controllers and to be successful in the kubernetes world, we need to be able to write controllers, so what we are trying to do is to implement a tool that will allow us to write controllers or you can call it operators simply and effectively, but by using those methods we are used to as sysadmins, and we think we quite succeeded in that. Let's look at an example.

A

We've got a kubernetes cluster with the default namespace and the secret in it. We also got a lot of other namespaces, and some of them are marked with a special label. What we'd like to do is take our source secret and copy it into these namespaces.

A

Some namespaces get added to the cluster later, and some of them might have a special label if it has we'd like to take our source secret and copy it into them as well. If a label is removed from a namespace we'd like to remove the secret from it, if a source secret changes we'd like to propagate these changes into labeled namespaces, if a secret is removed from a namespace by accident, we'd like to immediately restore it, but before we take a look at the implementation of such a controller in shell operator.

A

Let's take a look at what shell operator is. Shell operator, like all other workloads in the coordinates, is deployed in a pod. In this pod, we've got hooks directory into which we place executables written in bash in python. In ruby you name it.

A

We name these executables hooks.

A

Shell operator subscribes to kubernetes events and objects, and if something of interest to us changes, a shell operator executes these hooks now, how does it decide what hook and when to execute to solve this problem? A hook has two phases. When shell operator starts, it executes each hook with a config argument: each hook outputs a yaml with its configuration once the configuration phase passes each hook gets executed when the relevant tweeted event happens, it is executed without any arguments and it gets passed a binding context, special json about that.

A

We'll talk a bit later now, if you use bash, we highly recommend that you use shell lib, which simplifies writing hooks in bash. You'd, want to implement two functions, one for configuration phase in which you output, your binding configuration and one for the normal execution, which contains your logic.

A

Now we've discussed what a hook is and how it subscribes to kubernetes events. But what objects are we interested in? We are interested in a source object, source secret since we'd like to know when it changes and whether it exists or not. We are interested in all our namespaces in the cluster to check whether they are labeled or not, and we are interested in destination secrets to check whether they are synced with the source secret.

A

Let's start off with configuring a subscription to the source secret binging configuration for it is extremely simple. We define that we are interested in secrets in a default namespace with the name mysecret once configured in such way. A hook will get executed if a source secret changes and it will receive a binding context, which has a name that we've configured and the full object.

A

Next up the namespaces, we specify a kind namespace and also include a new field. That's named jkfilter.

A

Now what is a jq filter, let's get to the console and find out if it could ctl get namespace and output it in json format and pass it through a simple jq filter, we'll get a new json object that contains two fields that we are interested in once configured in such way. A hook will receive a binding context that contains an array of filter results for each namespace that we in a cluster.

A

This filter results will contain a namespaces name and the boolean value, which describes where whether it has a label or not. In previous example, we received a full source object, a full grenade subject, but since we are, we aren't interested in it. We can remove it with a keepful objects in memory, false statement in a minimum configuration and it will be missing in binding context.

A

Last but not least, destination objects, destination secrets. We subscribe to all secrets with the manage secret: yes annotation label and pass it through a jq filter which contains its secret namespace and a resource version that we pass into a annotation when create when we create a destination secret. We use resource versions and store it in the annotation of a destination secret to compare secrets.

A

Yes, we can compare secrets bite by byte, why? Why do so when kubernetes, already stores resource versions for us if configured in such way, a hook will execute when some of these objects change? If any of these objects change, it will receive three binding contexts that we've described previously.

A

It will be like a small snapshot of a cluster. Using this information. We can devise a simple algorithm, we iterate over all namespaces and we check whether it has a label or not. If it has a label, we compare source resources, version to destinations, resource version and if they are similar, if they are the same, then we have nothing to do and if they differ, then we execute coop, ctl, replace or create.

A

If a namespace does not have a label, then we have to make sure that no secret is present in the namespace. So we check if a destination secret, if a destination object exists, we coup ctl, delete it and if it does not well, then we do nothing. A bash implementation is simple as well. We iterate over namespaces. We check whether it has a label or not, and if it does, we synchronize secrets and uh if it does with synchronized secrets and if it does not, we delete secret.

A

The delete secret function contains three lines of bash. We check whether a namespace has a secret and we delete it by static name, my secret.

A

When we synchronize secrets, we compare the source secrets, uh resource version to destination secrets, uh source resource version that we've placed in an annotation and if they differ, we construct a new object and we execute kubctl create replace to make sure that the destination secret is a carbon copy of a source. One.

A

As you can see, we've skipped some parts of a script. You can see the full version on the url below and that's how we've implemented a simple controller in 35 lines of yaml 20 lines in bash and a shell operator that binds them all together.

B

Secret copying is not the only thing you can do with the shell operator. Few more examples, let's imagine deployment having three pods and this deployment parts of this deployment use config map for some configuration.

B

They might use environment variables or they might mount configmap as a file, but that's not important. What is important is that when these pods were starting, config map was in some state. Let's call it version one.

B

So all our pods are running with this version, but if configmap changes to another version, pods will continue running with the old version and what we usually do in such cases.

B

Yes, usually we add something to our pods template. Let's add the annotation with configmap exam. So now our deployment has this checksum in it and all the pods also have it. So when we change the config map, we should not forget to also update the annotation or we can use the shell operator and program a hook that will subscribe to the config map and change the checksum.

B

So when a user modifies the config map, shell operator notices the change and updates checksum and then the kubernetes auto magic happens. Kubernetes removes one of the pods, creates new one waits while it becomes ready and then repeat the same with two other parts. So our deployment becomes fully in synced fully running with the updated config map.

B

We are not going to delve into the code for such a hook, but you can find the whole examples by following the link below it should be like here, or it probably will be bigger so like here, um okay, another example and let's jump back to the beginning, to the slide with yaml file server.

B

We should remember that kubernetes allows us not only use the object kinds that are embedded, but it allows us to create our own kinds. So we can create the kind with the name mysql database. Why not- and this kind will have only metadata, only name and namespace.

B

So basically, we have our kubernetes with our namespaces and we can create mysql databases in them and we can use shell operator to watch for resources of kind mysql database connect to the real mysql database and synchronize the list of databases between what we have in kubernetes between the desired state and what we have in the mysql server.

B

The observed state between desired and observed, state okay example number three uh or four. We want to monitor the cluster network and one of the easiest way to monitor the network. We know it from our previous sysadmin life is by using ping and to do that.

B

We of course use shell operator and we subscribe to nodes and we only need the node name and its ip address, so we can go through the list of nodes and just ping each of them.

B

But I want to add a few things to this binding configuration. First, I don't want the hook to be executed on any event, so I pass the empty array. So when notes are added changed deleted or whatever the hook will not be executed, but then when it will be executed every minute, it will be executed every minute and every minute hook will receive the list of all nodes because we have subscribed to them.

B

Okay, uh that's that looks cool, I suppose. But how do we report problems? How do we report packet loss?

B

Let's look a bit into the code and find out, so we walk through nodes extract, node's, name and node ip address pins, the node, and then we write prometheus metric shell operator allows hooks to export metrics to promisios, and all you need to do is to write to the file stored in pass to the file is stored in environment, where, where variable metrics path- and you should write the metric name, um what to do with it in this case we increment a counter and add the labels.

B

You need that's basically it so I don't know how many lines of code, but not many, and you get simple network monitoring.

B

Okay, I suppose now your uh imagination is running and you have in your mind at least a few things you want to automate with shell operator or maybe even a dozen and probably that's time for us to finish. But I want to explain one last thing but extremely important thing. Without knowing it, you will not feel comfortable with the shell operator.

B

Kubernetes shell operator event happens, shell operator executes a hook and it's running, but then another event happens. Will shell operator start another instance of the same hook but off? But what, if five events happen simultaneously? Will she'll operator run all of them in parallel and what will happen to our resources to our memory cpu? How can we manage them? No shell operator will not do like that.

B

Shell operator has cueing mechanism, so all events are put into the queue before being processed. Let's imagine we have two hooks and having two hooks, our cue will have a vans form for two hooks.

B

The first event is for one hook, so it's executed and the event is passed to the hook and when it's done, the cue proceeds three nicks three. Next events are for another hook and they are popped out of q and passed to the hook as a badge so who receives the array of events, the array of binding contacts, or these events can be even combined into a bigger one, and we have been actually using this all the time. Through all the examples, the group field is responsible for grouping.

B

Moreover, you can have as many cues as you want and you can have as many hooks as you want, and you can use one cue from two hooks or you can use two cues in one hook or create any combination you want, and all you need to do is to add just one field into the binding configuration if the queue name is not specified hook is just run in the default queue, and this queueing mechanism totally solves all the resource management problems.

A

A shell operator has an older brother that is named addon operator. If you recall, a shell operator subscribes to kubernetes events execute hooks. If something interesting happens, these hooks can change something in the kubernetes cluster or affect something in an external world. If we replace shell operator with addon operator, we can bundle a helm chart with it.

A

Then the hooks that previously affected the kubernetes or the external world can set modify or delete values in a value store of an add-on operator. If such a change is detected, an add-on operator executes helm and passes it a helm chart and the value store the rendered chart is applied to the kubernetes cluster.

A

If shell operator is a collection of hooks that are triggered by events, an add-on operator is the same thing plus helm. Integration.

B

Okay, that's it for today. Thank you for your time. We hope you enjoyed it we'll be happy.

A

To answer your questions right now and we will be answering in the slack channel for the next few hours.

B

And yes, just one thing: how can you try shallow curator? The easiest way is to open the shower paradox. Github page and you'll find the getting started guide there and.

A

If you liked it don't forget to start.

B

And subscribe to our youtube channel, we just have started it and uh check out our other projects. I think you might find them useful see you see.

B