Kubernetes Meetup Videos, 9 Jun 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: k8sOM#13: Extending Kubernetes - Gigi Sayfan

Description

Kubernetes/Cloud Native Online Meetup - 06.09.18 - Extending Kubernetes - Gigi Sayfan

A

Can you see my screen?

A

Yes, I can see you ski all right. Let's see everybody can see my screen so I'm, sorry also about yesterday. They should have checked with a mirror and they make sure we on the same time, but now we're here. So my talk will be about the extending kubernetes.

A

Kubernetes is a huge system, as you know, and it has tons of ways to a extended and extended and make it an even bigger and more customized, so it will be a very kind of it will win tool whether we go over many many many things, but with just a mention them and their each slide. In this talk, you can do a whole other area, just a talk or even a workshop about it, so don't be too overwhelmed.

A

This is more about the kind of getting the idea where what's possible in water is a levels kubernetes. So first let me.

A

And I've been as social engineers for many many many years. I started the thousand years ago in the previous millennium and I wrote a book about the head, kubernetes two editions I actually muster in their kubernetes and also I, don't tell hundreds of a technical articles I like to write- and this is how I learn a new things. I also have several years of hands-on experience with architecting this witted systems, on top of a kubernetes and I used the Google Play command at this engine jke.

A

So a lot of the nice features that Google provides I was able to a benefit from and currently I work for, a startup company called the helix dot-com, which is in the space of a consumer genomics, DNA, sequencing, etc. And my actually my next project is studying- maybe man next week or in two weeks, is it to evaluate the AWS es, which is the elastic kubernetes system, from a from an Amazon to a transition or infrastructure into a kubernetes and I'm, also a jujitsu blue belt. So this is another way to identify me.

A

So let's say: let's get on with there I'm from talk and what we'll cover today. So you can see. This is a pretty dense a slide just there.

A

So we talked about the kubernetes extensibility landscape water can be extended with kubernetes. Then we talked about custom resources, which is the most common work by an easy way to extend abilities, we're going to more advanced topics like a API, so repetition different types of a plug-in, so CNI plugins allows you to it completely control and customize the way a kubernetes network he works, CSI plugins, allow you to plan any type of storagee to company this.

A

On top of all the built-in storage options device plug is, is something get pretty new, so you can actually manage using kubernetes own kind of additional devices that could be use at the network cards and FPGA. So this is a still very young area but they're very promising.

A

Then we'll talk a lot about the access control, Web books, so there are different ways to control access to a kubernetes resources, or is there the triple a tenth, occasionally ization an admission, we'll talk about the web mujer option, then custom scheduler, plugins, so the main function of kubernetes is a schedule owns to a different air instance there's nodes, but it could be a physical machines or a virtual machines, but it is their primary function, all the way kubernetes and you can actually replace the scheduling algorithm with your own algorithm or even run side-by-side.

A

Then another area is custom metrics for a horizontal pod, auto scaling, which is another very cool feature of kubernetes, that it can automatically scale your infrastructure based on a different, different metrics. So the common ones are a CPU or memory alright.

A

So if you run out of a CPU or 100% CPU, cost 1% or men running out of memory, then kubernetes can automatically add more nodes to your to your service and then I extend the capacity but as well see later there are other metrics that could be useful for the peon density of memory and finally, I'll talk a little bit about a cube, City and like in so keep CTL is the command-line interface to kubernetes and extending it with your own.

A

Plugins provides a really smooth and streamlined the working environment, especially when you extend units itself was there other resources and components? So, let's move on and let's look at the components, extensibility landscape, so kubernetes say it's made of a several big parts. There is the kubernetes control plane. So this is the API server itself and it stores the state in a CD. They distributed a database and the actual layer workloads. They run on a different nodes inside pods, so on each node, each machine or virtual machine. There is something called the couplet.

A

So couplet is an agent that the kubernetes installs on each such node and then through this couplet kubernetes can schedule reports to run there and the logic here and all the extensions and plugins that run in the cupola that they control. What happens on this particular note, so a couplet can now interact with a big hook a back end. So this is one option for extending kubernetes, which means that there, when the couplet leads to a verified check, something with the examples later it actually hit says some remote.

A

The end point and the decision happens there that the couplet applies. The decision to what's going on on the node itself in the option to extend the kubernetes is to have her a binary plugin that integrates with the couplet itself, so we'll see also a different options for plugins our plugs that are built into our plugins at the dynamic plugins, and then there is also a GOP, C, plugins and another option.

A

This is a very common is to run on the node, a controller, so controller runs they actually inside their iPod and kubernetes manages it just like any other air ball, except that it's locked in to the to a specific node. So there is always a one running on each node. It could be done through a diamond set or through a other communities, features like tainted toleration, etc. But the disk controller can watch and listen for different events and then reacted in this way extended functionality and what kubernetes actually does so.

A

Those are the different dimensions to air extend kubernetes and, let's start with custom resources, so custom resources are resources that you define and the cool thing about them is that you pretty much get an API server and a database for free. So your own resources will be stored by kubernetes in xcd and you will get automatically various a crud, the rest endpoint, so you can create get under the list and they list a your own resources. You get versioning, so you can version your resources.

A

If you have a multiple, then the person that you want to evolve, your own custom resources, and then you have a watcher, so you can watch changes to and respond to them. So this is a pretty much goes a end in hand with their controllers.

A

So a very common pattern with people that extend covenant is that they define some a custom resources and we'll see some examples later and then they also create controllers that would for events on their own resources and then I'll respond and react, and there is a nice project called a queue builder that allows you to build your own api's and they give to your local variable a plate. Okay. So let's look at the an example of the custom resource definition.

A

So, as you can see, it's a ml file is similar to other a kubernetes manifest and we got some air metadata here. So in this case, I define a resource, that's called the candies and the name.

A

It has a some F format, a requirements, so the name of the result should be human, for example, and then you have their expect. The specification of the resource that they identify, the particular group and versions, and the reason for this kind of a well-defined format is that it can work a later respondent kubernetes itself, because it follows the same naming conventions at the same way attributes.

A

So we see here, for example, the name so the name, the plural, is Candice, and then the single R is candy can specify the single on customer that this is not a necessarily very good at grammar and he can't tell how to convert the tour name to a single name, and then there is a kind which is kind of the type of the type. And finally, you can even have a short names is the way who examine kubernetes.

A

If you want to do a good city, a region to do I get bored, you can also do get Pio just a short neck. So you have the option to give a fraud names to your own, a custom resources too. So that's part of their streamlined integration with the rest of the components experience so here for candy the selected CN is a shop name, and now we can see so working with a candies, always custom resources.

A

So you can see I define and define is Candido tml fine and here I define one as specific candy right, so they will have the API version. We have the kind which is candy. We have some and mental data that can be here. Anything in this case that shows a name which is chocolate and then the spec I define the label sweet, and then we use just the normal. A good CTL create command to create this resource.

A

So then we see that the result was created, and now here you can see that there I can use coop CTL and now it's already integrated with my custom resource I can do get candies, and then it gives me back my my chocolate.

A

A

So we can, we use here the other capabilities of coop CTL, so here, in this case, I use the command with the short name, so I do get CN and as output I request the JSON output. So that gives me a much more detail. The output in the normal, just the default, a you get command, and you can see here that we have. The version will have rare items. We have a self linked in case. We want to access it there.

A

Those resources are programmatically through a some rest interface and there is a unified ID and also we can see that the kind here is a list. So this is a list of their resources.

A

Okay, moving on so customer resources are cool, but they have some limitations, but they are very easy to use you just pretty much work with EML files, and then everything happens, they're just there on its own. If you want more control and more more customization of the resources more operations, for example, maybe you don't want just clutter? Maybe you want the different operations like a exact or scale. Then you can use a API server application here.

A

You use the same API machinery like a kubernetes itself, so it's still managed by kubernetes, but you write your own API server, so the integration is very similar in the user point of view it's going to be almost as if equipment is running everything, but you control the storage of the resources. So it's not going into a HDD automatically and you have to actually write some a goal code to implement it.

A

Another benefit is here: you can have multiple versions of the same resource when you manage it yourself with customary sorcerer, there is just one version of the time and finally, the payload of your resources. You can use it protobuf, so Google protocol buffers protocol buffers, which is a very efficient and there performant the format, while with custom resources. You have to use a JSON that pretty much so that's a API server aggregation another option and now moving on to a to plugin, so say my plugins.

A

So what's really say my plug-in, if you want to add the network interface to your container network, and you want to control how information flows between your containers, so you can do various types of bridging between the host and the container itself, because the whole point of the kubernetes networking model is that pods are really a can see each other.

A

So it's a completely flat networking space, even though the actual physical network and the configuration can be very complicated but from a a kubernetes, a uses, the developer point of view or pods can just see each other. So in order to go and control it in different their networking environments, you can implement your own plugin that defines how those IP addresses for the pods are assigned and how they the traffic, is routed between those different containers and pause.

A

So you can see here in this diagram that the container run time it talks to the scene I plug in and then underneath the cni, sorry, the cni network interface and through this interface, many types of plugins can be can be defined and they only have their own specific attributes and they are suitable to different area situations. So some of those already built into with kubernetes, and then you have also a third-party plugins that they, you can define yourself.

A

So what's really the interface of a scene I plug in. So it's very simple, you can add a container to your network. You can remove a container from your network and you can pull the version. That's all it is and command-line interface and then the interface is very interesting. So it's a command-line interface.

A

You run the you plug in as you execute it and you provide the input through a just standard input and environment variables, and the main and the most important argument that you pass is your network configuration which is a adjacent block that there is system.

A

So here are some of the environment variables that you define in order to control your plugin. So there is the operation that you want to perform. So if it is it a add or delete or old version and then say my past, is there were kubernetes? Should they look for your scenario, plug-in executables and then a network namespace file and interface, and a few you can have even a extra arguments so pretty much. You control things with the environment, variables and.

A

And this is an example of network configuration that he Prosser through a standard input. So you can see there is the version here and the name at the touch this case it's bridge and I parent. So that's their IP address management, so you can see all kind of a networking level at parameters he writes, owns some meds in a gateway and the TMS etc. So there is a lot of nuances and details here.

A

So if you're interested they're encouraged you to a read up on it here with just a skimming over the capabilities, we're not going to get into details, so that's the networking configuration and the next. The next type of a plug-in is a CSI plugin. So this is a common storage interface and this is an industry-wide standard. So it's not just for kubernetes, but kubernetes is the probably the most prominent user of CSI plugins or this is going to be, and the idea is that their kubernetes already has a lot of plugins for a storage.

A

But the new storage type or a vendors of a storage equipment may want to define that their own plugins. So before yes, I was adopted by kubernetes, you had to integrate it into a string kubernetes, so you had to create your cool, then follow all the requirements of a integrating new code and get it reviewed by a kubernetes engineers, and it was like pretty pretty tiresome and long process.

A

So, with this new CSI a standard, you define your own plugin and there is a a way to communicate between kubernetes and your plug-in across your PC, so that the reason it's using your PC is that you don't have to add, compile it and build it and Link it with the kubernetes code base itself, which is both fair, could cause a performance than there security issues as well as it takes a more knowledge.

A

So by separating them and having a geo, PC interface between them that kind of more isolated and it's easier to implement and the kubernetes a engineers don't like to worry too much about the damage that some, a storage planning can do worst case. They can always just cut it off and say: okay, this storage is dysfunctional at this point. So it's a huge improvement over the previous way to do custom plugins, which was a flex volumes, and it will think it will have a great ecosystem of new era storage options.

A

I integrated there is a little kubernetes.

A

So here is a pretty complicated diagram of CSI plugins that you can see that they're here at the bottom, and you see the legend and the green components are a kubernetes, a kubernetes component.

A

Instead, a part of the SCSI infrastructure and the origin points. For example, here we have an NFS, a SCSI volume, so those components are created by a but by the vendor or whoever is the developer of this storage option and the communication goes over a gob C. So in this case we stay on the left side which same option where the storage goes to a some central. A network storage on the right sides of this is Ed even said. In this case.

A

On the right side, we see a stateful set where the storage is actually a local storage. So every empty do volume and it only communicates and they're coordinates a destroyed option through the Java C API and through the ver si si infrastructure, and here, on the completely on the right side, you can see how this SCSI attacher talks to the API server. So the kubernetes is completely a completely unaware of what's going on with storage, it only understand the CSI talks to a CSI toucher, which then talk to Joe PC with the actual storage plug-in.

A

So it's a very elegant design.

A

So device plugins I couldn't find any real examples. So there are some vendors that already created device plugins. The idea is to it, have kubernetes manage all kind of custom resources beyond just a poles and nodes and the network which those are resources, that the kubernetes is internally a world but other more specialized components like GPUs and then different. The network cards and FPGA is so similar concept to the CSI applicated with so before.

A

So the couplet provides a RPC service that they're all kind of devices can be registered and then the device plug-in is deployed a on each node that they wants to utilize those special resources, so they are deployed as a daemon set, which means there is a one per node in the cluster and then the pods can just request this resource in their spec. So we can see an example here we have a code called a demo code and then hearing their resources.

A

It's a because. Obviously that's for limits. I need to go to the next page.

A

You, oh okay, so actually this is the wrong one. Oh here we go yeah as a vendor. You have that federal domain and resource. So this is a where it's requesting some custom resource based on their dependable men. So that's not a built in kubernetes resource and there must be a plug-in device plugin, underneath that knows how to service those requests so very new, and it may also be a great way for kubernetes to extend its ability to handle different workloads of different devices.

A

Okay, so now, I'm moving on to a access, control and kubernetes was always always had a very sophisticated with a access control method, but it was very difficult to integrate custom, a custom access, control methods because the details were very gnarly and they went away really understand both the domain and also how to build it into a community.

A

So it required that building your own controllers and having a lot of in-depth knowledge of kubernetes, mechanics and internals of the API server, so a new option to make it that much easier is to use a web hooks. So you were looking at the first kind of the first pillar of a access control, which is the authentication and by having a remote web hook, you can have a promotion, a web server or an access authentication server that runs completely independently of kubernetes, and you can use any other application method that you want.

A

You can adjust implement that what everyone and through this side, when hook a interface you can every authentication request, goes to this remote of application service through the web hook, and then it returns just the response. If there is a particular individual or service or whatever agent, it's trying to authenticate if it's a dedicated or not. This is jonathan difficulty, fine, pacific agents that want to interact with kubernetes and the way the kubernetes itself authenticates there.

A

The remote server is through a is through a just a standard XSL, so we have a certificate authority and client search, etc. So it's using it best practices, because if they're your own authentication server is a compromised and obviously the entire cluster is compromised.

A

So this is a authentication and you can configure the API server to where to work with a remote education server with AML fund, that's similar to their comm config file and Camilla T's. That they're supported the transcript abilities are a impersonation, so you can configure the remote education server to allow users to impersonate other users or other services and also anonymous request, which is also a useful, sometimes so here's an all of the config file for the authentication and weapon book.

A

So we can see here that there are clusters which cluster are we talking about, and then we have here the file for the CA, the certificate authority. We have the URL for the the URL for that's. Jumping up can test select, so the URL for the remote authentication, server and then- and so the users section, is the user of the remote authentication server. The client of this remote application server is kubernetes itself. So it's not the end-users here, but it's the API server and then we have here the client certificate and the client key.

A

That kubernetes will present to the remote education server to authenticate itself. And finally, we have the web hook and there, which is which they collect the content. So we have the context of their of this authentication procedure and, and then the user is the class. So this is pretty much repeating the context they're putting information there and the next face. So once we authenticate the user right, the request comes in. We authenticate the user who's requesting to perform this action.

A

The next thing is an authorization all right, so maybe a we know where who dis a user? Isn't there? We can authenticate it, but now is requesting some operation that there is not allowed to do so. This is the authorization phase of the the whole acts control and it's really a pretty similar to authentication so with web hooks before it was very complicated and you had to integrate title music when I this. Now you can just implement your own, a dedication server and you can see here an example of all authorization requests all right.

A

So where the kind is a subject, access review- and you can see the date going into this authorization API group- so then we have here a so. The spec of this request is there okay, what are what are we asking to access? So this is the resource or which namespace? What are we trying to do what verb? So it's a get and then here we try to also trying to get pods from this particular group and then we'll have you the user.

A

So this is already there authenticated user and, as part of the authentication, we also got there the groups that the user belong to. So here we see that this user mark belongs to the group developers and testers and based on formation. The remote authorization server will return a response which pretty much is very simple. You have the status and then there is your allow that it's either going to be true or false.

A

So that's all there is to it and then kubernetes say if it's a true, the kubernetes will allow it to move forward and if it's false, the names will reject. It was there for three and finally, so this is something that's not very commonly don't say too many access control systems, but the kubernetes also has a phase called admission so admission once a request is already authenticated and authorized still, and there is another phase and there it applies only when you're trying to mutate their resources.

A

So if you're, just making a read-only request, are you trying to get some pods once you authorize? You're done your quest will be granted and you get a response, but if you're trying to create or delete or modify a some existing resource, you still need to go through this way. Admission phase, and here there are two types of a admission. One of them is a validation so validating that you can modify this this resource and the reason that validation can fail.

A

It look because you not a lot, because this part is already done during the authorization, but maybe there is nowhere for exam. Maybe won't you try to create a new pod and the cluster is at their max capacity. Just a you are unable to create this reso that you asked it to create. So validation can fail and the mutating admission is another phase where your request can be changed.

A

For example, you may be you asking to create a new pod, but then then the admission add plug-in with the admission remote server decides that that some other person needs to a take place. In addition, for example, maybe someone needs to be notified, or maybe some records should be stored somewhere for audit purposes. So that's a additional additional things that they add omission can do so again, it's optional, but it has been useful for many real ones and Muriel's. So there is this third we should face and the mechanics are really a very similar.

A

It's just again with modes requests and then components. We do something as a response.

A

Ok and then the next thing is the custom scheduler plug-in. So I talked about a little bit at the beginning, but scheduling pulled the two nodes. This is the core function of kubernetes. If you want to define kubernetes in a one in one word, it's a scheduler for for container workloads. So the built in a kubernetes scheduling algorithm is a super complicated and advanced. It takes into account it only say that it's the apology, aware and it's resource aware and so many ways where you can kind of just do the basic scheduling algorithm.

A

We can control it with many different, their attributes and their policies and defining their limitations and constraints on different they're. All the different election scheduled, but still some some users need have needs to go even beyond dead and for that kubernetes gives you the option to completely replace the scheduling algorithm with your own algorithm.

A

So you can take all those all those resources that are stored in the API server and just decided on for yourself when a request comes in for a credit, any odd when a an existing existing folder crunches, they need to schedule it into a, and you know, then it can be done by your own customer algorithm.

A

But this is not all you can also have it run a side by side. So, for example, you can have that the different communities- scheduling algorithm schedule most were closed, but then you just cook your own algorithm in and when, when you decided there, a particular error request comes in that there.

A

You want to a schedule this, a particular poll based on their different method, data options and tags that comes in with the post puts up so right now we're you still have to integrated just kind of write, your own a go code that implements a particular interface and then integrated. They build it in to kubernetes and provide it and when you start a kubernetes itself, but there are talks about having a web hook. Option also in the future will be easier.

A

Okay, moving on so I think this is a last last item, which is a custom metrics for horizontal. They put auto scaling. So this is a one of a thing of the coolest features of great kubernetes. Every non-trivial it devotes operations. We're scaling and also scaling is always a very hard task and you need to a bit a lot of a support and infrastructure for it.

A

So with kubernetes a there's really a very solid foundation, it already works a great for a CPU and memory, but if you want to do a your auto scaling based on something else, for example, it, let's say or a workload, is the DB constraint all right. So when I, for example it when you cross some air thresholds over requests per second, then your whole system begins to slow down, and maybe you want to schedule a more a database node or maybe a schedule more, a caching depending on your architecture.

A

Do this is where those custom so matrix a come in, so you can define it, but it's to a scaler different pods based on your own air custom metrics, and there is here a a good example of the custom metrics, a API server, which is a bunch of fifteen, the server that lets you wear and your own metrics and very easily. So here is an example of the custom metrics interface, so we're at the end can see some a go code just for fun, so you can see there.

A

You have your different methods that you need to implement in order to select to between two latech kubernetes select those add metrics, and we can even list all your metrics and based on this interface. If you implement it, then kubernetes will be able to it automatically scale different pods and without your explicit involvement.

A

So that's great for a self-healing and self auto-scaling systems.

A

Down: okay, oh so we still have yeah. This is the last one, almost there one before last, so we're coop, CTL plugin, so coop CTL, if you're familiar with kubernetes. This is your go-to tool to control everything from your local machine. You can interact with kubernetes a very easily with. There are many good UI and web UI options, but this is really a indispensable tool when develop in working with kubernetes.

A

So since we talked a lot about the extending it and they're adding a new capabilities, then you can also extend coop CDL itself, and this is great, let's think about. For example, let's say you have something to add provision a new and notes for the cluster right, so kubernetes can a auto-scale pods with in your cluster. But if you need more capacity, then this is something it goes beyond the the kubernetes itself, so you can integrate the disability into a coupe city air.

A

So we have a single uniform interface and you can add your own command to do it pretty much anything and you get the whole coop CDL a command line experience and it's very easy to to add new plugins. So we'll see an example here. So just a another Yammer file- and here is something just just an example: I took from a kubernetes itself, so here's an hello we're hello plugin. So you say this is just it. The name is hello.

A

So if you write cook City and hello, it will run this command, which is just a echo, echo, hello, plugins and that's all it is, and then you get the help. So the short description is part of the help. You could try to look for help on this command and it will give you I, say hello and super easy to implement and make Quad City a work for you.

A

Okay and then this is the final plugin, so I'm going to plug in my own book. So if you want to know more about kubernetes and really dive deep into all the advanced stuff and then many of the things that I just covered briefly here, then you can get my book and I go into a lot of detail there and it's pretty up-to-date. The second edition is a I think it's up to version a 1.10, so it's very very current and.

A

Is it so if you have questions or any other comments, then let me get back to them. I'll stop sharing. Now I'll go back to the chat.

A

Are we living on time pretty good.

A

A

So I'll just read the questions and then I answer them here. So we're one question here: do you know what is the difference between a builder and the upper predatory SDK for Mac OS? So the answer is I, don't know exactly the difference says: I read a little bit about both, but it didn't really dive in I. Definitely ticked away any work with them. So I always hesitate to talk about things that I don't have a hands-on experience so we're, but in general and I think both of them are valid tools and people use them.

A

So if you need to stay, definitely I think you can go wrong with either one, but I think that in general the operator kind of paradigm from Korres say it becomes more prominent than people are really a gravitate more to to this a model of creating a operators on top of abilities.

A

Okay. Another question Oh is the same question. Okay, so do you get the request payload from the user on the admission web book organization? Where book, so you you get, you can have access to the to the payload and it's necessary during authorization, because sometimes the authorization decision is going to be based on the actual payload, because sometimes the pill defines their. What really will happen there when processing the request so yeah that the answer is a you can get access to the pillow and.

A

Don't see any other questions, that's it.