Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2018 (Shanghai), 21 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Configuring Your Kubernetes Cluster on the Next Level - Lucas Käldström, Independent

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Configuring Your Kubernetes Cluster on the Next Level - Lucas Käldström, Independent

Are you tired of passing way too many flags to your Kubernetes components? Annoyed when flag defaults change or flags are suddenly removed? There’s a better way around the corner, called ComponentConfig. Put your configuration in a file, tag it with `apiVersion` and `kind` like any other Kubernetes resource, and point your component to the file using the `--config` flag. As the general Kubernetes API machinery can be reused, version upgrades of the configuration are handled seamlessly. This talk will dive into how ComponentConfig works, how you can adopt it for any (!) Go component, and how you can start configuring your Kubernetes clusters smoothly with this new approach. By the end of this talk you should be able to: - Write your own Go program that adopts the ComponentConfig practice - Write ComponentConfig files for any Kubernetes component.

To learn more click here: https://sched.co/FuKB

A

And welcome to this session of how to configure a kubernetes with components config a new technique in kubernetes, I'm quickly about myself, I'm from Finland I, just graduated from high school now currently I'm doing my military service, which is mandatory in Finland. But after after that I'll do my university studies I've been in a project for 3 years about and doing a lot of nice things with the community. Most recently I've been doing cubed iam working on that and now it's a pleasure to laugh we're going GI the next release. So that's that's!

A

Really fun and I do contracting, for we works working on open source, Y, cubed, M and now also from this summer, I thought. Well, we gotta get rid of the flags sometime from components from components, because we don't like the negative sides of them so and well: let's do it, and now we've done the first work and there's more to come and that's why we're all here today here today.

A

So what are we going to talk about? First? What's the current status of components, config? Where are we at, and that means, for example, limitations. What can be done better?

A

Where do we want to go in in the future? With this effort.

A

And also I'm gonna try to show a demo here about it's on this github site, so you can check out the code yourself, but basically implementing this code that are I'm showing on the slides. It's also available on github, well, where I'll talk about and have an example of this practice and the roadmap for for the future.

A

Also, it's important. So what what's really nice is most of this coding has actually like implementing this component. Config has been done by Chinese contributors that I've collaborated with. So that's that's worked really well and it's nice to see okay first, where are we at today? And what is this thing?

A

Basically, we all know that kubernetes releases as a fart at a fast pace. There are new changes. There are new features coming all the time, but it also comes with a not-so-good side, because if you have to keep up with all the new features that are added- and there are also many that features that are removed or deprecated at some times and when you then migrate or upgrade for from version one to version two or like 111 to 112 or 112 113 similar, you really have to begin to all the changelog.

A

That is the huge release, notes and see if some flag has changed in the in the components and take action as required before upgrading.

A

So that can be very dangerous if you just flip, the version and defaults have changed. You're now running with different a different set of configuration than you used to so it's evolving, evolving, very fast. So when backwards incompatible changes are made, they can break your cluster pretty easily with with the flags so right now, if you want to set a parameter today: API server, for example the service subnet or similar. What address it should listen on then.

A

Well, you specified bind address or another similar flag, but if that flag was removed in a version, then, if you upgrade it's gonna exit, the component is gonna fail because it doesn't recognize that anymore and when you do upgrade, you need to build like. If version is. This then include this flag. If worse version is this other thing then have this replacement, and if the version is some third thing do some hacky workaround.

A

But we all know that the kubernetes api has worked pretty well in defining our applications and defining how our program should be run. We all I mean not. Everybody may be like siamo, but at least it's a representation of the objects and it's kind of the cabana standard. How to do things at the moment. So what is this is yeah. The flag names and functionality between them is kind of insane consequent between the API server controller manager, scheduler and similar.

A

So there's in consents, inconsistencies and we're using flags where we could use instead structured and version configuration, because we also flags are only using key value. We have no structure there and there's no version. I can't say: I want to use the flag schema version, one or I want to use some other thing. So instead we want kubernetes components to ideally have just one argument: one flag which is config so.

A

With this we want to read structured and versioned configuration and it looks like a normal combination. Apiary object, but it's just a file. It's just like similarity to the kubernetes api server object, so the current the current situation is that most components of kubernetes thanks to the efforts we did this summer now at least have dedicated kubernetes repos. So we have case at I/o, like you, slash cubelet, contain that repo contains the struct.

A

The configuration strikes for the cubelet, which you can use to write cubelet component config and the cubelet is the most tested component with component config at this time, the other because it's in beta the other structures are in alpha, so I wouldn't use them, maybe quite yet, but when we go further, we hope to graduate these as soon as possible and make it consistent for all components.

A

So the cubelets queue proxy and cube scheduler can all read the config file. We have work to do with the controller manager and the API server. They can't so there's some internal refactoring to be done. There.

A

There's also this problem of when you're using flags or have have very different component like where different structures may be for like for both the proxy cube, scheduler and cube controller manager, and the cubelet as well are similar to that that they host a very lightweight HTTP server.

A

So this HTTP server, so some basic endpoints, like like health checks, southern version endpoints and similar for debugging metrics, such small things, but it's all the same code that is behind the scenes, used still this code, that is, implementing the metrics implementing the version checks. Http server is using totally different configuration structs because we have no standards here in between these components. Just like how the author of the this code liked it at the time.

A

So now, with this effort, we also want to extract what, for example, leader election configuration for leader election is a generic thing. You want to do later. Election with kubernetes then include destruct in your component and, for example, now they can be shared between the scheduler and the controller manager right now, these these fields and trucks are duplicated and that's unnecessary same for debugging configuration and the HTTP server stuff.

A

So what this could look like to visualize is something like this and then this is kind of the end goal, the most important part and why this starts to look like a kubernetes api style object. We have the api version and we have the kind you can. This also is the same format as, for example, C RDS are using so it's very familiar to all kubernetes developers and pretty straightforward to use all components. Configs in kubernetes are under config that gates that io, so under that the component has its own API group.

A

So in an API group there are a set of structs that make up this configuration schema and each component gets its own API group. Then the kind should be the component. The component, the name of the component and configuration we'll see more about this later, but what's great about this is when we first have like we release 113 or say we release 114, we go GE a stable with the component config feature we would have, then all versions would be we one.

A

But if we do some huge, like change in the structure, then in one 117 or something we can bump, the version have a different schema. We can bump the version we. So that means when the user upgrade to 117, they can either specify this file, which is we want or they can use the new more advanced schema, because the the component of the new version will know both formats and converts between them automatically so and then we also get some structure.

A

We see that under the controller struct, we have different controllers, every controller can have their own key value pair and if there are, for example, there's a generic controller manager thing. If we want to put generic controllers, we can have those there, but then you can also implement your own at that. An example is, for an example, is the cloud controller manager which has the similar things to the normal cube controller manager, but- and it should have the same, look and feel here, but it should still be different per cloud.

A

So the yes, this configuration that you write should comply with most urban areas, API conventions and have that kind of UX, and that makes it familiar to other people when upgrading you must be backwards compatible.

A

So if you release in a new component v2- and you already supported- we one- you have to be able to still read we one, so users can upgrade without any like downtime or any any like problems, so that is, that is really important, and that is what components config is solving that Flags can't do, and with this we can also start to do things like.

A

When we put the configuration in a file, we can also start appending patches. We can have one base cube black configuration for the whole cluster, for example, and we can then, if we have a special cube like that, should implement some GPU feature. Hardware, accelerated thing: I need special configuration parameters. We can, for example, use customize or some other patching framework of llamó json whatever now the examples are showed here in Yama, but Jason works equal well like any kubernetes components in a kubernetes resource.

A

So now that we have these kind of things and have the file in a configuration in a file, we can start doing higher higher level systems and these get-ups patterns, as we call them like you, can store the ultimate truth or what is in your cluster is storing it in some gates repo. Then you propose changes to that gate. Repo to the configuration like I want to apply this kind of patch to our cluster. Then it's easy. It's! It can be easily seen in this llamó.

A

Well change that value to that and then can be done like any other Cabana kubernetes objects like we deploy our apps today and, as we said, we can now share a lot of the common things. So if you want to do a component on kubernetes that supports lead election, you don't have to like write all that code reinvent that wheel again, you can use the shared types in kubernetes, and now we also get consistency for free.

A

So this talk is a bit on, like the cig plus lifecycle side, I'm a lead for cyclist lifecycle, but it also is pretty much API machinery. So it's good. If you have prior knowledge of how the kubernetes api machinery works, because it's this is gonna dive into how it works, but I'll try to explain a sim as simple as possible how the structure is so now with the new refactor.

A

We store the external versions in a component repo so case at I/o, which is a elias for github calm, slash kubernetes has new repos. It has cubelet, cue proxy, cube controller manager and cube scheduler. These repos right now only contain these types. These components config types. So it's easier to it's easy for you. If you are building a higher-level system, utilizing component config, it's easy for you to vendor these.

A

You don't need to vendor the whole of kubernetes, which is very annoying for many and impossible for some, but the internal versions will we'll get to what internal and external means in a second are stored in still in the kubernetes package, and all the rest of the cubelet code and whatnot is in the kubernetes. So we've only extracted this one API package to its own repo for easier ven during that is what's been done, and here we have the example again.

A

The API group, as said, should be the components name dot config that case at I/o. If you implement your own components, you can have whatever you want, but for this kubernetes specific ones. That is the pattern and then the API version should be should work like any other API version. Here's an example, some example values and the kind name should be written like that, like in the example here so here we have the shared config problem. So, as you see here, these are real-world examples of what we have right now.

A

So the cube proxy configuration group- if you are a kubernetes client, if you're talking to the API, so you need some parameters like where is the cube? Config I should access. How often may I talk to the API server with request per second burst and how should I talk to the API server should I eat juice is plain JSON or should I use new, a proto buff or whatever. So these are configuration parameters that are similar to any kubernetes component like globally, I thinkyou proxy. This is okay.

A

This is scoped under the client connection struct, but that is not the case with the cubelets, the cubelet host, all of these fields just on the root and have other names as well. We have cube config with a little C there and cube config with a capital, C and so on so forth, cube API, burst, burst, cube a API requests per second etc. So this is pretty much the same, but it's written in different ways, because it's been different authors and now there's no consistency. For that mean that is configuring.

A

This I have to remember that if I'm told using the cubelet doing stuff with the cubelet I have to write stuff in some way and if I'm, using the new proxy I should use a different pattern. So this these kinds of unification, we we want to we're striving towards by making these shared types. So, instead, the shared types will exist in a generic kubernetes repo like case that I/o API machinery or case at I/o, api server or whatever.

A

We might create a case that I will slash component or something we don't know, but somewhere that that is easily accessible and easily vulnerable.

A

So I don't know if you're familiar with what an external version and internal version of the api is, but here I'll try to explain that so an external type is some set of structs that can be written to a file and that can be read from a file so that by definition means we need to set like when I, when I'm reading a file, I need to know what kind of version is this? What kind of object is this, and what group does it belong to?

A

Is it like a deployment, or is it a component configuration or is it like? A CRD I need to know what kind of thing it is. That's why we have API version and kind, and that's also that's always in that's always the case. With external versions.

A

They always said that, and an external type must also have JSON tags, because you know that in go lower case, if a field start with lower case, it's private if a field starts with upper case, it's public so and in when we're writing these Jason or like yamo files or whatever we always kind of always start with a lowercase. So that means, if we we can't read them to go because the go interpreter, things they're, they're private fields, and it doesn't recognize them.

A

So that's why we must set all always JSON tags which are lower case which override the default field value so an internal type. There should be only one internal type in the system, so in the component there should be this one internal struct that is never written anywhere. It's never read from anyplace, it's only used by the component itself and why we should not have Jason tags.

A

And yeah, so why we have. This is because we might- or we want to support many external versions. We want to support loading, your configuration from we 1 and V 2 and V 3. We all we want to support loading. Those and using those in our component in our application, for example, kubernetes supports many many versions of the apps API. For example, you can specify deployments of type version, 1 beta 1 version, 1 beta 2 version 1, so we we should have this functionality.

A

So then we create many external tags and create conversions, many external versions and create conversions from that external type and its schema to the same internal type and that same internal type. We use everywhere in our code. That also means, if defaulting changes between versions. We also can track that by having only the external pipes should be defaulted. So you we have a default of 10 requests per second in our application in version 1, and we realized that is not a good default. We need at least 100 requests per.

A

Second, we can bump the API version and now, in the new API version, we use a new default.

A

Yeah, so we register conversions from external, so we read the file we decode it into the external type. We then convert it into the internal type and when it is the internal type, we can start using it in the application and to reduce the need for writing. Conversions like this field, is in the external type. Is this field in the internal for everything? There's a lot of auto-generated code.

A

So I've written a sample go application for this to demonstrate that it works and it's accessible in the github location under github.com, slash, luksus, mmm I know. I'll now show what different what you need in order to implement this. So if you want to do this, for your application.

A

You can you can implement this using this pattern. So here's the here's, the structure we have a CMD folder. We have a packaged folder in our package in the top-level config we have the internal type. That is just one and then we have many external types we support. Now we won and we be the one. So we be, the one was our first version.

A

We thought it was pretty good, but we realized we had to make some changes before we release the stable version, which is we don't we want, and then we have a scheme package with which contains references to boat to all all the types.

A

So and it's important to note that the structure of the internal type and latest like in this case, we won, is pretty much the same. There's no extra conversion code there, it's just like it, has exactly the same schema, but between we be the one and the internal type. We need to do conversions, so that's where we have conversion goal in we'd, be the one but not in we one and we'll we'll go walk through what this. What should be existing in these go files.

A

So the the sample application here will basically will have three flags. It has one config flag which allows you to specify a component config and it has some overrides as well. So on a graphical level, it seems like this. So if we set a config file, I realize I've missed the flag name, that it should be config an config file. So if we've said that we, the application, reads it into the external type, when we have it in the external type that might be fields that are not set, so we default it.

A

We when we then have defaulted, the external type we convert it in the inter internal, so we can use it, but if we instead specified just a flag like I, don't need much extra configuration I just want to for a HTTP server. I, don't need a lot of TLS stuff, for example, I just need to set override the bind address, so I'm fine with just having one flag.

A

Then we start with an empty external type. We apply the flag value in this case like bind address.

A

We default it and convert it in the right internal type, and now we can continue using the same code path there and just Marshall this bites again, so you might have seen how kubernetes api types look, what they look like.

A

This is basically the.

A

What is what it looks like normal go-go configuration?

A

Then we have a scheme which registers all these, like that we set, we one be the one here is the internal type, and we have conversion factors between these.

A

Then we specify some tags for order generation, so there will be automatic code automatically generated code here, and we also reduce the defaults that if a field is not set, then we set it to some default value, and here, for example, here is a sample conversion code. So a field that was earlier renamed server address has now been moved into a dedicated server object and address, and then we have a scheme which registers all internal and external types.

A

So the scheme does the automatic conversion, and now what it looks like when we have this setup, we can read a file by just calling runtime decode into and give the bytes that we want to decode and the internal object. We don't need to do any special. We don't have to write more code for for it like do conversion. We don't have to do if this then that we just that is all done for us. We just need to say here: are the bytes? Here's, the internal object, parse any external version to me.

A

If we want to populate defaults, we create the object, we call scheme default or if you want to convert it between objects between versions. We just call scheme converts and if we want to marshal this again two bytes, then we just create some kind of serializer. It can be llamo. Jason are made by default, but we can do whatever like if I want to have a I, don't know.

A

What's what casein netting or whatever kind of format I can create my own C or Eliezer, but then what's important is that we create a version specific sterilizer. So we say that I want to output bytes again with v1 structure or I want to output it with we to structure and then just call runtime in code. So, what's done and what's going to be.

A

Better in the future, we've done the initial work for kubernetes components. Now we have this framework like you can use this so like say, you're, creating an operator, then it's very good for, like the consistency and for your users. If you need configuration for your operator specifically, then you can adopt this same pattern. That's shown in this repo. It's probably probably going to create some better documentation here in 1, 12 and 13 as well. How to write your own component configuration and.

A

Graduating the schemas.

A

Of these two alpha to beta, it's also going to be important. So, like the Q proxy is now alpha. We want to make it better. We have no API server components. Config at all. We should fix that. We should create component config. So now the cloud provider is, for example, I'm moving out of three. Then they should also have the same look and feel as a normal controller manager. Things like that.

A

So if you're interested in the actual code, the like deep details of how it works, you can check out the repo linked in this presentation, but, as a recap, components, config is about this pattern. It's a convention that we use. For example, there are conventions for CR DS as well. That is very similar. We just apply this these kubernetes api conventions on any configuration and reuse, the benefits of Cubana.

A

Quite what makes kubernetes so great, and when we have configuration in a file I said now we can start applying instance overrides we can start doing patches. We can start doing this, get up style deployment because that's very hard to do if you're using flags, then, if you need a lot of if-then-else configuration like so that it gets complex as a new program itself that it's applying flags, then it's not good.

A

But if you have one file the definite source of truth, then then that's pretty convenient and we're trying to unify the UX of using kubernetes trying to make it easier for admins to use it.

A

But this can be done for any combination, extension or any application. Any go program can can do this.

A

And yeah now we have the external types in so that you can render them so now you also can write application using these. These trucks.

A

And yeah: that's it.

B

A

B

I think you are the leader of the cluster lifecycle, so you probably know the cast API and how do you plan to apply this on the class API and maybe also the cloud provider and make sure all then are following the same pattern in terms of configurations.

A

Yeah, so the cluster API does this already. It is using this convention since the beginning, but with regards to the cloud providers, yes we're working on that or also we like help with that.

A

That's in the roadmap, as Ann said right now, the we don't have. We haven't plumbed this true in all places in controller manager, but we're planning to do so so that it gets consistent.

A

Any other questions.

A

Okay, thank you.