Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2020 - Virtual, 4 Sep 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: ComponentConfig Technical Challenges - Michael Taufen, Google & Alexander Knipping, Noris Network

Description

Don’t miss out! Join us at our upcoming events: EnvoyCon Virtual on October 15 and KubeCon + CloudNativeCon North America 2020 Virtual from November 17-20. Learn more at https://kubecon.io. The conferences feature presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

ComponentConfig Technical Challenges - Michael Taufen, Google & Alexander Knipping, Noris Network

Michael Taufen and Alex Knipping will give a technical deep dive into some of the trickier problems our working group has encountered while implementing ComponentConfig. The talk will focus on recent areas of work such as strict decoding of ComponentConfigs, the "instance-specific" config problem, and how ComponentConfig applies to hierarchical components like kube-controller-manager. The working group deals with broad issues that affect a large number of users, but you don't need to be an expert to start helping. Alex will share how he was able to quickly make an impact on these problems as a new contributor, and how others can get started too.

https://sched.co/Zeun

A

Welcome everyone to our session um for uh working group component standard. uh My name is mike tauffin. uh This is my co-presenter uh alex kniffing um today, we'll be presenting on some of the technical challenges that alex encountered while trying to implement various pieces of component config.

A

So just quickly before we get started a little bit about the working group. We formed this working group a couple years ago with the idea of trying to develop a standard foundation for uh core kubernetes components to build.

A

On top of, um we have many core components such as the api server, the scheduler, the cubelet uh cube proxy um and many common functionalities that are distributed across these components, but we didn't necessarily have standard implementations of these functions across the project, and so that's the idea behind the working group is to provide a place where we can talk about uh and design and standardize these solutions. So these are a few examples: um how components are configured, uh how they sort of report their status.

A

Various integration points and and sort of logging in metrics are the main areas. So I'm going to turn it over to alex and let him talk about sort of how he was introduced to the working group and some of the projects that he's worked on.

B

Thanks mike uh yeah before we get started just a quick overview of the agenda today, so we already heard the introduction uh from mike um I'll be talking about points two and three. um Basically two of the problems I encountered or I dealt with uh when working with the working group and then we'll close up with some uh shout outs, other works with durian and, of course, q, a alright.

B

So uh this is me my name is alex, I'm a site, reliability engineer at x-ray in munich, we're building a german iptv platform, and it was around mid-may or admit last year um that I got interested in contributing to the kubernetes project. I think it was um for kubecon in europe around may.

B

When I like went to a couple talks- and you know I saw a couple of uh like working groups or special interest groups and themselves. I think one of them was a c-class life cycle and it was really a nice and open environment and a very like welcoming community, and it was around that time um where I was trying to get into the project. I tried to contribute a little bit. I started with some golden golin issues, um although that got a bit boring after a while.

B

So I was kind of looking for a new challenge, but it was a bit daunting like kubernetes being such a big project, it can be quite tough or like at least at the beginning, um to really get a grip on on this. If you want to start out contributing so um the guys from sicar's life cycle, they pointed me actually to the working group, and I didn't have any prior experience to that.

B

I've had little go experience, um certainly no open source experience, uh and uh he just told me like you know just just talk to them, um talk to mike and lee at the time, and uh I went there at a good time. Basically, they just had a contributor onboarding coming up and we discussed a couple of things: they're, really really nice and welcoming um just didn't really matter that I didn't have any uh prior experience.

B

There's a lot of issues actually um in the community that can be tackled by people with a fewer technical skills, but they walked me through the code base uh regardless, and you know we talked a little bit about the projects I could be working on. uh Why am I telling you this? Well, I want to structure this talk a little bit uh like this to to take you on a journey.

B

um It's like my journey through uh through the project, at least like over the past year, um and just really, you know, show you what kind of problems are dealt with and how I learned step by step, and hopefully you know if you're looking to contribute or if you're thinking about it, but you're also a little bit scared as I was you know.

B

Maybe I can take away some of that and show you know it's it's really, not that not that of an issue, and there are loads of friendly people who can help you with it all right. So um back when I started uh coming to like the meetings of the working group, there was just this pr was just merged, basically codec factory options, which allowed some options to pass through codec factory, mainly like the pretty encoders and the strict decoders.

B

What those are will explain in detail, but basically the problem was that when you load a component config, which is the ammo file um which configures a component like the api, server or cubelet, those yaml files can be malformed, so they couldn't have duplicate fields or typos and field names in the ammo, and those arrows would be ignored right and this could lead to undefined behavior. So we can't really assume like which of those fields are intended or not um and back then you know that wasn't really happening.

B

Like anything straightforward, I think it was just like the first field I encountered it took, but you know, as I said, it could lead to unexpected behaviors and um with this pr merged. Basically, it was creating the prerequisite to deal with this problem to actually load the config file um and arrow out on those you know um like malformed fields, so this was like the first challenge the guys suggested to me it's like. No, why don't you deal with this? It's a good.

B

um It's a good point to you know, get familiar with the code base to learn a bit about api machinery, so this is what I did. uh This is what I you know told him I will tackle on. So you know it sounds quite straightforward. You know we can. We have uh male 4 gamma files and if we load them um it should throw an error right. So uh sounds quite easy right. Well at that time you know I didn't really know anything. uh You know what was the component conflict?

B

Well, obviously, it was the ammo file you passed to the component, but you know how does it work like? What are the concepts behind it? um Also, what's the codec? What's the codec factory, um I didn't really have any clues, so you know, let's start at the beginning, and you know walk a little bit through the process.

B

All right, so this is the configuration it's called cubic configuration, and this is a component config and it's for the cubelet all right and um when, if you're, if you installed a cluster before, if you did uh kelsey high towers um kubernetes, the hardware you've probably seen this before created it yourself, um this config file, you pass it equivalent. As I said when you start it now. This goes alongside to right now, the cli flags as well.

B

Now the cr flags uh configure the cubelet too, but um like in the future, we want to like phase out the cli flex completely. Why is that? Well, there are loads of problems with flags. One of the bigger ones is that you can't version them right. It's really tough to deprecate flags in a way that it doesn't break. You know older clusters, or things like that. So this is just one of the problems.

B

um I won't go on in detail here if you're interested in that and there's still lots of work around this section like deprecating cli flags, um I recommend you to talk from the working group last year at cubecon, san diego. I think, um and those uh some great detail into that, so this all happens on the note right all right so and this is the problem, basically we're trying to solve.

B

So, if you have, you know a config here and you have a duplicate field like one's cluster, local, the other one is my cluster local, like which of those fields, you know do actually, you know want to load. You know it's not really clear and uh the same as this area is this like a field which doesn't really exist in the individual beta one api.

B

So what do you do with it right um and, as I said, you know, the idea is to to load these configs and if you know, um through the process of loading it, if you encounter any of these errors, like you, you error out all right. So the first thing I did basically, I loaded up the the editor. I jumped right into the code um and digged a little bit through the gcublet codebase and um basically I I found this function right. You have s loader.

B

uh It takes a file system and a cubit file as a string, so the file system is just an abstraction, basically, if you're for testing, uh whereas the cubelet file is the path to the to the actually q black camera uh yamo.

B

And uh what does it do it's the first line here in 148, it returns a cubelet scheme or like uh basically a new scheme and codex like here. Right, um didn't, really know what that was. So you know, follow the function and in scheme dot go you see, it returns a scheme, an encoder factory and an error.

B

So there are a couple points here which you know. Obviously I didn't know exactly what they were. So we have a scheme um which gets returned from from the runtime package. Then some things are added to the scheme and then we take the scheme and then we turn some codecs right. So um obviously the question were here like what's the scheme, you know- and you know what's at it and um looking up like digging into these questions, really let me down the akai machine rabbit hole.

B

I won't go into too much detail here, I'll just try to give you the overview, but these are like core kubernetes concepts. It's really good to know about them, especially when you want to start out.

B

So, let's start with the scheme right, um let's say you're like cube control and uh you fire up a get pods and it's like you know, ask the api server and the ibi api server responds like here are some structs. You know here you go now. How does cube control figure out if those were actually parts which were sent over here right? um These are just like, for all intents and purposes, just you know some some ghost trucks it received and this job actually from translating it from like some struct to a known type.

B

uh This is what the scheme does uh one of the things it does actually, as I said like the scheme, is uh one of the core concepts in kubernetes and a bit of a like a swiss army knife for api machinery.

B

But one thing it does: it provides a mapping between an object so like a ghost truck for the part, for example, um to a specific goal. Type, which is called group version kind of gbk in short and you'll, find this a lot around. The code base this terminology now, in order for it to do it, you know we need to tell the scheme uh what kind of goal types it it knows. Basically, we have to tell it.

B

You know okay here, this is what a pod looks like with version v1 um and as soon as we have registered into that, we can pass on an object and we will turn this to group version kind if it knows about it, um so it's really an elemental process which happens all the time, this kind of conversion and if you go and dig through the code.

B

Basically, when you look up the scheme, you'll find you know a comment that says it's the foundation of the version, the api and configuration over time, and, as you might know, you know, every api uh in kubernetes is version or like pretty much. Everything is version and the scheme basically provides us. The tools to, I know like work with these versions to convert between them um and to do yeah, conversions.

B

Basically- and this can easily happen- you know if, for example like the client is, uh has a different version than the api server, uh like, I think, with the version sku policy, we allow like two major versions to differ of. I remember correctly, so you know we definitely need to have some tools to convert between these different versions, all right. So this is what the scheme does. You know it gives us some some tools to convert between versions and speaking of versions. This is exactly what we do here.

B

You know we tell the scheme which versions are available and we can see here is an even better one. It's the one! We you know it's the the api version. We see here, it's an even better one. This is the basically the external api it's called, and then we also register another thing in here and it's just called cubic config without a version. Now this is what is called the internal api, and this was very confusing to me at the beginning. Like internal excuse me, internal and external versions, you know what does it mean like?

B

Why do we need this? And um it's basically like the internal version is like a hub type right and it's used to convert between any different versions. You can obviously you convert between external version and the internal version, but you can also convert between two external versions going through this hub type called the internal versions. So let's say we one alpha. One has you know fields a and b and b, one better.

B

One has fields b, c and d: the internal version would know about all fields, a to d so and it would, it would provide you with some tools to you know. Convert between those versions. Do some defaults, um all kinds of things you know.

B

So in the end, you know you don't lose any information when you do these conversions all right, so we have a scheme. Now we register some versions into it and then we pass it to a codec factory. uh We get some codecs back all right. What's the codec factory, let's have a look now, if you look through the code, you see that for codefactory it says it provides methods for retrieving codecs and serializers for specific versions and content type.

B

Now you can see here, the codec factory has a scheme, it has serializers and has a decoder a universal decoder. Now there are a lot of things to it, but on a high level, basically a serializer or deserializing. What we do in our case is we convert it from a wire format, so a yaml file into a memory representation which is the go struct.

B

Then we take the decoder and take the go struct and map it to an internal or any external version doesn't really matter, um and basically this is where the the actual conversion happens between the between the actual uh versions. Yeah. um And if we look down here, we can see it also has some options right. So uh in this case we have strict and we have pretty and strict is what we want right. So configures all serializes, the strict mode, and this is exactly the behavior we need in this case.

B

um So if you know the serializer encounters any errors um or like any any unknown fields, you know doesn't know about. You know, overthrow and error all right, so we have everything covered. We have the scheme. We have some versions which we which we need. You know the internal version and the external version we've on better one in this case and we have the codex. You know which you know we can use to actually convert and uh and load the file all right. So the idea is just has the strict option here.

B

You know behind the scheme and should be fine right. Well, you know technically, yes, it's it's just how it works, but there's one more thing. We forgot- and I forgot this at first as well, uh so I implemented this pr got merged, but then you know, then um we, you know we got some some ideas that you know we actually might break some clusters, some older clusters.

B

How does this work? You know what what happens. um So, let's say you know we introduce this change from. uh Let's say 119 is the current uh current version. Let's say we introduce this for 119., um but the thing is when uh 119 gets released. This doesn't actually increase this version here. So you know, the version of the api of the cubic configuration doesn't go in lockstep with the kubernetes release cycle. So what would happen you know people would upgrade. Let's say someone has, has this config here and has a duplicate field?

B

The change would get into the tree and the the user would upgrade his tubeler uh and what would happen the cuber would fail. You know which would break the cluster in the worst case, and this is not something I as user would expect right. So because this version didn't change. You know, why is my cluster not working? Basically, um this version kind of especially like better and upwards, uh implies some kind of stability in this case, so you know we don't want to break any user clusters, so we need to figure out.

B

A way to you know only strengthen this when this version changes- and this is exactly what we did. You know we strengthen our the validation on the version boundary um and basically, we want to for v one better one. We want to decode with some warnings, but on v1, better 2 and upwards. You know we want to have the expected behavior that it will throw an error when the ammo is now formed and what we introduced here is called the new linear scheme and codex.

B

It's like a helper function- and this is used to you- know, return a codec factory with disable strict, so like the strix here. That was disabled just for a single version, and this gets called with the scheme of the version for even better one, and just for this version. It returns um like some codecs, which uh you know have strict disabled and everyone upwards. Every version upwards will have the strict serializers enabled all right, and we basically did this for every um component so for, like the q proxy, the cube, scheduler and cubelet.

B

As you see here, the first list we implemented these first um and then we came. You know upon the issue I just talked about, and then afterwards we shipped basically the lenient paths as well, so we call it so you know this was all in the same release cycle and um so the user didn't really. You know uh recognize any of the of the issues um you know unless they were running from master but yeah. So this only like all these changes got in into the next version.

B

I think it was 1, 17 or 118, I'm not sure anymore, all right yeah. So we did this for these components. uh Now, as you can see here, the cube controller manager is strikethrough right. So actually- and I didn't- I didn't know this uh before so- the cube controller manager doesn't use a component config. I think it's one of the only components which doesn't.

B

Why is that? This is what I uh asked myself.

B

Why doesn't the controller manager use a component config, and I asked it to the guys, and now they told me well that's difficult. The cube controller manager is like works, a little bit different than um the other components in the new kubernetes cluster or ecosystem, and let's do it as we did earlier. So, let's start from the beginning, you know first of all, what's what's controller manager, and this is from the official documentation.

B

um Basically, the controller manager embeds core control, loops ship with kubernetes and a control loop, watches the shared state of a cluster through the api server and makes changes attempting to move the current state towards the desired state right. So um basically it it looks like to the api server and it checks you know is: is my cluster in the state? It should be right if someone starts a deployment and the pod isn't rolled out on on this on any node.

B

Yet you know the uh there would actually be a controller making sure that would happen right, and it's like this- this basically uh infinite loop, just always checking if a certain condition is true or conditions, and if not, it will try to reconcile that. So this is a controller and the controller manager is is a binary host. Basically, it is um also like a hub or a host, which holds all the controllers, basically or almost every controller uh and starts them, make sure they're configured and provides a home for them more or less. Now.

B

The idea is that individual controllers are interchangeable and our users should be able to. You know build their own controllers or put their own controller managers for industry card, and so this is what we have to keep in mind. You know. um Well, probably not many uh are doing that, but you know it should be possible. It's one of the core principles of kubernetes that you can exchange things and build your own stuff.

B

All right, um and you know, if you look into the code, uh you know you can see. You know what the controller manager does. One of the things it does. uh It has. This function called new controller initializers and it has, you know a map of basically if the key of the name of the controller and the value is a function which starts the controller right. So we have, I think, over 30 here- and this is this looks quite daunting.

B

You know there's quite a bit of complexity in here, but you know in the end, you know that we're just starting individual controllers and this can be uh configured from the command line. uh Of course you know we don't have a component config and um it takes a list. I think I'm not sure what the flag is called, but it takes a list and you can turn on or turn off uh specific controllers and by default it has a star. So basically it turns on all the controllers right.

B

So next question would be okay, so it starts the controllers. But how are they configured right? As I said, control manager deals also with configuration of of the controllers or provides uh primitives for that, and basically the controller manager configuration exists and it's also it's a version api right and it is uh like uh yeah, a version api holding all the configuration uh necessary to start the controller manager and the individual controllers right. So in this case, it's what we call a container config.

B

So we see here it has a type meta which basically means it is. It is versioned. You know it has like an external version, but then we have the individual controllers as fields in the struct, and each of these controllers basically holds the or like sorry. Each of these configurations holds the configuration for the individual controllers.

B

Now we do have something you know curious here. It's called cube cloud shared and it holds configuration for shared related features, both in controller manager and cube controller manager. So actually the cloud controller manager used to be part of the controller manager, but it moved out of three because it was just too much complexity and the individual cloud controllers were differing. uh You know too much after a while, so you know the maintainers decided to take it out of tree and maintain it as an individual controller, basically which is not maintained by the controller manager.

B

So this is the first. You know the first example you see of everything should be, should be plug and play in a way. So uh you know the controller manager doesn't really start or care anymore, that much about the cloud controller manager, but there are still some configuration fields which are needed by both parties right. um So this is why we have this configuration uh substruct, basically all right. So, as we said, the container type, the controller manager configuration is version.

B

But if we look at the individual controllers, um we can see that they are actually not versioned. You know they only have an internal api and the first thing that gives that away.

B

Is you look to the import paths and if you see that these are coming from like entry, so the kubernetes slash package, um basically directory it's where the internal apis are like when you go to slash config in the end- and this is what it looks like right conceptually, so you have one container tile and this is one version, but it loads all the all it like um embeds, all the um configurations of all controllers.

B

So if anything changes here, you know uh it goes into the same version of this, so we basically version it more or less and step, although these are all internal right, these are not really externally versioned and if we go into you know any controller or like a specific controller configuration we can see, we don't see a type meta here indicating this is just an internal api right all right, so you know actually there's no real argument against not having you know a configuration or like component config, at least not technically.

B

You know we can easily just create a mo file. Give it a like gvk, a kind api version, an api group, and then we just list all the fields, and now we can. We can load this and we get a conflict from that. So it's not really a problem technically right.

B

So if it's so easy, why don't we have it? Yet? You know that was my question. Basically and uh well you know it's difficult. It's it's not! You know it's. It is a technical challenge still, um but it's also an organizational challenge.

B

So we need to think about couple of things you know um so, for example, what happens if a controller moves out of tree? So we saw this with the cloud controller. You know we had to introduce a shared configuration field. Now, if we have to do this for every controller which we move out, you know it gets a bit busy in the code base and it's not really. You know a nice solution. Probably and then again you know what happens uh if control the share config fields.

B

Actually I think I looked at it they're about 10. So, like a third, a little bit less of a third of the controllers, we have, in the controller manager, actually share some configuration fields. So we need to deal with this. You know what, if you know, if we have a single controller, which we don't want to get uh start from the controller manager? How do we, you know, build and build a component config around that or what does it mean for our yml file?

B

um Then what happens uh if a con, if the owner of controller or like the group owning a specific controller, um you know, doesn't want to follow this uh versioning? You know they're kind of bound to it as soon as we load it from a file um or what happens if users want to implement their own controller manager. Now, how do they like?

B

Can they write their own component config, like what kind of primitives do we provide for that, and these are all like, like kind of tough questions, you know it's very much an architectural uh problem uh and also like a community. You know organizational problem, I would say you know, there's a lot of you know um a lot of targets and and brainstorming which has to go into that, and it's quite difficult.

B

Now um one option would be, for example, to you know just give each controller its own external version. So we still have you know the same api group. Maybe we can share, we still have a container type, which is version, but then we also have individual versions. You know for the controller, and you know they can move as quickly or as slow as you as they want.

B

um You know this. This could work, but the problem is like. We have a lot of controllers. So, if you think about having external apis, maybe several external apis for each controller, this can get quite messy. You know there's a lot of lot of code, a lot of duplicate code, probably which has to be written um it just you know, quickly, blows up in proportion. So it's it's a possibility, but it comes with some serious tradeoffs as well as everything basically, and it's just something you know we need to get a consensus around. You know.

B

If we want to do this, so actually like the community has been trying to figure out this problem, you know, since I think 2018 uh there was first students who tried to you know, merge the pr which introduced a conflict for the controller manager. This one got closed because it was missing a cap, kubernetes enhancement proposal.

B

So shortly afterwards it was lucas who started the discussion um around the component config for the controller manager uh start with some brainstorming and a design document. I have it linked here if you're interested by the way, I'm just on slides you can, you can go to all the links um and you know there was. There was the initial discussion around it.

B

Unfortunately, you know it kind of got a little bit stale. um There hasn't been much uh around this topic um still tried again in 2019, uh failed to gain some consensus. um I think it also, you know slipped under the radar of some people. uh This cap. uh Unfortunately he uh in the end didn't have much time and uh had to deal with other projects.

B

So um basically, this kind of got stale a little bit abandoned as well so and yeah, and- and this basically shows you know the the problem- uh you know one of the big bigger problems of such a big community or like open source community. You know it's quite difficult to get these changes through now, if it's not like the most shiny feature, you know all the core main contributor companies want. um If it's like some cleanup, you know and some some housekeeping you want to do.

B

It can get really really tough to get consensus on these guys on these things, um especially if there's you know no one real, you know with no no big contributor driving it. It can get really really hard.

B

um Okay. So what's the plan? Well, I I thought you know. The current plan is: let's pick up where uh stuart left and um you know try to get the conversation going again. um You know. Take his cap, maybe extend it a little bit and try to get everyone on the table, and uh you know you might think now so, okay. So what what's? What's the plan now like how? How should it end differently? Then? Then, what's your day, you know, honestly, I don't have a. I don't have a answer for that.

B

You know, um I'm definitely not not more qualified than uh than stuart was, but you know at least you know, I think we have to try. This is really an important um issue and um you know we just have to you know, get the conversation going again and I'll. Just try to you know, stick with it or like we as a community. That say we should try to to really you know, stick our heads together and try to come up with solutions to this problem um yeah.

B

So that's the current uh situation right now, of course, you know always looking for some help or we are always looking for some help on this. I just mentioned two issues or working on there are loads more. um You know the working group could use help with. So, if you're interested in any of that, I think it's a really good way to learn about the code base to learn about some core concepts in kubernetes in api machinery.

B

You know, please don't hesitate to just you know, join our slack or you know, come to come to the working group meetings and uh you know we'll be happy to onboard you and help. You uh become a contributor all right.

A

B

Right thanks alex.

A

uh For the presentation, um and thanks for all of your work in the working group um as we wrap up here, I just wanted to quickly highlight a few things. So these are some of the other projects and areas of work in our working group. um Some are being worked on actively. Some are not uh so. We have a lot of component config stuff, as alex talked about in the presentation.

A

um There's some work that overlaps with that around command building as far as sort of dealing with flags and that migration to config there's a bunch of work going on uh a contributor whose handle is cerrathius is leading that around standardizing the logging subsystem in kubernetes and uh kind of redoing. A lot of that um and finally, there's there's some potential work around uh standardizing component servers and some of the common endpoints like healthy um and configsy between them. um So if folks are interested in that, please feel free to reach out to us.

A

We'll have some information about that right at the end of the presentation um and then the next thing. If we go to the next slide, I just wanted to give shout outs to all of the folks who have contributed in some form or another, maybe just a couple lines of code, maybe a major project, two working group projects, whether that's contributing to things in the core kubernetes repo or in our component-based repo in the past year. So thank you to all these folks really appreciate all of your hard work.

A

And finally, this is how you can get involved with our working group. We have weekly meetings on tuesdays at 8, 30 a.m. Pacific time, there's also weekly office hours for folks who are working on things and have questions or need help, and that's the same tuesday, but at 10 am we have our mailing list, our slack and our github label as well. So please feel free to reach out to us. The best way to reach us is typically on our slack channels.

A