Cloud Native Computing Foundation KCD Berlin 2022, 15 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Managing Multiple KubeConfig - Simon Bein, Kubermatic

Description

Maybe you have run into this situation: You have started extensively using Kubernetes and now manage multiple clusters. But every time you want to switch to another cluster, you manually need to change the file or context.

Website: https://www.kubermatic.com/

Organized by @Microsoft @kubermatic7173 @SysEleven

Thanks to our sponsors @CapgeminiGlobal, @gardenio, @sysdig, @SUSE, @anynines, @redhat, nginx, serve-u

A

A

We try to make it really easy for you to spin up a cluster on any kind of hardware on any kind of platform you want well and as a result, I just have lots of clusters and lots of configs, and today we're going to talk a bit about how you can manage these I also have to say, in addition to a T I also have a background in economics and in economics.

A

What we usually say is, if you want to look smart use a chart, so I did exactly that, for you guys, here's a chart and, as you can see indicated by the beautiful blue bubbles, we're gonna take a look at five Solutions today, five completely different solutions, and, as indicated by the x-axis of the chart as the stock progresses, the setup effort for the solution is going to become more complex or you at least to have to set up more.

A

But I think there is a trade-off in the sense that you will get more flexibility and more functionality.

A

um With that being said, after the five Solutions, you will also hear me complain a bit about shell and the way some stuff is just really horrible.

A

um With that um I want you guys to remember that this is an opinionated talk, while I will present five Solutions as objectively as I can, uh of course, I have an opinion in this. I also have a personal stake in this, which we'll find out later more about, but overall, um it's important I think if there's any takeaway from today's talk for you guys, it's that you need to find what works for you right. You need to find a solution that fits best.

A

Your situation and everyone here in this room is in a different situation in their native's Journey, but also in general, how they would like to organize things um with that. um Another part is Life's, too short to always type Cube CTL. So if you, in any case in this talk, see a KC, that is just what I have Alias Cube CTL too, and that's what I'm going to go with.

A

So whenever you see Casey, it is short for cube CTL in this talk with that, we can hop into some basics for cubeconfig, um usually hoop CTL manages access to its clusters and also the whole manage watch user access which cluster in one single file. It's the you config by default, it's in your dot, Cube directory under at the file at the file name called config.

A

However, this file can store multiple clusters or multiple users, and it's using like a helper construct called a context. So context essentially is a mapping of a user and a cluster, and by that you can basically have all your clusters and all of your users in a singular file, and you can, by default, switch that using cubicity error, config use context.

A

One thing that's interesting, though, is that the file is structured in a very specific way, and that does not allow you to just append new cubeconfigs. So if you just had two files, you cannot just concatenate them together and we're going to have a quick look. Why that is here. We can see a sample, a structure of it and you can see it's divided into three major top level. Keys clusters contexts, users, and you can also see it is a yaml file by default like even though it doesn't have the file extension.

A

The format is by default, a yaml file so, but that means of course, if we say one functional cubeconfig consists of a cluster, a context and a user. We cannot just append the files next to each other. We have to insert at these three locations.

A

um In addition, also, the current context of that file that is currently used is also stored in a field called current context, and by now you can guys can already kind of see that we need some sort of tooling to merge these into, and the good news is actually for all the purists out there. You can actually do it with Cube CTL um and how that works. I'm going to show you so the first thing is that the cubeconfig you use the DS. The default is dot Cube config.

A

However, you can overwrite this to any file path you like, by using either the cubeconfig environment variable or each cubecdl command also takes in a cubeconfig flag and guys we still have some seats.

A

You can go here in the front if you guys want to so um what maybe some of you or what's new- and it was new to me- is there is actually a little hack that you can use to merge multiple cubeconfig files into one and um I would like to you guys to pay attention now to the cubeconfig variable at the start of this command. You can see multiple files are separated with a colon, and if you do that, it only works by the way for the cube, CTL config command for all the other ones.

A

It doesn't work, but for this command you can actually use that to concatenate them like running, keeps a config view with that as parameters, it will actually show you a merged View and well, if you pipe that into a new file, that is your new merge config.

A

um Similarly, you can also split them if you use the Minify flag in combination with a context flag, so it does work so everyone, just if you want to stay on the pure side. Maybe you can Alias also these commands, but obviously these come with some downsides right. One downside is, if you decide to merge all of them in one file, you will get one gigantic file and you need to be aware of what you have named. What or if you decide to split all of them.

A

You would always need to set cubeconfig environment variable, and you would also then need to know, of course, always the full path for that or you would have to build some sort of areas. So, in the end it reminded me a bit of this situation right here.

A

um And I just wanted to show you guys, but you have heard of it. It is possible, um but with that I think, let's dive into some more advanced Solutions and we did have a survey internally at Cuba Matic.

A

What what like kind of our developers use and a good colleague of mine uh voycheck, he said well I just have three lines for this and in honor of him uh the next solution is called voice, checks, genius, reliner and I'm, going to show you first what it does and then we're gonna look into how it works. So a little prayer to the demo gods and we'll get on with our first little demo, um so just to show the setup that we have real, quick, so I've just created a directory called cubeconfigs in that directory.

A

There are three sample Cube configs. If we just look into them real, quick. So, for example, if you look into the Simon's fun cluster cubeconfig, you can see it's just a standard config and it just contains a single context. So if I go in here, if I go in here, you guys can see. There is only one context and it's a very simple free files, simple configuration for this now. If we have a look into this, um so I've called vertex solution, kcsw for qcq config switch.

A

If we hit enter what we can see, is it pops up a little dialog that shows exactly these three files that I just showed you and that does support some little fuzzy search. So if I, for example, type in Simon here and I hit enter he's gonna set this so for the purpose of this demo um oops my Mouse ah there's my mouse and for the purpose of this demo, I built in a little prompt that shows you the current cluster. Okay, just to speed this up, but just to prove it to you guys.

A

If we would uh Echo cubeconfig, we can see it points exactly to the file that we just selected in the dialog and well now. How does this work with Reliance um oops.

A

So um yeah this is it usually I, don't like showing code in presentations, but I think the lines is perfectly fine. Also the slides will be available, so you can just copy these later on, but essentially what we do is we give it the search directory. We run a find, only search for file types and then we pipe it into fzf.

A

Fzf was the one that showed us the fuzzy finding and that little pop-up dialog, but you could pipe this into any type of command line, prompter that you would want like it's up to you and then it just sets Cube config to whatever you chose so I found this pretty neat, um if you say ah I, don't really like splitting all my Cube conflicts into files, which is a requirement for this.

A

So if you want to use this solution, you would have to split all of your Cube configs into multiple files, and each cubeconfig can only contain one context because it just searches for the files and sets it to whatever is in the file. And if you say I don't really want this I want everything in One, Singular solution, um then the next. The next solution might be something for you. It's Cube CTX, um since it has like 13k stars on GitHub I'm, assuming it's rather familiar.

A

It was built by Ahmed during his time at Google and I'm, going to demo to it first. So we'll have a look at the next solution. um Real quick for this solution for this solution. I have created um uh all in one Cube config, so just I've merged just these three configs, because obviously, if we only want to use one config, we need to put in all the contexts for this. So if we now run um Cube CTX, which is the command again, it also integrates with fzf.

A

Maybe that will be the red lining everything integrates with fzf, but that's what we do and if we go now to Simon's fun cluster we'll see it has a switch to contact successfully.

A

um I should mention that this also comes with another command. That's called Cube NS to switch to names, spaces just to be clear in this demo. It doesn't work because it goes to the cluster and checks. Actually if the namespace exists, so I cannot show it to you here, because I didn't want to run three clusters for this demo now, um just to be clear, it can also do that.

A

um However I- and this is now where this talks good, really opinion. It I have struggled a long time with this functionality, and the reason for this is that I like to be in multiple clusters.

A

At the same time, that was the one thing that I would really like to do and if we go just into here- and let's say you can see now here in the bottom- I've I've spun up a new shell and, let's say um oops, so we are in Simon's, fun, cluster, sorry and now, if I run Cube City oops qctx here and if I switch to let's say uat environment and what's going to happen, is it has switched down there perfect?

A

But if we go up here and we just refresh the prompt we can see, unfortunately, it has also switched to uat environment over here, and the reason for that is that these two shell sessions, that we just see that we just saw they both share the same file and when you run Cube CTX it modifies that file.

A

It just sets the current context field to whatever was selected and because both of them depend on the same file, it will change it for both of them, and that was something that also is true for namespaces, so you could, for example, in these two shell sessions. You could not be in different namespaces, at least with the board, with the tools that are on board. uh Oops. Yes, so for this um there then, is another tool that has become quite popular: it's called QB.

A

um It's also you can see 1.2 K stars on GitHub, so it's also rather popular and I'm, going to show it to you oops how it works now so solution number five is QB. um Qb also takes in a directory and just reads in all the files I'm not going to show that one that part again, but if I just run q b c t x, um it again integrates with szf I. Think at this point. If it comes a bit of a meme, almost let's say we switch to uh to uat.

A

Maybe, um and now we see something weird has happened right. Suddenly, our shell prompt looks completely different right. It's just now: it's not the nice green and blue anymore. It's just suddenly just a white prompt um I want to show you guys what happened here. So if we opened us another shell and if we just run.

B

S e f and we crap for cube.

A

We can actually see what has happened here so, even though it looks like here, we are in a brand new shell and QB has already finished it hasn't, because what QB does is QB lets the qbctx command running. It leaves it running, and that command then spawns a new bash which just takes in all the settings from the old one uh and the reason why it does that is so.

A

It has control over the new bash because it wants to manipulate The Bash for you or the session itself for you and it in order to do that, it needs to be the parent. So basically you have then this hierarchy chain, where you have the normal shell, where you started that started QB, then you have the QB process inside and then that starts another bash and it tries to hide that by default by making just look the by making the batch Loop the same, but for this demo I just wanted to show it to you.

A

So this is what actually happens, and here we spawn a new shell session. But, of course, the cool thing with that is because it does control the shell. We can do something like this, so if we uh do now um so cubes.

B

A

So we can see we're in uat here and if we also start now, QB CTX over.

B

A

We go into Simon's fun cluster here, so just again: Cube CTA config context. We can see the bottom one is in Simon's fun cluster, but the top one will stay in uat environment. So this does allow us to be in different clusters or different name spaces at the same time using different shell sessions and that, of course, from a functionality perspective, at least for me, was a huge win.

A

However, it really bugged me because first of all, is I, usually have quite a long RC file like bash RC or Z shell RC file, so it has to source that file every single time, so that takes that time again, it's for the demo, I've quickened it, but if it takes two seconds to Source your bash RC well, this prompt is also going to take two seconds.

A

The other part is, um it also is, in my opinion, a bit resource hungry, because if you think about it you have the parent shell, then you have the QB process, and then you have the child of QB, just for like manipulating a config or like an environment. Variable I find that not nice and then there's also stuff like and you have nested shells.

A

You need to set your history so that the history is shared across NASA shelves and it's something I didn't really like and then in I guess a night where I felt a bit over the top I felt a bit hubris. I decided Well I'm just going to build my own, so I just go out there uh and build my own and I.

A

Try to solve this problem um now, I'm going to demo it to you first and then I will show you how I solve this problem, but let's go for the demo first, so you can see uh conf and by the way, I would really love a star. It's at 15 Stars, the other one had her like at a thousand. uh That would be really cool, but um I'm gonna show it to you now.

A

um So let's say we are so we are. We are in the top pane here. So if we run um concept, you will see I actually built in a little pop-up dialogue. So no fzf required on your machine. If you don't like it, but it can also do uh fuzzy if I am remember correctly yep, so Simon's fun plaster. So we are in Simon's fun cluster that work. That also went reasonably fast, and it is because, if we uh Cube config now it just sets that variable like it doesn't do anything else.

A

It just runs an export after running the command and that's it and we can also see down below.

A

um If you run out concept here and let's say we go to uat environment down below here we're in uat down below, and if we go up here, we are still in Simon's fun cluster. So it does do that. It allows you to have these two parts um and it allows you to be in different shells.

A

um This maybe now raises the question: how is it so fast and we kind of have the answer? It doesn't need to spawn a new shell and then I think everyone who has children knows what comes next is why don't we need to spawn a new shell and I've already mentioned that? Usually a child process cannot make changes to its parent process.

A

So you cannot just say: hey my binary, that is running that cannot set the cube config for the parent and the parent is the shell that you are using, and that is where you would want to set the cube config. So it cannot do that right. Well, yes, but actually no, there is a way to do it and it's if you abuse a bit bash and that shell functions and what that means more precisely, is Bash or that shell functions that are sourced in the parent.

A

These can make changes to the parent shell like it is actually possible for this and By changes, I mean at least it is possible to set environment variables, which is essentially all we want. So let's have a closer look into this. What exactly is confident the binary I just showed um there is the binary part um that is named confgo and that does all the heavy lifting.

A

In the background that you see like manipulating the files copying, the files merging different files, there's all done by confgo, but there's also conf, which is like a bash or that shell function- that all it does is it calls conf, go waits for the output and if the output contains a certain string, it is going to export cubeconfig to it. So whenever you run conf set, it's actually just going to talk with uh conf. The shell wrapper is not gonna talk directly with the binary, and that is the big trick.

A

But um before now, I just keep spamming you with new slides and complicated graphs, I'm just going to show it to you what it looks like and what that actually means, then for the end user.

A

um So what this looks like if, if you run conf go and I've I've just nicely packaged it into a command and to sew it in inside the demo.

A

um If you run this, you will actually see what the conf function looks like and we're going to go through it uh step by step, I'm going to explain it a bit. So this just says: okay, we want a bash function or that shell function, that's called conf. uh It's then just gonna run confgo and it's just going to pass in all the arguments that you gave to conf, and it saves that in the result of the um of this variable so um and then what we just do is we look.

A

Does the result container string, cubeconfig change, which is just something I, have coded into the Quantico binary to return. Whenever we want to change the config, then we export cubeconfig and yeah.

A

This here is just some shell stuff, but basically it's the it says that um split out, cubeconfig change out of this one is is I would just take in the path that comes after a cube, config change, so confgo essentially returns Q, configs change, then a colon and then a slash, Simon's cluster.yammer, for example, um and it also has like a cleanup function to clean up unnecessary uh parts of it.

A

um In addition, I would like to mention that conf also needs to maintain its own store to do some of the magic that we just saw and for this I've shipped a con import command. That by the way, can also import just multiple configs, globbing or all of these parts, so we just run content import.

A

You can see we've imported now, all the freeze and you can see like it, maintains its own store and it also maintains its list of active configs and it needs to maintain a list of active configs, because basically each shell only points to one config, which is what allows us the whole Magic that we can switch between contacts that we can switch between namespaces.

A

For that um so yeah, that's what it how it works, however, I think it's rather elegantly done for the end user, I hope, at least because if you want to install conf today, which of course I recommend that you do that. Next, uh you just install the binary and then basically in your RC file, you just run source and then the output of the command, so I think that is for the setup effort reasonable and after that you can just run a conf.

A

um In addition, there's some other things. I would like to mention that are optional, as we have like a couple more minutes left. um So you can also it also supports Auto completion like the tab completion you saw. um So if you do like cons, set oops set and then Simon and then you see it autocomplete, so you can also tap tap through it. Without any issues or maybe I do uat, and then you see it just tap completes and it works. Also for that shell and Bash.

A

It also allows you to set a default Cube config. Every time you open a new session or set the the cube config you used for the last session. So if you just if you just do something like this, so if let's say I spawn a new split here, this will also be in uat environment, because that is the country last used.

A

um And lastly, just some aliases oops.

B

A

um Yeah with that I like to announce conf is now in beta. uh There will be a lot of things, I think to iron out over the time, just because different clusters do take different yaml formats, different operating systems, pars like colors differently in terminals, past tables differently and terminals, but um yeah I would really love for you guys to check it out. um Yeah, put up some issues uh or like leave some feedback. Of course, some PRS.

A

If you would like to or give it a star on GitHub and with that I'm happy to take questions.