Kubernetes SIG Cluster Lifecycle, 15 Dec 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: kubeadm Cluster Creation Internals: From Self-Hosting to Upgradability and HA [A] - Lucas Käldström

Description

kubeadm Cluster Creation Internals: From Self-Hosting to Upgradability and HA [A] - Lucas Käldström, Student

kubeadm is the Kubernetes tool that helps you set up a Kubernetes cluster quickly and easily. kubeadm is different from other Kubernetes setup tools in that it doesn’t assume or depend on any special infrastructure. It assumes that you have one or more machine available and those machines can connect to each other via the network.

The master plan is to make kubeadm work both as the “fast path” to getting a best-practice Kubernetes cluster with a couple of easy-to-remember commands and as a toolbox for higher-level solutions like GKE, kops and Tectonic.

But how does kubeadm actually set up a cluster? How is it so easy to add a node with the Bootstrap Token? How does it self-host the control plane? How does it upgrade clusters smoothly with only one command? What is the plan for achieving HA without relying on any external infrastructure?

After this talk, you will be able to describe how:
kubeadm runs the different tasks in different stages
the network traffic between the cluster components flow
self-hosting of the control plane works
the Bootstrap Token works
the `kubeadm upgrade` command works
kubeadm will support multiple masters that are dynamically rotated
you can extend kubeadm to build your higher-level Kubernetes deployment tool

About Lucas Käldström
Lucas is a passionate Kubernetes Maintainer and CNCF Ambassador that is excited about all things cloud native. Lucas has been engaged in Kubernetes work for about two years now and been involved in work like porting Kubernetes to multiple platforms, getting Minikube off the ground, being a core contributor in SIG Cluster Lifecycle and maintaining kubeadm. Besides Upper Secondary School Lucas runs a consulting company for Cloud Native tech programming tasks.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Hi everyone, I'm Lucas, shelter and right coming directly from Finland and I'm. Today, gonna talk about qbm and how it works under the hood. Things like that I'm, not a secular school student and doing some cool stuff, with combinators upstream contributing to some things and working on this. This year, most like you, barium and driving my own company that is contracting for weworks and such things it's nice to be able to contribute to this community.

A

So what are we going to talk about today? Well, how cube area what cube enemies and how it fits into the ecosystem? Overall, what is the scope? Why does it exist overall, then getting a bit more deeper into? How does the different kubernetes components talk to each other?

A

How is the cluster secured, what is self hosting, and how can we benefit from using such a technique? How does like m18 cube area metal supports for easier upgrades, and so what does that actually do under the hood, and then how can hedge, a or high-availability someone could call it multi master? How can that be achieved with Q barium as a building block and well? This is a deep dive. I hope, you'll enjoy it.

A

Okay, so, let's start with the cube: am the scope of cube am what what should cube Adam do where how does this fit into the ecosystem? Well, it's at the bottom of the stack. We have some kind of instil infrastructure. It could be some of the public clouds shown here. It could be your bare metal cluster. It could be my raspberry PI's on my desk or what? What? What any anything that has some machines and then we're all here today, because we're excited about kubernetes.

A

Yes, then we want to have kubernetes installed somehow to our machines and before qbm was created. This could could be done, but it's it's kind of hard or you could use some other end-to-end tool as well. hmm So we create a cube area that bootstrap kubernetes and it executes locally as a CLI tool on each machine. It only sees the machine, it's a kid executing on like the file system and or other local resources and the kubernetes api above it. So what cubing cube Adam gets. You is a kubernetes cluster.

A

It stitches the parts together and makes it makes a best practice, but minimum viable cluster by design. So we we don't install all the things for you. That's that's left up up to the user to lay a tree that could possibly include things like cloud provider items a thing we're currently wanting to extract out of core. Currently, cloud providers are built, a torso are built into the core, but that has some severe limitations, so we're actively trying to move those out to run as normal controllers on top of kubernetes or it could be load balancers.

A

So monitoring logging, whatever is still but that's, left up to the user, and then we have a new, exciting API coming up an API spec called the cluster API. You might have seen Robert, Bayliss and Krishna was talk earlier today about on this topic. If you didn't, you can check out check the recording out on you in the coming weeks. The cluster API is a declarative way of specifying. What should my cluster cluster look like?

A

I can say things like I want the control plane configured like this and then I want a machine set of these end nodes running on this cloud or whatever, and then we have some so the spec is generic, but then we have an implementation that is cloud specific or environment specific that is running on top of kubernetes and reconcile to create these nodes and all masters.

A

So cube item is a tiny part in this ecosystem. There's a lot of different a lot of moving parts but cube atom. Is this like turtle at the bottom, that does it sting executes locally and has a very small scope to just do the minimum viable thing needed for a cluster people often ask me: what is the difference between cubed, M and cups? Are they competing, which is better things like that they aren't competing as they have totally different scopes.

A

Cubed M I said only executes locally on the master, that's a very local view of what's happening. Let's cops. Have this global view of the cluster and manages the infrastructure, all the machines, bootstrapping kubernetes and atoms as well, so there are two total different things and should be used in by two different kinds of uses. If you just want a cluster up and running on AWS or GC or whatever, and want everything like just one command to do all the whole cluster. For you, then cops is your thing.

A

If you want to build your own terraform, whatever bootstrapper on bare metal or your custom, something then cube area might be your better. A better choice.

A

So what are the key design takeaways with cubed em? Well, when we started this effort, one and a half years ago, or so we decided it should set up a best practice cluster for you, and it should have a set of very small scope. The user experience should be simple and the class a reasonably secure. For example, we, you can add notes. With this token, the cubed M join, which is a trade-off between security and simplicity.

A

You can you, can you don't have to use the token you can use other ways as well like copying over a file from the master to the node, and it's intended to be a building book cube any minute. That's a lot of things which we'll see in a minute, and you can run these separately as well.

A

It's a cube, Adam doesn't care about how the cube lattice run. So we do provide package deb and RPM packages. Sigler's. The lifecycle provides those for you, but if you want to run cube area, Mon like core OS or Rancher or whatever other always that doesn't use them setup rpms, you feel free to run the cube. Let yourself and everything will just work: it's only a template. Then we made a design decision that we should not favor any specific provider like flannel or weave or calico, or something like that.

A

Instead, we chose to not do do such things and it's up to the user to install the CNI network. We are a cube, CTL apply, so that also has a consequence that if you run cube at a minute only you won't get a working cluster as there is no networking.

A

The Cubans are Kotak sure is composable and everything's divided interfaces as we'll see in the coming slide who's the Cuban audience, then, who should use Cuban? Well, a lot of users are happy with this. Cuban I mean it flows, they're typing, into the command line, then joining nodes as they go, and this is a great way to get started to kubernetes start like tinkering about like with how it works. Seeing all the API server running things like that or it could you be used in an automated manner by tools like cops or cubicle or whatever.

A

So here is the high-level component architecture for kubernetes. We have the master, which is an API server, which is stateless, basically a REST API in front of a TD. Then what makes kubernetes kubernetes is the controller manager which runs a dozen dozens of controller loops to make sure that the solid state is the actual state and reconcile on that.

A

Then we also when you create the deployments. For example, the controller manager will create replicas set and it turned ports as a consequence, and then the scheduler will kick in and bind these pods to a node which will then trigger the cubelet to run the container image of your choice using the container runtime, which will well then execute on your host, and we have the all these beautiful interfaces with C and I, CRI, LCI, etc.

A

So qubit a minute does a lot of things. The first thing it does is generate the needed certificates. They are a couple. We have the root CA cert, which is the boundary of trust for a cluster from that root. Csr the API serving cert is generated with when I years well and it's valid for a year.

A

Then we have the cubelet client search so well, we'll see in a later slide that the API server, when calling when you're calling cube CTL logs or cube CTL x'q, you are talking to the api server, which then is talking to the cubelet, and then the api server needs some kind of credential to talk to the cubelet.

A

Then we have a service account private key and some things needed for api aggregation support. Then we have. We generate some identities for the initial actors in the cluster, for example the cubelet client cert, or search for the admin or for the KCM and scheduler so far so good. Then we post a TD. You could do this externally as well. I mean that's probably preferable, but then the easiest way is using a static port and letting the cubelet host it.

A

For you basically babysit it, then we have the API server, controller manager and scheduler running a static ports. In from that, it's EQ Benares manifest directory.

A

The running cubelet will then start these static ports and we'll have the control plane up and running.

A

Then Cuba remarks the initial node to be schedulable with this label or key value pair, and it also taints the master to not schedule workloads there by default. So- and this is because of security boundaries, you shouldn't really run normal workloads on your master, as that might lead to privilege escalation pretty easily as well. Hence we have the taint so that your your normal work laws won't schedule on the master. Then the first queue Batum specific thing is that queue, bearing will upload the configuration the internal state of the world.

A

It's a config map so that later, when upgrading it will know what is the current state of the world and act based on that.

A

Lastly, we create a boot up token, so you can add, you can use the cubed M join functionality to authenticate to the master when, when adding this node and then deploy the mandatory add-ons in order to pass conformance tests, so that is that is the current bar of cubed M cubed M should create a cluster that makes it possible to pass the conformance test, but nothing more I mean, as well as other things like heap store, which is now matrix server or some logging solution, or whatever other thing that is normally used in kubernetes cluster should be installed by the use of themselves or this higher level solution, and one thing is one cool thing is core DNS support is now alpha in 1 9.

A

You can enable that with a feature gate, but if you don't want to do all these things at the same time or in like this full meal deal, you could run these phases as we call them these atomic sub steps of cluster creation.

A

You could run them separately, generate the cube, configs certificates, the control plane, static, pawn, manifests and etc, or you could, if you have some good replacement or don't need, for example, the bootstrap token you can just don't do that and do everything else and provide some other like copying over the file. There are credentials from the master to the node if Stella.

A

So what does a like in a TLS secured cluster? How do these components talk to each other with what credentials and so forth? Well we have the API server, which serves on HTTP by default six, four, six, four, four three, and we have this cubelet client cert as I talked about.

A

Then we have the other components with their certificates that have have to have this like photo control manager, for example, it has to have the CM of system cube controller manager in order for it to be able to pass that well, have the right or back privileges. Of course, if you have something else you have to assign, you have to create a cluster role binding between the normal, well, the control manager, privileges and dessert.

A

The controller manager runs to, in this case, significance controllers, which are the CSR signer and the CSR approver, which we'll see in action a bit later, and here a CSR is a certificate signing request.

A

Then we have the initial master node and now we're about to join the second one, we're now basically typing in cubed M join with. With the token the cubelet we'll start with a self-signed HTTP server, this is still an area to improve. This is basically true for all companies deployments right now that the cubelets art is of a self-signed as we go.

A

Hopefully this feature will graduate to be the next title in 110 so that it can use the same cluster CA, but as that is working progress with singles, so we give cubed em the boot up token, and it will figure out the trusted CA, which is you which the cubelets uses to talk to the API server, will then get a CSR with no node to which then the CSR approver will automatically go ahead and approve.

A

If you don't want this link, you can disable it it's just an Arabic rule, but by default in order to have a smooth flow for you when you're joining notes that CSR approver automatically approves which the CSR signer then signs.

A

So that means we have the C a key on the master. We never give away the CA key anywhere and.

A

Then we have a single. We have this reconciliation flow where we just post these requests and whoever has the C a key can then go ahead and sign them. This is also true for external cs. For example, one common request with cubed M is I, have some kind of corporate policy that I don't get access to the CI key or I can't move its own cluster? How can I Milla gate that and still use cubed M?

A

Well, the answer is you run cube, RM, face certificates and cube config, which will give you all the required identities while you're having the CI key. Then you'll just go ahead and copy everything by, but the see a key to the cluster and run this CSR sign or somewhere else. That will give you the same effect so cool. Now we got the client search for the node for the second node and we have and its identity is unique.

A

Then we have an old another feature introduced in 1:7, which is the node authorizer which will make it which scopes down their cubelet privileges to only what it actually runs.

A

So when the queue when you type in cube CTL logs, the API server is going to insecure or unverified connection to the self-signed HTTP or so the this request is using HTTP, but the API server can't be sure that the cubelet is the one that it's actually pretending to be. But this is working progress and contributions. Welcome as always, then, when the tublat gets this call or gets any call, it will send a subject: access review, request to the API server asking. Is this actor now the cubelet client authorized to call my API.

A

Basically, it's it's asking. Is this person who is trying to access my API root in the cluster and in this case? Yes in other cases? No, so in one four and one five and earlier most clusters didn't secure this cubelet api bit, which essentially made it possible to access. If you could ping any node, you could exit in to that node, because there, when there were no authentication in this step, but as of cubed n16, this is all secured.

A

Okay, let's move on to self hosting. What does it mean? Well, we're using kubernetes primitives to configure and host the control, control, plane itself and.

A

This was in it. This concept was developed by Korres and has been vetted in the boot cube project. Now we're working the Cygnus lifecycle team is working on upstreaming, this kind of functionality into cube ATM into this building block the cube enemies, so you can still run your cluster is in static pods, but the chances are you might want to use self-hosting because it leads to easier upgrades eventually and stuff like that.

A

There's also another self-hosting talk tomorrow by Diego from korra's make sure you check that out.

A

A

So how is a self-hosting cluster created? That's kind of a pack to many people as well. Here is the sequence diagram how it works. There's quite some detail as well, but this is the deep dive, so I guess it's fitting.

A

First cube areum was has to know like what's the actual state. What's the current state, it figures that out by reading the local file system for the static force, it then mutates these static ports a bit to work, a self-hosted components. For example tolerating the master taint is one thing, and then it just goes ahead and creates the exact same, the nearly exact same resource in the API server.

A

So you could pretend and think of it, like I'm running a new workload, it's an API server that should be run on my master, then the API server is run, but it can't bind to the port it's trying to bind to the 6 4 4, 3, port or whatever.

A

But then the cell, the static pod hosted API server already listens to that port, so the self hosted API. So we'll go ahead and trash loop. When Cuban and detects this crash rule looping, it will remove the static port, API server and the self hosted one will come up cleanly and now we have a stage where kubernetes can manage itself, which is kind of cool, and then given does this for all components and there we go.

A

Upgrading clusters with cue barium.

A

We made it easier with cuban upgrade plan and cuba time upgrade apply. These helper commands have a good UX and makes it straightforward to upgrade the cluster but and then hides the complexity under the hood in 1 1 8, +, 1 9. This basically shuffles around these static ports on disk and make sure to catch edge cases and rollbacks. For example, if the 1/9 manifest doesn't come up, cleanly will roll back and but then the promise is with self-hosted up with self-hosted and aj will be able to like roll it out like any manifests.

A

Well, we'll just edit the daemon set. It will use the the controller manager will roll out new pods as we go and replace the old ones, and if something fails, it will roll back itself. So that is a cool thing and in 1:9 we're also supporting automated downgrades in a future release. We might look at run time, reconfiguration or basically cube CTL apply for a cluster. So let's say you started.

A

You started your cluster with some config or like normal cubed, a minute. Then you, a week later or a month later, found out that I should actually have done this kind of thing. But you have use of workloads there and don't want to tear everything down. Then you could just do a cube. A time upgrade apply with a new desired state of the world and Cuba and we'll roll that out for you.

A

We don't officially so support this yet because there are some parameters that could be dangerous to to reconfigure that we'll get into weird state. So we probably have to catch that. But technically it's already working and possible to do. If you know what you're doing.

A

Yeah, so we have heard a user request that AJ would be nice to have, and yes it would. The the ideal flow here would be Cuban a minute. Then we get to tokens.

A

We get one master token that we can use to add the master and we get one node token as usual to add nodes by definition, this means that the the master token is equal to a net CD private key to a TD right key, and that is the thing like you may ask yourself: is that production grade, or is that secure to have 22 chars string, Abel like if someone else gets to know this 22 char string, then they'll have access to my Etsy d2 rock-like can basically do anything.

A

So in order to achieve this, we have three primary challenges. First, we have to manage state, and this is probably the hardest one. How do we run a TD, especially in a secure manner, with unique identities? Dynamic identities like we could start cube at a minute run. One master for a week add some notes as we go, then a month later find out that well, I actually want to have two masters or then three masters or five, so we can't know in in beforehand what will the desired cluster state look like?

A

Instead, we have to dynamically reconfigure, which is challenging then sharing and rotating the common certificate between all masters is another challenge we have these like, for example, the service account private key has to be the same in all controller managers, otherwise we'll get in interweb state where pods will start erroring.

A

So these these have to be like on all masters, but they also have to be rotated rotated at the same time, if we or when we rotate things and then how should the cube, let's be able to access to address all masters dynamically, so we started with the one master all cube. Let's talk to this one master, then we scale that up to three and now suddenly they should be able to somehow discover all three masters.

A

In a secure way to be added well, you can achieve cubed M AJ today, and that is one of the most common misunderstandings which I'm glad to clarify here.

A

If you run your own, a TD AJ cluster to start with that might not be easy, but we'll provide documentation for the one nine release, what it could look like and you could build some some kind of automation yourself, like ansible, terraform or whatever then run cube animate on the first master copy over the shared certificates to the second one run, cube Adam in it again and do the same for the third one so basically or you could preach ener, eight all certificates and just distribute them to all masters that might be preferable.

A

Then some kind of external load balancer. It could be a load, balancer or DNS server, or something like that just to make the cube. Let's address all the Masters dynamically.

A

What static, therefore, I thought also work, and then the API service should talk to a TD. So this is all doable and I said will provide documentation for you how to do this. There's there are more commands to to execute than a cube radium joined by Chef master, but at the same time it takes a lot of a lot of pain away from you anyway, because if you're about to do or create a normal age, a kubernetes cluster there's a lot of things.

A

Cubed I'm already just for you, as said here with with things like certificate generation and control, plane, spec generation with good defaults, etc. So this is going to be documented. Then one possible solution to create this cubed M join master flow could be this we're using the cubelet as the babysitter sitter for all components, self hosting the api server and control manager scheduler in daemon sets using putting all the cluster certificates in secrets and having self hosting at CD on top.

A

This is extremely challenging, though, and hasn't been proven to work. So I mean this is one idea that we could do but and then use something like envoy as a daemon set to address envoy as a proxy to address the Masters.

A

In the meantime, we might do something in between that you can do cubed MHA, but you still have to set up a TD externally and we'll do the other two parts for you or whatever, as Cuba diem, has this local scope it's really hard to not the global scope and and that it would know of all machines. It's very hard to achieve this. Cubed M join master flow just like that, but joins in lost lifecycle, and you you'll be a valuable resource for us to speck this out.

A

It's possible in that case how, if it isn't possible with one command, maybe two or three and yeah. That was my talk. I'll post, the or this presentation is posted on scab, so you can check it out there and look at these links and yeah. Let's talk about this in three class life cycle and continue to improve cubed, M, Gi, Gi and beyond any questions we have one minute.

B

So as an alternative to a che, sometimes it feels like it would be nice to just like have a snapshot of its eg and if something goes, bad I'm like a panic and I kind of restore with that with. Is it easy to get Cuba DM to do that, or is that not at all the right direction? I think.

A

You think helped a lark or something similar will would be useful there or a dedicated backup tool. Of course, you could at CDC TL snapshot what.

B

I was assuming I, have the backups like it, let's say: I have a backup of HCG and I.

A

B

Mode and I try to revive my dead cube master, can I use Cuba diem for that or am I using the wrong hammer on the wrong nail. I would.

A

Haven't thought through it, but spontaneously I would say you should use another tool. Okay, thanks. Let I'll go.

C

Can you use cube admin to manage existing clusters? Let's say if I have an existing production, kubernetes cluster, which is not created or managed using cube, Edwin can I use cubed win to take over that no I wouldn't say that.

A

Okay, I mean it all depends if you happen to use the nail in the same semantics. Maybe, but there are so many parameters that that's probably impossible.

D

So cube EDM has a had a slide where you can do everything in phases. Yeah.

A

The pay functionalize there since 1/8 so and unfortunately not now the times up so we'll have to you, can come and talk to me after this, and also we're gonna host cyclist lifecycle in person meet up the tracker project at Hilton in Hilton Austin, just right now after this so feel, free to come and talk to us to Clough lifecycle guys after this. Thank you.