Kubernetes SIG Cluster Lifecycle, 23 May 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive: Cluster Lifecycle SIG (Kubeadm) - Fabrizio Pandini & Lubomir I. Ivanov, VMware

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Deep Dive: Cluster Lifecycle SIG (Kubeadm) - Fabrizio Pandini & Lubomir I. Ivanov, VMware

The Cluster Lifecycle SIG is the Special Interest Group that is responsible for building the user experience for deploying Kubernetes clusters. Our objective is to simplify creation, configuration, upgrade, downgrade, and teardown of Kubernetes clusters and their components. In this deep dive, we will take a look at recent changes in kubeadm, examine how kubeadm is going to implement support for high availability clusters, and finally peek through the window to see what will come next. We’ll reserve time to talk about how to get involved with SIG Cluster Lifecycle and kubeadm, for your questions, concerns, and feature requests!

https://sched.co/MPj5

A

B

Hello, everybody- this is the sequence. Life cycle cube ATM, deep dive session. Today we're going to talk about cube ATM. What have we done with the project recently and also where we going in the future.

B

Okay, so, first of all, who are we I'm blue medieval off I work for VMware the open source program office. There are we contributing to cooperate? This is 2017 I'm.

C

Fabrice, the panini everyone thank you for being here: I'm, a secret, a lifecycle contributor since two years and I work for a unicredit in Italy. So.

B

Before we begin, I want to ask the question like how many of you have used cube, ATM to create a question: Wow, really solid presence of cube, ATM users.

B

Okay, so we can adapt the tool base so that I guess so before we begin for those that do not raise their hand, we have to explain what cubed M is so cubed M is a 2/2 provided with a best practice fast and secure cluster, and also we focus mainly on the user experience, and you know making it easy for the users to create these questions. But we have to mention that cube. Atm is not a question that is not a tool that manages the notes for you.

B

As a cursor, you have, you can run cube a them only on a single node and as part of the UNIX philosophy we're following here. So you have to provide the machines. You run cube ATM on these machines to connect them in a cluster, and then you install the C&I plugin, which is something that we also don't manage, so some some other takeaways from the design we are following his the design.

B

Basically, there is an experience should be simple: the cost reasonably secure the scope is intentionally limited and we basically only deal with the local file system, the kubernetes api, and we are agnostic how you can run the couplet.

B

Almost setting setting up basically of every in a certain see a nice plug-in is out of scope, like all of them have some benefits. All of them have problems, so we are not favoring the CNI. It's up to you to decide, so we are going to outline quickly. Some recent changes in cube, ADM, cubed Maga for those that do not know GM is general availability or basically, we are stable.

B

The stability means that we have stable command line UX. So if you are already using cube ADM in your you know, in our courses, you know infrastructure. We guarantee to you that we are not going to break the command line and we, when we deprecate something we have to keep it around for 12 months. That's the GI deprecation policy of kubernetes and we are following it.

B

Something else is that we have a stable underlying implementation. We have no plans to move away from static pods for the control plank operates. We have corporate config for the cube, ADM config configuration itself and also we are using it for the cube root in cube proxy and also we have both our tokens for the join and init workflow, and we don't have any plans to move away from that as well.

B

Also part of the stability we provide is we provide upgrade paths between kubernetes, my languages and, if you ever wonder it like, do you support jumping between a couple of minor versions? Well, that's that's a topic for kubernetes as a whole and we are following the best practice of supporting only from minor to the next mile. That's about degrees. We have a configuration file.

B

It's currently the version is view on beta1. This psycho, actually we're going to upgrade it to v1 v2. True, the changes are minimal, but basically our back-end, like API / configuration file, is stable. It is broken into multiple, basically sub configurations and it's ready to be used like a Swiss Army knife pretty much also. We have cube alien phases and the Q beta phases.

B

A basically a mechanic where some of our main commands like Q beta immunity and join, we have them broken up into soup phases where you can, for instance, run skip the pre-flight of the Cuban init command, or only around the you know the fast at the face that is cube, a cube config for the cube, a minute work for or for join. You know you can completely skip some of the phases for the control plane or only around the complex time frames.

B

It's basically, we are making it as customizable as possible and in the red here is we're continuing to develop the phases by the way, so in 1:14 we extended in it to support a certain future. This is going to talk about that's appalling certificates and also in 1:14. We didn't have joint phases at all, so we before 114, so we added like the whole phase, runner, basically Washakie 114.

B

Next steps is probably basically phrases for reset or a break. We still these two kinda works. Also with a survey for those who participated. We would like to take you basically, the overall one of the main questions was: what is the general uxe and we got a average score of 4 out of 5, which is pretty good, and also we asked the the the big question at what is the most desired feature that we you want us to work on, and people responded teaching. They want, have availability, posters and the past couple cycles.

B

We've been focusing on that much so this part is going to take okay.

C

Great so, let's have a first check about the survey, so how many of you are using cubed mean I availability feature already: okay, a few of them. How many of you are planning to use it?

C

That's great! So today we are I'm going to talk about two different aspect of the high availability. The first one is certificate copy and the second one is the knight dynamic workflow that you can achieve with cubed mean for I availability. So, let's get started. What is a certificate copy? In a nutshell, when you create a availability cluster, it is important to keep in mind that you have to to share some certificate, your certificate authority and account signing key among all the control plane in order to get your court.

C

Your cluster work- and there are many ways to do this, but the approach that the best fits with the Kuban mean workflow is to create the certificate on the first node within it and then keep the certificate and move to the second control plane or to the Terkel plane before joining it to the to the cluster.

C

These this operation, before we 114, was basically manually and in charge of the cluster operator, but what we found out is that this operation was error-prone, and this is why, in 114 with kuben mean provided a new feature, that is called cuban, mean automatic copy certificate.

C

Now the question is: why should you care about this feature, even if cubed mean is doing all the heavy lifting for you? There are. There are few reason, but I think that the most important one is that you have to take care of how you're certain your primary, your private, private key interface, is managing how your cluster certificate in manager, because if someone gets access to the certificate, basically they can destroy your cluster. So how does it it's work?

C

It is pretty simple: when you do in it, you have to pass one player, which is Minos, Minos, experimental, upload, a search and then before Cooperman creates the certificate as usual and store them in the TCD kubernetes PKI folder, then coburn mean detect that you are asked to upload cells. So what it does it research it encrypt all the certificate and store the certificate in one secret. Okay, then, finally, Cuban mean gives you the instruction for joining a new counterpane control, plane node using the certificate.

C

So what is important to notice here is that Cuban mean at any time is not actually copying certificates around. It's just keeping this certificate, storing them in in a place that is us that will be acceptable accessible for the joining node and up riding and applying a little bit of security.

C

On top, the the actual copy of certificate happens at joint time when you join using the the commander flag, that cubed mean provided you basically, what happened is that, immediately after after pre-flight kuben mean download the certificate decrypt certificate and store in in the in the etcd, kubernetes pick a folder, and then the the joint workflow goes on as usual, so first set of takeaways when we think about a certificate copy, you have to remember that at any time, certificate share it across the counter.

C

Plane are encrypted and uploaded in in the in a secret, which is called the combat, mean cert at the joint time, certificated are downloaded and the creep that uses the certificate key. This is how it works. Another thing that is important notice is that really really important. The certificate key must be kept safe, because if someone gets access to to this key and to the secret, they get access to the secrets that control your cluster and it can be destroyed.

C

Another thing that is important to note to notice is that for for additional security reason, the cubed means our secret gets deleted after two hours. So if you need to join another contemplating afterwards, basically you have to go to one existing counterplay node and run upload as earth again, and this is defining ok. I live also the other two, no sir, for you to read in it a slider that are already available online and I move to the second topic of of today. That is a dynamic workflow.

C

What is dynamic flow dynamic flow is a term that we use to refer a behavior that is in Coober mean since its creation, a behavior that you get because due to the work flow in its joint work flow and and basically we are referring to the fact that we could mean you can change your cluster dynamically. So you can create a cluster you do in it and then you can decide to join a first set of worker node. After sometime you can add a new node and so on and so on.

C

So you, the topology of the cluster, can can change dynamically.

C

Why we should care about kuben mean dynamic workflow. Now that we are talking about I availability. The answer is that, because basically they were flow for creating a hierarchy. The clustering could mean use exactly the same pattern that was that is already known and was already used for joining worker nodes, but there are some additional considerations that should be keeping mine. So, let's start beginning into those consideration.

C

First consideration is that you need an external load balancer in between your nose and your contour plane. Also, you have the another load balancer in the middle. Basically, what you can do you can init your cluster and then you can grow your contour.

C

The number of your contour palette with the plane nodes by using cubed mean join and passing the the flag, with the only difference that you have the past the flag, Nino's Nino's, experimental control, plane, and it is it is the exact complex, same workflow that you use to worker node to grow the number the worker node. What is important to notice is that there is no predefined order, so also you need to cluster.

C

You can join contour play node or worker node at any time, and in any order you can start with a single control, plane, node, join worker after some time turn your cluster in a real H, a cluster, adding contour play node and move on move on move on. So this is a dynamic your flow. This is what we define in with the term dynamic workflow.

C

Another consideration that you have to keep in mind when you think to this workflow is related to it. Atc D, as you know, which could mean there is, there are two modes to work for ABCD. The first one is that you provide- and this is the cluster external built somewhere, and you connect your contour pain to this ATC D. If you are not doing so, could mean takes care of creating it and it is hidden node on the machine where you do in it.

C

How does this change with the new feature? For you a reality? Basically, if you are in this condition, you are using a locality CD. When you grow your control, plane kuben mean dynamically grows, also the TCD cluster. That means that growing the number of control plane you are seeing. You are also moving from 180 CD member to a decision, member and and so on, and this is also the reason why, when you are in this configuration typically, you end up with three or five control plane nodes, because we have taken.

C

You have to take care of the quorum problem of the data CD cluster as well.

C

So roppa for this part, in order to set up on in a cluster, you need before an external to set up before undisturbed, load balancer.

C

Then you can in it the cluster and after you can join the control, plane, educated, join the quarter, the the control plane, nose worker nose in any order, and it any time some consideration for for these.

C

For this workflow in case, you are not providing an externality CD Azteca DCD cluster is automatically generated and is span across all your control plane nodes. Another consideration is that we could mean, and in the dynamic work flow. Basically, certificates are specific by node and- and this is this can be easily explained it.

C

If you think that when you create the certificate for the first node, you does not know in advance the node that will follow, so you cannot add the certificates and all the IP address and whatever so could mean when, when create a new node, basically create certificate, with all the required information for that node.

C

For the same reason, the API server instance on one node is connected only to added to the city to do the APCD node, which is local to that machine. Because again we don't know in advance all the t3d member that will be created in the future and last point, which is really important because we received many issue about these.

C

Is that if you want to preserve the dynamic workflow, please don't override the address field for Koob API, several or eighty CD, like the best aya advertising address or list and client wars, because if you overwrite this flag, cubed mean basically preserve what you're passing and, and they note in the classes, topics growing dynamically.

C

C

So last couple of slide is just are a summary of all the work that that we did to implement each a and just to give you a picture of all the effort that that is behind this. This work, so the first slide gives you basically represented a starting point, that is cubed mean creating a single, not a cluster traded.

C

A single node cluster in Cuban mean was a process that basically means to create some artifacts, like the certificate files like Cooper config files or static port, manifest those those artifacts are stored on the disk of the local machine and also to create some artifacts in the cluster. As soon as the khubaib API server restarted, like created, the cubed mean config map, the adorns like or DNS or Cuba proxy, or our back rules, those those artifacts are deployed in the cluster. This was the situation at the beginning, creating an assay cluster.

C

This is the grand theory of a changeable mean. What does means means? First of all, you have to copy from the first water plane into the second control plane all the certificate, a service account that must be shared at all the control plane.

C

Then you have to sir to generate only the certificates that are specific for this node for the new model for the join node. After that you have to generate all the Cooper config files that are necessary. We read exception of coup ballot account. Why? Because we want to use the TLS boost wrapper, so there is a additional layer of security on the joining node, and this allows for instant class administrator to prevent node for joining. Without permission.

C

After that, you had to Janeiro sorry, you, you have to generate a static pod, manifest for the control, plane, API, server, control and manager scheduler, and then you have to do two action that has to be synchronous. The first one is to add a static pod, manifest for annuity CD. The second one is to effectively to actively join this new TCD member 2012 ec class, a cluster.

C

This is a critical operation, especially when you move from one one member to tune a member, because 80 CD is a short blackout when, when it is it arranged quorum, but we managed to get it well and finally, you don't have to deploy again.

C

The the in cluster resource is like a common config, Mart or whatever, because the cluster already exists when, when you join the second, the second control plane- and this is the history on how implemented these in cabal mean it is an effort that started in 111 and the first thing that that we did was to split all the set. All the settings which are cluster wide, like I, don't know the the seed report for the network from the setting that are specific for for one node.

C

Now you have the cluster configuration object and you have the init configuration object or the joint object.

C

Then we implemented the first version of join contour plane, but at the time that was 112, we had only manual copy of certificate and we were able to support only cluster with externality CD, the next release. We implemented the support for scaling, 80 CD, but still with manual copy of certificate and, finally, last cycle we implemented automatic copy of certificate.

C

Hopefully, with the 115 release we will go in, we will go to beta with all this feature so back through by yeah.

B

I just wanted to point out that all the features of a bridge we talked about is these: these are automatic. So if you want to transition your question to HJ cube ATM can manage the a question for you. So that's you know. Desk has been a very important goal for us, so I wanted to talk about what quickly. What is the road map for you know, 2019 and the next year's it like the bridge will get a change support. Incubate the m2 beta.

B

We currently only worked on adding a single field in the config and getting the tests to run to basically have end-to-end signal for AJ, but it's we are pretty close to getting this feature to beta. At the same time, get the V 1 beta 2 configuration of cube, ADM, true to be in place this cycle. We. This is something that we work with the sig Windows folks to get bring back windows mode support in cube ADM. We as a you know. We already have this working in a way. We just have to fix some parts.

B

So hopefully we can have some sort of alpha release. Possibly in this release of community, we have to consider consolidate the story with you know: certificate renewal, externals, a support and location of certificates. This has been a bit of a problem in Cuba diem, but hopefully at least we can update the documentation. This cycle we have to like I said we have to improve our CI signal. We are working on this and something that this is the clean up of artifacts is something that we have been battling.

B

All sig release a little bit about that, but hopefully we can improve the picture of your depth set up. We have take up in a bit of a problem for cube a diem in general and also something that a lot of people have been requesting is evaluate like customize how we can customize. You know life coastal configurations, but we still need to take time.

C

If I can add something about this feature, but the meaning is historically a trade-off between the request to support many more customization, many more scenario and the need to keep something stupid easy to test and manageable, and this is a way that that we are looking for for basically a load.

C

The two scenario, the the the simple managed scenario and the advanced at the scenario of user, doing whatever they want- and this is something that we would like to have a feedback for from the come from the community before and also help, if possible, before starting to execute. In order to get it done right.

B

This way it is about who is maintaining Cuba diem, basically how you can contribute. We have a list of core active contributors. That's Tim myself, Ross Jason, listen check from VMware. We have Marquis Rafael from Susi. We have Alex Annette from Intel, not last but not least, Lucas Fabrizio and Iago, who are independent. We are you time, zone friendly, which is not very common. For you know: community six.

B

We also recently recorded an onboarding video, so if you're new, a new way want to contribute to, you definitely should check this link. We also basically want help with any labor in it. It tickets that are labeled a good first issue here wanted or sequester lifecycle.

B

We also we are productive on the cube, ADM and sequester lifecycle channels, and you can also join our weekly meetings. We meet every week in the cube, a diem of sounds and also you can, of course, contribute to drama dogs, so this is our logistics again with this overlaps with the previous slide. Basically, you can check these links to get involved, and this is a quick picture that outlines that you know. Cuba. Diem is part of a much larger picture.

B

It's it's a piece of a puzzle that sequester lifecycle is trying to build, and it's if you look at the diagram cube ATM is the only GA item. So we need a lot of help. It's about some of the other tools that we are currently working on, or basically a plant like, for instance, HC, didm Westeros. These are like in the planning stages. Pre-Alpha we have cost. Rapier is awful. It's super active. Currently we have component config sort of Europe is needed there.

B

Cuba diem has one row, sleepy obviously has one, but we need this across all the components and on the top we have some, basically any provision. References like cops or cube spray that can utilize the whole set of tools that the sick is developing. So that's pretty much it question. Thank you.

B

We have four questions.

D

Hello, do you have any plans to support declarative update of clusters as well plus a note so currently I see cubed M in it join per set and if something has to be changed, I guess I can remove the node and esri added again, but if something could be updated, do you have any plans for that I.

C

Can answer so in moving a shade to general ability? We are also working to upgrade for a Shea. Currently it is already supported, but basically the user experience is not ideal because we have to command one. One is could mean upgrade configure that upgrade the conflict for coup ballot and the other is could mean upgrade experimental control plane that upgrade the control plane on the secondary control plane north ok, so it is already supported, but we have to call to comment on the on the secondary control plane. Not what what we want to do.

C

I, don't know if will fit in this release on the next one is to provide only a single command that is Cuban mean upgrade node. That gives you a more simpler experience so also for upgrade. Basically, what you have to do you have to run could mean upgrade apply on one on the first control plane node like in it and then all the other node you run could mean up upgrade node. This is the target that we have in mind. Does this Ascot request now.

D

Actually, it answers that question which is quite similar to mine. My question was about updating configuration without changing the version. Ok, ok, I mentioned one specific example that I had to cope with, which was changing the IP address of a master. Ok,.

C

Ok, changing the cluster is a long discussion. We are discussing a lot about these. Basically, we have two options. mmm One is to follow the path that you use. Cubed Mina upgrade passing a new config file with the same they are shown, but something else change it. Ok, this is one option. The other option is to create it. It has Pro some pros and also we know some cons of this approach we consider at all. So there is also some document. I can point to to you that we consider also some another approach.

C

That is, that is to create a new command. Let's call it cobalamin change cluster that give you change plain and change apply, so we start a figure out the tour flow, but the problem, the problem. The problem is huge because there are change that basically are different. If you are changing a cooperage configuration, you have to act on all the nodes. If you are changing API server flag, you have to you have to act to only to the contour plane node.

C

mmm So so we discussed that this year yesterday evening in in a face to face the plan, is to try to scope down the problem to some more simple and manageable use case and try to find out something a tional. So we are working on these and we will go step by step.

B

Multiple notes, at the same time, cause sleepy eyes your project want to manage one dolls customize it. The cube. Adm eventually is going to support one load at a time. Questions yeah.

E

So I understand how you build an HJ cluster, there's also a scenario where I've got a cluster everything's running. Finally, one of my control nodes, one of my control, plane nose blows up. Is it possible to replace that notice, cube admin support that? Yes,.

C

It is possible, so we using the join Minos experimental control plane. You can create a new one. There is a currently there is a unknown problem that if you are stored, if you are using the the stake of the TCD and and you lose the the control plane know that you have to clean up the TCD membership. Otherwise you can have problem. We are trying to get this fixed. Also this in this in this release. But there is this: is we are going slowly because there is back.

C

There is a an issue in the ATC D client, and so we are trying to get this fixed set up in a TC D. So so, instead of implementing that work around some other question.

F

So it's not another presentation, there's also the ability to bring your own certs, but when you're doing the upload certs have you considered pushing those into like another sort of secret vault rather than storing them in the cluster itself? You.

C

Know that secret in.

F

Another in another vault like if you've got you know Hoshi court fault or key vault, or something like that. Where you've got machine identity already there. Could you use that to delegate access to those secrets rather than store iam in the cluster? So, yes,.

C

We have considered many option this one that we implemented is basically self-contained. You don't have to depends on something else, but we still support other workflow, because you can create your certificate in advance or on on an order and then copy on the first node and that could mean repeat or or make them available on each node.

C

Those those are also scenario supported so and there are uses that generates earth and and and do this way, what is tricky is that if you want to use the dynamic workflow, you don't know in advance all the certificates, and so my suggestion, this scenario is just generated certificate authorities and let Cooperman generate the certificate which are specific for the nodes, which are the client certificates.

C

G

Thanks so my question is a little bit hypothetical, maybe so a cube EDM can abstract away or or codifies the workflow for setting up kubernetes cluster like has there haven't been discussions on maybe abstracting away that logic into an operator so that, like I'm, not Cooper net, this cluster can, from from like a from a resource definition, can create another.

G

C

And this discussion, basically some ups to the work or that we are having to the cluster API I mean their world tomorrow. There will be a talk about cluster API and we talk about also a lot of in the past day. Cluster API gives you an abstraction that allows you to create machine and to spin up a kubernetes cluster and all those machine. Just to give you a very high level overview. You have machine machine, says and machine deployment, and this is a declarative.

C

There is some tricks in order to get this machinery started because you're using kubernetes to create it itself. So there is a chicken-and-egg problem that should be solved that we see solvent the work is there is a lot of work in the cluster API stream of six cluster lifecycle and then cluster API use operators dose operator use, can use, Google mean or can use whatever else to to create the kubernetes cluster.

C

So it is also extensible so lot lot of work to getting there, but we are already working at this higher level or structure on to lock up admin. You remember the UNIX philosophy, kuben mean. Does one things and another things take care all of the higher level.

B

That's all the questions we have I.

C

Think that we are one more.

H

First of all, thanks for the discussion, it was nice presentation, I have a question because you mentioned that right now, it's hard to define in advance the IPS and all the information about the control plane. So how you clustering, the etcd!

H

Sorry, how you cluster the etcd node and as you mentioned, that is hard in advanced. You know all the informations of how the and you said that the connection from the control plane is to local host, so how the etcd and the relief is clustered. How.

C

It is it is, it is connected with the API server know.

H

Know how the etcd to each other's.

A

C

No actually other question, so API server is talking to the to the local etcd, but each ATC D are talking to to the other, and- and this is possible- because basically, even if you don't know the ATC remember- that will come afterwards- ATC sinks that that point between the member. So when you create the cluster, you have to provide only one existing and point, then II join and they sink and they find out all the new one.

C

So we we are you we are leveraging on an internal machinery of a CCD to get all the decedent member synced.

C

Great, thank you. Everyone enjoy the party.