Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2018 (Shanghai), 26 Nov 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Deep Dive: SIG Cluster Lifecycle - Di Xu & Alexander Kanevskiy, Intel

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Deep Dive: SIG Cluster Lifecycle - Di Xu & Alexander Kanevskiy, Intel

The Cluster Lifecycle SIG is the Special Interest Group that is responsible for building the user experience for deploying and upgrading Kubernetes clusters. Our mission is examining how we should change Kubernetes to make it easier to operate. Since the group's formation we have primarily focused on creating kubeadm, a streamlined installer tool and building block to simplify the installation and upgrade experience, and enhance kops, the easiest OSS way to get a production-grade Kubernetes cluster up and running in AWS. We have recently begun building a Cluster API to provide an abstraction of machines across different deployment environments along with a common control plane configuration. In this Deep dive session, we will review recent accomplishments, and discuss our future development plans, where you are very welcome to contribute to the discussion.

To learn more click here: https://sched.co/FzI2

A

All right welcome to seek cluster lifecycle deep dive session. I know we are last for today and last for conference, so bear with us for last half an hour, we'll try to fit in our time slot.

A

So let me first to introduce ourselves: I am Alexander Kerensky I work as a cloud architect in Intel, open source, Technology Center. We are contributing in different parts of kubernetes like signal resource management, network and so on, and my heart is actually closer to sequester lifecycle. But what's one of the reasons why I'm here today I'm talking about it and.

B

This is DC from China and helping with communities for more than two years now. I am a top 50 called contributor to the coop net. In my repo now I'm working at on the financial as a software developer, the TGT XE is my handler and also much like ID. So if it cause and questions feel free to be me as like, a sesame github.

A

All right so sequester SiC cluster lifecycle. Our mission is actually to create and maintain documentation and set of tools which simplifies, where creation, update maintenance, downgrades, tear downs of kubernetes clusters, so what it means in reality. In reality it means like dozen or even more projects. What is part on the umbrella of zip cluster lifecycle. Today we have only half an hour, so we definitely we are not able to go through all of those and it's deep dive session.

A

So we selected just two things which we want to talk in more details, so we will be talking about what hue medium about best practices and how it works and plans for near future and about cluster API. Again, we go deep, so we will just say how it works, and maybe, if you have additional question, we will take it afterwards, so Q by DM best practices. We are constantly working on improving user experiences, how you get with you baby, M, how to install cluster and when some things are changing from time to time.

A

So and of course, like every environment, every setup it has its own small differences and so on. So what we have we have cube ATM, which is packaged as Debian and RPM packages. It's all it's available from our repositories. The thing is the world is not video, so you have different distributions. You might have problems with accessing our repositories. You might have different policies. How do you want to upgrade or mind or stay on particular version of the packages in your setup, so packages not web packages?

A

What we provide might not be the best solution, always in your particular setup matter, narrative what you need on your note. You need to have just few binaries. You need to have cube idea myself. You need cubelet cubes et al. You need to have reference network plugins and you need to have a CRA tool to interact with runtime.

A

So if, for some reason your distribution is not supported or you have setup where you need to have like some more manual tweaks and my recommendation would be like follow instructions which we have in cubm documentation for container linux. It will describe like practically like 10-15 shell commands, which binaries in which location you need to put, and you will get where existing setup without packages.

A

Second part is actually fine-tuning, so you install the binaries what what's next in previous version of Cuba Geum, it was saying like we have a system deep drop in config file, which describes all the parameters. How we run cubelet and with set of parameters was quite long. This experience has been improved in past six months, so don't eat it anymore. This file, don't trip, don't put your additional system, we drop in files. Instead, we have so like our normal cubed iam systemd file.

A

It actually reads on Debian system eg see the world couplet or in RPM system a TCC's can pick couplet. It looks for variable for couplet extracts. So if you need to mind, if modify some of the flags, this reallocation way, you should put it another part which has also improved in latest versions. It was about to have config map in cube system namespace, which contains like the cluster global configuration parameters. So again, once you need for cluster and the few later one wants to adjust, we you can either do this.

A

Config map and newly added nodes will be reusing it so how it works behind the scenes. Actually, we could bleed sorry. The cube idea is able to detect some of the parameters which is needed for couplet. It fetches well cube config from this namespace. It not just were information from with atc default for files and when creates two system files which I could cube. Idm is maintaining like one is common line parameters and whereas cubelet dynamic company and config.

A

So what's next of line installation, which is also something would has been improved in past couple of release cycles.

A

Previously we had practically two pieces to two parts of the code which was dependent on when network operations. One was flock of Cuba diem, which results were symbolic label like latest 1.12, to actually release like one point, twelve point to it: use it both in unit comment and in upgrade commands.

A

Nowadays we have a bit better code which actually understand what like. If, for some reason, we cannot reach what download servers like because of firewall issues because of something else, what Cuba DM has good fallback defaults? What you still get with appropriate stable version of of your cluster second piece, which was doing my network iteration, was actually to fetch my images from registry like well kubernetes components and one couple of infrastructure containers like post it easily and add-ons like for DNS. So nowadays we have additional two Commons one command config images list.

A

It will allow you to to see which images is needed for your particular configuration, and actually it will show. Also this, which version of those images is needed and second comment. It allows you to pull those images. So if you have, for example, like air-gapped install how you can use it, you can have like separate machine which have direct access to Internet.

A

You pull those images where, when you do like docker export, Rios images, transferable files in your secure environment and when dr import, those images into your well either your hosts or your private registry and Risa's just helpers how to how to do it.

A

Beside wed. Sorry all the things what what I mentioned like web cuba, DM, binaries, couplet, binaries and so on. All of those binaries are available.

A

Doesn't so files in our download server and URL scheme is always simple, so you have like releases name of release and when Mike architecture and so on, so usually you can get all the binaries. If you don't know what is actually behind with URL, you can use Google cloud to kill. It is to actually list with storage. I know it's problem here and China, but most probably you know, you know how to how to get wet information next thing again: network network problems. So, as I mentioned vault is not video right.

A

Most of like old-fashioned techie companies. We prefer two separate networks, so you have probably like internal network where you have all when you did connectivity. But if you try to go to outside world you guys don't have connectivity at all how you have in best case scenario by proxy, so how to tell with kubernetes to use particular proxies. You need to have it specified in two places. One place is your container runtime, so we contain your runtime like dokie rows here.

A

I should be able to fetch images, so we're first underwater stone here is when no proxy variable. You need to make sure what, if you have local registry. So it should go, not true a proxy because otherwise, like public will be working internal, not be working.

A

It's a bit more complex for cluster itself, so here in no proxy variable, you need to take care a bit more, so you need to have like your local domain. Like my company comm, you need to have ranges for this particular network segment when you need to have ranges which will be used within the cluster and if you have a cluster domain also needs to be specified.

A

So too long, don't read with us, for example, like local network local domain cluster segment, cluster domain next thing, Cupid iam handles reparation for practically all the configuration parameters for you when you initialize in the cluster, but in some scenarios you would like to use your external PKI to prepare were certificates for your cluster. The obey diem nowadays allows.

A

What only thing is what cubed iam expecting those certificates to be named specifically, so you can look at where face comment generate certs, so to see if like which certificates should be named it as mid file and then get a group private, the key I export or certificates put it into a directory man just instruct with you begin to to this directory next thing, also: change in user experience of Cuba diem. So over time when we develop it with Cuba diem realized two things.

A

One thing is what the kubernetes itself actively developed, so some flux I changed some parameters, a changes via things going out in and out. Second thing is what, if it's start to add more and more parameters, configurable by cube? Am we simply end up with I?

A

Don't know like thousand character long common line to cube am so we can't add more things into a flux, so it resulted as idea of component config, so you can provide to Cuba DM the file we ammo file, which contains one or more objects like similars semantics, like anything else in kubernetes it it's worth showing it. So, even though, between different versions, that will be some changes of how particular flag of a variable is named it.

A

We have a bility to actually migrate between versions of worlds, configs without actually asking user to be changing it and McKiernan config spec. It was alpha in 112, but for 113 we are trying to get it into a better stage, so what kind of configs is wet? It's in yet configuration it's cluster global configuration, its global cube, let parameters and it's Joanne kinetic parameters where were used.

A

I will show a bit later, but what you need to know is actually what global configuration will be saved in your cluster in cube system, namespace and likewell config, which was used to create the cluster, will be stored in mechanik map, cube, cube, IDM come here and global couplet configuration will be saved as well were shown at were config map, couplet config, for example like 1.12.

A

So let's look like what those objects are like. So with this example of init configuration here, you have like very simple things which you customize when you're initiating, where the first master so, for example, like well address what you advertise may be. Some defaults for bootstrap tokens may be some theorize okay, and by the way you don't need to remember those files. By heart we have a helper command cube by DM, config print default, and when you can specify when name or object, it will print it out with the false.

A

So you save it in the Yama file and when just modify or remove for parameters what you don't need to modify and when you you use it second object, is a cluster configuration as I mentioned its global configs. So what is interesting here is where the things which really affects a whole cluster, so you can specify which version you want specific for China.

A

You can specify what my images are actually in my local registry, your network and when for each control plane component like for scheduler for API server and so on, you have the ability to add either additional common line parameters or you can mount additional volumes.

A

If you need to make some custom files available to to go sports with control, plane, components, the couplet again global configuration, so, for example, if you want to enable let's say CPU manager feature globally in my cluster, you can just define it in in couplet configuration and when every node, which joins will be having with configuration, enabled during configuration.

A

It's used during to join comment, why it's useful it's usable from few perspective. So if you have a custom CRI, so you can specify with your eye socket you can overwrite when names you can pass additional extra flags to couplet specific to a particular node again, if you have some specifics to it to be handled on on a node right.

A

So what's next! Yes, last but not least in the latest versions of cube ADM, we are deploying the couplet proxy sorry cube.

C

A

As a daemon set, so we also be able to provide web config what company and config forward component, so you can specify like if you have like usage of five years or some defaults, what you want to adjust for IP tables.

A

Next thing are Don's.

A

If you know what could be cool by the Emmys, you probably already know what we are handling only the things which is really mandatory for certification of kubernetes installation. So we deployed practically two adults. One is DNS and we have ability to specify which one. So previously, it was Cuba DNS nowadays by default with its core DNS. But if, for some reason you have reasons to deploy it. Otherwise you have a config options which you can switched on so.

A

Everything else it's up to you, there's boards specific networking q by DM doesn't touch it. However, as a scope of a whole cluster lifecycle, we are thinking about how to streamline and how to generalize usage of add-ons and how we do it and we are currently looking at the project called cluster bundles. This project was contributed by google kubernetes team.

A

Not official Google project is just like community ones. It looks promising. Let's see how it will go all right. So what was my part and now reward to my colleague, okay.

B

Since Alex folks show you know so many best practices on yodeling could be there, so copa-data actually is simple and he fish on the way to RP swing, that about kinetic cluster. In fact, money on the popular kinetics dollars. How Richard took took over the area to cool area as a as T for the installer, like kuba strike, Hofstadter X that work and, however, if you're interested in understanding what's going on behind the hook under the covers, the new commander could be alpha phrase come into your skin.

B

So actually we bring this new feature in one point said, and the main the main targets for this new victory is too making Kuby area more modular, and this modularity enables everybody to invoke dynamic. Several steps will input stream in the cluster and inhale that could be a DM true to some some parts and the filling your customizations during the prescribing and actually for some months. So you enjoy the some coupons here so actually the first is just.

C

To the community.

B

They have several sub mode here, just like a trade flag, so you can draw, could be them are from face pre-flight master, so pre flower must node and the wrong could be could be a DM, our prophase pre-flight node to Rome pre-flight other no unknown.

B

And there are so many facets here, so you can't write a letter after decision to try our new featuring one point. You let one point to twelve and.

B

The phone could be knit. This is the most complicated the step in in qiyam, so there are lots. Oh no there's about eleven steps here. So I will just to pick up some some steps to illustrate how it's going on behind the screen. So first, we already have a pre-flight here, so the pre-flight here actually the pre-flight.

B

We will draw a series of pre-flight checks out the node, for example, and master to check whether it is whether it is ready for post wrapping a master or con trip lines such as whatever check about the power usage. Whether there is the pot conflicts during prescribed in the API server, the scheduler or the under control manager and the you know, we're also checks.

B

The weather, ET city is running in local, a to make sure that the data really not escaped or delayed it during the new static spot, and the next step is to convey a couplet. It ever arrives complete complex to the disk and try to restart it complete, and this is the most important part.

B

This is money deposit in D, so the certainty perhaps restarted complete and the next is to generalize all the certificates, because every components exactly such as IPS server, scheduler I use a TRS certificate so that this step we have generous all the needed has to have prescribed the concho plan and the next one is to help you generates. The co beacon, victory.

B

And the next way is to generate copy config fires. This fire be needed by the culture plan such as the controller, the scheduler and the the fifth step is the most important part during the community. It will have generates all the static static files to the manifested folder and the Cobra tried to watch that manifest product reload and to recreate all the other parts that defined the United States.

B

In that the manifesto folder and later the the copy editing will try to writing straws a whole culture plan to be ready because the gas, if the consumer is not working so the later steps will not going on. So when the controller is ready here, the controller includes the a pastor or the controller manager and the scheduler and the copier name. We try to respect all these three components to be ready.

B

Next means already, so we can go down, they will try to upload all the COO medium and the Kubik on complete configured to config map, and this is a new feature. Choosing one point term right, yeah.

D

B

Config, my worst are all the configurations that we use that you're in the coop in each step. So it don't worry about. You were missed any configuration steering step, so you can just restore so any any versions that you a convict the next and this is Champlain- is ready. So the next step we need to attach this know. Is it tender this note as master, because it's all master to be scheduled, so this will herb at the tent Susan note unscareable, to prevent other post release scheduler to onto this note and the next.

B

This is the most interesting part or could be them so actually cook them will try to generate a positive talking to her your clusters of to Harvard your nose joining until the cluster, so in a virtual eraser positive token. So then those videos is talking to join the cluster and since the community has you force tribe, you are right to the API server, so we need to set up some our break rules to into a cluster, including to allow either as a note, to send six hours to the API server to to do Ted Hayes.

B

Oh, it's a certificates right and the last step is to install some ions this ass, as includes the normally we just test Cody s so and the could be proxy to draw as a team instead in the cluster.

B

Actually, if you are drawing a hybrid running running, the cluster is different architecture. So don't worry about the the proxy or the demon says here, because we have use the multi multi are dug images to to enforce all the other. Those can pull out the rights right, toggle image with the right architecture, water.

B

So the text product could join so when we post wrap our culture prime. So the next step is to join a note to the whole cluster and the creepy join is much simpler than the Queen could be need.

B

Normally, we have always three or four steps to joy or no to the cluster, but you'll have so see somebody insuring you make a complete confused actual. We have near either new effects. One point: each of the experimental comes a plane, an average planet and our explain it later and the phone nobly showing a note so cluster, the Rubies four steps here.

B

The first step will still be preflight checks from a serious sir pre-flight checks on the as note, and then it will try to fetch the unique configuration from the IBS server so to make to make sure the the note can be connected and ever never write. Some prefer specific checks based on the configuration we fetched and the next step is to directly put strap the cub lid.

B

So the step- it's quite clear and very short, so we really see the node will join to the cluster and we will do cat nose means under the control plane. You'll see this nose is showing the cluster, but we are joining yeah. We are joy, adding a new, fake, experimental control into. What's the cookie idea me join command, as some extra steps will be added here. The first a step is to check the cluster. If the cluster supports adding such a new culture plan to the cluster, so don't worry about.

B

You are adding your own notes to add in your own node as a contra plan, so could be, enemy will help you to make sure it all works. Yeah and the next step is to draw could be an amine its pre-flight check. So this pre-flight check is money. You can consider it as the enemy are for pre-flight master, so it is, Rania has control, and so it you can consider it as a mass node, the next step, the generates static port, so we were generates four components here.

B

A pastor was scheduled contaminant control manager and the ATC D. So when this for well visits for manifests, are generated for posts will be created during the API server, and this way is still preserving that coke, so all the scenes down. So we need to upload some configurations. We man we have met. We have changed that you're in could be joining so the API server. So this is why the stay we here and the last way is still to tainting the node as a contra plan.

B

So we were at a 10 here to some bug the node as unscareable.

B

So this is a upgrading, so when talking about abrogating, we were split into two pathways about the controlling. The other is about the load. The first thing is the come to plan, so there will be still the pre-flood here and we will check the cluster is where the car stays healthy. So if the car universe a river, not any upgrading, stop actions here, so slider below tried to get a configuration from the the config map here, so bad your changes or uploading.

B

Your chinese charges to the api server in this could be admin, config config may be so, and the copy and me will help you to try to read to fetch all the configurations here and they try to enforce our version. School policies carriage currently could be anything thousand spots to upgrading from from major to minor versions. For example, if you are appropriating your cluster from 1.2 1.2 turf, so this is forbidding, so we are only supporting from one point certain to one point: eight eleven, so in upgrading cluster step-by-step.

E

B

The next step is to upgrading the respective posts because we we started out. This contra plan puts a stake fold static file, so this is take Phi over period between the right ad and the cob late. Work will try to request this post later and the falls and those actually could be enemy won't help you to upgrade to the copy the Werdum, because because this is tau by your package package manager.

B

So if you are, if you want to upgrading the coverage stuff, so it need to issue your package manager such as a meteor young, to upgrade to perform the upgrade for the corporates configurations could be anywhere try to download the config map here. So the copied configure one point, one point X, where X represents your complete version here. So for this feature we need to, we require the copy warden to be our present. One point: eleven.

B

So for copy them, we have several changes in our one point ro y is about. We are adding component config into Arabia, which is still considered alpha, but we try to make it better or higher version in the next release and in the most most significant changes here is we are adding a new fake, experimental control plan to have you join a new control plan into the cluster.

B

We have also turned every province during the endpoint trophy, such as the Sierra spot and a fly and a curved spot, and we are going to deprecating server, hosting and LJ, because we have already have this fake here and this flag will help you to to make a hyper birthday and the for the next release. We are going to make my copy damn general general availability. So this is what we are trying to to in the next release, and this Ruby coming through in the next pass.

B

The December- and we are also going to to promote the component convict from the are frustrate red one. So we can have a better version to the component config, and we are also going to create a fascist should better, which is alpha now, and there are several bug fix here so could be. I could be adding with trying to improving, using pure and still resists three or four releases yeah.

B

So we are trying to make kobe them are paid to chew for four years of use, and you can rely could be Allium to as a as a building block for your customizer. You Staller.

A

All right, so we actually, we just have a couple of minutes left so I'll try to very briefly describe a cluster API. What you need to know. It was a few sessions already here during this cube corner here, which was talking about cluster API, past tense, and so I thinks what you really must know about what last rate day. So first thing: it's declarative.

A

It describes the cluster machine machine, said and machine deployment, so so to draw analogies where machine is your port machine said it's replica said deployment its machine deploy, and so we have controllers, like we common logic which operates the same thing, so you can created machine deployment describe what kind of template of machine it needs to be done and machine controller will speed will will create machines according to a specification logic is common between our providers.

A

You know, of course, like some providers will need to implement exact, specific, like plug-in and piece which handles like individual machine for particular environment. So, for example, like Amazon VMware, whatever you have, we need to create my own provider.

A

Ready is what like, we will get a declarative API, which we can reuse across multiple tools, so, like 10, 10 kilometers view how it works. First of all, you need to have some cluster where you deploy CFDs, which I just described, but you need to deploy a controllers which will be talking to your cloud API, and you have couple of utilities with which you can interact with this cluster API like one is to bootstrap Withings and one cube CTL to do like all normal activities.

A

So once you're done with your local cluster, will it be like your management cluster or, like mini cube on your laptop, doesn't really matter? You can copy, will subject those controllers immigrated to a real cluster, what you have and once you are done, what you can do you can practically like scrub, your bootstrap cluster, and when you operate normally with your cluster, so you can create more clusters. You can add machines, scale, scale them up and down up to it's all api's.

A

So the current set of like the whole cluster API is, we don't have to even get to a real like alpha release. So we are prepared to have a form of alpha release providers which is implemented right now you like few of where I'm already did small religious but majority of remnants you under heavy development, where actual list you can find on web project page of cluster API. If you have some specifics for local providers here in China, for example, $0.10 I, don't really know where we're publishing way code. So probably you can ask to.

C

Share what part.

B

So so don't fill in many to make any changes to our rebel, so even a small change in mechanical a big difference. So if your cotton question children's are the participating, so you can just pin me ass, like or sesame on github and we were charged a Harvey get started is sick. So thank you. Thanks for all your coming and we have everything can help us to make it better thanks. So much for today, yep.

A

So what what now we really want to compare contributions like a small box or a participation in our channels like we know with its many of them, but big ones which you really like and which is close to your heart and get involved?

A

A

Think we have a couple of minutes for questions if, if anyone would like to ask.

D

I know we have already called provided its, what a different cloud providers like a device or Alibaba. So you mention here we will use some kind class cloud provider right. So what's the difference between the cloud class product and the cloud provider so.

A

D

Basically, it's based on it Oh.

A

Excellent question: so where is the difference so what current cloud provider it's usually about integration between couplet, sorry, kubernetes, control, plane towards services? What provider provide like load, balancing networking, ingress controllers policies? Maybe maybe networking made like persistent storage and so on cluster epi cloud provider? It implements one simple thing: well, not so simple!

A

Actually, so it interacts with provider API to create a new, let's say virtual machine or physical machine, and when bootstrap were like basic operation system on it and when ghetto access, what say, SSH to wet machine and when, where this piece actually will go and execute on what newly provision that machine, what huge idea so.

D

There are different systems, yeah.

A

Independently DD different purpose, so a cluster API cloud provider responsible to create delete and bootstrap with new machine. Thank.

D

E

You mentioned the feature flag, experimental control, plane right so can we use that for initialize a cluster with multiple but multiple masternodes I mean yes.

A

E

Okay, so what about the certificate? You didn't mention that it will hand out the certificate from the first master to the others.

A

So, yes, we have documentation how to do it. We we have well when you create, in notes, use an experimental master, join. You need to copy few files from your first master to any notes. Ok,.

E

A

When, when you execute, you were a cubed, iam, Joe and experimental, and so on, it will pick up those files and when okay.

E

I see so there are some menu steps to transfer the certificate files between.

A

Note yeah master knows yeah. Yes, if you go to kubernetes dot, io / dog /, cube IDM install. Where is a special section which say it's like how to use word feature and we.

C

A

Like really I, don't know like five six lines of shelfs niblets, which would.

C

E

Hope you paste and it.

A

E

Work: okay, thanks I.

F

You mentioned the big strapping and like needing maybe another cluster to bootstrap your first cluster. What is there in terms of like disaster recovery? Maybe you've destroyed that big strapping cluster and your main cluster has a problem.

A

Well, why you need to have existing clusters, because we whole object. Definitions is series, so you need to put them somewhere and when, like with the API storage and when controllers like cluster controller machine controller, we need we need to be run at somewhere. You can run them as a standalone binaries. So, but still you need api server, where you store with whole objects for disaster recovery. Of course, you can have like my own management cluster and like permanently hosting those components. But again it's it's. Just it's just objects and database.

A

You can have as many copies as you want just just.

C

To matter would like don't create a race conditions for multiple controllers: try to do the same thing: use of the class, the cloud API, okay,.

F

Say best practice is to probably keep that sort of each track cluster around or.

A

Create a proper like management, cluster yeah, so you have like management cluster as you as a cloud provider infrastructure provider and when for customers you just provisional clusters. This is something for example. What consent was showing one hour ago on web presentation? Okay,.

F

A

Think we are done. We ran out of time. So thank you very much for visiting our session.