Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 14 Nov 2015

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kubespray Intro – Antoine Legrand, Red Hat, & Matthew Mosesohn, Virtustream (Any Skill Level)

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

Kubespray Intro – Antoine Legrand, Red Hat, & Matthew Mosesohn, Virtustream (Any Skill Level)

Join the Kubespray Intro session to learn about deploying a manageable Kubernetes clusterAntoine and Matthew will help you decide which options to consider when deploying Kubernetes. Included in the presentation will be coverage on how you can use Ansible to extend what Kubespray does out of the box.Kubespray can deploy Kubernetes clusters from small to large with an intuitive and logic based approach. This session is best for users who have tried out Kubernetes in a virtual machine and are now ready to prepare for bare metal and/or production Kubernetes.

About Matthew
Matthew is a software engineer at Virtustream developing platform services on top of Kubernetes, providing containerized infrastructure. Matthew is one of the maintainers of the Kubespray project and participates in the Kubernetes SIG Cluster Lifecycle.

About Antoine
Antoine Legrand is a software engineer at CoreOS co-leading the working-group 'app-definition' and works closely with major actors in the ecosystem to improve the user experience in managing application in Kubernetes. Antoine is involved with open source and cloud native software projects, and for the past two years has helped the community and led related projects including Kubespray, kpm, and app-registry.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

Alright looks like we have pretty much everyone, okay, so my name is Matthew Moses inthis is Antoine Legrand before I start getting into cube spray. Let me ask first who here has already deployed their own kubernetes cluster. Raise your hand pretty much everyone. Has it been multi node everyone, multi master, that's cool, it's excited, so I wasn't expecting I feels like I've, never installed, kubernetes. Let's, let's get an intro, so this is kind of a basic talk, but we will have time for questions at the end.

A

If you want have any questions about some things that might be kind of deep, but maybe we won't go too deep, I'm, Matthew, Moses and I work at virtustream. We do kind of a storage and virtualization platform on top of kubernetes kind of everything. At the end inside in helm, charts renting in kubernetes, it's very exciting, Antoine I found it yeah.

B

Three years ago, with a smile on prediction, use case and I'm also very interesting- is a application deployment of the manager workload also working on self definition and also monitoring team inside open ships.

B

Yeah, so why we care about like how we deploy communities so we go. The end goal is like to run application and to keep them running to get out of them idea available. So this go down to the cluster infrastructure. So when you upgrade your cluster, you need to make sure your precursor and keep running as a row. So this is like just to refresh why we care about the infrastructure so to make sure your applications still get resources and be able to to run in an ID available fashion.

A

So kubernetes deploying it seems kind of easy if you have a script, that does it all, but really to think about what are the building locks of a coronaries cluster. There's masternodes, there's node nodes because slave isn't a cold term anymore, there's sed, which is commonly combined with the master, but it's usually kept separate in case of scale deployments. You know over a hundred nodes. You need to contain a runtime which is usually docker and our plugins.

A

If you have multi node, you need CNI, there's other ways, but most people use CNI a way to make your certificates for the kubernetes api and for a CD. Unless you don't care about security which shame on you and persistent storage. If you want your containers to keep their data after node restarts or migrating, you really need persistent storage, which is usually offered by your cloud vendor or something you can deploy yourself like clusters have rook whatever you want.

A

This is a really not ideal diagram, but just kind of an idea of how kubernetes deploys there's masters and then there's a nodes.

A

Everything routes through the API code, miners, API server, not the controller managers, it seems so the API server talks that see directly the no-strike directly to api server, the other master components route through the api server. It's just a like a client-server architecture, but so a lot of people ask: why do I work on a Cabrini's deployer? Isn't it really easy?

A

Well, it's not, and why is it divisive so there's the old argument of which can may figuration management platform is the best I kind of pick danceable for a few reasons, but there's others out there, or should you go alone and make your own, or should you orchestrate with something written, natively and golang? That's just a topic that brings a lot of contention and then there's, of course, that not invented here syndrome where people are concerned. If I didn't write it from scratch myself, how can I trust it?

A

So a lot of people are still going to write their own. A lot of vendors are gonna write their own because then they can put their name on it and sell it, and my offer support keeps vary. It has no commercial support, it's just an open source project in the kubernetes ecosystem. That's not official, but it's in the ecosystem and of course, deployment is divisive because people want their deployments to be reliable, reproducible and maintainable.

A

So that's another reason. Some people prefer to deploy their own homebrew, the other items we compare ourselves to our peers. If you would would be que ops or cops, which is written purely and golang, it's a very good piece of software. It happens to be geared more towards Amazon Cloud, sir AWS, and it has offerings for bare metal. We kind of differ ourselves in that it's deployed everywhere, including other clouds or bare metal or virtual machines, and so on.

A

There's cube ATM, which is very popular, very official, but it has some shortcomings because it doesn't prepare your nodes completely for deployment. So we do have a mode of deploying you keep spray with cube ATM, but it's just a small part of the deployment actually and then, of course, there's commercial offerings which are either open or closed source. They have their ups and downs, the down, maybe being cost and openness.

A

We try. Not to compare ourselves too much with them, but they do exist and we coexist with them, because people will consider them yeah. So.

B

Cubes break quickly on the go, so it's a crystal lifecycle management manager. So it means like to create your cluster, but also after how to manage it so are to upgrade from communities 1/9 110, but also to add node or to don't scale so resyncing about the management which is like creating cluster, it's very easy but then being able to upgrade them can be more complex.

B

One of the initial goal was like prediction: really: I say what like for our own use case. So the pro commit is a prediction setup. So idea available, it's like a key goal, so a lot of work in it and flexible and composable meaning to work on many ways to be able to choose a network plug in to ignore it like there is a lot of components in the ecosystem that we can just change. So this is like one part, moving focus and yeah.

B

So project has not nearly three years now and we've been committee drivin, so there isn't like no, no one shot behind and just the committee drawing the project.

B

Okay, so we choose to use so keep spray. Is an NCBI playbook. We choose and shiver because first its cover or component, so we can just have like a plain operating system just with SSH access and just setup it from scratch. That's a very key element. It's very easy to read. Even for people who don't understand is very easy to follow so she's. Like me, many user Ranger community never known about NC bed before but just very be able to copy the people and get started very quickly.

B

So I said it's flexible. It works on many ways and it's actively maintained by the committee and I read that I know so. Yeah.

A

Thanks Red Hat I've added a lot of useful modules since their position, so project keeps getting better, which is a good sign if it's something you're intend to be invested in for a while. Nobody wants to deploy a production cluster that has its software out-of-date quickly.

A

So, in addition to just being ansible based, we have a strong focus on cross-platform. You can have heterogenous os's across your cluster, preferably same docker version. You might have interesting results. If you try to mix, we do container as everything the static pods for the master components to containerized at CD and some other services there's no packages that are installed, except for Python dependencies for ansible and docker, which is nice, so that makes the cross distro compatibility layer quite easy.

A

We just added sousei support a few weeks ago with no issues, no thing that was installing docker, so yeah bring any OS or any Linux flavor in that we have bindings for and it should just work. We do have a compiled by matrix.

A

I won't iterate everything here, I'll also list that we have three different modes for generating certificates. Our own script, bastes, which just runs open, ssl x.509, create you know very boring traditional approach. There's the cube ATM approach, which does this all and golang, but basically the same, and then we have hashey core vault support as well, which stands up vaults, and then you can hook up coronaries back into the vault so that all those secrets come out of vault as well. It's quite exciting stuff. Just.

B

To reflect also this composite matrix is, why keeps res being like highly adapted because it just like fit whatever your infrastructure? Is it's mostly going to be able to work on it like we, don't ask you to change for any particular OS, any particular setup or cloud is just like mostly going to work with your actual infrastructure, which is good, good.

A

Thing we're not biased about network plugins; these are just the ones that we have configuration for. We've had added more recently. Content just was recently added, so there's room for more. If you happen to work actively on I ain't, never plug-in what is going on: okay, cool, it's a clicker.

A

They got to click okay,.

B

Yeah so I say it's: the project I've been like talking about the community I think when she grew like twice in the last few months, and it's very supportive, like mostly newcomers, just say how can people always upon the selection air, so you can join, join the communities cube spray Janelle and you get like ton of air.

B

It's like more than now 240 contributors, which is a lot and means it's really easy to just many people just do one PR and sometimes they are big but see it's very easy to get studied like to add your own use case and just fix a bug. So thank you if the risk on Twitter in the room- because this is like I- was a project move around, so they like just quickly also to thank you, companies and contributor, we'll just keep continue to to contribute so yeah yeah.

A

Some of these companies don't work on cute fare is their main project, but they have members that work on it for some smaller projects, but this is just representation of people who have worked on the projects so really its diverse. It's not any. Just one company, that's dominating the project. We do run continuous integration.

A

We do only three full deployment scenarios by default, just because of time constraints and cloud credit constraints, but before release we do run all twenty plus CI cases, because we have so many options and configurations, and if we didn't test them all we might miss bugs. So we definitely need to run all these scenarios, but we do make sure they all work, which is good for users rather than saying oh, it passes, you know some sort of syntax check, I'm sure it works.

A

No, it's it's really tested. It's used in production and many companies already.

A

The CI tests is a good way to enable first-time developers to see if their code actually works without having to maybe test it locally, but that's not actually really good. This.

B

Was not one? The question like the kombucha metrics like oh, you can be able to maintain such a huge front. Pass. I was like one of the comment feedback, and this is like why we focus too much on the continuous integration, so we're sure, like we try with work aid, we try with sues and someone change on reddit, which just triggers a change for the reddit OS, etc. So this is really oh. We can move forward quickly because we have like, or many paths are tested.

A

This is a really busy slide, just all the main steps we go through for a deployment. We have to prepare the OS get Python on there. If it needs it. You know a lot of cloud images. These days are very small so that they boot up faster or it can be pulled down faster. Then there's some pre-installed steps, you know maybe set time zone, maybe create some directories, verify cloud config parameters and then we set up docker sed itself.

A

We build up certificates and then the the master, the nodes, that's a typo and the network plugins and then other any other add-ons like DNS, which is optional, but you should really use it.

A

How Prometheus and a few others I missed out Balt in there, if you're, using vault it would get set up before it CD our component up component upgrade workflow is pretty simple: it's just how you would do any component, upgraded ansible, just change the configuration update, your launching scripts and then restart system d-unit static, pod upgrades for like operating masters would just happen by updating the config file itself.

A

It's nothing magic.

B

On that we do test each time force, so we're really is we tack, keep spray for when we test everything and also for we take the letter 3ds and we tried we deploy it's on the CI and we try to agree to the letters commit on master. So we also time test that awkward, it's working, if you use a master, so we can keep track about make sure it would be always able to a gradual just, oh yeah,.

A

There's to upgrade strategies: one is the unsafe method which just upgrades all the components and if cubelet happens to be down for over a minute, your pods get shuffled around. But if your posts are really fast, you might be lucky and just upgrade fine.

A

If you have nothing in production there on your cluster, just use the default cluster yang will play a book, otherwise use the upgrade cluster playbook, which will upgrade masters one at a time and then Cooper news nodes in batches of like 20%, but that's going to figure ball that will shuffle your resources around then upgrade the node and then allow those house to be scheduled again.

A

B

A

Have a che for the two main components that needs to be a che sed: all the clients can just connect with a comma-separated list and the cube API server is actually a very different divisive point. We can't rely on users having like a dedicated IP for a load, balancer, so kind of came up with a backwards solution which is install a load balancer on every host, which means just right engines locally, that proxy's the traffic to all the API servers.

A

It doesn't really add much load, but it does reduce the requirement for a dedicated load balancer. But if you do have one available and you do configure it in your settings, we'll use that because it's it's always better when you have dedicated load balancer, so coop spray has been tested scale. It's been scaled to a thousand nodes deploys in about seven hours. Ansible is kind of slow, but we still are happy with the results. The readability is definitely worth it.

A

When you do scale kubernetes, you do need to tweak the polling time for cubelet and make su denotes dedicated just so that you have enough disk IO for sed to catch up. And of course, you want to run SSDs on your master components.

B

And if you're interesting in the sched there's like couple are easily fine blog post about also details to scale thousand node, so what's what's next on on queue spray first, like add more so it's working on a degree base as you and the meter, but we don't allow track this tested in CI, which can be useful to just be able to say to checkbox is working.

B

One thing we want to focus also is to be able to test again master branch of committees so be able to build communities from master and then deploy oscillators. So we can actively develop on communities and just be able to deploy our raters. One thing that is me I would say missing compared to public helps on Amazon is auto scaling, because right now we need to SSH. Oh, so we need like endpoints to configure the node, and this is like really something we would like to change so to be able to.

B

You know to the nod, to fetch a configuration and be able to bootstrap himself, so we can just scale point to a CCD or whatever to fetch a configuration.

B

But that's that's like one things really need to focus and the different thing, but also there's like I demand for how to deploy primitives or me to support our the log management and different add-ons, and we try to figure out what we can do it in psyche. You spray, which is compatible with the community regarding.

A

This auto scaling there is the cluster API project, which is still sort of in draft state which seems to enable this sort of technology. You would define a change in your cluster definition and then it would run the steps to deploy in whatever cloud you're running, to bring a new VM and then run whatever deployment tool.

A

You want to put communities on it, it's kind of a vendor agnostic approach which would be exciting, but it's not ready yet and, as always, we're looking for new contributors and thanks for coming to our talk, do you guys have any questions you in the green shirt.

B

Yeah officially commit this. You bought one one certain I think so.

A

No, it's supported now 1703, so we try to keep in. The question was: why do we use 1703 in cube spray? The answer is: we keep with recommendations of the release notes because we, if we go to a newer version, we might introduce more issues. We only upgrade as necessary when we run into bugs the same thing goes with the Etsy D.

A

This is going to apply to all of the players, so it probably will work with the latest and we do support installing the latest version of docker, but we default to what's recommended by the community. It's a good practice. Next.

B

So, as the question is I, do we support which multi-cloud so no funnel we deploy one cluster? We don't do so. We can deploy on different cloud, but it keeps Fred doesn't manage aggravation. It's.

A

It's more of a problem of the network plugin if it'll cross, the the hybrid scenario, I believe calico with IP IP enabled my but I wouldn't do it.

B

Yep and but just yeah.

B

So the question is like: can we use existing it city or vault? It.

A

Was done as a proof-of-concept, you could skip the role at CD and explicitly said to add CDN point variable it does. It did work when we tested about a year ago, but it hasn't been repeated.

A

A

Let me explain that first of all or your question is: why do we do eyepiece change and how does it work with a local lynchings load balancer? First of all, we don't support changing IPs of masternodes because we generate certificates based on those IPS with subject alt names. So if you change, the IP is better to rename the node as well. In addition, we it with ansible, create an Etsy hosts file with entries for every host on every node. So that's how we do dns ID event, you back there blue shirt, yeah.

A

Well, you create, the question, is: do we support multi master and keep spray, and answer is yes? From the first day, multi master means first of all, setting up multiple master nodes in the control plane. It's just as simple as adding your hosts in the inventory to the group you master and the the the point contention is a.

A

Second, adding connectivity to all three met, all the Masters, usually three or five wave, a load balancer, so the local engines load bouncer- does that it just adds the all the masters in the list and they connect. So.

B

To touch on this, you can really easily define the topology like say: 3t, CD, remaster, tree node and on different nodes. So it is like Jesse Ventura with groups, so you can define different nodes or use the same node, etc. So you can really define what kind of setup you want.

A

I can answer that your question is to require proxy. Do we allowed write, require direct internet access? The answer is No. Perhaps the docs are just old, but no for every component. There is a proxy support and you can change the repo config for any docker image as well. So you can point to your local registry artifactory or whatever you have.

B

Quick questions like can we done great downright.

A

Downgrades aren't officially supported, but you're very lucky.

A

Upgrades from every major from every released to next release is guaranteed to work, but we don't actually test down grades. Yep.

B

A

The questions like.

B

Do we use and supposed to work so no, we have like. We never try to use just one say yeah in the back.

B

No, so the question is like about the scaling, so we speak about note, auto scaling. So if you want to say I want one more note to manage the workload yeah, we can't do that.

B

So you're using cups- and you want to know what. Why would you change to give spray? This is a I.

B

Mean it it depend of your use case, usually if you're on AWS I will recommend like and you're sure, you're going to stay, for example on the ATA, so we recommend to use Gabe's, usually so what I say. But if you want like some everyday setup like at some point, you want to be on bare metal. I would say, like you, spray, it's much more adapted to get on between different clothes and not every parameter. In.

A

Addition there's more options and keep spray for deployment modes, so everyone has their own personal preferences and not even our tool offers every option possibly available, but I like to think we offer a large variety again.

A

Well, so is there any revert mechanism, not specifically, but if you were deploying from one version to another version of keep spray, you should just go back to the previous version and see if it works and.

B

Also in the gray strategy, I said it's going to a great one master after now, sir, so if you want everybody set up different master, if one master fail, you get the other end same when it came for the note. Let's go like tree node by tree node, for example, you can set up so and as it rains I'm before. So all the water was running on the node, getting a great reschedule to also note that alive.

B

So if this tree get dance, you still have your workload, so you can just try to reapply the upgrade and you don't you don't lost any. Are you ready, like your applications, still running, even if it's failed in the middle of an upgrade.

A

The question is, can you remove or add nodes? Yes, you can add masters you now you can add SED nodes and cube. Now it's easily adding and a master requires regenerating certs. So we don't do that automatically. But it's an option manually.

A

So it's just a matter of modifying your inventory and running the playbook and it will add nodes or move nodes.

C

A

Can take that so I need tips on security enhancements. Well, in general, you make sure SSL is enabled there is a utility that was based on some security analysis of kubernetes. We implemented a lot of their best practices there, so they're already inside of keep spray now so as far as I can tell, and from others, we've covered all the major security gaps that should be locked down in kubernetes in the user project.

B

So questions like: do we try on a sure? Yes, we have like a mini contribution on Asia, so we have like also a directory named country where so we don't bootstrap note so you need to get, but we have like in the contribution script to bootstrap on Azure and it has been tested quite a lot. So nature and I don't know about your stack but yeah.

A

We have no integration with, like you know the automation of deploying nodes within you know, Amazon, Cloud or Asher, and so on. Just you create you, bring your own VMs, build an inventory or use dynamic inventory and then just run that into a playbook. It's not it's just for deploying kubernetes, not bringing up VMs. That's still up to you. You have to choose a solution that works for you, because everyone wants their own.

A

So the question is: is there high availability for access to worker nodes, correct know if you want that it sounds like you probably are more interested in in bonding, so that you connect your node to two switches.

A

Yeah but I just can't think of a scenario where one VM can can't paint another VM on the same network and you you need a load balancer just to get to the nether note on the same network: Oh ingress. Okay.

A

So generally you set up a load balancer yeah for ingress.

A

That's still up to you because we can't manage that externally.

B

So this part of the second steps- and we found like there's too many opinion about how to set up so you speak about Asia proxy and talk to someone else. We want to use a traffic or nginx so, but at some point like we already have many many possibilities, and so we just part of the load balancing anyways. We decided to not go on this path. We can you can ask questions. Akimichi can help you out, but not in the main core project. Now, okay,.

A

Last question, and then we should break for lunch okay, lunch time, thanks for coming. Thank.