Flatcar Container Linux Tech talks, 19 Mar 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Kinvolk Tech Talks: Infrastructure components for Kubernetes deployments

Description

This talk by Sergiusz Urbaniak, senior software engineer at Gesundheitscloud, and Alex Somesan, software engineer at CoreOS/Red Hat, was first presented at the Pre-FOSDEM Kubernetes event in Brussels at the beginning of February 2018. It was so informative and full of useful information that one can apply when setting up and running Kubernetes clusters, Kinvolk requested that Sergiusz and Alex do the talk again. The first hour of this extended talk is led by Sergiusz while the second hour is given by Alex.

This talk tries to demystify the concepts behind a Kubernetes deployment. It outlines the necessary building blocks for networking, storage and compute for a cloud-based Kubernetes installation and provides a concrete example using Terraform.

Speaker: Sergiusz Urbaniak
Sergiusz is a Senior Software Engineer at Gesundheitscloud and formerly of the CoreOS Techtonic installer team, and before that Mesosphere.
Twitter: https://twitter.com/_surbaniak
Company: Gesundheitscloud ,https://hpi.de/open-campus/hpi-initiativen/gesundheitscloud

Speaker: Alex Somesan
Alex is an engineer at CoreOS/Red Hat and is on the CoreOS Tectonic installer team. He previously honed his Terraform craft at AWS.
Twitter: https://twitter.com/ASomesan
Company: CoreOS, https://coreos.com/

Meetup: Cloud Native Computing Berlin Meetup
Event: https://www.meetup.com/Cloud-Native-Computing-Berlin/events/247961017/

Host: Kinvolk
Web: https://kinvolk.io
Twitter: https://twitter.com/kinvolkio

A

Nonprofit organization in pastas on tonight's cloud and I'm doing software.

A

Literally, like half of my life already, I think in total, like professionally.

A

Already like nearly eighteen years, which is way too long- and you can see I he lost all my hair along the way. So that says something yeah so currently senior.

A

Software engineer are helping out a nonprofit organization here in berlin.

A

I've been doing various stuff in the past way too many to enumerate here. The.

A

Stuff that is relevant for you is like I'm an x choreo.

A

As we say, internally right, I've been helping to build the record project a.

A

Very, very nice alternative to docker. If you, if you're into these kinds of things,.

A

I really love rocket, so try it out and also at korres. I've been working.

A

Together with alex chalmers on on, because I've been also closely related.

A

To the company, this project on what we call the tectonic installer, which is.

A

Sort of like a kubernetes commercial communities, distribution since the.

A

Inquiry of reddit, I have no freaking idea. What will happen to the project.

A

Alex probably also doesn't exactly but the reason we are standing here since we.

A

Built this project, you know in the last, almost like one and a half years now or.

A

Is it just one year a little bit over a year right, we have some learnings along.

A

The way that we like to give you along your way when you have the challenge of.

A

Building your own kubernetes cluster right I mean we, we made a product, but we.

A

Actually had the challenge of building communities clusters on various platforms, so we know the pain, we feel your pain so and maybe hopefully you.

A

Will get out of this with a couple of learnings before that I am. I have also.

A

Worked with mesosphere is also a very good friend of mine and I've been part. I.

A

Have been part of the prod of the team that was trying to replace the.

A

Kubernetes scheduler with the mesa scheduler so and that's also the reason.

A

Why I'm sort of heavily also involved in the community's ecosystem? But mostly you.

A

Know I'm a software engineer, you know I'm. I have no background in.

A

Infrastructure I came to infrastructure as a software.

A

Developers so to say, and along the way, I learned a humongous amount of things.

A

Which were super valuable to me in my two days? Work as the backend, like mostly.

A

Back and software engineer I think like this is the canonical idea of what we.

A

Every everybody calls it devops, and I think this is exactly it when a software.

A

Engineer enters the infrastructure. Well, so I really like go the.

A

Programming, language, that is no the underpinning of rocket as well as of.

A

Kubernetes as well as of terraform the project that I will show you alongside.

A

With me, alex is amazon, so alex is an awesome guy. I had the big pleasure to.

A

Work with him over the last year he's an aws veteran. He has been working on aws.

A

Directly, so if you spin up vms, you can blame him on aws. I know.

A

So alex a couple of words, I wrote you.

A

Like set up, but oh mike, mike mike mike sorry I'll just so yeah like suresh.

B

Mentioned we met roughly a year and a half ago, as I joined kauravas, I.

B

Initially joined to do monitoring, but they very soon discovered that I have a.

B

Little bit of background and infrastructure couple of years at aws, so.

B

They moved me to the installer team. We've been working together on trying to.

B

Get people to have a nice experience installing kubernetes basically think we.

B

Did a good job at that until kauravas got acquired before that.

B

I was a circus mention that aws for a while at opsworks, which is the service.

B

That's being developed here in berlin, that's why I mentioned that if you can.

B

Blame me directly for your instances: that's not the service. You want to use.

B

There are better options out there, but what aws, what what four years of aw has.

B

Taught me is how to not do things and how to do things the right way. Basically,.

B

So it was a huge learning experience just running the service and having all.

B

Those s la's like behind your head, pretty much all the time and trying to.

B

Achieve them, it's a really enlightening experience. So a lot of that came into.

B

The tectonic product as we develop the installer.

A

So you know, working at korres is crew has been crazy at times like you know, a.

A

Typical monday, at korres, more or less looks like this hey good morning alba.

A

And the argo, let's build two container run time or like hell, okay, um you know.

A

Let's build a raft based key-value store, which is today backing up communities.

A

You know like working at korres has been really crazy under the challenge aspect.

A

So a typical monday for me and alex was, you know: brendon one-one once they.

A

Stepped into our office literally, he came from san francisco and said: hey alex in search. Can you please build a new kubernetes installer called tectonic.

A

For aws it seems like easy, I mean he's been working for aws, but brendon. Then.

A

Also mentioned you know, io, please also make the install available for a zoo or.

A

Openstack, the em were a gcp and you know like oh and don't forget bare metal.

A

Right so you can imagine like my face and in this very meeting like I was.

A

Literally, like I knew I can spin up the ends on aws. That was a world like that.

A

Much of knowledge that I have back at the time, so you can, you can think of it.

A

Of a really nice challenge back then so yeah, so that was that was more or less.

A

Our election reaction back then, over a year ago, you know we were sitting in the.

A

Careers office, I'm thinking about how to solve this freaking problem. You know.

A

It's not only like, probably who a few has to install kubernetes on more than.

A

One platform, okay, so that's at least two that's great okay. So, like I really.

A

Really really really feel your pain and I think the sort of recipes that we can.

A

Give you her along the way will help you out a little bit so how to solve this.

A

Well, you know like me, and alex after having a good sort of bottle of wine.

A

Over the weekend, so there's no problem with: let's have a look up in the repository of calcium in he has all the answers right. So who of you knows.

A

Kubernetes the hard way, okay, some of you great, that's a very good.

A

That's a very good thing to follow, actually because kelsey.

A

Does a pretty good job at explaining sort of like the lower-level intrinsics.

A

Of kubernetes, when it comes to deployment, you know, like all the parameters of the api server and how you setup things and things like that. But.

A

You know like, after looking at this, a little bit were like mad right. I mean it was like a like a set of written documentation. Oh.

A

Praise the lord okay, at least I can say I have the same.

A

Haircut here that's pretty awesome. That's great.

A

Cassie has been talking here too right, yeah, exactly right, exactly okay, so I.

A

Think kelsey does a great job at you know explaining the manual steps that.

A

You have to perform in order to set up a cluster, and I encourage everyone to do.

A

So, if you're in that business, nevertheless, it doesn't really solve our.

A

Problem because you know we need to automate everything, so we have to take.

A

All these recipes that are out there, I mean the experience from from kelsey and.

A

Also, our own experiences and somehow make this available in an automated way.

A

Across many many platforms using some abstraction that at least allows us to.

A

Use a common code base, essentially right. So when we looked at the problem, you.

A

Know it pretty quickly came clear to us that we must use some abstraction that.

A

Allows us to abstract away from the concrete proprietary ip is that the.

A

Cloud providers give you right so, for instance, on aw. Yes, you have cloud.

A

Formation on azure, you have what, how do they have an official name for the api.

B

What yeah resource manager, okay, so ezra, has.

A

Resource manager, whatever openstack, is interesting, okay, so, but they essentially.

A

All provides api's which are proprietary in in the nature, and it's very hard to.

A

Like find a nice abstraction that covers them all terraform helps you a little.

A

Bit by providing you two big sort of abstractions, one of them is called hcl.

A

Which is the abbreviation for hash accord configuration language which is a.

A

Declarative only language, it's like if you want to say so, it looks a little bit.

A

Like json feels a little bit directly yama.

A

And it sort of allows you to describe your infrastructure, your target in a.

A

Declarative manner- and this is what everybody says you know like.

A

Infrastructure as code more or less rather than being imperative in nature,.

A

What we know sort of like from everyday programming, it's declarative in nature,.

A

They understood that a pure declarative language, unfortunately, it's not enough.

A

To solve all problems at hand, so they invented like a little sort of nested.

A

Language with which they called icl interpolation configuration language.

A

Which does the imperative parts that hcl can provide you? Okay, so, for instance,.

A

Like a very, very, very simple thing that yet that you need to do is like.

A

Branching if-then-else right, if you have just a purely.

A

Declarative language like json, there is no notion of saying things: okay, if I.

A

Only have one virtual machine do this. If I have three virtual machines, do that.

A

And in order to sort of solve this problem, they added this icl, which is.

A

Like a mini language by the way really badly designed, but it gets the job done.

A

Somehow, ish, okay- and these are the sort of the two primitives that terraform.

A

Gives you along the way to to construct your infrastructure, and this is the tool.

A

That we used, you know like if terraform wouldn't be there. I bet that would have.

A

Been another mundane project that core has, but since it was there- and you know.

A

It is a pretty major project. This is the solution that we went with so speaking,.

A

Of building blocks to build a kubernetes cluster on your own on any given platform, essentially like I could conclude the talk and finish it off here.

A

You essentially just need three things like who was shocked the first time when.

A

You're locked in on the aws console- and you like, saw this huge amount of list of.

A

Api is like, I have no clue what they are all doing. Well, like you all, essentially.

A

Just need three things: to get your kubernetes rolling, okay, so it's compute.

A

Storage network, that's it end of story: let's have a beer, okay, so more or less.

A

So these are the three major ingredients to get running and honestly, we didn't.

A

Use anything else right as far as I remember yeah, exactly.

A

Networking, so exactly so yeah. These are the three building blocks.

A

That we use from any given club provider- and essentially you don't need more and.

A

The like the smaller the list is up here, the better for you, because one thing you.

A

Learn across the club provide is that they are very drastically with the.

A

Amount of features, even in those intrinsic s-- that I enumerated here so.

A

For the compute, at least from what we did in the installer also- and I make.

A

This slide, like very consciously very simple, just to point out how sort of.

A

Internally, very simple that the tectonic installer is we essentially only.

A

Maintain three sort of profile of compute machines: one is that city which.

A

Obviously, who doesn't know that hcd is sort of the key value store of.

A

Communities you don't know that okay, so for your information edd, is nothing more.

A

Than a small database sort of not relational, not like my sequel but like.

A

More like mongo, why, I would say right: yeah: it's a document store right so and.

A

Like edd is being used by communities to store its data, that's essentially yet.

A

That and obviously you have to you, have to somehow instantiate that hcd database.

A

You have to provision it in terms of compute resources and that's the first.

A

Thing you have to do, then you have what we call in tectonic. Installer master.

A

Machines they host essentially the kuban, ares control plane, whatever that means.

A

If you don't know what the cuban is, control plane is, doesn't really matter.

A

Look up in calais this tutorial. It will will tell you everything, so the master.

A

Machines are sort of not targeted at your workload like your micro services,.

A

But rather these master machines are only meant for hosting the control plane.

A

The kubernetes sort of backends itself and then the third type. Well, that's you.

A

That's you workloads and that's the workers and there is no more machine.

A

Profiles that we sort of envision in in the detect, tecktonik installer and like.

A

For me at least, there was one visible like one evident sort of like drawback.

A

With this solution, like I, don't have only one worker profile, I have probably.

A

Many right, depending on your workload, if you do machine learning any gpu workers.

A

You need, I don't know. If you have your services, you need more workers that do like io stuff, so the world is obviously a little bit more.

A

Complex, so you could classify those worker profiles into more, but I think.

A

Like to get the big picture running and honestly, the installer didn't know.

A

Multiple worker profiles- I don't know how it's just we don't still write, so we.

A

Only know one worker machine profit, so it's very simple.

A

That cd very very easy, essentially there's only one service running on the.

A

Hdd machines, namely the hdd database itself, as some of you know, hdd is like.

A

Sort of a little bit of a special database because it needs like a very.

A

Queer very strange number of machines to operate successfully and that's always.

A

Like two and minus one machines, which is strange, but it's heavily sort of.

A

Justified by the underlying algorithm, that is, that it is using, which is like a.

A

What we call the raft algorithm developed here, which accomplishes a.

A

Majority with a majority quorum of votes in in a given cluster, so you need like.

A

It's like a strange number of its see of those machines, only one three, five and.

A

And so and so forth, so these machines need to need to know each other. Okay.

A

So, in order to format you at cd cluster, let's say we have to spin up three at cd.

A

Machines, those three machines need to know their respective sort of ip.

A

Addresses they are sort of, they need to be able to resolve each other at cd.

A

Gives you three possibilities to accomplish that and I would like to.

A

Enumerate them, and also tell you about the solution that we went with and that.

A

Currently is still sort of the recommended way to bootstrap at cd.

A

Machines, the first possibility that city gives you. If you look up literally in.

A

This documentation link down here is, first of all,.

A

Discovery of other @cd machines via dns, most notably srv records, who has used a.

A

Servi records before okay cool, so you all know more or less what what it is.

A

Doing srv records for those of who don't know what is doing. Essentially it's a.

A

Special dns entry, which contains dns names plus ports, which is pretty cool.

A

Because when you hair, like in a service world, you can enumerate your concrete.

A

Service endpoints and that including their target ports and the only thing.

A

You need here is a dns server, which is pretty cool because dns service, like.

A

Like they're everywhere right, the second thing that you have at.

A

Your disposal is what we call the hcv discovery service. It's just still.

A

Operational the thing in production- I don't know.

A

Cool, so it's still valid to be on the list. So again those @ çd machines need.

A

To be able to discover themselves so, instead of using dns, this discovery.

A

Service literally uses http s, so you have some rest based endpoint, where you.

A

As an @ çd machine pose your ip address into and all the other seedy machines.

A

Post their ip address into and then under a certain url those set of ip.

A

Addresses are available very simple, semantics, very easy to understand. Then.

A

The third way well very easy if you instantiate your @cd machines and you.

A

Know the host names in forehand, like at cd, there's 0 1 2 dot by corp org.

A

That's good enough to instrument parameters at at cd to tell them where.

A

These guys are sitting next to each other, so pros and cons. Dns server based.

A

Discovery obviously the biggest pro is. You have only a dependency on dns, the.

A

Biggest condo and that's the reason why we did not go with the solution is.

A

Because a server-based discovery, you know, allows you only to bootstrap one.

A

Cluster per domain, which sort of sucks it could be easily fixed in their city.

A

Code base, I guess or you could you could have subdomains but exactly but.

A

Especially like because of this, and because of the induced sort of complexity, deployment complexity, we did not go with that solution. The second.

A

Thing is just the discovery service here you don't have the problem of you know.

A

Bootstrapping dns, first and setting up as or v records and all the you know.

A

Sort of jazz, it's sort of like is sucks from from a provisioning perspective a little bit, but you need a discovery service and.

A

Guess what who is hosting the discovery service? It's a cd, so you need at cdr to.

A

Bootstrap a cd and then who would strap that hcd that is bootstrapping at cd, so.

A

You see there is like a recursive problem right, so korres today, redhead.

A

Offers this as a service which is cool but not so cool if you're in a strictly.

A

Corporate environment with closed locked down networks and honestly, I don't want.

A

My internal hdd server's ip is to be you know, posted somewhere on some service in.

A

The united states, you know I work for a company right now, which is now heavily.

A

You know, like germany focused like this is a total no-go right.

A

If you host this at city, discovery service yourself, you're cool, but then.

A

Again, you need to bootstrap again a net cd. So that's what you try to accomplish.

A

Here in the first place, so it's sort of like problematic, so we went with this.

A

Very simple solution, and I think it's the easiest- you simply assume that those.

A

At cd machines have stable dns host names. That sounds easy, but on some true.

A

Pro cloud providers is actually a challenge, because not all cloud providers provide intrinsic sort of internal, stable host names, digitalocean.

A

Is one of them vite? Sure I think is another one of them, so this is this. May.

A

Sound like a low-hanging fruit, but if you're on a very restricted cloud.

A

Environment, that might even like, be a small of a hindered entry level here, but.

A

Nevertheless, since the tectonic installer- and I guess you all rather.

A

Deploy on like bigger club providers like aws, essaouira and so on, and so.

A

Forth, I think this is still the most sort of recommended way of bootstrapping.

A

At city machines, so how does this look like if you use sled car linux or korres.

A

Container linux, that's essentially all that you need to declare as a systemd.

A

Service unit, so every flat car linux, as well as each color, has container linux.

A

Ships, what we call the add cd wrapper or a cd member, we sometimes call it and.

A

Essentially, what you have to provide here is this endpoints thingy, and these.

A

Endpoints encodes the dns names from all the other ad city machines. That's it.

A

Boom done ready to go, and that's exactly I copied this today from the tectonic.

A

Installer code base it's open source by the way you grab it, while redhead did.

A

Not delete it yet, there's also a good recommendation and that's literally the.

A

The way we solved it, that's that's the whole system disservice that how we.

A

Deploy at city and tectonic questions so far.

A

Great, so let's come to the second profile of machines that we need the.

A

Second profile of machines that we need is what we call master machines that.

A

Host the communities control plane- and this is like the all the whole beauty of.

A

The community is the hard way thing right, so these master machines how's the.

A

Api server, the scheduler, the controller manager, obviously also the cubelet and.

A

The proxy, if you're hosting parts on those machines and additional control.

A

Plane daemons, whenever you need them on tectonic, for instance, we go all the way.

A

Self-Hosted, so we also are hosting flannel demons in like network demons in.

A

There, which are responsible for instantiated the the network over life.

A

Necessary for communities- and you know when you use self host, that it's better.

A

To have some recovery mechanisms when the whole control plane goes down so.

A

They are even additional daemons hosted on this monster machines, which we call.

A

The check pointers right for the master machines. You have essentially two.

A

Deployment methods of methodologies where all the rage out there is is about.

A

If the first one is self hosted- and I think cube atm- got support for that.

A

Recently, a couple of months ago right, it sounds like, and there is static.

A

Deployment question: who knows this distinction? Who knows what this actually.

A

Means okay, so it's just very few of you, so I will just shortly explain what.

A

Self-Hosted means question to you: what can you do with kubernetes hands up? Who.

A

Can answer the question: what can you do with kubernetes what what problem is communities trying to solve continue orchestration, so it can start containers.

A

Right, okay, cool! So imagine the api server is a container so who could.

A

Theoretically, start the api server docker and who tells docker to.

A

Start the api server, sorry who exactly the cubelet and kubernetes itself. So you.

A

Could use kubernetes itself to start itself sounds crazy right. Another sort.

A

Of alex paul, we had this stupid today, it's not stupid apps. I had this nice.

A

Saying crazy idea time and the self-hosted idea was exactly like this.

A

Is like a little bit recursive, we use cuban eddie's to start kubernetes. What.

A

We do, is, we start a super minimal control plane in one process and then we.

A

Deploy all these guys here up there as literally as containers, and then we shut.

A

Down this temporary mini communities and then the actual start communities takes.

A

Over administering itself, most notably the most known project for this herself.

A

Hosts itself is boot, cube check it out. It's pretty cool, it's by korres nahor.

A

Head head cube, edm also got support for self hosted if you're into that.

A

Semantics, that's the way to go another question towards you: why do you think.

A

Self hosted could be beneficial. Does anybody know the answer? Hands up right.

A

So you have somebody controlling that exactly what would be another reason.

A

Exactly another reason: the most important reason actually for us is.

A

Tectonic, I'm wondering who did update communities eventually in the past. Okay,.

A

So that's another cool thing that you can do with communities. You can do rolling upgrades and you know you kubernetes cluster. You cannot assume.

A

That you will run with one version like for five years. That's like illusionary,.

A

Right, so what we you can do, if you go the self hosted way you can use the.

A

Intrinsics of kubernetes, most notably deployments daemon, sets and rolling.

A

Upgrades to upgrade and and yeah and like bump, the versions of the control.

A

Plane containers themselves and it just works.

A

Sorry, let's have it: let's have a talk at a beer exactly sometimes. Obviously I.

A

Mean I totally agree. We also had troubles with that, but I can at least we.

A

Had a pretty good integration test suite around that right, yeah to answer these.

A

Questions, if you have more detailed questions like we can talk about check.

A

Pointers which help you in a disaster case when all the control plane is.

A

Totally down, but essentially, if you go with the self hosted way, essentially you.

A

Have this benefit of upgrading the control plane using intrinsic kubernetes.

A

Methods, obviously, you are also dependent on the scheduler and you have to trust.

A

Him that he's doing the right thing, which sometimes can fail. Obviously so if.

A

You're, like super cautious and are a little bit like, I wouldn't say afraid. I.

A

Would definitely go with the static way? What the static way does is exactly the.

A

Same that I showed you with that cd. You simply set up a system d service unit.

A

And declare the api server to start as a service unit, and that's it right so.

A

A

That's actually a very good point. So if you don't trust communities.

A

Itself, you rather should not use it at all right. Okay, questions around those.

A

Two semantics self-hosted versus static, or is it clear, okay.

A

If you're interested in two projects doing this sort of thing, you check it.

A

Out either boot cube, which is in the incubator of of the kubernetes.

A

Organization, that's one project accomplishing that thing and that's what we use that also what korres developed in the past and the other big project.

A

That sort of catches up slowly is adm with the current limitation, I think, to.

A

All be able to deploy only one master machine, so that's quite limiting right.

A

Now, okay, so there is one big limitation: when.

A

You go to the self-hosted way and learning from our side, because, with the tectonic installer, we went all self-hosted. Okay, the problem is the boot.

A

Cube can run only on at most one machine, that's sort of a limitation if you.

A

Deploy in a chase setup of the kubernetes control plane, you have three.

A

Master machines, for instance, boot cube, can only run on one of them, so you.

A

Cannot, like start boot, coop on all three masters and hope that this cluster will.

A

Eventually bootstrap the problem is, it can run only on my one machine and what.

A

We call this machine internally and the tectonic installer was the bootstrap.

A

Node, okay, so that's a big limitation and how do you actually solve the problem of.

A

Accomplishing that let's say you have three master machines dammit. I wanted to.

A

Ask the question to the audience, because it's a low-hanging fruit.

A

You have three master machines and you only want to run or let me go back. It.

A

Was just a teaser and you want to run a boot cube only on one of the machines.

A

Like very naively what what you do hands up? Okay,.

A

So what would you do hands up? Who knows the solution? Sorry, you dot! S3.

A

Okay, well, that's the another solution, that's cool! Who knows another one.

A

Intuitively exactly yeah exactly so once the master machines are up there.

A

Is nothing running yet on them very easy. Your first execute terraform, whatever.

A

Form counts. This accounts this internally is terraform apply. Then you.

A

Literally ssh into the machine and start the boot cube, systemd service unit and.

A

You decide as an administrator or as a devops engineer, which of those three.

A

Master machines is the boot strip node and call it a day. So that's possibility.

A

Number one: it's what we call the active sort of self holster deployment right.

A

Because it is done proactively from the sort of machine that you are.

A

Provisioning from or second solution is what we.

A

Called passive, let the master machines decide themselves who is going to.

A

Bootstrap, the cluster and that's very interesting and that's the s3 solution.

A

And what we accomplished that by doing what we call a poor man's master.

A

Election algorithm, okay and the algorithm is very simple in is, as.

A

Follows get my machine id, so every cloud provider has some notion of ids, which.

A

Get assigned to machines and you can usually query them via metadata via some.

A

Metadata service so like get my id transform it into an integer which is.

A

Sortable then get all the other machine ids, which have the master sort of label.

A

Attached to them, so I have all the ids of all. The ema are the master machines.

A

And then figure out on the machine that I'm currently on is my id. The minimum of.

A

All available master machine ids. If so, I am the bootstrap machine else. I skipper.

A

Makes sense: okay,.

A

One objection to this: you need like a two step process for.

A

This right, so that's the that's disadvantage of this of this approach. I.

A

So remember our discussion on this: the beauty, not the beauty, but the advantage.

A

Of this algorithm is that you simply instantiate like launch three machines.

A

And call it a day, whereas here you need two runs of terraform. The first run.

A

Which first would the first master machine and then the.

A

Second terraform wrong, which would strip the other ones: okay,.

A

So, let's be very okay, I simply give you the metadata service.

A

So usually, when you launch virtual machines on aws.

A

You can attach labels to them, and then you can view the metadata service.

A

Exactly that's! That's like the implementation detail of it. It's literally for aws. It's like describe the I to auto scaling group of the master.

A

Profile then you get the set of all machine, ids right and then you know your.

A

Own mesh machinery and you do the algorithm two images dance exactly.

A

Access, it is specific to the cloud provider. Yes, so we deploy this master.

A

Election script algorithm to each cloud provider- redundantly. Yes, exactly! That's.

A

One disadvantage because for this cloud providers don't give you any abstraction.

A

Cool any question to this self-hosted: oh and etsy. You also have a new id. They.

A

Only have the problem of how do you propagate the you ids of the other.

A

Machines to all the other machines, so that's the only problem you have to.

A

Overcome there, if you accomplish that problem, then you can do the same dance.

A

Obviously make sense exactly answer: question: okay,.

A

Any more questions, okay, cool, so workers, not much to say here the only.

A

Two things you have to launch on these profiles is the couplet and the actual.

A

Proxy, if you don't know what that means, please look up in the kuban. It is the.

A

Hard way, cal c does a great job of explaining the semantics of this and.

A

Obviously, other control playing demons like flannel or whatever is necessary.

A

Just rip your network awfully so silly example. Now how hcl looks like okay and.

A

This is actually like for those who don't know terraform also a good example.

A

On the icl plotter of it, so you see like this whole sort of machine and I copied.

A

This over from the actual install tectonic installer code base, you install.

A

A virtual machine called master, you encode some custom data into it, and this.

A

Is an example of the icl of the interpolation language, which is.

A

Imperative, because base64 encode is a function implementing a goal by the way.

A

Which you simply invoke, and then the result of the of the function is being.

A

Injected into this custom data of the operating system profile- and this is.

A

Also, examples of icl and simply injecting certain variables from the.

A

Outside same thing, oh, this is viper. This is my private project, because I just.

A

Picked a stupid stupidly super cheap random provider where I wanted to host.

A

My play kubernetes own, but essentially it's all looking the very same thing.

A

This is how it looks out on aws the biggest pro on aws compared to the other.

A

Providers is that you have this notion of auto scaling groups, so we can spin up.

A

And spin down automatically master nodes and or workloads depending on your.

A

Workload and everything just works, azura has this too, but it had a huge.

A

Limitation regarding storage- I don't know, did they overcome that already or.

A

We did, we didn't catch up exactly cool, so this is essentially how it looks like.

A

In terraform, so nothing really magic in here. So now, how do you tell? How do you.

A

Tell aws that this guy here is a master machine, and how do you tell this guy.

A

That it has to launch the following system, these service units right.

A

You need some configuration abstraction for that. You need to tell the lcd.

A

Machines that they have to launch at cd servers at cd demons. You have to tell.

A

The master machines that they have to launch api service or whatnot right so.

A

You need to somehow declare the system these service units that you want to.

A

Launch on korres container linux or flat call linux.

A

And caresses answer to that problem is ignition and before I delve in like dive.

A

Into what ignition is, I asked the question who knows cloud in it or has.

A

Ever heard of clothing it before okay, so ignition is essentially the same.

A

Thing but better, okay, and the reason why why it is better, at least an in the.

A

Opinion of korres is because it runs way earlier than then cloud in it cloud and.

A

It has the biggest drawback that it is run after the machine has booted into.

A

The root filesystem into the target root filesystem right. So it's when you, when.

A

You launch up dbn with clothing is something sometimes I do these things to.

A

Automate stuff, you see, you can do like inspect the cloud and it locks. You know.

A

And voila clothes in it block or something like this, but for many many.

A

Operations, this is way way too way too late. Imagine you have to format your.

A

Route hot route partition: how do you do that with cloud in it if.

A

The system is already booted the problem right so and how do you, for instance,.

A

Instrument, lower-level intrinsics, like network cards, or something like this.

A

Before the actual root filesystem before the actual network cards are being.

A

Initialized by system d, so cloud in it is way too late. In this whole process,.

A

And ignition rewind sort of the semantics idea and has been executed.

A

Actually, in an environment which we call init or d, everybody who knows in it rd, I.

A

Want to see all hands up here, okay, great, so I will ask the experts yeah go what.

A

Is in it or d exactly, and you can actually run things there right. You can.

A

Run programs in it or d right exactly. You can even run light system d in it.

A

And actually it runs and you do right so you have like a mini mini system before.

A

Your actual target system runs where you can run real programs before the actual.

A

Root filesystem is running and we wrote a small program called ignition and go.

A

Which is run exactly in that environment right and the cool thing is that the.

A

Init or d is being executed before the root file system is mounted, so you can.

A

Do all sorts of crazy stuff in there this.

A

Is how ignition looks like it is jason right, and this is the buff isn't as an.

A

Example of how you would declare a system deservers unit inside ignition.

A

And lid looks like pretty you know ugly, because jason exactly so, please please.

A

Don't write this by hand rather use the thing below which is sort of like a.

A

Llamo representation of the above and then you can use what we call the.

A

Continue next config transpiler to compile this llamo into json, ok and then.

A

Down here you can declare your system d service units, but there is more because.

A

I told you there is terraform, so ignition tester form equals laugh. Okay,.

A

Which is pretty cool because terraform understands ignition natively the same.

A

Way, you declare your virtual machines, which I just showed you a couple of.

A

Slides ago, this is the way you can declare ignition profiles so inside.

A

Terraform, you can declare the system d service. What is it? Ok, I'll ten-minute.

A

Break soon, ok, cool you can and that's perfect, because we will come to my end.

A

Of the first half and then the second half will be taken over by our alex. So.

A

You can declare inside terraform all the system, this service unit that have to be.

A

Deployed via ignition, ok and it's all supported, natively via terraform and.

A

That's pretty cool what what the above does! Actually, when you execute terraform.

A

Apply, it simply generates this ignition jason today that I just showed you.

A

Before and we use the super heavily inside the tectonic installer to sort of.

A

Like declare all the services that we need that have to be run on those.

A

Different kinds of profiles, so, if you look at this, we have an ignition.

A

Declaration for masters for edd and working machines, they are all different.

A

And then they are being provisioned to the actual machines. So the question is.

A

And that's maybe again another riddler question to you once I have this jason generate it. How do I transfer this jason to the actual machine that has to be.

A

Booted riddle who knows they're over who can sense the answer: what who, what.

A

Use the data exactly so: what is user data? Can you explain to the others.

A

Exactly so, it's it's like a storage thing right that you can query via our.

A

And http endpoint: yes, exactly yeah exactly so it's it's something that you.

A

Can put on your blog called provider like a huge string like the ignition.

A

Json string and then when the machine is booted, sorry I will come. I will come to.

A

That I will come to that. Don't worry the! I still have a couple of minutes left so.

A

You can post this json, whatever the result of that terraform apply run is to.

A

To your cloud provider, you can post it like to an rest endpoint and when the machine boots, it can curl it again from this user data. Endpoint right and it's.

A

Not protected by any means it's just curl http, some magic ip number and then.

A

You curl the user data down and then ignition curls it down.

A

Execute the operations that you declared here and does its thing.

A

Problems and that's where we, where we can unlearn x, okay, so ignition json, is.

A

Distributed via cloud user data, which is awesome, but there are concerns first of.

A

All the biggest concern that we had and still I think, are scratching our heads.

A

Against security, because imagine you will also most likely distributed things.

A

Like private keys for tls for web servers and stuff like that, right so and.

A

You will most likely also distributed like secrets via this user data, and this.

A

Is honestly what we do so you must treat user data like a secret and that's a.

A

Very important message for you and that's sort of like, in my opinion, the.

A

Biggest limitation, also on this, we treat user data as a secure sort of.

A

Environment, so you must do everything to protect your communities parts not to.

A

Break out of that environment and to be able to pull stuff from user data right.

A

We consider user data, a secret which is, in my humble opinion, the biggest.

A

Weakness in this model, second concern is immutability. Once you declared user data.

A

You can never ever change it again that well, you can, theoretically on some cloud.

A

Providers, but on some on most providers you can't and that's also put.

A

A problem right: what, if you're like sort of ignition profile changes over time.

A

Like so this, for that, you would have to instantiate another machine profile.

A

Another big problem is size constraints. I have no freaking idea why aws has a.

A

Limit of 16 kilobytes in the year of 2018 on user data, sorry yeah! It should.

A

Be exactly right so come on. I literally have no idea but, like I was.

A

Fire, you don't want to know what tricks we apply it like to overcome this. This.

A

Problem like including gzipping and like insanity, like, I, don't want to tell you.

A

So sometimes you will have to find ways to distribute user data via other means.

A

Then yeah use the user data endpoint. For instance, you mentioned it already. You.

A

Spoiled it, you can use s3 or other sort of object, storage mechanisms that the.

A

Cloud providers give you that's one thing: the cool thing about the semantics.

A

Is that you can change it, it's not immutable right. You can't simply replace.

A

An s3 object and optionally: you can delete the user data after installation.

A

For whatever reason, for security reasons, for instance, and then it is not.

A

Available anymore, downside is, when you delete it, you cannot boot a new machine.

A

But you know you will have to solve the problem somehow, if you want to see.

A

Examples of this again like at the end of the session, we will show you pointers.

A

To the tectonic installer, it's still open source. Hopefully it will stay open.

A

Source, how are all of this is accomplished, and this is essentially how.

A

We solve the problem on the aws side of things. We moved out ignition.

A

Totally this big json into an s3 object and host from there, and that's it like.

A

This is the storage, and that concludes my last slide, because the next thing is.

A

Networks and then you know, for the storage part, there is not really much to.

A

Say I showed you already, some of those sins for distributing your sort of.

A

Machine profile, user data via means of object, storage right, you can use object.

A

Storage for distributing ammunition, data and distribution of deployment, as it's.

A

Like systemd service units and then obviously block storage, that's sort of.

A

Like the ebs stuff that you can mount into your machines here, beware.

A

Of support inside kubernetes cloud provider, support and bugs and.

A

Vulnerabilities because we some saw some if you stuff in azure, I need something.

A

To drink, I hope you liked the first part, let's have a break and then alex will.

A

Talk about networks.

A

Hi guys, like sergei said, I'm alex no more introductions. Just the huge.

B

Thanks to suresh for inviting me here to do this walk with him. This was all his.

B

Idea, yeah, so the maybe the most important yeah, not.

B

Necessarily the most important, but one of the biggest differences that you see.

B

Between different deployments on different platforms for kubernetes is.

B

How you handle the networking right, kubernetes does a great job of providing.

B

You an abstraction for your workloads in terms of networking, so there is this.

B

Thing called the pod level network so that your application actually sees the.

B

Same networking, environment, no matter where kubernetes runs on, but that's not.

B

Necessarily the case when you actually try to deploy kubernetes, you have to.

B

Implement the the glue there, so that's what I'm gonna try to go over, but first.

B

Of all, let's get a little bit some some concepts out of the way kubernetes.

B

Realized heavily on tls for securing the network traffic, and this is important.

B

Not necessarily from I mean not not just from the point of view of secure traffic.

B

But it has a lot of implications in how you provision your networking and some.

B

Auxiliaries around that, because tls in its authentication form so client.

B

Certificates relies heavily on, I mean you, do have to have an identity to.

B

Present that the certificate can be can be validated against right now. The.

B

Identity is most of the time based on the fqdn of the machine that is making.

B

The call right on the host name together with the domain, so that immediately translates into the fact that you need to have dns.

B

Your environment, where you are trying to use tls right tls, is like there's no in.

B

There's no hard connection between tls and and and having a working dns setup.

B

But it really does make little sense to try to use tls and authentication so tls.

B

Client certificates without having control of your dns environment, because.

B

Your alternative, if you don't have dns, is to use ip addresses as identities or.

B

Something else right ip addresses are not pinned to the machine lifetime. They.

B

Can change over the machine lifetime depending on? Are your dhcp, the http.

B

Setup behaves and so on and so on. So your best bet for providing identity in.

B

In a tls authentication scheme is fqdn stable fqdn, so you need dns working and.

B

You, more importantly, you need to have control over the dns control plane. So if.

B

You have if you're trying to deploy kubernetes but someone else in your.

B

Organization owns the dns servers and you don't have good communications or.

B

Api is there that can turn into a very messy setup right, because you're going.

B

To be going back and forth with them to provision the the host names and the and.

B

The identities for your machines that you need and if that's not automated,.

B

It's like it's. The whole installation process becomes a mess. Okay. So what kind.

B

Of tls materials right so there's a.

B

Couple of different there's actually two sides to tls: there's tls server.

B

Certificates which encrypt your traffic, that's their only job right and provide.

B

An identity to the clients as to whether the machine that they're talking to is.

B

The real thing that they're trying to talk to and then there's tls client.

B

Certificates, tls client certificates act as authentication credentials for the.

B

Client towards the server so they're, basically they're working in.

B

Opposite directions: you need both in kubernetes and the way you their life.

B

Cycles are different and the way you provision them may or may not be the.

B

Same so in at cds case, which is illustrated here, you need client.

B

Certificates for the add cd clients talking to the add cd cluster, so that.

B

Would be your kubernetes components like the api server and sometimes.

B

Historically, for example, flannel would talk to you directly, but that's no.

B

Longer the case, but there whatever is a client where td needs to present an in.

B

Tls client certificate, which is the identity of the client towards the.

B

Server right, then, you need peer certificates, peer.

B

Certificates are used by the various machines that make up the at cd cluster.

B

Not the kubernetes cluster, but just the sed cluster to authenticate and encrypt.

B

The client, the traffic between themselves, right because the ativy being.

B

A distributed system, it obviously runs on more than one machine and it needs to.

B

Trust the other machines that are part of the cluster, so that's how it, how it.

B

Does it and then there's a td server certs, which basically tell your ad cd.

B

Clients that the machine they're talking to is actually part of the ad city.

B

Cluster they're expecting it to be, you can skip some of them all of them, but.

B

Ideally, you do the complete set up and the cert, and ideally you have a.

B

Dedicated certificate authority for at cd because at cd is or should be, a.

B

Different level of criticality than your control plane is. This is basically where.

B

All of your state stays so if the control plane machines themselves are.

B

Compromised, you can rebuild them and not lose state.

B

But if these machines are compromised, you cannot easily rebuild them. Alright,.

B

So for the control plane this this is the client to kubernetes right. You.

B

Ideally have yet certificate authority because, like I said they have different life cycles and different lifetimes, and we will talk.

B

Later about tls rotation, so that kind of makes sense when you think about this.

B

Certificate has a specific life span and how do I change to a new one in the in.

B

The sense of ever running cluster, so if you have an intermediate ca dedicated to.

B

A specific, let's say, life cycle domain of these certificates. Then it becomes.

B

Easier to rotate them and also, if they're compromised, you can invalidate.

B

The ca and then all of the compromised certs are no longer valid and you can.

B

You can distribute new ones so who actually uses certificates in the.

B

Control plane, it's the api server. Mostly, it does use a couple of certificates.

B

Which we haven't explicitly listed here so, first of all, it's a server so it'll.

B

It'll have a tls certificate for so a tls server certificate for providing.

B

Identity guarantees to its clients, so the clients will know that they're.

B

Talking to the actual api server, not a man in the middle thing, then it does.

B

Have it does make use of the sed client certificates that I talked about earlier.

B

To talk to at cd as a client, so you will have to distribute the sed clients to.

B

The control plane machines right and then, depending whether you're, using.

B

Aggregated api servers there's again a different set of certificates for that.

B

Because the aggregated api server are similar long processes, okay,.

B

This is for the internal. What we've been talking about until now is for the.

B

Internal kitchen of the of the control plane now you do have to have access to.

B

Your cluster from the outside, because otherwise it makes little sense to run.

B

It right so somebody needs to somebody.

B

Needs to call into your services or your website or what are your whatever you're.

B

Running on the cluster right and this somebody is your random client from the.

B

Internet, so you do have to use certificates that are rooted on a public.

B

Certificate authority, as opposed to the other ones, where all of the certificate.

B

Authorities that I mentioned earlier can be private ones issued by yourself or by.

B

Your let's say organization, so for ingress you do need to have an oracle.

B

Certificate, like you, would be using for an http server, which is signed by a.

B

Public certificate authority, http right, okay,.

B

So I've been saying issued a couple of times now and I mentioned that. There's.

B

More than one issuing authority involved for the public one, it's pretty obvious.

B

You use whatever you've been using for your web server ever since. Just get.

B

Public certificates from your favorite certificate authority for in cluster.

B

Communication, so the hcde certificates and the api certificates. You have.

B

Options, you can still use public certificates, but they cost money and.

B

It's not mandatory that they're public, because nothing public should actually.

B

Be talking to your lcd cluster, whoever talks to your lcd cluster should be well.

B

Defined within your organization, so you have control above what certificate.

B

Authorities, it knows about so those can be self-signed certificates, but not self.

B

Signed in the sense I'm just gonna like make up a ca now on my machine and sign.

B

With it, you need to have a process of distributed, distributing internal.

B

Certificate authorities within your organization for that to make sense and.

B

Somebody has to own the the security of those cas.

B

Right, ideally, it's not you and somebody else which deals with security, as I.

B

Mentioned earlier, this makes little sense. Not yet we do have a vault.

B

Operator, but it's not part of the installation process is just for client.

B

Applications that could be yeah. It could be because of the self hosted.

B

Bootstrapping process, we can't run vault that early because it's being managed by.

B

An operator so we would have to like make it part of the bootstrapping.

B

Process itself- and that was like too much complexity to handle, but that.

B

Actually is a valid option if somebody else runs well for you, okay, so yeah use.

B

Dns and you make use of the fkd ends as your identities, not ips terraform for.

B

Convenience has resource providers that can run their certificates for you or.

B

You could use your favorite certificate generation tool and just import them as.

B

Files in the tecktonik installer, which is our reference implementation in this.

B

Talk we use both so by default. If you don't supply any certificates, they all.

B

Get generated during the installation process, you have the option of providing.

B

Just the ca, or you can say I have somebody issuing certificates for me so.

B

Here's a bunch of certificates, use them and obviously you have to specify which.

B

Is which this is all nice? As you install the cluster, but certificates have a.

B

Limited validity period, or ideally, they should have so when that validity period.

B

Is over or ideally before it's over, you have to replace them with a new one.

B

Now this is basically a much more complicated problem than.

B

It sounds on the surface because you also want to have your your control.

B

Plane, your kubernetes cluster be running and maintaining work like driving your.

B

Workloads, while you do the rotation, so we don't have a recommended solution for.

B

That, in the current incarnation of the installer, it's being developed for the.

B

Next version of it, and but basically the idea there is that instead of.

B

Distributing them as individual assets, what we will do is we will just.

B

Reconfigure the whole instance or replace it with an updated with an.

B

Updated configuration of it, which will in turn, include the the new certificates.

A

Request from some customer yeah and like we did a very silly solution.

A

That was like to simply make it the validity from one year two years and we.

A

Are proud of that at all, there was like the the stupid, not the stupidest, but.

A

Like the most simple request that I ever approved- and I was really ashamed of- but.

A

It's a really really hard problem to solve in this sort of setup right, and I.

A

Think this is currently no solution out there. No tectonic, no typhoon, no cupid en.

A

Is currently a good solution for the certificate rotation problem for.

B

Intrinsic cluster certificates, actually, the problem lies more in in trying to.

B

Make it generic or cross-platform or cross environment so coming up with a.

B

Universal solution, but what chris just said is actually your best bet right now.

B

Use volt or some other distribution mechanism volt actually is pretty mature.

B

In that regard, and has quite some nice distribution feature, so you can have.

B

Let's say virtual volumes: moated mounted in your filesystem that carry the.

B

Materials, the secret materials that vault managers, so you just let say.

B

Expect the files to be at a certain path within your file.

B

System- and they will be there because you're running the vault distribution.

B

Mechanism or you can have clients call into specific api's, but that actually.

B

Means your your application is tied into the distribution mechanism for legacy.

B

Applications or if you want to decouple them, I think distributing them as files in let's say a mounted volume that.

B

Something else manages, namely vault, is actually a nice mechanism. This is I'm.

B

Recommending that, because I've seen it work really nicely internally at amazon,.

B

Where we had something very similar to what vault does with these volumes, where.

B

A daemon, a dedicated daemon on each machine would take care of mounting a.

B

Location on the filesystem and dropping certificates that were allocated to that.

B

Machine there and that worked well for for the whole organization. So it's kind.

B

Of the same idea, it works at amazon. Most likely will work for you too, but you.

B

Knew you do need to run vault and manage it and for bootstrapping, both vault.

B

Needs to be available before that. Ok cool. So that's that's tls, but we are.

B

Actually here to talk about network, so these are a bunch of things that you.

B

Need to reason with, when you're choosing your networking layout for kubernetes, so.

B

Basically, you are most likely gonna be running on a pass system like aws like a.

B

Sure, like openstack or you would be running on bare metal where a bunch of.

B

These things go away and other problems come to replace them. So let's talk about.

B

The past situation, first right, the the past, usually provides you with those.

B

Primitives out of the box, so routing is taking care of load balancing is taking.

B

Care of, unless you're on a very simplistic path,.

B

Firewall or access control is usually taken care of in dns, most likely is also.

B

Taken care of in the sense that you have api is to manage them right. The most.

B

Important thing to know to to be aware of actually, when you're running on a.

B

Cloud provider is the the actual base network of your provider is not a.

B

Physical network, like what you're seeing there is a network, may actually be an.

B

Sdm, a software-defined network that is managed under the hood by the platform.

B

Provider, so do not assume that your that your vms are directly tied to layer 2 in.

B

In their in their host system right, the whole system itself may actually be on a.

B

Software-Defined network that has implications when you layer it with the.

B

Other abstractions that kubernetes comes with, namely the overlay network for pods.

B

And services and we'll see how- but this is really important- do take a look at.

B

What the md recommended them to use our in your cloud provider, documentation.

B

Because if you exceed those funny, things start happening because of the fact that.

B

That's a that's an sdn under the hood we've we've been bitten hard by that.

B

Ornatrix specifically, so what kind of networks do you have to reason with when.

B

You're installing kubernetes first there's the usual machine-to-machine.

B

Network that makes everything possible. This, like I mentioned, is not always a.

B

Level two at at your vm level right so do not do not assume that unfortunately,.

B

A doublet doesn't document explicitly what the implementation for.

B

The vpc is, but I can tell you from from seeing it from the other side.

B

It's actually encapsulated in udp and that's how it transports so everything.

B

That goes into your network interface in in the vm is transported via udp and.

B

Funny things happen like arp calls are not actually rp calls beyond the vm.

B

Level they're actually http api calls to control a so do not assume those kind of.

B

Things right because you don't know how they're implemented under the hood. I.

B

Don't know how a sure does it, but they may be doing just as crazy stuff right.

B

Then there's the pot to pot level. This is the point where you start controlling.

B

Things so port to port network is really nicely.

B

Explained at that at that page on the kubernetes documentation website.

B

Basically, they have a couple of assumptions there that you need to.

B

Reason with first of all, each pod needs to be or kubernetes guarantees that each.

B

Pod is addressable by every other part in the system by viable same ip address.

B

And this is the same ip address that the application sees for itself when it.

B

Looks at at its network interface, there is no netting or kubernetes guarantees.

B

That there is no net in-between like in the on the way between two pots. So it's.

B

A flat it's a flat network there there is not so much magic involved in.

B

Implementing port to pod networking, it's actually just a very clever use of.

B

Routing and ip tables, but there's no encapsulation or at least not if you're.

B

If you're, not using vx lan vx lan is a different beast. Tecktonik.

B

Actually runs with vx lan by default for uniformity reasons, because we did not.

B

Want to support the current implementations for different.

B

Cloud providers, but the exelon is, let's say the most generic and most simplistic.

B

Abstraction you can do better than that. So, if you're, if you're interested in any.

B

Kind of performance overlay via vx lan is not what you want to do, which sort of.

A

Was initiated by former engineer at korres yeah? Maybe you was smelling the.

A

Typhoon, which is a nice performance comparison on those different, the.

A

Calculation, yeah techniques- if you want to go with the low-hanging fruit, we.

A

Explained is sort of the easiest because, because it runs over udp yeah, it is very.

A

Easy to configure it very easy to get started and that's the reason why we.

A

Choose it as a default pass I like to mention it's not yeah. It encapsulates as.

B

Your traffic leaves the kubernetes node and then that gets again encapsulated by.

B

Your top provider so yeah, your performance, your mileage, may vary. Then.

B

The third network, you have to reason with the service network service network.

B

Is actually a bunch of virtual ip addresses that are not actually attached.

B

To any interface, they're rather resolved through ip tables and they're being.

B

Trafficked these are pieces being forwarded to the appropriate pods that.

B

They front so that they behave like a load balancer, but they are implemented.

B

Just as as targets for some ip table rules that they don't actually exist,.

B

Attached to add to an interface, these are relevant because if you start.

B

Looking for them, you won't find them anywhere, but in ip tables configuration.

B

It's it's actually, a very clever mechanism for.

B

Very, let's say: lien load balancing scheme within the cluster, so the minute.

B

Traffic leaves your pod and is addressed one of these service ips. It immediately.

B

Gets translated to the randomly to one of the ip addresses of the service of.

B

The pods fronting this or backing this service, so you actually talk to the.

B

Application behind that service, with just the simple ip tables on.

B

Transformation.

B

Ingress as in incoming traffic to your cluster, not the ingress.

B

Controller is actually two ways: there is traffic to your workload. Applications.

B

And there is traffic to your control plane, they are actually handled by.

B

Different components: the api can be externally exposed if you want to manage.

B

Your cluster, let's say from a from a different network than the one. It's.

B

Running in ideally, this is not exposing directly the master machines, but rather.

B

A load balancer in front of them and then the other kind of traffic is normal.

B

Workload, traffic, which kubernetes calls ingress whenever it's being handled by.

B

The ingress controller, which is like layer, 7, reverse proxy thing that you can.

B

Run to distribute workload to certain services or you can actually access the.

B

Services directly, so if you're running an api on the cluster, you don't have to.

B

Run ingres controller necessarily, you can directly expose the service as in number three service to your clients and, as he.

B

Mentioned the iptables is just the epital. Implementation is just one of the.

B

Options- and it's actually just used inside the cluster whenever you need to.

B

Take traffic from the outside. You have options, one of it is actually spinning.

B

Up native load balancers in your in your cloud provider, acceptance of a load.

B

Balancer so kubernetes for each service that you deem externally accessible or.

B

More specifically, you assign a type of load balancer to it. It'll spin up an.

B

Elastic load, balancer or an aws, or a load balancer on azure or whatever.

B

Concept applies in the platform that you run on and which kubernetes supports. I.

B

Think right now it's just as your aws and gcp that have implementations in.

B

Kubernetes, but you so the important thing to take home here is that if.

B

You're running api's or services that are consumed by other programmatic.

B

Clients, you don't necessarily need to use inggris, you can directly expose them.

B

And obviously you have to take care of securing them at the application level.

B

Okay, so at the cluster level, what is your, what is it, what are some typical.

B

Layouts for the network between your nodes, the most simplistic one which is.

B

Usually, what happens if you use one of these demo tools like cube a table as.

B

For example, or something that is very simplistic, is you get a bunch of.

B

Machines spun up that all get public ips, so essentially, your cluster is entirely.

B

Public with all the implications that come with that, I'm not going to.

B

Like insist too much, I mean because I think it's obvious for everyone that that's not good. This is just for demos right. So if it's.

B

Easy to spin up a couple of public instances or like assign them public ip.

B

Addresses you may be able to demo your application running on kubernetes like that, but actually wouldn't consider that not even for staging in an actual in an.

B

Actual environment, where you develop for obvious reasons, right, there's no like.

B

All traffic between nodes goes through the public internet.

B

You have to really work hard to secure the each and every node they may not.

B

Even be concerned with outside traffic by themselves, but you still need to.

B

Secure them because they're exposed right so don't do that. The other option.

B

Is to have a private network which sounds, let's say trivial, but depending.

B

On the platform that you run on, may not be as obvious as it seems so, for example,.

B

The v pc in aws is obviously exactly what they call it. A virtual private.

B

Network, like, I said, don't assume implementation. So this is not.

B

Implemented, let's say like you would expect in a normal data center or.

B

Scenario like, for example, like a villa- and this is it's not a villain, it's just.

B

A couple of assumptions done at the routing level in the coop in the aws.

B

Networking control plane, but all the traffic goes through the same channels.

B

Right there is no physical separation between that traffic and another traffic.

B

From another v pc, just it's being handled through routing rules, so even if.

B

It says virtual private cloud. It actually goes through the same channels.

B

As other traffic, unless you use specific settings which are like there are some.

B

Advanced features in v, pc where you can sort of isolate yourself from.

B

Neighboring traffic, but that that's actually significant.

B

On top of your instance, if you're in your own bare metal environment, that's.

B

Obviously, a physical network, which you can reason it reason with like you.

B

Normally do with with any kind of physical network nodes, get assigned.

B

Private ip addresses in vpc- that's pretty much all you get so I'm not sure.

B

How many people have been surprised by the fact that they have a public, ip or.

B

Like an elastic ip and actually doesn't show up in ip address list, have you seen.

B

That have you noticed that that you actually don't see your public ip.

B

Address on the instance itself, can you imagine why.

B

Exactly it's handled at the border, so it's it's a nat feature actually so in your in most private private network, like or virtual private.

B

Network environments: this applies to azure as well and sync to neutron on.

B

Openstack, you don't actually get to assign public ip addresses to those.

B

Instances they are handled through border routers that knapp those ip.

B

Addresses to the specific instance that they're assigned to so bear in mind when.

B

You when you try to deploy our application, for example, this applies when say you deploy a web server that tries to do like ip based rules, and you.

B

Expect that the ip address that it's called with will be available on the.

B

Machine, that's not the case. The most important thing is that the traffic is.

B

Actually isolated within the network, if you take the appropriate precautions.

B

That I mentioned so in a ss by default, it's not, but you can use the dedicated.

B

Networking, I think they call them, they call it an enhanced networking or.

B

Something yeah there's a setting on the instances that you can enable and then.

B

Basically, you yet separate your your network, your.

B

Virtual network interfaces on the machines get attached to physical.

B

Network interfaces in the in the dom0 and then in the host machine and that's.

B

A different level of isolation that your normal vp see like I mentioned earlier.

B

Outside traffic only happens through not or load balancing you, you shouldn't.

B

Expect on any cloud provider to to see like actual ip traffic coming from the.

B

Outside directly to your instance, even if it has a public ip address, separation.

A

Of private and public subnets like new clusters, that's very encouraging to see.

A

Because, like when, you see those those tools, it's like who strapped your.

A

Cluster in five minutes, your kubernetes cluster actually have make the illusion.

A

Towards you that, like they do all the things correctly yeah, but I would like.

A

To emphasize what alex just said just be very, very careful if you do use like.

A

Those demo like tools, but they actually don't care they just care about having a.

A

Working kubernetes cluster, they don't care about strict network separation and those security implications that they have, and this is like the important.

A

Message that we were like to transport be very aware and and productive setup.

A

That you have to separate those private and public subnets and even if the cloud.

A

Provider gives the illusion of having a separation yeah like be aware that this.

A

Is just a virtual illusion? It's really. There is no real hardware. Separation.

A

Except like for this exactly and and do research the documentation as the.

B

Network implementation, that specific provider- because there may be things in.

B

There that you did not expect so obviously, most cloud providers have this.

B

Feature in some shape and form aw says the v pc. As I mentioned, there's.

B

As your virtual network, which pretty much works in in the same way, openstack.

B

Does this concept through neutron, they also have a more simple.

B

A networking stack that I think what is it called the public one. Don't doesn't.

B

Come to me, yeah nova, exactly a nova is basically it's. It's not isolate its non.

B

Isolating so it's a flat network environment where you don't have the the.

B

Virtual network concept, everything has ip addresses from the.

B

Same range, and they can all see their traffic and again do not assume the.

B

Implementation below is layer two. I do not assume that you have the this direct.

B

Physical layer under the vm, this is an illustration of what let's say a.

B

Simplistic one of those simplistic tools, so those tools that circa sh warned you.

B

About they do this right, they may spin up a private subnet for, for some.

B

Machines, they usually put the control plane in what they call a public subnet.

B

For easy, accessibility from the outside, they usually assign public ip addresses.

B

To the to the master nodes themselves, you don't have to do that. You shouldn't.

B

The master nodes, but at the very most, should only allow you to access the api.

B

Server, you can do that with a load balancer, which is more easily secured.

B

Any you have the ability to control which services are exposed through a.

B

Little bit, sir, so don't assign public ip addresses to masters. If you have load.

B

Balancers at hand just put them in in a private network right, installer.

A

Yeah yeah, so this is what we started off and then we realized that this is.

B

Actually, we can do better than this. This is what a friend of ours, johannes.

B

Simcha came up with for one of his kubernetes deployments, and this actually.

B

Better illustrates the fact that, if you, if you try harder, you can actually like.

B

Remove all your kubernetes components from the public side of the network so.

B

Those those squares up there are the public subnets, the ones that actually.

B

Can take traffic from the outside? You don't see any kubernetes components.

B

Being deployed there, so everything is being either netted or load balanced.

B

Into the into the private side of the network to basically this is the better.

B

Approach yeah also, you can.

B

Make some some very, let's say, common sense analogies if you work in a reasonably security concerned organization, you most.

B

Likely have to vpn into your organization's network to access any.

B

Kind of internal tools right, so you won't directly even if they're.

B

Authenticated and authorized you don't have them exposed through the internet.

B

Why would you do that with your kubernetes control, plane right.

B

Don't put it don't even put it behind a public load balancer. If you have the.

B

Option to to vpn and and access it like that right, if, obviously, security is any.

B

That's a reasonable concern. There there's no need.

B

I mean it's convenient to expose it and it's it's authenticated and authorized.

B

Via tls, which is a fairly strong authentication mechanism, but still like.

B

If you don't trust your other network services and your other network tools or.

B

Your organization's tools to be directly exposed, don't trust the kubernetes api.

B

To live on to to be exposed to the outside, and just don't do that, okay, so I.

B

Mentioned that dns is actually a pretty critical component in the whole scheme.

B

For, if only just for the reason of implementing solid tls schemes right so.

B

The kind of dns resources that you need usually are an api server endpoint, which.

B

Actually should be the first one, because it's the only thing that is really.

B

Really mandatory because you want to secure it with certificates and.

B

Certificates need the stronger identity of an fqdn right also edd can benefit.

B

Like serge explained from dns srv discovery, and not just the srv record,.

B

But you do need to have stable host names if you're using discovery via dns.

B

So you need tns entries for @cd. You need dns entries for master nodes if you're.

B

Gonna access them directly in any way or if you're gonna have tls between your.

B

Load balancer and the master node you do need dns records for worker nodes. If.

B

Again, you're going to have tls between the actual service load, balancers or the ingress and the nodes itself, but not.

B

Neither the master nor the worker actually require you to to have dns.

B

Records for them it's just if you want to have a a solid security scheme by a.

B

Tls you you want to issue certificates to them, then you need records for them.

B

Or if you wanna go ahead, I think we played around with it at some point, but.

B

We never actually made it into a feature, I think mostly because there was not.

B

Enough expertise, iran, dns sec, yeah, some some colleagues of ours did some some.

B

Proof-Of-Concept things, but actually never made it into a feature right. I.

B

Move the slide so for actually implementing your dns setup. What you.

B

Really want, if you're using, let's say the tectonic installer or any kind of.

B

Automation tool is you want a dns server that is api accessible, somehow right so.

B

If you, if your dns configuration, requires you to open a file edit, it save.

B

It and restore the server or even worse, past, that file to an admin. That's gonna.

B

Be kind of complicated to integrate in an automation scheme, but luckily there.

B

Are options for pretty much any platform that can can be tied directly into your.

B

Provisioning tool, so what we're doing is we're using terraform, which has.

B

Resources for all these dns providers- and we generate.

B

The records dynamically as the resources come and go so as terraform creates.

B

Instances load, balancers and so on. It also keeps the records in the dns.

B

Service in check aws has route 53, there's an api. You access it with your.

B

Typical aws client asier has, as you manage dns, which is part of the azure.

B

Which was manager api? You can manage it through your typical editor clients, gcp.

B

Has it designate as openstack solution for a managed, dns server? All of these.

B

Are only api based, so the your only interaction point with them is an api.

B

For bare metal, for example, a good option is actually powered dns and, in my.

B

Opinion this is very underrated, like this, this tool is very underrated. You.

B

Can do a really nice really nice things with it specifically because it has a.

B

Couple of api is not just one, so it has an arrest api that you can talk to with.

B

With pretty much any http client and manage the records and manage the.

B

Lifecycle of the server and everything it also has. It also has support for rfc.

B

2136, which is dns update, dynamic, which is what dynamic dns as in as you know, it.

B

From your home, router uses, basically it's its write operations via the dns.

B

Protocol so you can actually publish records using an extension of the dns.

B

Protocol or dns supports that a bunch of other services, including active.

B

Directory support that so worst case. If there's no api on your dns server do.

B

Check if it supports this mechanism, it's really useful and there are. There is a.

B

Terraform provider, which makes use of this, and you can actually from your.

B

Terraform templates publish records using the dns.

B

Protocol to your dns server so either way you have options. You you, it's not.

B

Hard to deal with dns as long as you as you have programmatic access to it right.

B

So this is important when you're planning your cluster deployment, make.

B

Sure you check what the state of dns management is in that specific.

B

Environment, where you're trying to deploy it, if this is managed by a.

B

Different team, a different part of the organization and so on, and so on. Try to.

B

Talk them in advance and see if you can get programmatic access to those servers.

B

Because then it becomes a lot simpler and you will also need it during the.

B

Life cycle of the cluster- this is not just for provisioning, exactly yeah yeah.

B

But on top of the subdomain you still need to talk to the server itself and we.

B

Actually did that we underestimated it and that's that's the reason why I'm.

B

Talking you so much about dns, which seems to be like. I can already discuss.

B

Topic right do think about dns before you start building the cluster, because.

B

You're gonna you're gonna need it there's. If, if you want to work around it.

B

Or you, you don't have the option of controlling dns.

B

For your cluster you're going to be severely limited in your security.

B

Options and and again power dns, is a really cool tool, even if it's like 25.

B

Years old or whatever, then there is this strange idea of split horizon dns, which.

B

I would kind of I was expecting that most people are aware of, but a lot of.

B

Our colleagues were actually surprised when we try to do this. This is actually.

B

The only way where you can keep traffic within your cluster entirely so each.

B

Node each worker node is going to want to talk to the api server and vice versa.

B

There's bi-directional traffic happening there, so the cube, let's call the api.

B

Server to get workloads for themselves, the api server calls the cubelets for.

B

Statistics for getting logs and a bunch of other things, so there's there's.

B

There's communication going both ways and all of this communication relies on.

B

On host names because you're using tls. So if, if those host names resolve to.

B

Public ip addresses, for example, then, basically, you've, you've blown up all.

B

Your network containment right if you're, using public dns or some third party dns.

B

Provider or whatever scheme that resolves to public ip addresses because.

B

For example, because your api server needs to be accessible from the outside.

B

And you only have one one set of records for it right all. The traffic between.

B

Your nodes and your api server goes through the internet right with all the.

B

Implications that that carries, and I'm just going to say, dns outage and and cloud provider connectivity issues to.

B

The internet, which otherwise wouldn't have affected you if you didn't use.

B

Public ip addresses behind those records, so split horizon solves that problem by.

B

Allowing you to have different sets of records for when you're accessing your.

B

Clusters from the outside versus from when nodes or applications internally.

B

Need to access various other machines in the cluster. This basically means that.

B

The same record is published twice to the dns server and depending on where.

B

The query comes from you get one or not not twice, but you can have multiple.

B

Incarnations of the same record and depending on where the query comes from, you get either a private ip address. This is only accessible within the cluster.

B

Which is what you want to do for your node to a api, server communication or.

B

You get the public one, which is what you want to get when you're, when you're.

B

Remotely accessing your cluster but do look into split horizon dns. Most of the.

B

Tools out there, I think tectonic installer is the only tool that actually.

B

Makes use of split horizon dns for yeah cops, publishes public ip addresses for.

B

Your master nodes by default for simplicity and pretty much all other.

B

Tools do do this kind of stuff look into this. It's it's really useful.

B

Also in aws the private side of the of the split.

B

Horizon setup is handled differently than the public one, so it's resolved.

B

Within the vpc boundaries, if w us has a connectivity outage to the outside, your.

B

Cluster is not affected if you're using split horizon right, because the.

B

Resolution for the internal queries happens before traffic goes to the to.

B

The to the internet, so it does not have to go through the whole dns farrar key.

B

To resolve right, it'll immediately resolve on the.

B

First, hop on on your first resolver, which vpc provides you, so your cluster.

B

Would be resilient to aws, independently of your of your actions or your your.

B

Your set up, but temporarily loses connectivity to the internet at least.

B

You don't maybe you'll be affected by your workload, not being accessible, but.

B

At least the cluster won't lose its head, because the nodes won't be able to talk.

B

To themselves right, if you're, using public ip addresses for your masters as.

B

In your deployed with an out-of-the-box configuration with your random.

B

Installation tool your nodes will no longer be able to access the api server.

B

Which actually lives in the same v? Pc with them? But because it's publicly.

B

Resolved that wouldn't work anymore, so you will have a basically an impaired.

B

Cluster, this is this is an example of how you publish the same record to.

B

Implement split horizon on aws, so it's a matter of just having two resources to.

B

Record resources that point to the same name. The name is basically what becomes.

B

Your your dns record, so you can see there's exactly the same name, but in one.

B

I don't think we're actually yeah exactly so there they're being published.

B

Two different zones right actually there's an there's another attribute.

B

Which is not illustrated here, which would be the different ip address or.

B

Like the different value of that record, that would be published to those.

B

Specific zones right, so you would have the public zone resolve that that name.

B

To public ip address and the private one would have a different one: a private ip.

B

Address right, load-balancing right: this is a solved problem in pretty much all.

B

Cloud providers with some caveats I'll, explain later: simple providers like.

B

Vulture, for example, or even pack it they don't, have load balancing solutions. So.

B

You're pretty much on your own. There you do what you normally do in bare metal.

B

You deploy something that acts as a load balancer or you use cloudflare in bare.

B

Metal most likely you, you are already doing some sort of load, load, balancing.

B

Scheme hardware load balancers work best. If not you deploy a proxy or reverse.

B

Proxy nh a proxy whatever nginx, but please deploy it redundantly I mentioned.

B

Caveats on the cloud provider side so, for example, in azure, so all cloud.

B

Providers allow you to have public facing load, balancers and private ones,.

B

The private ones act as I kept saying, as in like the traffic stays within the.

B

Boundaries of your vpc, they have private ip addresses assigned if the outside.

B

Connectivity of the data center is compromised, at least you can use.

B

Whatever that load balancer is fronting from within your cluster, so that's what.

B

You want to have for your api server facing the workers. You want to have a.

B

Private load balancer there, the private load, balancer private-public scheme.

B

Applies to load balancers as well as dns. So if you're implementing split horizon.

B

Dns, you have to match that with load balancer facing your private network.

B

Different load balancers facing your your public network for exactly the same.

B

Reasons if you lose outside connectivity, your cluster stays functional right now,.

B

This works really nicely in both in aws. The only difference when you provide.

B

Provision them is, actually you set the private flag on the low pass and it.

B

Suddenly becomes private in escher, it's just as.

B

Easy to provision them, but they don't work like that.

B

Asher has a bug where, if you, if you make use of a private load, balancer fronting.

B

The same set of instances that need to call into it so in in kubernetes this.

B

Translates you put a load balancer in front of the master nodes, but your.

B

Master nodes they run cubelets as well, which need to talk to the api server.

B

Which is being fronted by that load balancer in this set up those cubelets.

B

Won't be able to talk to the api server, because the load balancer has a bug and.

B

It does not allow traffic coming from the same ip address, that is behind a.

B

Load balancer to loop back into it, so that doesn't work on azure. It's a it's a.

B

Well-Known feature: it's a well-known issue with them. We've actually talked to.

B

Them about it, they know they're trying to fix it.

B

They may have already fixed it, but I'm not sure I didn't double-check. Basically.

B

We were forced to not use that in azure and we did something uglier, which is.

B

Basically put the master, no, we did. We still did not go through the public load.

B

Balancer, because that's worse, what we did is we we registered the machines.

B

Themselves directly under a multi record, multi and three a record, so they yeah.

B

Which is that which is actually not load? Balancers just a discovery scheme, it.

B

Does not guarantee any kind of load balancing, but it works for now until.

B

They fix it, yeah, that's kind of the story with loeb answering most of these.

B

Actually, most of these hardware load balancer actually have api's or remote.

B

Management systems, so you could theoretically.

B

Oh, that's a pretty cool trick. So let's say you like. I use cheap digital.

A

Provider a lot of club providers to deploy to our communities to us to make.

A

Larger and digitalocean like this louis, has a load balance auction. But for my.

A

Sort of communities experiments where I want to launch $100. You know I don't want to spend another 20 bucks on the little pen, sir, so cloud fair has a.

A

Pretty neat trick to balance traffic, and that is clear, protects to be a dns.

A

Provider right so you submit dns a records or, if you're good, like ipv6 ike,.

A

Records to cloud fair and that when you do way and this lookup on tig, they look.

A

Up on your records against klopfer, you actually don't get back your ip records.

A

You get back close for installed balances so like, if I submit for a.

A

Records to cloud fair and I will clearly after the submission cloud free, I get.

A

Back there all consoles, and what happens after behind the scenes is that your traffic gets being gets being low, balanced by a code, fairer affray of load.

A

Balancers and they are dirt cheap they're, like 5 bucks a month or something that's pretty cool, and this is the neat trick that I use.

A

When compromises, where they don't have like load, balancer primitives right so quite sure, for instance, doesn't have a local printer primitive and there was too lazy to deploy my own law principles. You know, like I'm lazy and then I simply.

A

There was a provision er for terraform for cloud fare and you simply submit.

A

Your public a piece over there and they do the noticing, for you make sense this.

B

Is just for public ingress operating? This is not suitable for all the private.

A

Jets that alex just explained to you that's totally, not applicable, but if you.

A

Have like ingress, like your block, you are block by public dns names and you don't want to spend the money for open thing you can use. Confer that's essentially, I think that's a pretty good idea and they think this sort of semantics is a little estimated. Nobody implements this semantics except clause.

A

Fear and I wonder why I think, there's a pretty cool trick action.

A

No, you, you submit to tls certificates against dns names right so again, the.

A

Importance of dns right, so their dns certificates have the dns names.

A

Registered, so that's it so again, thanks.

B

To search for inviting me and for actually coming up with the idea and.

B

Putting a lot of work into this, if you don't have communities provide.

A

Integration, you more or less screwed. There is a addition inside abilities.

C

Called flex volumes, which gives you a nice sort of hook in such abilities to.

A

Write your own script. Whenever you want to attach storage to your parts, yeah.

A

Flex volumes just google flex volumes with kubernetes, so the tld rs. If the.

A

Cloud provider integration is missing. You can use that including some sort of book scripts to get a native integration. I was playing around with that to get.

A

Watcher storage, integration ready, but that's sort of like the teal.

A

But you're totally right, we skipped over it. That would be like another, like ten.

A

Minutes of slice or something like this, but that's like the sort of experience that I can give you along the way if provider integration is missing.

A

Roll-Your-Own using flex flex provide flex volumes, make sense.

B

I'm assuming you've already researched persistent volumes and persistent volume.

B

Claims right so those are basically being hmm so if you want to have native.

A

Support for exactly this sort of jasmin volumes, then the flex volume stuff is.

A

For you, and even with the cloud provider, integration, research them because for.

B

Example on aws ebs volumes are bound to the availability zones in which they.

B

Were created so butt pods can migrate the availability zone. So if you have a.

B

Persistent volume attached to a pod and the scheduler suddenly schedules it.

B

Because you did not constrain it, it gets scheduled to a different availability.

B

Zones the attachment would fail, the volume attachment would fail and there's.

B

All sorts of subtle things like that the metal is yet another story like yeah.

D

You have to come up especially roll-your-own solution which, like use.

A

Something intrinsic like and don't over engineer it. A lot of people are trying.

B

To come up with very fancy storage systems or distributed storage systems.

B

For communities, if you think about how we fish it, how easy it is to deal with.

B

Ebs in the classical or like vm kind of setup in aws and how well they work.

B

Think about the fact that they're actually implemented with just like scaasi with an api attached and the hardware matters. So the for any kind of.

B

Serious performance for storage, to use a dedicated network for storage and not.

B

The same one as used for regular traffic.

B

Something that one thing you can do to make your life.

B

Easier in that setup is make sure the operating system that you go to has the.

B

User system deep because then it's easy to transfer the ignition configuration.

B

Into something else that provisions those system d units, if you don't have.

B

System d or you choose not to use it after we write all the logic that we.

B

Have in system of units lowest hanging fruit here, if you don't want to go with.

A

The chorus so pretty much all all of the.

B

In on instance, provisioning and deployment of tools happens via system d.

B

Drop-Ins, if you have system d on your target system, just copy whatever.

B

Ignition renders as droppings and distribute them somehow.

A

Things red hat coming, you know, provide internal.

A

Intrinsics, which allow you sort of autonomous upgrades of those systems in an atomic way which, if you use like classical.

A

Systems like db on it's like, if you did you ever come to the situation, you have.

A

Credit, a debian system and it's failed in the middle and then the whole node.

A

Becomes unusable right so there are modern alternatives to those sort of.

A

Problems like caress, continual nick's, like atomic like car, whatever, which have.

A

Countermeasures against these sort of like disaster failures, when you do an.

A

Upgrade of a node and it fails in the middle totally I totally get there. Yes,.

A

I can say from a totally neutral perspective, as of today, I wouldn't use.

A

Personally, tectonic anymore, because of.

A

The unclear future mere conjecture with redhead. There is actually a very interesting project that those guys investigated called typhoon, which was initiated by a former engineer of korra's, which sort of want to have.

A

Like a debian like distribution of kubernetes using terraform, so that's.

A

Something I would recommend right now and I would definitely recommend going with terra for because, if you look of all the other sources that cube adm cops.

A

And whatnot it's super much opinionated and it's so hard to sort of deviate from.

A

It and terraform, for me at least personally, gives you like this sort of the only bit freedom where you can deviate in your own topology and your.

A

Own requirements set right, so I would go definitely with the terra firma set up.

A

From another source perspective, I would probably to take our typhoon if you go self-hosted, because typhoon is self-hosted all the way through.

A

If you roll your own, it's not super hard. A few follow this guy's recommendations.

C

And then coat them in terra firme. I did this like for two different providers.

A

For head's, in a cloud and for macho it's not like make a complex, but you have to.

A

Be aware of those compare like of those trade-offs that we does it make sense.

A

But I want to give the ability for a ratchet engineer to answer that. Yeah.

B

So well, you wouldn't be able to use tectonic anyway because it can write as.

B

As a current master or latest, let's say for releases, we don't deploy when our.

B

Kubernetes anymore, so you would you'd be deploying the whole tectonic stack, which.

B

Is kubernetes as you get it from upstream, but also a bunch of other.

B

Operators and components that are proprietary to tectonic, which are.

B

Optional, you still get a functional kubernetes cluster, but you will see filling pause there because of missing license for.

B

Example, right see it's it's a commercial product. You would have to have a license.

B

For those level components, if you don't have it.

B

Those things would be crushed, looping in your in your cluster. Of course, you can.

B

Like disable them, because it's all open source, you can just like forked, a few.

B

Versions ago, we actually had a flag to deploy to deploy just kubernetes as.

B

Upstream kubernetes, so you may be able to fork at that point and and maintain.

B

Your own copy of tectonic, like serge, said the fate of tectonic in red hat, is.

B

Not yet decided we're still discussing it, it it's going to merge with openshift.

B

But in what shape and form we don't yet know, so we don't. We certainly know that.

B

The operators of tectonic are going to make it into openshift and the.

B

Self-Hosted installation approach, but not the actual tool that I mean we're.

B

Still discussing how much of the the installation tool is going to be kept so.

B

If you want to use it for kit, I wasn't aware that cool questions.

D