Flatcar Container Linux Tech talks, 28 Mar 2017

Previous Meeting

⏯

youtube image

►

From YouTube: Tech Talks @ Kinvolk: Kubernetes AMA session w/ Kelsey Hightower (March 28, 2017)

Description

Kelsey Hightower held an "Ask me anything" session about new features in Kubernetes 1.6 at the Cloud Native Computing Berlin Meetup which took place the evening before CloudNativeCon+KubeCon EU 2017.

About Kelsey:
Kelsey Hightower is a Developer Advocate at Google. Kelsey Hightower has worn every hat possible throughout his career in tech, and enjoys leadership roles focused on making things happen and shipping software. Kelsey is a strong open source advocate focused on building simple tools that make people smile. When he is not slinging Go code, you can catch him giving technical workshops covering everything from programming to system administration.

Meetup link: https://www.meetup.com/Cloud-Native-Computing-Berlin/

Follow us on Twitter: https://twitter.com/kinvolkio
Get more information about Kinvolk: https://kinvolk.io

A

So this is my first time in like a hundred talks not having a slide deck or a computer okay, so I'm nervous all right.

A

I'm super nervous, I don't have my my crutch or my neck suit, so I will talk.

A

About a little bit about kubernetes and then the goal here is, you can ask me.

A

Anything. Okay, I won't give you any google trade secrets, but I promise to give you my honest opinion about everything for.

A

Example, I don't really like joe bata, but he happens to be one of the creators of.

A

Cooper neighs, so I'm forced to get along with them.

A

Okay, I'll! Be that honest okay, so one that sticks is out the the goal of 1.6.

A

Is to really show up the core things that we made promises on things like our.

A

Backs of your past provider, like openshift, you've been trying to handle all of these things yourself at a higher level.

A

Than kubernetes, which is really fighting the system, the core system itself needs.

A

To have the ability to have detailed and granular auerbach roles and permissions.

A

Against the resources that you manage in your cluster, how many people here have play with 1.6 and turn our back on, so my biggest fear.

A

Is that our vac may turn into the selinux of kubernetes, where people just.

A

Figure out how to turn it off, and I think you all should join that.

A

Conversation with 1.6 ships because there's going to be easy way out, but we.

A

Hope that the community doesn't take it because you won't learn anything about.

A

How a system like kubernetes should interact especially start to think about.

A

Add-On services, today, all the things you run in a kubernetes cluster, pretty much.

A

Run in god, mode right, you download something from the random internet. How.

A

Many people are doing this pulling things from the ransom internet, raise your hand, because you know you do it a lot right. So all those things pretty.

A

Much have root inside of your entire cluster, so you pull down a controller. It.

A

Actually has the freedom to create anything that it wants inside your.

A

Cluster today, for most enterprises we're starting to get into a maturity phase.

A

With kubernetes, that's not going to work anymore, so discipline is starting to.

A

Enter to platform, so you're going to have to actually start to deal with things like our back and things won't work by default. Unless you give it.

A

Granular permission, so this is a good thing. We hope people don't abandon ship.

A

On it, the last thing is around some of the cluster bootstrapping stuff. You guys may have heard of hefty. Oh, so this is one of.

A

Jo betas startups that he started and one of them like he's, going to be a.

A

Mogul but their goal is to make ku bernays much easier by doing the work.

A

Inside of the core, so the last couple of years I have personally had the feeling.

A

That coral or kubernetes itself is experiencing a little bit of the journey.

A

That linux kernel has experienced right. You have things like slack, where red hat.

A

Debian ubuntu we're in this world of kubernetes distros. Now the danger here.

A

Is that if you have too many of these distros and they start to fragment from.

A

Each other, this thing we call kubernetes starts looking different depending on.

A

What distro you're using so it's important that we actually start to focus back onto the core to find what it is and keep it simple.

A

Part of that work is making sure that it's easy to install a secure, kubernetes.

A

Cluster so having the cluster nose, be able to come up and bootstrap themselves.

A

By minting a certificate and using that for future communication, so those like.

A

The two big things around one batch six that I think will change the way people.

A

Operate and think about a cluster, and then the other things to me are more of.

A

Moving things from beta or alpha to beta and beta to stable and you'll, see some.

A

Of that in the release notes, so that's it for now, but I'm going to open it up.

A

For ask me anything all right so my background here is I've been dealing.

A

With cabernets for almost three and a half years since the project came out.

A

Yes, the first person to get contribute access that wasn't at google and since.

A

That time, I've been building things around the edges and working with lots.

A

Of large companies dealing with kubernetes for the first time or scaling.

A

Beyond their initial use cases, okay, so I'm happy to share all of that learning.

A

With you guys open it up for questions, don't stand there. This is a meet-up, so.

A

You're actually specific there you go how's that going to be good, so the.

A

Question is there's this movement around the open service. Brook. Do anyone know.

A

What that is where's your happy note, that is how many people have heard of.

A

Cloud foundry, okay, so in the cloud foundry world, and you can also think.

A

Another path of the service like heroku. This idea that when you write your.

A

Application, the way you connect to things like a database there's a service.

A

Middle layer broker in the middle, so things like credentials and what you.

A

Connect to that should be largely hidden from the application developer. Now the.

A

Goal with this design is that you, as a person creating a service, a database.

A

Service, a payment service, so ideally, what you can do is say hey in order to.

A

Connect to that payment service, here's the bit of data that the application.

A

Needs okay, so that pattern proved to be popular in the enterprise world right.

A

Maybe, where people don't have a much a lot of time to spend on doing the.

A

Integrations right, like you, and I would say how about you just use popular chef.

A

Put in a config file and off you go problem is that in that world there's no.

A

Auto team there's no central place to gate who has access to what and when. You.

A

Have that central broker, you start to get some of the stuff that you see with.

A

Like hashey corpse vault, where you can as an application, knock on the door and.

A

Be mented a set of certificates or credentials just to connect to that.

A

Service and it all be tracked, so the idea is to bring some of that ecosystem.

A

Into the kubernetes world right now, I'm of opinion that this shouldn't be a core.

A

Concept right to me, this moves criminais is too high of the stack. This will be.

A

Useful or something like deus openshift, you know cloud bound. You just.

A

Announced it there vashta will be able to install kubernetes so they're.

A

Starting to touch on those edges so to bridge some of these systems that.

A

Service broker pattern becomes critical for a lot of people who are used to.

A

Working that way right my opinion, I think it will add value to people who.

A

Like that model, but for people who are more will roll-your-own who will use.

A

Something like etsy, b or console for service discovery and they're. Okay,.

A

Pulling their own configs and secrets, I don't think it's for that cloud, but we.

A

Remember, kubernetes has grown up now right, so it's being used by a lot of.

A

Enterprises that work is driving lots of things in kubernetes. So the last thing I'll talk about on that question is there's been.

A

Work around being able to give have policy inside of kubernetes inject.

A

Things being a mission controller into a pod, so the idea would be you as a.

A

Developer, you wrote an application and you're not really familiar with the.

A

Environment that you're being deployed in the cluster administrator could.

A

Define that any pod that looks a certain way is injected with, let's say some.

A

Certificates, some secrets and additional volume right, something that we can do is.

A

Rewrite the pod on the fly based on the administrators policy, so that keeps you.

A

Out of knowing what's being injected, maybe there's something that we know you.

A

Need like tls certificates or when I was playing with this particular pattern. The.

A

Idea would be all of my apps are be small, go bynars that don't have a base.

A

Image but then they don't have any tls certificates, so the root ca doesn't work.

A

So in this case, instead of me going into every pod and writing this little volume.

A

Section where I'm mounting certs from the host, I can actually have a different.

A

Thing do this at at runtime right, so you wash the pod at between the.

A

Scheduling process and running another node you're going to be injected with.

A

All of these things, okay, but that work was to help support the service broker.

A

Model because when you say hey, I need to talk to my sequel. Well, all those things.

A

Can be generated on the fly and then inject it into your container as an.

A

Environment variable or as a volume, ok, so that's the thinking with the service.

A

Broker we want to open up the ecosystem right as a community, but it's not.

A

Necessarily something I think everyone will have in their kubernetes cluster. I.

A

Fantastic question, so this service broker thing is big.

A

Right in order to make it all work, as flutist people have in their minds.

A

You're going to need work around the edges, so this thing I talked about will.

A

Enable the service broker to be able to inject things via policy right, so all of.

A

These things will come together. It's going to be a lot of work to get that same experience that you see in something like heil foundry, but just.

A

Know that the community is working towards this step-by-step and.

A

Identifying bits and pieces that are moving, so you as a developer, let's say.

A

You had a service broker inside of your environment. You can have three different.

A

Views to this or multiple views, one would be like push the deploy. I need.

A

Postgres, but I don't care how you do it, maybe that pulls in a container does.

A

Something create some secrets and then register is it and then you can just.

A

Deploy your app and then the service broker will do all the binding stuff.

A

Necessary for you to use postgres or you may be a producer, you may be building.

A

The payment service at your organization and you provide a service broker binding.

A

No one needs to know anything about your app other than how to get through it to.

A

The service broker, so to me this might be a bigger story. I.

A

Don't know if it's all fleshed out, but for me personally I never work in that.

A

Model and if I have it's been completely transparent to me, okay, any other.

A

Questions so the question is: what is the roadmap for secrets all right, so this is.

A

A good one, what sequence is built contributed mainly by the community? We.

A

Need a way of injecting secrets with our application. What were people doing.

A

Before you're copying files, what were you doing? Copying files to servers.

A

Puppet or chefs right, it's technically kept glorified copying files to servers.

A

And most people were ok with that, but in the current age world is a bit hard.

A

Because you don't know where things are going to land, so we have the secrets: api.

A

So the secret api- I think people are happy with the api.

A

Meaning store something and kubernetes refer to it and a runtime. We bind it to.

A

The particular pot that needs it great problem is that most people don't have.

A

Any security whatsoever, no tls between the components right. So then those.

A

Tickets are passing being plaintext, some people don't have the distant.

A

Cryptid at rest, so if you're, storing a net cd and eating the snapshot to the.

A

Disk, it's just laying there in plain text cloud providers not so much for.

A

Instance, most cloud providers with a bootstrapper kubernetes cluster, like we.

A

Do ogg we walked out all that stuff. So what's the remaining problem with.

A

Secrets docker rightfully pointed out that, and.

A

They solve the problem half way, because I challenge them on this one. So in.

A

Kubernetes, when you make a secret object by default, all the nodes in the cluster.

A

Are free to pull any of the secrets that they want? Alright, that's just a default.

A

Setup before we have our back ok, so now a pot is placed on any node and any node.

A

Can look at that pot manifest and go grab the secrets? This is all being well.

A

Behaved but what happens when there's a malicious actor on the node and they.

A

Have this token or this access to be able to pull any secret that they want.

A

So now stickers become something that every node can see whether they need to.

A

See them or not, that's the first problem. This is because we have a essentially a.

A

Pull model for secrets right the nose pool on demand. So in order to make that.

A

Work, we need very rich, our back right. I actually have a prototype that I built.

A

On my laptop where the node can never get to any secrets by default after a.

A

Scheduling decision is made, and I know that it's running on that note, I can.

A

Dynamically go to our back and say this node with this name can see these exact.

A

Secrets very explicitly so then you get yourself in this very dynamic model that.

A

Allows that pool model to be consistent, be audited and be locked down and match.

A

What you expect now docker solve this problem by doing a.

A

Push model so the node itself is pushed its work run this container, oh yeah and.

A

Here's the secrets that go with it everyone's, like that's, perfect! Well, I.

A

Should encourage just do the exact same thing? The truth is, it only buys. You have.

A

The story: what happens when you write a service someone's going to have to do.

A

Rich are back at some point. They can't push everything to everything right so.

A

At some point they know they trust the node, because it's been bootstrapped and.

A

Then they can push to an endpoint that they meant it a certificate for so what.

A

Happens when you start to have third-party tools and clients, what are you pushing to alright so eventually they're going to have to.

A

Solve the rich are back problem to make this actually work. So for us we're.

A

Taking it serious so for us, our back will probably start to have a little bit.

A

More capabilities on who can see what secrets that's step, one obvious thing to.

A

Do joe actually has a lot of insights, who I talk to a lot about this, because.

A

He knew how we did it in google, and just has experience with this stuff is really.

A

It's more of an identity problem right. What is your apps supposed to be doing.

A

It's not about pushing around secrets and who can push secrets around them.

A

Secure what you really want is when vault kind of hinted at, if you have an.

A

Identity then I can associate policy to that identity and then secrets and.

A

Database actions access the implementation detail. Alright, I just.

A

Need to know who you are, we don't have that. So that's probably the problem we.

A

Need to start thinking about in the kubernetes community is that when you come up inside of a pot, how come you just don't give you an identity that.

A

Could be trusted by other systems? Ok, so there's a doc that eric tune.

A

He's running, I think, weeding the security sig. They have a doc about all.

A

The phases so I've built some prototypes around bulk integration. I think a lot of.

A

People are writing to consensus that we don't want to just expose the bulk to.

A

The entire kubernetes system, I will break a lot of api's and guarantees. We.

A

Have so when you look at that formula, there's low-hanging fruit and there's a.

A

Bar discussion about how do we make it easy to identify the thing that needs.

A

Secrets in the first place: ok, so hopefully that answers the questions.

A

About secrets, but a lot of people are working on this allows you to actually.

A

Know what they're doing or working on this, which is the good part, the hard.

A

Part is convincing the entire community that we should move in a certain.

A

Direction when people have other views about how it should be solved.

A

Ok, fantastic question: I'll take two more.

A

So the question is: what do I think about? Go clients against the kubernetes api.

A

Man, this is what I get in trouble for all the time, so, instead of the.

A

Kubernetes core codebase, the client to me, grew organically right. We were using.

A

It internally and of course, it makes sense that people want to build things.

A

On the outside and google does a good job in my opinion of api first design, so.

A

You least have some sort of contract between you and some other components in.

A

Kubernetes, the problem, though, is that there's a lot of implementation detail.

A

In the current client, so if you've never built anything on top of kubernetes, if.

A

You import the kubernetes clients into your application. Let's say your.

A

Application was about five mix, that's where you start it. You import this thing.

A

Wet 50 site not laughing my people import it in they build it like. This is.

A

50 meg's, what the hell is in here right and you essentially imported all of.

A

Kubernetes into your application, so the team is working on it. One one goal was a.

A

Stepping-Stone is to at least have a nice swagger spec that we could generate.

A

Clients from or at least conform to what our promises are. The second thing is we.

A

Start to move the client to its own repository the problem, though right now.

A

Is that we're effectively just copying it from here to a third-party place.

A

That's not the solution me when I build tools, I don't use a client because I.

A

Actually, like use the http semantics, the problem with the http interface today is.

A

That some of the stuff is implementation. Detail like the math on scheduling like.

A

What is the cpu a milic, or how do you subtract the miller core from some other.

A

Unit that koreans have that's coup bernays own implementation detail, so I'm.

A

Stuck either reinventing that for every language that I want to write in or.

A

Bringing in that 50 meg's only for going so we need to fix that. We need to define.

A

Some more clarity to me around all of these parts of the client and they also.

A

Have the ku ctl factor, coos ttl does a lot of magic.

A

You say: coo ctl delete a deployment and it's like wow. I cleaned up everything.

A

Life is great and then you go and build against the api. You say: delete.

A

Deployment and a bunch of positive laying around okay.

A

So we need to unify what it means to be a client of kubernetes and a lot of.

A

People are thinking coops ttl, not the raw api that we actually present. Okay, so.

A

Lots of work going on there there's a sig wondering that around the api.

A

Machinery pop in there, but people have good ideas and the community is moving.

A

For by generating climbs out of the swagger spec, okay, that's all hand up.

A

Here, no, oh yeah go ahead, you got it all.

A

Right so we talked about this core thing, so the question is we have this thing.

A

Called ingress, that's out of kubernetes right, so you think about kerbin a as we.

A

Take infrastructure and we extract it away and you just give us a mo file.

A

Today and you say hey, I want this many containers running, we make it so and.

A

Then people want to do things above that. How do I get traffic to these particular.

A

Things so we have service discovery that lets you give us labels to identify what.

A

Belongs is backends, so we collect that list right. We have this thing called.

A

Endpoints, that's where all the healthy pods are registered, but then we have.

A

This idea about how do you integrate a higher-level components like a load.

A

Balancer so there's a question here: what is a little bouncer engine xh8 proxy.

A

Google cloud elb, none of them work the same so to me, one flavor flaw was to.

A

Make an ingress controller that attempted to normalize them all. I just.

A

Want to have this path based routing, maybe do tls termination and for a lot.

A

Of people you get kind of that heroku model and it's a good starting point.

A

Also, the customer the other day and they're trying to do reg x, matching on.

A

The pass there's already a problem with reg expansion on the path, but the thing.

A

Is nginx supports it, but we don't do it in google load balancers, but the problem.

A

Is now you're stuck with the ingress controller that won't? Let you do that.

A

So then some solutions would be that's added. Annotation.

A

Right so then, if you put annotation there that will let your load balancer.

A

Know that it should interpolate these paths as regular expressions but be.

A

Ignored on another platform, so now we start to get into this world where we're.

A

Now, configuring, what an ingress controller is through this abstract.

A

Concept and no one knows what that is, because most of it is a documented. It's.

A

Ours, to be specific, so ingress was this attempt to let you integrate with load.

A

Balancers, depending on your provider, nginx edge a proxy or the cloud provider.

A

But focus on the common denominator because I would solve some percentage of.

A

The use case for everyone what's turning out is that we prior are going to be.

A

Better off saying this is the engine x, ingress controller and here's what it.

A

Supports right- and I don't know if these things will be able to be fully portable.

A

The way most people really use a load balancer in practice. Ok, all right I'll.

A

Take one more, you really have one other than that go hand all right. Thank you.

A