Flatcar Container Linux Tech talks, 28 Mar 2017

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Tech Talks @ Kinvolk: How Kubernetes Drives Organizational Efficiency by Joe Beda (March 28, 2017)

Description

Joe Beda stopped by the Cloud Native Computing Berlin Meetup which took place the evening before CloudNativeCon+KubeCon EU 2017. His talk focused on clustering technologies, such as #Kubernetes, and how they drive team efficiency. He also compared and contrasted different operations and organisational maturity models.

About Joe Beda:
Joe is currently the CTO & co-founder at Heptio. He was a founding engineer of the Kubernetes project at Google. Before that, he founded Google Compute Engine and was the lead engineer for the project. Previously, Joe worked on the Google ads system where he built tools to help advertisers pick the right keywords and bids. He also connected Google services to the public telephone network (PSTN) and defined and launched Google Talk.

Meetup link: https://www.meetup.com/Cloud-Native-Computing-Berlin/

Follow us on Twitter: https://twitter.com/kinvolkio
Get more information about Kinvolk: https://kinvolk.io

A

So I've, given this talk before, but probably a lot of you haven't seen it.

A

And I'm happy to sort of you know riff on stuff. If you have questions or if you.

A

Want to go into some detail on stuff. Some of this stuff is probably the old.

A

Path for folks that are through you know, understand this world pretty well, but.

A

Hopefully I can bring it all together and maybe give a little bit new perspective on things. So I'm going to talk about government operations and how.

A

These things work together and how it teams can work together. I'm having.

A

Trouble with feedback, okay, bob, and so it's all just one operation to get those.

A

Things together, you get devops right. I honestly don't even know what they opted.

A

So no one knows what it is. So kelsey tell me still here huh so kelsey we were.

A

We were having, I don't, know we're going to drink artwork. You know soft drinks.

B

And kelsey said that devops is group therapy for large corporations, but I.

B

Need to totally get on you know it's a things even flip about the stuff, and I.

A

Think there is a kernel of truth here and that kernel of truth is that we have.

A

To move to a world where developers have to give a crap about how their stuff.

A

Runs in production and folks who are running stuff in production have to take.

A

The learnings and they have a feedback back into the development process. These things have to work together. You no longer acceptable to throw stuff away.

A

Well, so you know, I don't hope that devops or not, but I think that's one of.

A

The things that I think comes out of this rethink around sort of how do we.

A

Actually relate operations and developers all right now, let's mix in.

A

Micro services, what the hell do micro service! I don't know that concepts.

A

Really fuzzy, you know, I could, you know, stand up here and have a bulleted list.

A

But everybody bulleted list would probably be different at the end of the.

A

Day, I think it's really come taking applications breaking into the.

A

Smaller parts and then have to send them over the network.

A

Now, how small are your services? I mean to be a pinko services right, you know, do.

A

We have like every single function is its own thing. I think part of the the.

A

Technique of getting this right is sizing the services and figuring out.

A

Where to actually create think that makes sense for you and your company and.

A

I don't think there's a right answer for this stuff, but we need to build building.

A

Blocks and so you need to pick the the right building blocks for you, my.

A

Experience here with respect to this up is architectures grow organically. You.

A

Know there's a job of architect. You know this guy's. He like.

A

Draws up with a gazillion boxes on the whiteboard and draws lines and like I.

A

Know tony has some sort of place. You know to use red pen or blue or whatever.

A

Right, you mean something, and then it's like okay, we just implement this and.

A

Then we're done right and that always works right. You can plan all that stuff.

A

Up the reality is: is that applications grow organically? You start something you.

A

Don't fully understand the problem, as you start to develop, you understand the.

A

Problem or any other and hey this thing that I thought was really one thing is really two things right and that's how applications grow, and so I think you.

A

Have to always be read back to always be looking for these opportunities to.

A

Actually take your application sort of, like you know, mitosis the the you know.

A

Cells dividing right: how do we have to take these services internally to and.

A

Have the application architecture grow as your understanding of the problem.

A

Group, so one thing that I think is interesting here is: we've always done.

A

Computer people taking big problems breaking down into small chunks solved.

A

The thompson and put it all back together again right: this is how we build big systems. We've done this before with you know, you know, moving from.

A

Assembly to like multiple source file moving to object, oriented.

A

Moving to you know, message tasking, depending on your flavor of crazy.

A

Language, one way to think about microservices is that we're essentially.

A

Taking the link or the thing that actually takes all these different things and binds them together into one thing and we're moving into productions.

A

Announced ed of linking bean at compile time say the linking is now a run time thing and we're not talking dll we're not talking, like you know, a lay bound.

A

Object object code, we're actually talking about across the network's right.

A

So this is what I think it's really fascinating when the des moines folks.

A

Came up with we cordy, they actually named it linker, because it really does.

A

Bring that thread through we're actually taking this thing that used to be done at compile time and we're moving it to be to be one time now. The real benefit.

A

Here is not really run the code about scaling your organization right. How do.

A

We take a big project and break it. Down to this. Team can actually operate.

A

Independently and so this is the real benefit for microservices again, this is.

A

Probably old hat for a lot of folks in this room, but I think it you know we get.

A

So enamored with the technology, it really pays to actually take a step back.

A

And understand: why are we doing this stuff and it's really so that these.

A

Teams can operate, they can make some progress, they can be loosely coupled.

A

From each other and and not b7i, each other's stuff, so the end result here is.

A

Philosophy right: you want to ship more products faster, more reliable and that's.

A

What all this stuff builds up to so so, when we talk about services, I think it's.

A

Interesting to say that not every service is actually created alike. I.

A

Think there's different ways to think about services. So the first thing here.

A

Is a service as an implementation detail, and this is what you'll see when people.

A

Have a model it they'll, take their big chunk of code, they'll carve off a little.

A

Piece put it across the network and then they'll call it a microkernel. These.

A

Things tend to be fairly tightly coupled but you're, starting to move in the right.

A

Direction, jack should sort of break these things up if you actually create the right abstractions the right contracts with.

A

The api you can have that the team now operate independently. The next level of.

A

Service that I like to think about it is what I've come to share artifacts.

A

Private entities, so this is where you have a service and the service is.

A

Generic enough that everybody that multiple people can stamp out instances.

A

Of the service databases are great pamphlets right, so you can have one you.

A

Know built primary four database on that, even a safe container by it could be as.

A

Clear but not being container, but everybody uses that same thing and they stamp out their own private instance of this thing right. So that's a great idea.

A

Of taking this and being able to this across across multiple domains, the next.

A

Step there is share is the sort of remote content. It's kind of one of these.

A

Things but when you think multi-tenant it actually sort of brings on this idea.

A

That you're going to take on sort of you know random people off the internet. That.

A

Are going to try and break you inside of a company, you can actually have.

A

Something that's soft multi-touch. We have multiple users inside of that.

A

Company using that service and they're using it in a way that you know if you.

A

Have something some boundaries but they're not actively trying to break you.

A

Right and now inside of a big company to start getting real value, because now you.

A

Can actually start taking expert teams concentrating it's not just in code, but.

A

Also in the operational expertise of running this thing and then the final.

A

One is including this one here for, for completeness is what I mean cause. They.

A

Have services, so this is like you know, you're putting an api out of the world.

A

You have to worry about security, for you know random people. You have to worry.

A

About charging them and getting all the documentation right being who's.

A

Superduper strict on versioning right so there's once you start going to like you.

A

Know big service as a product, you really import a lot of problems that you don't.

A

Have when you're keeping the company and mama sternums now the.

B

Interesting thing here is that, as you start doing more and more, this is what.

A

We saw at google, you actually search interconnecting mason's and, and you end.

A

Up with a service network, you have services, calling services calling other.

A

Services, you have a single service that maybe has a large span in and and it.

A

Ends up being really efficient because you have a lot of expertise that gets.

A

Leveraged widely across the company, but what it means, though, is that you don't.

A

Have anything that you can point to and get up an application at the stack? It actually ends up becoming a little bit of a tangle, but that's also.

A

Opportunities there also all right, so I'm going to switch gears a little bit a little bit of a non-sequitur kind on those most about.

A

Operations I mean I'm development. I want to talk.

A

About operations so um real, quick sort of like what's the difference between an.

A

Sre and and if this ops anybody have a good answer, google and straight-up yeah.

A

Okay, I'll give you guys actually so here's the thing so as who I am.

A

Everybody knows the same thing right. We just want to like make the bleeding stop.

A

And get back to that right, yeah, good question: what happened to tang in the.

A

Next, one right to sit on the water cooler and you actually say: oh man- is.

A

Sucked last night, like I'm, going to knock off early, go grab a beer because.

A

That sucks right or do you actually have the support of your organization and the.

A

Development team to say I don't want them to ever happen again.

A

Here's the ten things that we're going to do to solve that problem and, in my mind, that's the difference between eight things. It's really sort of internalized.

A

That you have to continually be looking at. How can we improve our processes so.

A

That we never see the same outage, we never see the same if you never see that same problem again, so that I think is in my mind the the.

A

Difference between just traditional operations and can sra, google, okay, so so.

A

There's another spectrum here I want to talk about in that op specialization, so.

B

One thought around devops is that everybody is in everybody's business.

A

Right, full staff, I'm doing everything from there like colonel on up look at me.

A

That can be pretty inefficient because there's a whole lot of knowledge. That.

A

Everybody has to know to be able to do this stuff and and when you think about.

A

It let's take cloud, for example, and I'm talking about for the infrastructure.

A

Cloud one way is this about this: is this is hardware off of a service, so we've.

A

Taken what has currently been an opera lacking, a staffing, repairing machines.

A

And we put that behind an api and we've taken that out function and we've been.

A

Able to outsource that that's great! Let me stop there.

A

Right everything that's happening inside the vms, all your problems, one of the.

A

Things with systems like kubernetes that we can do is we can actually start.

A

Creating more layering above that right, so we have hardware off of the service.

A

You know what can we do to actually create similar efficiency? L swear on a.

A

Stack so the first thing here is os ops and one of the things that we're saying.

A

Is this bifurcation of the traditional linux distributions right so there's two.

A

Parts to a distribution there's a sort of lizard brain- they get the kernel up.

A

And running and all the goop that goes along with that sexy bespoke system pays.

A

Tiffany and then there's a sort of package management like who's going to.

A

Compile open ssl and make sure that I'm not going to get right and all the stuff.

A

That goes along with that and with the world of containers, we're seeing this apply protection right, as you have you know, things like for allah.

A

Or project atomic or photon, or you know.

A

Cathy or whatever, or name your flavor you'll you're, seeing these os vendors.

A

Specialize on that lizard brain side of house right- and I think also with you.

A

Know things like how time you know the way that it's used is actually a package.

A

Manager, inside of contains their services specialization on the package.

A

Management side last and I could expect to actually see more and more of that happen over time. So what this means is that you can specialize on having a you.

A

Know a purpose-built os image and you can outsource sort of running that image.

A

To somebody who knows how to do that well right and you can make sure that.

A

You view that lizard brain as a separable piece from the rest of the.

A

Package management that you're importing now the next up gets here: cluster ops, so.

A

This is this, is you know, kubernetes is a service right and because we have this.

A

Nice api, a kubernetes because we've created a system, we can separate out the.

A

People who run this buffer versus the people who use the cluster and that's.

A

Incredibly, valuable line to be able to draw, and so the cluster of their job is.

A

To make sure that they can provide an experience for all the people remain on.

A

Top of that cluster that is predictable and secure and stable, and we can all.

B

Sort that so we're seeing stuff like dk we're seeing amazon solutions here, we're.

A

Seeing a juror collusion here and maybe even amazon at some point- there's a ton.

A

Of startups a book, including the salutes you guys who were we're talking about.

A

This earlier that are, are working to actually sort of enable to automate this.

A

Stuff, so this bus drop role ends up being something that you can sort of.

A

Make somebody else's problem, which is a huge efficiency and then above that you.

A

Have applications now this is where things get interesting, because we're.

A

Sort of coming back around a devops here I google, when you're launching a new.

A

Product the apps, the application development team is the one who writes.

A

The board configs and launches this thing on board and actually sees the.

A

Thing out the door does the monitoring all the stuff that you to actually get that stuff ready for production? So in that way, google does.

A

Devops right, but they don't do the full staff of devops. They actually knew the.

A

Ops part only goes down to work right and, and that's that that works really.

A

Well, because what it means is that the developers actually feel the pain of.

A

What it takes to run now as projects mature at google? You know at some point.

A

There's always going to be something going wrong at night. You want to hire folks around the world so that you know people aren't staying up and do handoffs.

A

And it becomes a professional job, so there are application level fre things.

A

At google for very mature applications, all right, let's take search right, so you.

A

Have to search develop a team you have to search sr evening. Those are two.

A

Separate teams, but you don't get the same conflicts that you get in.

A

Traditional enterprises, where you have the application scheme for the development team, you, the operations team hate each other. It's like one from venus.

A

One from mars and they can't get along, and we need group therapy right because.

A

The application ops in the application data teams, their goals- are so much more.

A

Closely aligned right, the application, ops team doesn't have 30 different.

A

Customers that they're all trying to make happy they just there were lines at.

A

One application it ends up being as attentions their end up being a lot less.

A

It can be a much more fruitful over relationship than you get with a.

A

General-Purpose obscene alright. So what are we doing with stuff like super nice.

A

So one way to think about this is there's a formula and have a longer.

A

Version of this talk where I actually have a capture the formula there, but.

A

There's a formula where there's none nonzero costs we're launching a service.

A

Right and as service is gaining complexity there day, there's an overhead.

A

For that that probably grows up like say: beware of it, the complexity of the.

A

Service right and then you have to find you have to.

A

Find the right sizing of services. Let's you balance the efficiency of.

A

Having teams run independently with the overhead of running multiple services on.

A

The operation side and also the communication between those because.

A

You're, coordinating more things with tool-free committees, we're reducing the.

A

Content of that of that formula, we're being able to actually run more.

A

Services cheaply, which actually creates the dividend- and this is what I mean.

A

When I say the operations dividend, if you can run more services, cheap, more.

A

Cheaply, you can take your operational capacity and you can use that to run.

A

More stuff or you can give the operation folks a weekend off. I don't know it's up.

A

To you right, but but if we reduce those constants, we create efficiencies, we can.

A

Plow, that back into the business and actually ship more products faster and.

A

So, that's really what I mean by the operations dividend. How can we use tools.

A

To make operations run more efficiently to enable developers to do more and.

A

Better things, so there we go. Thank you.

A

Oh, that's my blog and I got the domain. I feel a lot of things. It's the main base.

A

Name right, I got this amazing us what I named it I it tells me was talking about.

A

The identity thing a little bit earlier about identity. There's this other thing.

A

That I'm getting started up called city, secure production, identity framework for.

A

Everyone trying to think about, for how do we actually bootstrap getting.

A

Certificates everywhere in production again that's another example: domain.

A

Spacing I have two domain so that way you don't have a specific here. I do now.

A

Actually, I should modify that yeah yeah, let's encrypt an awesome.

A

Was nowhere the teachers, but I did yeah, I polish, isn't happy other, since that's.

B

What egg now but columns? Alright, okay,.