Cloud Native Computing Foundation KubeCon + CloudNativeCon Europe 2018 (Copenhagen), 4 May 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How to Get a Service Mesh Into Prod without Getting Fired - William Morgan, Buoyant, Inc

Description

Want to view more sessions and keep the conversations going? Join us for KubeCon + CloudNativeCon North America in Seattle, December 11 - 13, 2018 (http://bit.ly/KCCNCNA18) or in Shanghai, November 14-15 (http://bit.ly/kccncchina18).

How to Get a Service Mesh Into Prod without Getting Fired - William Morgan, Buoyant, Inc (Any Skill Level)

You’ve researched your service mesh options, made your decision, and are ready to deploy your very first service mesh. But how do you add a new communications infra layer across your entire production stack without bringing the whole thing down? In this talk, William Morgan (Buoyant) shares war stories from two years of helping users get their service mesh running in production. He shares examples of how teams with hybrid infrastructures adopt a service mesh between modern and legacy systems, how they use a service mesh to migrate production services from on-premises to the cloud, and other alternative approaches that don’t require a high-risk revamp of your entire architecture..

About William
Talky person @ Buoyant. Service mesh connoisseur.
Join us for KubeCon + CloudNativeCon in Barcelona May 20 - 23, Shanghai June 24 - 26, and San Diego November 18 - 21! Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy and all of the other CNCF-hosted projects.

A

All right, let's do it so this talk is called how to get a service mesh into production without getting fired. So.

A

Here's the situation, okay, you're standing at the edge of the precipice and off in the distance. You see service mesh land, where everything is wonderful, as we know having been at coop con for this long. We know that that's where all of our problems are solved, but before you can get there, you have to cross over failure, valley and you're kind of peering. Over and down. There are the bones of the the person who tried to do this before.

A

So how do we get from here to there? Okay, let me tell you a little bit about me. My name is William Morgan I used to write code in the good old days now, I, mainly write email, I work at a company called buoyant or based in San Francisco. We have two open source service mesh projects. One is called linker D, which is a CNC F project. One's called conduit does a lot newer, I'm, actually not going to talk too much about the details of what these things are.

A

There's a whole bunch of other talks about them today and I'll have a list at the at the very end, but suffice it to say that we have been putting service meshes into production for almost two years at a host of companies around the world. I've put some of the big fancy logos up there, including our friends from Mons, oh, but also a lot of small startups and we've kind of seen.

A

You know the the range of things okay and this talk is going to be about two things: it's actually the first talk, I've done for a long time that doesn't have a demo at the end. This it's just gonna, be words. So first thing is it's gonna be about manipulating human beings, okay and then the other part is, is going to be about good software engineering practices. A little bit of this is going to be service master specific, but in reality getting a service mesh into production.

A

That's not really that different from getting any other kind of complicated tech into production. It's maybe just a little bit more so because we're the service master sits, is in a very mission, critical part of the stack. So it's a little scarier okay and at the end of this talk you will be prepared. Do this or at least know why this is not a good idea. So.

A

Question number one: are you really going to get fired right? If you really work at a company where you try and get a service mission to pride, and it doesn't go well and you are actually going to get fired then just come talk to me afterwards, because there's a better company that I can talk to you about. Okay, it's not that's! That should not be a realistic failure scenario.

A

So what are the realistic failure scenarios? Well, here's the four classes, so, over the past two years, I'm gonna, try and distill. All of that wisdom into a set of easy-to-follow instruction guides all right. So the four ways that we actually see people fail with the service mesh is number one. You can't convince your colleagues that it's a good idea, so in most companies there are other people who you have to convince right: it's not just you.

A

Sometimes it is, and that's a very nice company, but most of the time there's other people and that's why a big part of this talk is about how to convince other human beings. Okay, the next way that getting the service mission to prod fails.

A

Is you manage to convince your colleagues, but then, when you actually try and roll it out to production, something breaks, and you have to stop okay failure scenario: three: is you get it into prod and things start going wrong and it's not your fault, but you get blamed anyways I'm going to talk about how this happens and how it's unavoidable, but how you can survive it and then finally, failure scenario: number four things go wrong and it is your fault, so we'll go through those one at a time, all right failure, scenario, one!

A

You can't convince your peers that putting the service mesh into production is a good idea. So first I think we need to ask ourselves. Is it actually a good idea? Why do you want to do this right? What problem are you trying to solve? And it's shocking to me it's shocking to me how often this comes up so I'm gonna play a little game. This is an audience participation game.

A

This is the only time that I'm gonna, you know, asked you to get involved until the the questions at the end, there's only time when you don't have to just listen to me talking talking all right. This is a game called good problem, bad problem, so I'm gonna state a problem, and then you have to shout out. Is this a good problem or a bad problem, and a good problem means? This is something that we can use the service mesh to address and that we should use a service mesh to dress and bad Bellamy's.

A

No, we shouldn't service mesh is not the solution for this problem. So, let's start we'll start with something easy: okay, here's a problem. The problem is, we need a service mesh. Shout it out. Is this a good problem or a bad problem?

A

Survey says bad problem? That's right! That's not a problem right! That's a statement of fact. We don't have a rocket ship, so therefore we need to buy a rocket ship. No right. The problem should be something like I want to go to the moon and therefore I need to buy a rocket ship. Ok, next one, our services are written in six different languages and we don't have consistent telemetry libraries across them. Good problem, bad problem.

A

Alright, you guys are on the ball. That's a good problem. This is a sort of this is a concrete problem is something the service mesh can address. There's other ways of solving this right doesn't have to be this all the service mesh, but this is an actual problem that can be addressed. Okay, here's a good one. We want to be cloud native cloud native is cool, we're in a whole conference about this thing, good problem, bad problem, I heard some goods and some dad's survey says bad problem all right.

A

That's not a reason to install a service mesh. All right, I think we're almost near the end of this. We have 70 service teams. Getting them all to add. Tls to their services would be not an impossible technical task, but an impossible organizational task. Good problem, bad problem survey, says good problem. Okay, so you guys are catching on here's. Okay, last one I added this at the very end, all right Google made kubernetes and we love kubernetes. Google makes a service mesh and therefore I need a service mesh.

A

This should be easy, good problem, bad problems, shout it out yeah bad problem, that's right! There might be great reasons to do it, but these are not the great reasons, alright, so believe it or not. Even though my goal in life is to convince you to install a service mesh because at the end of that journey, somehow I get paid a service patch is not actually always the answer. You have to make sure you're solving the right problem.

A

Okay, so here's my guy and I've put this in a very easy set of four steps that anyone can follow to convince human beings all right how to convince your colleagues and your peers, all right number one you got to identify who is going to be affected by this all right. These are called the stakeholders. If you're in the enterprise, you got to determine what the service mesh improves for them all right. This is the incentive. These are the carrots that you can dangle to make them excited about.

A

This you've got to understand what they're worried about right. Okay, because they are gonna, be worried about something and then you have to mitigate those concerns, extol the incentives and communicate alright. That's that's! Well, it's easier said than done, but this is called getting stakeholder buy-in and I'm going to tell you something: horrible, something that's just gonna make the rest of your life feel like it's just a sham, which is that Morgan's far fourth law. This is my fourth law any sufficiently advanced engineering work is indistinguishable from sales.

A

If you get really really good at engineering, you will become a sales person all right. So here are some examples. I'll go back, so you can take a picture of that Morgan's. Fourth law, yeah! That's a good one, all right! So here's some examples: I'm not going to go through these, but I just wanted to kind of give you you know a sense of who are the possible stakeholders. I might have to talk to what kinds of things would they be interested in? What concerns might they have so platform engineers right?

A

These are the folks typically who are introducing kerbin kubernetes, introducing the service mesh building the internal paths. However, you want to call it right and they care about a set of things and they are concerned.

B

A

Set of things another stakeholder in this process are the developers or the service owners who are going to be writing the code and mostly don't want this thing to get in their way right and then you have the security team, who's paid to make things secure and have a set of interests and desires, and you have senior management who cares about kind of higher level things like how fast are we moving, and this is going to cause more outages, I'm going to post the link to these slides and they'll, be on the CNC off like slide things.

A

You can go through this in detail. This is not comprehensive. It's just kind of a random sample of possible incentives and concerns all right. So let me talk about a case study. So this is a real life thing, all right, so 4,000 person, educational company and the way that this company works is that they have these extremely long-running feature freezes right. They deliver a product, it's for a school system or something like that right and the customers don't want anything to change for nine months.

A

Right, they're like trying to teach a class I, don't want features to change. You don't have to learn anything new, then I want the little pop-ups, let's say hey. Let me tell you about these new features. Take a tour like I'm, just trying to teach my class all right, so they basically have these nine months feature freezes okay, but at the same time this is all built as a microservices, and you got lots of different customers and you're trying to make progress on and go through.

A

You know advance the feature roadmap, so life is difficult, Ryan and the incentive that our service mesh champion said this was with linker D. We can run parallel service versions. Concurrently, it's gonna be really easy, because it's got this fancy routing thing and that way you can continue iterating and we've decoupled. The deploy process from you know the the fact that we have this or the development process from the fact that we have these long-running deploys. The concerns were around reliability like what's this thing as a blob?

A

Oh, how do we have to change our workflows and he addressed the concerns via testing and education right and and it's great now they are happy service. Mesh users they've been in production for over a year. So that's a happy one I've got some sad ones coming up. Okay, so that was failure scenario. One right! You have to convince your peers all right so now scenario: two! You convince your peers following my handy four step guide, but the rollout to production fails.

A

So we've seen this and often when we see this, it's because you're trying to take on too much right, you're trying to boil the ocean. You haven't addressed the risks that are going to be a part of this, or sometimes you haven't communicated the value or you're taking too long to communicate the value if it takes 12 months to do this well by the time month, 4 rolls around people might start wondering.

A

What's going on, okay, so quick case study here, multi-billion dollar financial company deploy the service mesh link, rudy, very complex roll out, multiple configurations, multiple environments, kubernetes non kubernetes, Bluegreen deploys and gen-x was in their hardware load balancers, just a very sophisticated stack and shortly thereafter started to experience problems and not just like. Oh, we got our like. That's great just got to change this config thing but hard to reproduce issues, issues that they didn't see in staging Ryan. It was very difficult to understand what was going on.

A

You could put mitigations in place what the mitigations really awful right and almost got to the point where they ripped linker, D out: okay, okay and apparently I have a typo on here. Luckily, we fixed things before this. Okay also emotion, emojis did not translate so those little squares are supposed to be sad, sad faces and then and then the cats are supposed to be kind of screaming cats. Alright, so the root cause was linker, D had some bugs. Yes, this occasionally happens, believe it or not.

A

Hardware load balancer also was misconfigured and had some like weird behavior, slash bugs, but those are actually the minor issues. The real problem was all the compounding factors: the fact that the environment was so complicated that there wasn't enough communication that was happening internally and externally, and also it was just difficult to understand: linker DS, internal state yeah. So what lessons that we learned from this number? One start small write: one configuration one environment, of course, you're testing this before you put it in the prod, but the reality is once it's in prod, it's different.

A

You never catch everything all right, so start small. There are product features that can help with this. So one of the things we've really tried to do with conduit is make it very easy to incrementally apply the service mesh to one service at a time. Okay, so start small, encourage communication. Sometimes it actually takes work to make different teams talk to each other, believe it or not.

A

Network team doesn't want to talk to the OS team, doesn't want to talk to the platform engineers, so sometimes you just have to like mash heads together and then good diagnostics. This is a real lesson for us, with, with Slinker d, you've got to be able to clearly reason about what the service mesh is doing when things go wrong. Okay, so this is another thing where product features can help.

A

So diagnostic I'd say: if there's one thing that we learned as service mesh purveyors as vendors service mesh vendors, the one thing that we learned was that Diagnostics are critical. You have to have this because you all the production bugs you're going to see are these hard to reproduce things that you know we. We can't take your environment and run it in our own. You know in our environment and and figure out the issue we have to rely on you to do so. You need Diagnostics, here's some screenshots.

A

If this is like the link, Rd admin, dashboard and here's like some of the state dumping features and conduit has a you know, a dashboard. That's just saying: hey! Here's the different parts of the service mesh and they're, all working or not working I've got a couple. Examples.

A

I don't spend too much time on this, but you know we have headers that kind of annotate when requests failed, whether that failure was internal or external, whether the party, a poor part of the service mesh we've got very extensive metrics that you know split things out by connection level versus application level versus mesh level. We dump as much internal state as possible because you don't need it normally. But when it's time to diagnose things, it's really helpful to understand what linker D is thinking in its brain. Human facing admin.

A

Dashboard got error, tracking fans, there's all sorts of fun stuff. You can do so this. This was a big lesson for for us as service mesh creators. All right failure scenario. Three, our thing is broken right, so you've deployed it to production. Congratulations! Our thing is broken. It must be the service mesh. Alright, this once you've introduced the service mesh. It's this new strange thing and it's it's in between all this communication and it's just gonna take a while.

A

It's just gonna take a period of time where people are blaming it, blaming it for everything, so I talked to a bunch of our production. Linker D production users and I asked them for examples, and every one had like three or four examples at hand of things that linker D had been blamed for that turned out not to be linked, reduce fault right. So a couple examples here: our app servers are failing. We've got this big prod outage must be linker D.

A

Well, the app servers running out of file descriptors, ladies slowing everything, down I, got this network lanes. It turns out. We had an MTU mismatch for a network sign. There's some network thing, I, guess that makes things slow services are dying, okay, well, the services had bugs, but they weren't logging, the bugs and linker D was logging all the failures. So, of course you blame, Langer, D connection. Timeouts turns out the hardware load. Balancers, we're running out of IPs and reusing ports are doing something totally wacky apps run regional unreachable.

A

We misconfigured our firewall, there's probably another 20 things I could put on here. You can't avoid this right. You have to you, have to just get through this.

A

You can be prepared for it right and, and the way to addresses is understanding what the service mesh does and what it doesn't do, use a service mesh that has lots of Diagnostics in it that you can introspect that you can debug and then, most importantly, you have to do some kind of outbound education right, everyone who is affected by the service mesh, all the stakeholders, in that you know in the first part of this talk, they need to be educated.

A

You need documentation, you should be doing luncheon learn, so you should be teaching teams not only what the service mission is doing about how they can diagnose issues themselves, and if you can accomplish that, then organizationally you'll all move past this period and everyone will become happy service mission. Thews iasts, like me, all right so far, so good we're on the last one all right failure scenario: four things go wrong and it is the service mesh. This does happen all right, and this is Morgan's third law to take down prod.

A

First, you must be in prod all right, so if you don't want this to happen, there's one way I guarantee you can avoid this permanently, which is never put anything in the prod all right. So this is gonna happen right, and this is not again, it's not serviced and meshes specific, any tech that goes into prod it's going to break and you need to have a plan for it all right, and we had a really great example of this.

A

Yesterday, with Oliver beedis keynote Anatomy of her production, kubernetes outage linker D took down a bank sort of, and if you weren't there, my brief summary, which is probably you know not quite accurate, but kubernetes, had a bug. Linker D had a bug. Manzo's mitigation for the kubernetes bug exposed the linker debug, and then they were human beings involved. So things got got complicated. What I think is really really good about. This particular incident is the fact that there was a public post-mortem and it was clear.

A

It was clear that there were lessons that they were learning from failure all right, so you've got to be prepared for us. You got to have real production failures, so remove the culture of blame right. If everyone is worried when something goes wrong, or if everyone's worried that they're gonna get fired, you're not going to have a healthy organization, learn from the failures and outages quantify trade-offs. The Google, a sorry book has a great section on this right. There you can, you can assign a cost to each successive nine of uptime.

A

You can make the system really really really reliable if you want to, but the cost goes up, you may not want to do that. It may be okay and it probably is okay for things to fail some percentage of the time and then understand the failure modes right.

B

A

Like you're doing with kubernetes, you got to do the same thing with with the service mesh right. Cooper days is gonna fail. You got to understand how that's gonna happen. You need to have a game plan for when that happens same thing with the service mesh. Okay, like I, said very little. This is actually service mesh, specific all right, so I'm gonna sum it up in six. Easy steps, number one make sure you're solving real problems. All right, not I, need a service mesh. All right, you got to have a concrete problem.

A

There find the stakeholders make them happy right by communicating by understanding what they're worried about by mitigating those concerns. I didn't talk too much about that, but there's a lot to be done. There roll things out incremental e, because you will find things in process. You've never found and staging or in testing get results quickly and communicate them. It shouldn't take 12 months to do this and then finally accept that things are going to go wrong and have a plan.

A

That's all of my wisdom. Two years of production service mesh experience distilled into one talk. We have a couple other real life examples for for linker, D and conduit. If you're interested in tomorrow during the linker D deep dive, we've had a whole bunch of customer lightning talks, Francie's, giving a demo of some of the debugging and diagnostic stuff that we've built into conduit.

A

The linker DC intro sig is going to have a kind of a hands-on workshop, and then we had a couple talks yesterday, which hopefully you caught but Israel shortly after this talk, I think immediately after this talk, Israel will be talking about linker D at Moulton, so we've got some real content for you. Alright. Thank you very much. I think we have some time for questions so please feel free to line up behind the microphones. If you dare, in the meantime, check out conduit that IO linker D that IO give them github stars.

A

We are funded entirely by stars at this point and yeah.

C

Awesome doc, so I have a question around separations of concern.

C

So you know when you, when you have a platform team, as we said that brings in let's say the service match and you're already doing micro services, and then often people like you know they debate the platform team debates, whether they should expose the developers towards, like the implementation of let's say, link OD like understanding D tab, and how does that work and all those implementation details or you can just tell them- hey like you were using DNS for service discovery between services, now you're, just using path based routing or something it should be just expose that it should.

C

We actually educate them on how link ID works, how the DTaP and syntax works and things like that yeah.

A

So the question was around kind of the separation of concerns. I think my suggestion would be first of all, it's very organizationally dependent, but my suggestion would be the more that everyone understands what's happening in the better, but the less that they actually have to do the better. So the ideal situation is the developers, don't even know that linker DS there right they understand I mean they know it's there, but it doesn't affect them they're, not like writing, D tabs or anything like that.

A

That's the ideal situation, but that they under, but that they understand that it's there and they understand the failure modes and- and so they understand when things go wrong, how to tell whether it's their stuff or your stuff. Oh yeah, that's a really good question cool.

C

D

In my mind, service meshes are kind of a paradigm change from layered load, balancing into distributed, load, balancing or reverse proxying, and that's something a lot of people. I talk to I have troubles, sometimes understanding.

D

How do you get people on board there I.

A

See so how do you get people to buy into the service mesh model when what they're used to press some buttons here when what they're used to is kind of hardware, level, load, balancing and and that sort of stuff, so the way that we've been most effective at doing that? That's a great question. Thank you. The way that we've been most effective doing, that is by talking about layers of abstraction right.

A

So when you're in a world that you know, there's there's like the layer, 3 layer, 4 world, we're what you, what you are thinking about, the kind of nouns and verbs are I've got IP addresses and I've got to get packets from here to there and so on, and then, if you shift up a layer of abstraction you're got to at the level services right and now your nouns and verbs are not IP addresses in TCP packets, but are rather you know, RPC calls or HTTP calls and different services, and there are lots of parallels there right.

A

So there is load balancing at both sides. There's flow control and you know at both layers, but it's at a different layer of abstraction and I. Think if you put it in that context, you know, and you draw the parallels and you say: look: the ideas are the same: we're still gonna have load balancing we're still gonna, have failure, handling we're still gonna have flow control, but it's at a different layer of abstraction. Then that helps to helps people understand because we're used to that as engineers.

A

We know that there's like seven other layers of abstraction and we're always adding new ones in there.

E

It's a broader question on a service mesh and potential security concerns which keep coming up so essentially service nation is a sidecar mode right, whereas if you use the client every is part of the same container, so one question that often comes up is: are there any additional security concerns due to introducing a sidecar versus if ever, if I'd just had the client ability? It's all part of the same container, yeah.

A

That's a good question: I, don't know if I have a great answer for that, so the question was: are there security concerns about having something run as a sidecar versus having it be part of the same kind of compiled? The same container I think there are implications to that I. Don't think that aspect is inherently more or less secure.

A

Probably a container security person will yell at me for for saying that. My my assertion is that by decoupling that code out of the application and by doing a bunch of the security stuff in the mesh itself, you actually end up improving things overall, because you have one place where you're enforcing things like policy, one place where you're doing TLS one place where you're consuming certs and and and all that stuff, so I think.

A

Overall, the picture is rosier, but specifically like what happens when you put stuff, you know the container boundary I, don't know, I'm sure it's fine.

E

Thank you probably.

A

B

Hi, thanks for your talk, I have a question: if I'm in the situation, where I've identified that my use case is probably a good fit for a service mass, you know multiple different programming languages, different environments, but I have a hard dependency in the form of legacy.

B

Cgi driven application, that's been in production for 20 years, a lot of micro services in the cloud. But there's like this on-prem product whoa. What kind of advice would you have for that kind of use case and getting that working with the service mesh yeah.

A

So that's a really common use cases, it's very rare. Occasionally it happens, but it's very rare to introduce any of this stuff in a vacuum or in the total greenfield environment. Most of the time you have existing legacy stuff and whether that's introducing the service mesh or introducing you know, orchestration or introducing containers. You always have to bridge that gap.

A

Do I have any specific advice about that. I think a lot of it comes down to. How are you getting traffic in and out of the service meshed kubernetes cluster?

A

So a lot of it comes down to kind of ingress and egress one of the things we see a lot of use honestly that off one of the things we see a lot of use when the features we see a lot of use of for linker D for the for this environment, for this situation is linker, D is able to talk to multiple service discovery, backends and kind of bridge them together into a unified things. You can configure link your D to you know when you ask it for service food. You say proxy.

A

This request a service foo, look that up in kubernetes first, and if it's not there, then look it up in zoo keeper and if it's not, there then refer to this file and it's or look it up in DNS, and then you can once you're in that world you you, you have you've kind of decoupled where services are from how to communicate with them and then, when you're moving things in and out of legacy and non legacy environments, life gets a little bit easier.

A

I think that's, probably the the most direct way of addressing this stuff. But it's a pretty that's a pretty general problem that everyone else alright. Thank you. Yeah.

B

A

Question. Thank you all right. Anything else, all right. Well, thank you very much for attending I'll be hanging out. There's a point and booth over there so feel free to stop by and tell me about all the things I got wrong and thank you for being a great audience.