Cloud Native Computing Foundation KubeCon + CloudNativeCon North America 2018 (Seattle), 16 Dec 2018

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Intro: Envoy - Matt Klein & Constance Caramanolis, Lyft

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Intro: Envoy - Matt Klein & Constance Caramanolis, Lyft

In this intro to Envoy, Constance will conduct a live demo in which she: - Builds a simple Envoy configuration and explains what each component does. - Runs the configuration using the Envoy docker container. - Sends traffic through Envoy. - Provides an overview of Envoy's observability and administration output. - Covers additional resources for further learning.

To Learn More: https://sched.co/GrcN

A

So hey, my name is Matt Klein I'm gonna. Do a really high-level overview of envoy it'll, be an extended version effectively of the keynote this morning. So if that sounds boring to you feel free to give someone else, your seat, fair, fair warning, so you know, let's go into a little a little history of how on go. I came to be at lyft, so this is lifts of architecture about five and a half years ago.

A

At this point, so is a pretty you know typical model attic architecture, clients, internet, lift, is entirely cloud-based, so cloud load balancer at the time. Lyft was a PHP Apache monolith and a single database. So you know this is how almost everyone starts. It's a it's a typical architecture and what's interesting here is that even though this is a monolithic architecture, it's not you know. Micro services there's still problems that people are having at this stage.

A

You know that are more common or more commonly talked about, and micro service architectures, so networking observability we still have different components in this system. We still have database. We still have the application, we still have load balancers. So, even in this very basic architecture, it can be difficult to understand where problems are occurring. Are they and the database? Are they in the monolith? Are they in the load balancer?

A

Are they in the virtual infrastructure so fast forward about two years later and again, this is a common progression for for most companies that are going through growth, so we still have clients. We still have the internet. We still have the load balancer, but now we have it expanded to micro-services. So we still have our monolith monolith. You know it's been a little slow, so let's add some a tree proxy and some queuing we're now using a bunch of internal load balancers for service discovery. We now have two databases so MongoDB and dynamodb.

A

We now have probably twenty or thirty different micro services and Python- and you know this architecture compared to the one previous is probably an order of magnitude. More complicated there more hops, there's more places that things can go wrong. There's more virtual infrastructure, there's some black boxes in terms of those load balancers. So it can be very hard to understand. You know where things are actually breaking and what we saw at this time in lips' history, which is something that I hear from lots of different companies that are undergoing their micro service.

A

Migration is that you know they don't they want to move to micro services for the agility reasons, but then what ends up actually happening? Is it stalls because developers don't trust, Mike receivers? They think they're too complicated.

A

They can't debug problems, they don't know where latency is coming from and then you know you end up in the worst of all possible worlds, which is you have a bunch of Micra services that no one trusts and then you have your monoliths and developers just end up, adding all their code to the monolith and that's a pretty bad place to be so.

A

Let's talk about the general state of micro service networking within the industry, so you know, we've moved from a time in which most organizations are developing using a single language to a situation, whether through just internal desire or through acquisition. We now have organizations that have you know developed in five or six or seven different languages. Lots of frameworks with which people write their their services, so you know we have a. We have a real expansion of how people are writing their applications, lots of different protocols.

A

We have HTTP one HTTP, 2 ERP, see we have different databases different caching, so people are building their applications and different ways: lots of different infrastructure we've gone through on Fram to virtual infrastructure containers as a service.

A

Obviously we have functions lots of different ways that people are building things different load balancers, so we've gone from Hardware load, balancers to virtual load, balancers from cloud providers to you know software load balancers, so lots of different ways that people are doing networking between services and, most importantly, observability, so we're trying to debug these systems through typically stats logging and maybe, if you're lucky tracing and what you see from the rest of all of these things that are on the screen is that it's very hard to get consistent, stats, logging and tracing because all of these components, your different load, balancers, your different infrastructures or different protocols, they're giving you different stats so from a cognitive overload perspective very difficult to understand.

A

What's going on in your system and that leads to people being frustrated because they can't debug, then we move on to best practices, so we're all told that we need things like retries and circuit, braking and rate limiting and timeouts. You know do developers have them. Maybe you know, are they implemented differently in different? You know different frameworks and languages and libraries, probably and again that just leads to confusion and cognitive overload, authentication authorization. You know these are key concepts.

A

Everyone wants them, but sometimes you know, let's be honest: they can fall to the the backburner. So these are things you know that might have partial implementations or rollouts and coming back to the multiple languages and frameworks, we have multiple libraries that people are typically using to solve these problems. So twitter has finagle.

A

Netflix has history, those are both Java based libraries and then you'll see that people will, you know, develop different libraries for different languages, but those libraries that have different stats, different logging, different features and that can be very complicated from trying to get uniformity and making sure that everyone is doing the same thing and that everything is easy to understand as we're doing this traffic between hops.

A

So that leaves us here- and this is this- is where I think most organizations wind up after they start doing micro services without a firm plan around networking and observability, and it can get bad pretty quick. So that brings us to the envoy project and servers mesh in general. So the idea behind the project is that we want the network to be transparent to applications.

A

Obviously, problems are going to occur so when they do occur, you know, let's, let's make it easy for people to figure out the source of the problem, and you like I, say in most of my talks. You know we're all here to talk about infrastructure and to hopefully build infrastructure that helps people do their their job faster. But for the most part any time the application developers are, spending working on infrastructure is time that they're not writing business logic and that's typically, not a good use of time for most companies.

A

So our goal to the greatest extent possible, is to abstract a lot of the virtual infrastructure from folks so that they can focus on their business logic. So this is a service mesh refresher service meshes very buzzy these days, so I'm sure that many of you have have at least been exposed to this a little bit. The idea behind the service mesh is that we have a proxy that is co-located next to every application.

A

So in this case we have, you know, service, a B, C and D and there's a sidecar proxy, that's co-located with that service and then, when a service wants to talk to another service, it talks to its local proxy. That proxy is gonna, do common concerns around things like service discovery and load, balancing and timeouts and rate limiting, and and then it will talk to the other side. The other side might do some tracing and some logging and some stats and the nice part about this is that we can have common concerns in that proxy.

A

They can be quite complicated and it'll work alongside any application. Language doesn't matter what language that application is actually written, and so, if it's a very powerful paradigm, so from a non-void perspective envoy is an out of process proxy, so it would be most similar to an H, a proxy or nginx and meaning it's not a library. It's a self-contained Dame that you run and it has a config.

A

It is a modern C++ code base. This was chosen for performance reasons. It is at its core, it's an extensible, what we call l3 l4 filter architecture. So what this means is that there's a core proxy that does a bunch of things and then top of that you can write extensions or filters and at the l3 or l4 layer. This operates on bytes, so bytes in bytes out you can transform bytes and what that means is that it can.

A

It can be used to build up on top many different protocols, so envoy can support HTTP Redis DB by sequel. Thrift whole bunch of different things. You know, so that's a powerful construct that has made envoy very popular is that there's that core code and then there's a bunch of extensions, obviously HTTP is very popular. We build lots of HTTP services, so envoy has a lot of HTTP codes. So not only do we allow filters and extensions at the l3 l4 layer, but we also allow at the l7 layer.

A

So we can allow extensions that operate on headers and bodies and trailers, and this allows us to do things that make it easier for people to write extensions around authentication and rate limiting and routing and stuff like that envoy was written as an HTTP to first proxy and at the you know, most proxies now, there's still not quite all caught up, but but most of them are heading in that direction.

A

But this is quite novel at the time the Envoy was originally put out and by HTTP two first, it means that envoy was written as a multiplexing proxy from the get-go and they can proxy transparently from HTTP 1 to HTTP 2 or vice versa, and that allows it to be used for GRP see.

A

It has quite a few different built-in service discovery mechanisms so, whether that be DNS and our dynamic configuration systems which I'll get to in a second and it supports both active and passive health checking. So active health check is you know, sending a slash health check pinging to a host and getting a response and passive health checking is monitoring in band traffic, say looking. If we have three 500s in a row and saying, oh, you know I think that hosts probably bad.

A

Let me boot that host and and kick it out and we'll see later, if it if it's healthy again, so the combination of active and passive health checking can be very powerful, because active health checking can give a baseline health status, but passive chuckling, passive health tracking can give you very fast reaction speeds to when things go wrong.

A

So one of the things that that you may have heard of with envoy is we have our configuration. Api is what we call our XDS api or star discovery service api, and we built up a whole number of these api's over the years, and the idea here is that envoy, which is very novel for a proxy, can be dynamically configured. So it's been built to have a separate control plane from a data plan and we've seen over the last few years, many control planes written on top of on voice.

A

You probably heard of Sto now a bunch of the cloud vendors or offering their own hosted control planes so Amazon, just enough at mesh or Google has traffic director and Ashur has service fabric. These are all control planes that use the XDS api to configure envoy and there's lots of other projects that do the same. So this API has been very powerful and that's been that's been pretty awesome in terms of load balancing. We have lots of different load, balancing algorithms.

A

We do priority failover and help checking and spilling, and all types of stuff and observability right. So, as I said before, really when we're managing these systems, we need to understand, what's going on and having you know, copious stats, logging, tracing superpowerful and finally, from an edge proxy perspective. You know we would ultimately like to be able to use the same proxy both for service to service traffic, middle proxy traffic and edge proxy or internet-facing, because in reality these things do the same thing right so 99% of what these proxies do.

A

Load, balancing service discovery, they're all the same so from an operational load perspective just from a cognitive load perspective, much easier to actually go through and use the same software and operate it. So this is roughly what lyft looks like today, so we have our clients. Obviously internet now, you know we have Amazon Network load balancers in front. We run a fleet of what we call our front or our edge-on voice, we've actually recently in the last couple of months.

A

Finally, after years got rid of our PHP monolith, no more PHP, and now we have lots of ghost services, lots of Python services, there's a bunch of random, one-off other services, C++ Java, a node and then from the backing perspective, still using MongoDB, Redis, dynamodb, a whole bunch of external partners, stats logging and tracing and Envoy runs everywhere. We have a hundred percent coverage of on, but all traffic at live flows through envoy and backing it behind it. All is what we call our envoy manager. This is an XD, a server or a config server.

A

So roughly the way that it works is an envoy boots up with it with a base configuration. It says: hey I am this server and you know this availability zone and this auto scaling group give me my configuration and then I don't quite have time to go into it today, but we have an entire configuration pipeline that will inform you know what configuration that on Boyd gets and then that configuration can be dynamically updated in real time without restarting the proxy, which is very, very powerful, very useful from an Operations perspective.

A

So this is generally what what lyft looks like today. So from a Envoy deployment perspective, we have you know many hundreds of services at lyft. Now we run envoy, depending on time of day on many tens of thousands of hosts anywhere five, ten million plus requests per second that run through the entire mesh. The majority of it is HTTP to service to service traffic, all of our edge service to service most of our external partners flow through envoy.

A

So those are things, for example, like DynamoDB we're just having global visibility of applications that are talking to DynamoDB across our fleet makes it much easier to identify problems. Is it specific to a host? Is it specific to an availability zone? Is it specific to all of DynamoDB and there are countless examples of times where we have gone back and forth with AWS reporting issues and being able to highlight you know either global information or logs from envoy that have given us confidence?

A

It's an Amazon problem and it just makes that debugging back and forth much faster, and you know we're still, obviously sorry, and so that includes dynamo. We do all of our Redis clustered Redis through envoy, and that's many many many tens of millions queries per second against Redis left is a very heavy Redis user. So that's been. It's been a very nice system for us because it couples envoys dynamic configuration eventually consistent mechanisms with support for Redis and consistent fashion, we're still working on our configuration management system.

A

Since we built envoy at lyft, we've gone through many iterations from static configurations to you know, template eight configurations to our dynamic configuration manager. You know, and that's just the natural way of things so yeah in terms of adoption, as I said this morning and the keynote for those of you that were there, we open sourced envoy about you know to two and a half years ago, and adoption has been truly incredible. I think beyond our wildest imagination, we're now being used by all three of the major cloud providers.

A

Many large peer companies, there's lots of companies down on the show floor that are building their technologies on top of envoy. So it's been, it's been a pretty pretty amazing ride from a you know.

A

From a why envoy perspective I talked about this this morning, but I'll expand on it a little bit more is you know, I think we've been able to keep quality quite high, while actually still moving fast, so through modern development philosophy is by using git up by you know, doing CI and all that good stuff we've been able to keep keep high quality, but move fast, we've really getting focused on extensibility and I already talked about. You know filters at l3 and l4 and else Evan, but we have lots of other extension points.

A

We allow people to build their own stats, plugins and their own tracers and their own retry policies and and I mean there's, probably seven or eight or nine different extension points right now and we're always adding new ones. The general philosophy is that we want to build a core piece of code. That people can, you know build into interesting vertical solutions are I talked about the eventually consistent API, that's been very powerful.

A

The API used to be in JSON with it, with a JSON schema we've since moved to proto 3 proto 3, in my opinion, is a real tech. Success story. It's it's pretty awesome in the sense that allows you to have typed api's, but actually convert back and forth between a structured file and JSON and yamo. So we allow people to call these configuration. Api is using JSON or Yama if they want, or they can use G RPC and have it be structured and whatever language G RPC supports.

A

So that's been really fantastic and, as I was saying this morning on boy there is no enterprise envoy. It is a pure community driven project and, unlike some of our competitors that has frankly allowed us to make technology first decisions and not worry about. Oh, we need to save this advanced feature for enterprise product. So that's been really really fantastic and I think has, you know, made us very, very competitive and very very compelling for people to pick up and form a community perspective. You know because of the know.

A

You know the know, like enterprise version and just because of our goal of trying to bring on maintainer xand and have everyone be involved. We've built up I think a pretty amazing and passionate community, which has been fantastic. Yesterday we hosted our first envoy Khan. We had like 350 people really amazing can't believe that all these people showed up. So that was great. So, thank you. That's a brief. Intro I'm stopped there. We have plenty of time like five or six minutes.

A

Little QA I can pass the mic around or you can call out questions and I can repeat them. So thank.

B

A

Sorry, encryption niblet, so the question is: can envoy be used for encryption sure. In fact, we have a lot of people that are building mutual TLS solutions and various types of encapsulation systems based on envoy. In terms of you know, data going to a volume, I think it depends on what protocol that is, and you know I mean I- think it would depend on a particular use case, but there's no technical reason that envoy could not proxy data to some service and perform encryption on on on that services. Behalf.

A

Question is envoy, is typically associated with service mesh or sidecar envoy. We see it now being used in a huge variety of use cases. We see it being used on bare metal. We see it being used as an edge proxy. We see being used as a middle proxy as a standalone server or as a sidecar or as a daemon set. I think what we've tried to do with envoy is we're not prescribing a particular use case. We're building a building block that allows people to build a whole bunch of different solutions.

A

Within I was going to ask with it, so the question is challenges of adoption, and this is the question within lifts or outside of lift.

A

You know within lift I, originally I thought that it was going to be difficult, like I I thought that we were gonna build this thing and you know we had deployed as an edge proxy shown a bunch of value. We had deployed it on our monolith to to help with MongoDB proxying and a couple other things and I had really thought that we were gonna have to take the stick approach of like you know, let's, let's get everyone in the list and, let's you know with them until they they they upgrade, and that's kinda.

A

Why I thought was gonna happen.

A

I was very surprised in the sense that we made a concerted effort to provide people value like immediately, so we made automatic dashboards that worked if they were using envoy and we gave them special logs and abilities to debug and all these kinds of things, and what we saw frankly at lyft is that once people saw the benefit of particularly the observability between two points, we didn't it was mostly a carrot thing like I was really shocked, I mean it was not hard like we had people, upgrading I mean we just gave them instructions we had.

A

You know, took some time, of course, but did not really have to chase down very very many people on the external front from adoption. I think. What's the most interesting is before we open sourced. You know, I didn't know what to expect so. I thought, oh, you know, it'd be great. I would be so happy if we got one of lift spear companies say like Pinterest or Yelp or slack, or you know similar size company wouldn't be great if like one of them used envoy.

A

So before we open sourced, I went around and talked a lot his companies and gave presentations, and everyone told me you know this is great but like we don't have time to go and do this right now and I was sad and we, you know open sourced and didn't know what would happen and what was interesting actually from a public adoption perspective.

A

It was, it was actually the biggest companies that came first, so Google showed up within actually before we even open source they found out about somehow and like Apple, came very soon after and then other other big companies I. Think the larger companies had the time to invest in in in this began, and they saw some of the potential and they had more people or more projects that were dependent on it and then I think over time.

A

Just with the momentum of getting more industry, wait behind it and then just the control plane and an sto, and now all the products I think it's a. We just have a snowball that keeps going.

A

Sure so the question is, we see patterns of envoy. You know finding kubernetes services. How do you do it outside envoy itself actually does not know anything about kubernetes. Really. All we offer is these generic discovery API.

A

So there's an API around the endpoint discovery service which helps you find endpoints yeah, and when we first put out envoy, we would see a bunch of queries come in about well, you know: will you support in CS, or will you support kubernetes, or will you support docker swarm rule whatever, and what we found pretty quickly is that people stopped asking a because I think the discovery API is are simple: to use, they're, easy to understand and they're compelling and to very often, people want to customize the behavior in some way so like just supporting kubernetes ends up not being that useful people want to have some intermediate translation layer that actually, you know, has some control over what is discovered, so everyone just ended up implementing our API.

A

So you know, there's tons of examples out there now of people that have envoy running against almost all of these systems, whether it be console or communities or something else. But the envoy project itself doesn't doesn't support kubernetes, there's a question no I think or there.

A

The question is, in sto, we've seen a bunch of 500s during ungraceful termination. What do you mean by ungraceful termination.

A

Right, I can't really speak to sto. What I would say is that that's where a combination of passive and active help checking comes in so typically at lift. We run with both active health checking and passive, and typically when something gets killed, we use passive health checking to fail it out very quickly, so I would follow up with the is do folks. There was some here.

A

The question is we're going to deploy, should we use sto or should we just just use envoy? Unfortunately, that's a really deep question: there's no there's! No there's! No one answer there. I will give you a general answer, which is that envoy is a Swiss Army knife that can do a lot of different things like, as was asked before, people use it for edge proxy for service proxy for middle proxy I.

A

Think whether you choose to build your own control plane on top of envoy or use sto or now, there's console connect, there's that mesh that I mean there's all kinds of things right, I think that's a decision that you'll have to think hard about in terms of where you want to spend your resources and how tied in you want to be to a particular control point implementation. I would say quite honestly that to use on Boyd directly requires a fairly high level of networking expertise.

A

So I think you know I think that's that's what you have to weigh in terms of whether to go one way or the other yeah mmm right, but sto is a combined system that offers an opinionated control, plane plus envoy. So unfortunately, there's no quick answer. There I think we've time for maybe one one more yeah.

A

Sure, yeah yeah, the question is perf in packs, so how much does does envoy add? We don't actually officially announce perf numbers and the reason for that is that, with this type of software very hard to actually accurately measure perf, it depends on the features that you're, using like the system that you're running on a bunch of other stuff, I'll, say two things. I think in many are most use cases. We see people measuring overhead in the sub millisecond range, so high microseconds to one to one millisecond range overhead per hop.

A

I also think that there are some people within the community that are doing some really incredible work around performance and envoy. So there's the cilium project. That's using BPF, there's talks at this conference about it and they're doing really what wild and crazy stuff to really minimize a bunch of that overhead. So I guess my standard answer here is I. Think everyone needs to measure for themselves to see what the overhead will be answer to. The performance is always getting better answer. Three.

A

Typically, the features so far outweigh the performance loss that I think for all, but a tiny number of installations. It just doesn't matter cool. Well, thank you. We are doing a deep dive in like 20 or 30 minutes in here, where we'll just be doing pretty much QA for the whole time. So, if people have additional questions, feel free to stick around Thanks.