Cloud Native Computing Foundation EnvoyCon North America 2021, 30 Oct 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Lessons Learned: Four Years with Emissary-ingress and Envoy - Flynn, Ambassador Labs

Description

Don’t miss out! Join us at our next event: KubeCon + CloudNativeCon Europe 2022 in Valencia, Spain from May 17-20. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Lessons Learned: Four Years with Emissary-ingress and Envoy - Flynn, Ambassador Labs

The Emissary-ingress (née Ambassador API Gateway) project got its start in 2017, with the stated goal of making it easy for developers to harness the power and flexibility of Envoy, in Kubernetes, without needing to be experts on either Envoy or Kubernetes. Realizing that goal has been quite a journey: to bring Envoy to non-experts, the Emissary-ingress team has needed to learn an enormous amount about how Envoy, from the basics of how it works and how to configure it to best effect, to how to build it, test it, extend it, and debug it. The experience has been by turns challenging, surprising, frustrating, delightful, and ultimately extremely rewarding: after four years, Emissary-ingress is a CNCF project running in thousands of installations around the world, bringing Envoy into production use for organizations that wouldn't otherwise be able to take advantage of it. Join Flynn from Ambassador Labs to dive further into the challenges we've seen, the many lessons we've learned along the way, and the things we think anyone working with Envoy should know.

A

I'm flynn from ambassador labs and I'll be talking a bit about our experiences working with envoy as part of the emissary ingress project, in particular I'll talk about some things. We learned along the way, some of the things that helped us out and, of course, some of the challenges we ran into if you're not familiar with emissary. It's an open source, self-service, kubernetes native resilient api gateway built to top envoy.

A

It can be used in a lot of different ways, but its bread and butter is managing access at the edge of your cluster envoy. Does all the heavy lifting of wrangling your data and emissary in turn, wrangles. The envoy emissary is a cncf incubating project and it forms the open core of the ambassador edge deck. If you're familiar with the ambassador api gateway name, emissary is the same thing. The name changed with the cncf donation to lay my biases for this talk on the table.

A

The way the emissary project uses envoy is that we take an envoy image, wrap custom control, plane, logic around it, and then that's the emissary image just to complicate things further. It's a custom built envoy, since there are still a couple of things that we need to use that are not in mainline envoy. So that's the lens through which I tend to look at envoy. However, I expect that this talk should be applicable to other use cases as well.

A

We started learning lessons about envoy actually before the emissary project existed back then, before ambassador labs was even called ambassador labs.

A

We used to host the microservices practitioners summits and, at our summit in january, of 2017, we're here to talk about this shiny new envoy proxy thing by this guy named matt klein that you may have heard of by now envoy came across sounding pretty cool, and when I got back to boston, I ended up taking a close look at how well it took to kubernetes turned out that it worked pretty well, but it was considerably more painful to set up than we would have liked.

A

So that was our first lesson about envoy. It was extremely powerful. It was extremely performant, but it could be painfully complex and, of course, you can say pretty much all the same things about kubernetes as well, so in pain. There are opportunities, and it looked pretty clear to us that we had an opportunity to make that experience of using envoy as a kubernetes edge proxy, really much more pleasant. That was the actual start of the emissary project.

A

We wanted to go through and set things up, so that developers could use envoy in kubernetes without having to spend time becoming an expert in either envoy or kubernetes, and really just making it possible wasn't good enough. We wanted to make it easy to state the obvious. This is a pretty big ask.

A

Taking a powerful, flexible, complex tool like envoy and making it accessible to non-experts is hard part of the reason it's hard just to state. The obvious again is that it doesn't make any sense to just take envoy slap, the name emissary on it and then lean everything else. The same that's kind of pointless.

A

Instead, emissary really needed to be its own thing, distinct from envoy, and given that envoy is very powerful and very flexible already, the things that could really differentiate emissary from envoy or from other tools built on top of envoy were not likely to be things dealing with our expertise with envoy or cool little technical things that we added.

A

They were much much more likely to come from our choices about the things emissary should make easy or the things emissary wouldn't support when what exactly the user experience with emissary was going to be like another counterintuitive thing we learned here was that in many cases, adding options really just adds confusion.

A

An early version of emissary allowed configuration using files on the file system or kubernetes config maps or kubernetes crds. We had all three choices, because we thought that people would appreciate having the options and, in fact, some of the more expert users did. But after a little while, we realized that overwhelmingly having those three options discussed was just confusing newcomers, and things really got much smoother with the product.

A

Once we removed the choices of files on disk or kubernetes config maps and just drove everybody towards crds, so, given that envoy is a complex, powerful thing, we were trying to make a complex, powerful thing accessible to non-experts in a meaningful way. That would actually add value for them, and we realized that that requires being opinionated.

A

So that was our second real lesson, although we needed to go out and gain enough expertise to form good opinions and to be able to act on them, the expertise was going to end up being secondary to those opinions.

A

That kind of meant that we needed to shift gears from thinking hard about the technology to thinking hard about the humans that would be using it. We ended up borrowing the idea of persona based design from the user experience world. For this we created a developer persona named jane in mid-2017.

A

Her ops counterpart julian came a little bit after that, but all throughout the project we've used them to help us think about how emissary is going to be used and to help us drive that process of forming opinions and distilling expertise down into value. We can add to this day the toughest thing that we do regularly working on.

A

Emissary is working out semantics that simultaneously make sense to jane and julian are explainable to jane and julian and still fit with envoy's semantics as well, and you know designing for the humans first always sounds obvious, but we constantly relearn how hard it is to really do it. Well.

A

So the first few lessons here, these are mostly things that we learned. While we figured out what our goals were going to be and put together some tools to give us a place to stand while gaining the expertise that we needed in order to actually execute toward those goals, actually gaining the expertise had its own challenges and its own lessons.

A

The first one we run into is that we, as an industry, have a big problem around the learning curve that we tend to throw at novices. Imagine for a moment, a new cook going over the range hauling out there cooking documentation and finding something like this. That gives them detailed explanations about what every control does and exactly what setting it's changing.

A

This is not what they need. It's all true information, that's very important to have available, but what the novice really needs at first is a sense of how to accomplish a goal that they have in mind. They need a set of instructions even better. They need a guide that can talk to them about the kinds of things that are even possible, with the thing that they're working with, because they don't have that.

A

Yet I'm going to show a simple example illustrating this with envoy, because this is a talk about envoy, but I want to be very clear here that this is not a place where envoy stands out. It's a ubiquitous problem in the whole industry. I could just as easily have pulled examples from kubernetes or from emissary itself or from unix or really any piece of technology. I've ever worked with, but grabbing one from envoy. There's a thing in envoy called a filter chain which is able to act on an incoming network connection within a filter chain.

A

There's a thing called a filter chain match which specifies which connections are going to match a given bit of envoy configuration.

A

One thing you can match on is the server name and there's excellent reference documentation on all of this stuff. What the reference documentation will not tell you is that the server name match doesn't work at all for http.

A

Once you get a more complete picture in your head of. What's really going on, this makes perfect sense. This level of thing this filter that we're working with is operating on a new connection, a new http connection that doesn't use tls and sni doesn't have a server name on which to filter. You have to wait to get the server name until you've read all the http headers and that's later, that's not the stage that's going on yet. So the challenge here is not that the information is wrong, because it's not it's perfectly correct.

A

The challenge is that it relies on concepts that a newcomer to envoy will not have internalized and really when you get down to it. One of the most important differences between a novice and an expert isn't that the expert knows how to do more things than the novice knows. It's that the expert has a mental model about what is possible, so they know where to look to figure out how to do it. The novice doesn't have any of that and again, I want to stress one more time. This is by no means unique to envoy.

A

We could pull examples from this straight out of the emissary documentation as well. It's really not about envoy or not just about envoy.

A

The lesson here is that again, the expert already has the mental model to make good use of the reference materials, but the newcomer needs a curriculum to guide them, while they form those models and creating that curriculum is really hard, although it's also really valuable first recommendation here, if you're, building on envoy, okay, so start with remembering that envoy is not likely to be something you swap into and out of your product, it's going to be more of an investment, so it'll be up to you to decide which areas you spend more time in and which you spend less time in make the docs a place.

A

You spend more time both in the sense of creating good documentation for your own product, but also in the sense of spending time with the envoy documentation itself to learn. What's going on.

A

With the documentation, envoys introduction and getting started, sections are really good. They tend to get overlooked. A lot you know go in, spend some time with them any time you're trying to build your own curriculum out of reference materials. It's an iterative sort of thing. You should plan to keep prowling around in the docs repeatedly, while you're learning- and you should expect to have your brain- feel awfully full every so often as well. It's all part of it. Building the mental models takes time.

A

Anything that we can do to improve the experience of building those mental models is a good thing for our users.

A

The next challenge we ran into is that envoy moves very, very quickly. It's not something where you just become an expert and be finished. You have to maintain your expertise over time yeah. This has to do again with the fact that yeah, it's an investment.

A

Just in the time that I have been working with envoy, I count 62 releases, we've gone through three major versions of the envoy api with a fourth in the works, and I literally have lost count of the number of new features that have been added. Things move very very quickly.

A

Given that speed, you will need to pay some more attention than you might expect to to deal with, bug, fixes and new versions, and things like that, if, like us, you're an envoy distributor, meaning that you build a product that incorporates envoy so that you're shipping, an envoy binary with your product, then this goes at least double, because in the case of a distributor, you have to pay extra attention to security releases, given that they always have a deadline, and they almost always require action.

A

So the most important recommendation here is that you always invest in keeping up with mainline envoy, especially if you're a distributor. I can tell you from sad experience that it's not really much fun at all to scramble to manage a security release when you've diverged significantly from the main line. In turn, that implies that it's worth thinking carefully before you commit to shipping a custom envoy build.

A

So that also raises a question: why did we decide to ship a custom envoy build historically? This is because of auth when we started there was not an auth service in envoy that did what we needed, so we wrote our own uh since then, our auth protocol has been added. It's the http protocol in the mainline x, auth, z, filter now, but by the time that happened, we'd added some other small tweaks, mostly around backward compatibility for our users and those can be kind of challenging to get into the main line.

A

So, although we're moving closer and closer to being able to use the stock, build it's going to take a little while, yet this permitted us to do a lot of things, but it has definitely been costly in other ways. If I were starting now, I'd probably look very very carefully at wasm before committing to hacking out on voice c plus plus code.

A

If you decide to go for a custom build anyway, keeping up with the main line goes at least triple it's difficult to manage security releases and go ahead and track new features, and things like that, if you have to jump across many envoy, commits at once much much easier to just go through and handle small numbers of commits at a time.

A

You should also be aware, when you're thinking about this one, that you will need to spend some time coming up to speed with on voice code base. It's pretty complex, envoy, uses c, plus plus quite carefully, but every c plus project has its own project. Specific idioms that you have to sort of get used to and envoy is no exception. There.

A

uh Envoy's code is also heavily heavily performance tuned enough, so that it makes certain things that you would expect to look really easy, actually look quite a bit more complex so that they can be faster. Another thing to be aware of is that building and testing envoy tends to really benefit from beefy hardware.

A

Within ambassador labs, we tend to run envoy, builds and envoy tests on a pretty beefy gc evm, with lots of cores and a ton of ram so that we can run the tests from a ram disk makes it really fast and it really drives home how painful it would be to try to run on all of my macintosh, and you know I said it before, but it's worth repeating think carefully about whether you really need to commit to diving into the c plus and shipping your own custom build.

A

um You can contribute to the main line without pinning your product to a custom, build that may turn out to be a better plan, especially as wasm and lua are better supported and more performant.

A

So finally, I would be remiss if I didn't mention a very positive lesson here, which is that the maintainers and the community around enboy have both been really really wonderful back in the time before covid ambassador labs was located closely enough to a bunch of the people working on envoy in cambridge massachusetts that we could and did get together with them. Talk over things get to know people. It was always a huge win.

A

The maintainers in the community have always been very gracious and very helpful and overall, a strong recommendation to get to know envoy's people it'll be a big help and, besides it's very pleasant. So to recap.

A

Lessons learned, yes, envoy, can be very complex if you want to use it without a team of sres devoted to it, you will need to be carefully opinionated and you'll need to be opinionated in ways that resonate with the humans that you want to support.

A

That is going to take a significant amount of time and energy to sort out all of that you're going to need to start right now by taking the available reference material and building a curriculum for yourself. That will let you develop the expertise that you need to do all this and you'll need to do it while the product is moving very rapidly under you.

A

On the other hand, you'll have a great set of people around you to help out with that in terms of recommendations.

A

First recognize that you're making an investment here with envoy be careful how you approach it obviously, but always invest in the documentation, both in the sense of studying the envoy docs being prepared to dive back into it more than once and in the sense of building your own docs to help your own users likewise always invest in keeping up with a mainline envoy if you're shipping, your own custom envoy, this is an especially big deal, even if you don't you're going to need to do it because the project is moving very quickly and it will help you a lot to not be behind.

A

Finally get to know the people they're great they're, a great resource, always a good thing. So, thanks much for joining in. uh If you are looking for more information, you can find emissary on github. You can always join. uh You can always join the ambassador, labs, public slack or just drop me a line at flynn datawire.io thanks much.