Cloud Native Computing Foundation EnvoyCon NA 2022, 28 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Leveraging Envoy to Implement Micro-Segmentation-Based Security Policies - Hermann Lueckhoff

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at https://kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Leveraging Envoy to Implement Micro-Segmentation-Based Security Policies - Hermann Lueckhoff, JP Morgan Chase

JPMorgan Chase application security architecture follows least privilege network and micro-segmentation principles. For instance, incoming requests from external users need to be validated in a designated security zone before it can be forwarded to upstream endpoints in a different security zone. Communication between these segments is highly regulated and involves various identity providers and different levels of authentication and authorization checks including token validations and exchanges.

In this talk you will learn how we deal with these complexities leveraging standard Envoy routing capabilities as well as Envoy filters such as JWT Authentication and External Authorization. AWS X-Ray Tracer is leveraged for added observability. For our token exchange requirements we utilize the External Processor filter with a Golang gRPC implementation leveraging Unix Domain Sockets (UDS) for improved performance and robustness. After validating the authentication status for a given incoming request we mint new tokens and inject them into the upstream request. The External Processor filter also us a clean way to logically separate standard routing requirements from very specific token exchange needs.

Envoy has become a strategic tool for operating in an elevated security requirements and the resulting additional traffic management complexities. We have been able to replace expensive, inefficient, and hard to maintain custom proxy implementations with Envoy and the External Processor filter. As our teams investigate Istio adoption, Envoy provides us added long term viability since we should be able to port our custom extensions into a service mesh environment. We have realized substantial cost savings on top of improved performance, agility, resource efficiency, and maintainability. Based on initial interest from other teams we see our Envoy-centric traffic management approach as an evolving pattern in our broader organization.

A

um Hello, my name is Thurman lukoff I'm working for JPMorgan I'm super happy to be here and uh before I forget it. If, if anything, I want to get across today is really saying, thank you for the community. Your work is so useful for us, and so many and sometimes really unexpected ways, and that's really the main topic uh for my talk.

A

um So this talk is not so much about technical Wizardry. It's really a perspective from an application team really at the end of the NY food chain and um I want to talk about a few application patterns where NY really is, is very, very helpful for us to do our job um since I'm working in the financial industry.

A

We have a very healthy sense of paranoia when it comes to security, so security is front and center and I want to talk a little bit about how Envoy is helpful for us to meet our company internally security standards, um but I also want to go beyond that and talk a little bit about other application patterns where we utilize Envoy and where it has been yeah again, very, very useful for us. So for us, it's really not only about I, don't know service match and and the things that are in the current discussion for us.

A

Envoy is really really helpful in in some more down-to-earth scenarios.

A

um So, if we look at our company-wide security policy, one really um very generic pattern is segmenting our service landscape into into zones and these zones came into place for different reasons. Sometimes you have historic reasons. These are applications developed, 10 or 15 years ago on different platforms, so they are physically located in different clusters and different server Farms.

A

You have applications that have different security requirements, regulatory requirements, compliance requirements, so they are separated out and if you have an application that wants to consume these Services, you have to go through a certain step or sequence of steps. So, if you start Downstream, we have um applications coming in human to app. We have app to app on the right side. We have requests coming in with different levels of authentication, with different levels of um authentication through different idps.

A

So we have a huge landscape at JPMorgan, with half dozen of idps or IDC based older ones, Legacy idps and we have to deal with all these kind of requests coming with different kinds of um tokens, and if we, if we start Downstream, we we get a request coming into Zone one. The first thing we need to do is to validate this this token and of course we can use a prox NY here, but we also need to actually exchange these tokens before we send it Upstream um and by going from a level to level.

A

Of course, we go through to Fireball, so we have a very regulated tightly regulated process to open up fire firewalls, um and so the the really the story is, if you go from Zone to Zone, you have to do that revalidation every time. If you start with Zone zero, you have to do the token Exchange um and uh yeah. So the question is really: how can we do that in an efficient way.

A

um This is a little bit going more into the details. um We see uh coming in a requests with uh with a with a jot, uh assuming that uh request has been already uh authenticated and then, of course, uh once the request hits Envoy. It goes to the filter chain and I just point out here: two different filters. The first is the jot validation filter, which is uh extremely helpful for us, because we have all these kind of different idps, uh different jot validation, configurations that we have to validate.

A

So we can handle that all uh without code, just with configuration and then the second step is really that token exchange and that's where we're using the external processing filter, uh which I think is a pretty recent addition to the filter Community.

A

um This filter has been really really helpful for us. It allows us to create a grpc server that communicates with the with the envoy core um and all our token handling token exchange logic is, is part of that grpc server. So we have different scenarios. We might want to create a new token from a different IDP or we have like custom self-signed jots. So it really depends on the use case, but we can really uh include all these kind of different variations um very, very convenient conveniently in that grpc server.

A

um What is really great is that um this Envoy supports UDS, Unix domain services, sockets, sorry and the really really nice thing about it. It's so simple to set up um I just point out here, the the the the Golan code is it's literally one line of code that you have to change going from socket to UDS um and in the envoy configuration it is very similar um and UDS is just a little bit more performant, just a little bit more secure, um just a little bit more resource efficient.

A

So since all our requests go through uh Envoy, it's very very um helpful.

A

um Looking a little bit um into this grpc server, I think if I just look at it at our requirements at at my company, um I think there's actually a potential to further streamline that um maybe having some kind of uh Envoy filter not only for the validation but actually for token Exchange, and we are. We have actually started to think about a way to parameterize token exchange like making it configurable, um for instance, defining and configuration how claims from incoming jobs float into uh the the outcoming jobs um or caching Etc.

A

So it is something that we are thinking actually very uh intensely right now to to um yeah, make that more more configurable and more generalized uh to make it more usable for a broader audience.

A

um But there are also other use cases they want to touch upon and they might look very simple, but they are really really useful.

A

So the first of all, first of all, that I want to mention is cloud migration, so we started out in a private Cloud, kubernetes environment um and we have been moving uh ours, our services incrementally from from private Cloud, to serve uh to public cloud and it's almost trivial but uh of course having all the traffic going through through Envoy and having I mean being able to configure invoice so conveniently to go from left to the right. It's been tremendously helpful to manage this migration, which was a half year effort.

A

um A little bit more interesting is a scenario where we actually uh unexpectedly saw a lot of value in Envoy, so we got a request from our product team, saying hey. We want to create a demo system for our application, um which, by the way, is a story at jpmorgan.com, our uh One Stop uh shop for uh Commercial Real, Estate Investors in uh in JPMorgan. So if you own uh I, don't know commercial real estate property, please stop by um and uh yeah.

A

So the demo system is actually very similar to the to the real system, but it has certain restrictions, like um a user, get invited to it with a custom job.

A

Certain requests are redirected where kind of static data is is, is provided uh certain postcodes modifying calls. The writing calls are Unite, and these are actually all things you can very conveniently handle just on a configuration level with with Envoy.

A

So we ended up to be able to implement these features very very quickly um with minimal changes to the original application code, which was great and then the third one I want to mention. Is it's more about our own development life cycle?

A

um So we have multiple teams working on our application, so we have more than 100 developers right now, working on that on that site um and we have different application teams.

A

We have also what we call a foundation team that builds services that are used across different applications and we're looking into ways to horizontally and also vertically separate these teams in a way that they don't step onto each other's shoes and what we came up with is actually approach where each team gets its own Envoy um like, for instance, the application team won, gets that team one Envoy and they can actually route their traffic to their own space, which is, in our case a kubernetes workspace namespace with some kind of predefined pre-loaded services, pre-deployed services and the team can then route.

A

These requests to different I, don't know feature branch deployments and more mature versions of the foundation, speed space and similarly team two um has its own envoid um and can do the same thing on the on the right side and finally, We have basically the the main Envoy, the main application Envoy. That really then uh points to the um uh um Quality Tollgate, past versions of the services in the different applications teams, as well as the foundation space.

A

So that has been really really useful for us to really separate these development efforts to give every team kind of a safe space where they can experiment um without again disrupting the work of other teams.

A

So to conclude, um I mean Envoy has been really really useful for us in, for us sometimes really unexpected ways, um but there are also challenges, and let me actually talk first about the challenges, and the one thing I really want to mention. First and foremost, is the complexity of configuration and I. Guess it's it's easily uh underestimated in a community that is so deeply.

A

um Yeah steeped into into Envoy, but for normal application developers it can be really really challenging to do even simple things in an Envoy configuration and since everything goes through Envoy, it's also very easy to screw up things very quickly and very fundamentally so. For us, it has been really challenged to find a way to somehow regulate uh the way. How we deal with our main Envoy configuration, which is by now more than 1200 lines of yaml code and that's by itself, is already a challenge.

A

So we thought about different ways. So, right now we are primarily looking into building a simple CLI that allows to to execute day-to-day commands like adding a service, removing a service, rerouting a service really trying to avoid a normal developer, really touching the the domain uh uh Envoy yaml file, which really becomes like, like a I, don't know almost like a sacred cow in our development environment.

A

So this is really the kind of main problem for us um and I'm really curious to see whether there's any uh other person or any other team that has similar uh issues or proposals to to get around it. But the benefits are really really very, very significant and um very, very helpful for us um and for us I think it's not really about wow um Envoy is amazing and um it's it's um mind pending is really the All.

A

The Small Things uh coming together well thought out things whether it's the filters, whether it's the routing rules, um the configurability, the extensibility, really the overall package makes it so valuable for us um and again for us as a as a team in the financial services it all starts and ends with security and dealing with our company internal security requirements. uh Envoy is just really Priceless um and we're using really a lot of uh the filters that are available there.

A

I mean for us it's first and foremost, that I would say the Trinity of security related filters, jot validation, um the jot external authentication, which we're using very very extensively and the external processing, with our own ways to to inject our tokens and looking at um beyond my own team, which again is an application team more into our broader company, Community I think there's a huge uh potential that is very easily overlooked to um to uh standardize reverse proxy and and security settings with with proxies.

A

Until recently, our our official recommendation was to use Zuul or spring cloud implementations, and we have dozens, if not a three-digit number of different custom implementations, so I think using Envoy and having that level of extensibility, where you can really put all the kind of f-specifico companies specific aspects is, is a very appealing uh pattern for us um and what we would like to see moving forward is a lot of things, but I really want to uh limit it to one point: there's a lot of discussion about NY as an uh Ingress controller and I mean all that stuff that we do with the filters.

A

um We could easily put that directly into an Envoy uh Ingress controller, but only if filters are actually exposed so I don't know the current state of the discussion, uh whether that's something um uh yeah discussed as a as a possibility.

A

um I think for the for the istio. There is there's a possibility with um um to to actually include filters, which is a little bit um complicated, but still it's possible. So that would be something that would be very, very useful for us. So then we don't need our own NY instance anymore. We could actually hook directly into the uh Envoy Ingress controller, so yeah, so that's, that's uh all I have again. Thank you. So much for all your hard work. um It's it's uh great for us um again!

A

It's it's not really at the Forefront I mean Matt's, saying uh is Envoy getting boring, I, don't get calls and Monday morning from my CDO say: Herman how's it going with our reverse proxy uh strategy. It's really something much more a matter of um saying sustainable engineering and I. Think in that way it's been great for us. Thank you very much.

A

Yes, we looked into webassembly and yeah, so the question is about whether we have looked into webassembly um and yes, we have been looking into webassembly for us. Webassembly is um a little bit problematic, because the languages that we're using, which is mainly Java and golang, is not are not that well supported, particularly if you want to use libraries. So for us the grpc server approach was much more accessible and and usable yeah.

A

You had a question.

A

Yeah, so um to be to be to be honest, uh we didn't really do um detailed measurements, so uh it it's more like um I would say common sense. If you, if you can cut out that that Network layer in in the IPC in the inter-process communication, it will be faster, but I don't have I, don't really have numbers.

A

What I can say, though, is that we had initially um kind of weird problems with with the filter, with the external processing filter before using UDS, so we were running out of uh sockets, which was strange because you have that setting where you can set the the um the the the socket threshold um and all these problems went completely away.

A

The moment we switched to UDS and we're running this uh this this filters in production since since half a year more than half a year without any hiccup, so we're super happy with with that yeah yeah. Oh sorry,.

A

Yeah, so the question is: do we run our own control plane and how users push configurations uh to Envoy? uh That's exactly the kind of challenge that we that we have so uh right now our NY is just a kubernetes part uh and we're running a static, config map configuration and, of course, the changes to this of these configurations go through so git and and PRS, but it is, it is yeah, really the the kind of weak spot, I would say right now and the more developers we are on board to our foundation layer.

A

The more um I think this problem is is is is growing, so this is really something where we yeah. Looking for inspiration.

A

Yes, yes, exactly 1200 1200 lines of yaml code right now and yeah. That's that's almost self-evident that this is not a good State and uh we actually expect uh going from 100 to 200 developers in the next one or two years, so it will only grow.

A

um That's why we're thinking about custom CLI that that might abstract things out, or maybe a custom crd that allows us to abstract out and and bring specific application semantics into that that crd, um but yeah, it's it's it's it's! It's still an open wound for us or a sore spot. Definitely.

A

Yeah, so we we don't do, and so the question is: how do we push our configurations into our runtime system? Are we doing um uh heart restarts or are we just pushing the configuration and I mean? In our case, we just push the the configuration into our kubernetes and we do a rolling update, um but we don't use x, our dats or or any of these kind of more fancy mechanisms.

A

A

um Because we really started simple um I think in our team might be also a a matter of uh skill set and and experience. So again we are really an application team and we had some folks who had some initial interest in in NY and got into it. But I would say 95 of our developers.

A

Don't have any uh knowledge or visibility for for um or Android doesn't have much visibility to them so um yeah we might, we might look into it, um but uh right now it was just I mean our primary concern was to get something up and running um quickly and and yeah Implement. Our our patterns over there.

A

uh So the question is: how does NY fit into our security posturing?

A

um Do we do anything with buff um or that matter, so for us um perimeter, security is is handled by our infrastructure.

A

um So we we have a company uh internal layer that that is managing all that stuff for any kind of application within JPMorgan.

A

um So the answer is no. We don't use NY, at least to my knowledge. Those guys are very secretive, so I don't actually know exactly what they're using so for us. Envoy really comes into play again when it basically hits um our kubernetes cluster yep again. Thank you very much, uh and it's great to be here.