solo.io SoloCon 2022, 5 May 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Improving Application Security with Gloo Edge

Description

SoloCon 2022:
Improving Application Security with Gloo Edge

Speakers:
Nick Schuetz
Field Engineer, Solo.io

Jim Barton
Field Engineer, Solo.io

Session Abstract:
Application network security flashes in the headlines on a near-daily basis in 2022. Gloo Edge is a cloud-native API Gateway that offers a number of security-enhancing tools: transport layer security (TLS and mTLS), rate limiting, user authentication via OIDC, and authorization with Open Policy Agent, just to name a few. This talk will discuss the security tools and techniques available to Gloo Edge Enterprise users, plus drill into a recent example in the headlines. We will discuss and demonstrate how Gloo Edge’s web application firewall (WAF) feature can be used as a key weapon in your arsenal to block log4shell attacks.

Track:
Edge and API Gateway

A

Hello: everybody welcome to improving application security with glue edge. I'm nick schutz, a field engineer here at solo.I o and I'm here with jim barton, who is also a field engineer here at zelda.

A

Together we are going to be presenting I'll start off with glue edge security at a high level, and then jim will move on to a real world use case and demonstration.

A

Then, finally, we'll go into a q a session.

A

So if, at any time during this conversation, you have any questions in chat, we'll try to do our best to answer them in the chat and at the end, we'll be doing a live q. A.

A

Blue edge is a modern cloud. Api gateway built with envoy proxy.

A

Why envoy envoy is owned by a neutral foundation? The cncf? This also has a huge, vibrant, open source community.

A

uh It's also built from the ground up to be cloud native and it's being built on c or built with c-plus plus, which makes it highly performant.

A

It's also been around for a long time now, and it's stable and mature.

A

uh Glue is the control plane for envoy um glue is responsible for managing that configuration.

A

With glue you can get enterprise support um for envoy, we address bugs cves and back porting at a much faster rate than you might see in the open source community.

A

Although we do offer an open source version of glue edge, we also offer an enterprise version as well.

A

We have a proven real world. We have proven real world experience, architecting implementing and supporting envoy environments at solo.

A

In addition, glue edge is git, ops friendly being that we are kube native we've built custom resources to assist in making complex envoy configs easy to implement and understand.

A

This decreases the the probability of human error, which can lead to downtime and potential vulnerabilities due to misconfiguration.

A

We also support a decentralized api ownership concept.

A

Gluex supports a delegation model that allows service teams to own certain aspects of the configuration while allowing a centralized api team to own some of the security and operation aspects.

A

Blue edge's api enables developers to focus on concerns that they most care about, such as, like routing tables, uh re, writing, urls transformations in offload the responsibility of configuring, security and operational aspects to another team, more suited for that compliance, etc.

A

Here are some of the topics we'll be going over today again, this is just going to be a very high level.

A

Network security- encryption is incredibly important, as you probably already know, especially for public facing or those services that carry sensitive data. Blue edge can encrypt traffic coming from external clients over tls slash https.

A

We can also configure glue edge to do mutual tls with external clients as well. On top of that, you can configure glue edge to use tls or mutual tls or connected to upstream services.

A

In addition, blue x can also auto discover ssl configuration for upstream tls connections using annotations on the kubernetes service. This can be used as a convenient alternative to the upstream to configure upstream or client ssl.

A

Finally, blue edge and envoy communicate through the xdxds protocol, which, by default, is in the clear envoy. Configuration can sometimes can say, contain secret data, so we don't want that in the clear.

A

So here we can use mutual tls to secure the xds communication between blue edge and envoy and validate both parties in in the session.

A

So authentication and authorization lots of workloads need to accept incoming requests from external users. That's just a fact: glue edge provides several mechanism for mechanisms for authenticating requests.

A

In particular, gluex enterprise includes an external auth or xoth service that has built-in support for authenticating, with identity providers over ldap or oidc. It also supports other forms of authentication, including basic basic auth and api keys.

A

You can also use json web tokens to authenticate requests and, finally, you can write your own custom authentication service and integrate with glue edge. There are lots of integrations for external authentication, as you can see in this slide.

A

Choose your poison.

A

So in kubernetes glue stores configuration as custom resource definitions or crds, because these crds are kubernetes objects. You can use normal kubernetes role, based access, control or rbac to create a policy that grants users the ability to create glue adds resources such as virtual service, such as a virtual service object.

A

However, rbac only allows administrators to grant permissions to entire objects with the open policy agent or oppa. You can specify fine grain control over blue edge objects.

A

um So if you look at this example here you have a standard. Kubernetes are back controlling a user john example.com, and it's allowed to create a virtual service.

A

If you prop opa on top of that use user, john example.com is allowed to create virtual servers services, but those that only point to the domain, example.com and note you can use both kubernetes are back in oppa together. If you like,.

A

Rate limiting api gateways act as a control point for the outside world to access the various applications running in your environment. Incoming requests could possibly overwhelm the capacity of your upstream services, resulting in poor performance and reduced functionality, instead rate limiting at the application of great weaponry at the application level.

A

We can now offload that onto the api gateway there, we can define client request limits to upstream services, ensure that limits are enforced consistently and protect the services from becoming overwhelmed all from a single control point.

A

Enhanced options are also are available on in glue edge enterprise for features like rule priority rule priority, simplify simplified api custom data, store, backing and rate limit, metrics.

A

A common use case for the api gateway is to produce access logs or audit logs. The entries of an access log represent traffic through the proxy.

A

The access log entries can be customized to include data from the request, the routing destination and the response proxy can be configured to output. Multiple access, logs with different configuration.

A

In addition, glue edge access logs can be enabled and customized on a per envoy listener basis.

A

Data loss prevention is a method of ensuring that sensitive data isn't logged or leaked. This is done by doing a series of regular expression replacements on the response body. Blue edge has a dlp feature that allows you to scrub sensitive information from these response bodies by default, dlp will run regex replaces on the response body. However, if you have access log or if access logging is enabled and configured, the dlp actions can also be applied to the headers and dynamic metadata that is logged by the configured axis. Loggers.

A

Cores modern, browsers use cores in apis to mitigate the risks of cross origin, http requests, cross origin, resource sharing or cores is a method of enforcing client-side access controls on our res on resources.

A

uh Glue edge allows you to configure your virtual server objects to inform clients of these access controls. As you can see. In this example, we have a virtual service that, through course, parameters will inform your browser that should allow get and post calls from services located on https wildcard.glue.dev or a static url https solo.I, o.

A

um Also note that glue edge enterprise offers binaries and images that are built with fips compliant crypto. Libraries, with our enterprise subscription.

A

And then glue edge, enterprise offers a custom envoy, filter web application, firewall or waff built on mod security. uh Jim's gonna go more into depth on waff in a in a valid use case for it during these times now I hope you enjoy it and with that here's jim.

B

Thank you, nick and good day, everyone. My name is jim barton and I'm a field engineer with solo.io in the first part of this talk nick surveyed, the larger security landscape within glue edge in the second half. I want to take you into a deeper dive on a specific capability that addresses a particularly hot topic, one that went from zero to 100 in the space of about 24 hours last december,.

B

In late november of 2021, the alibaba cloud security team reported a zero day, remote code execution exploit that targeted the wildly popular log4j logging framework by early december. This exploit had been weaponized and deployed widely across the internet.

B

Shortly after that, articles came out that began screaming that the internet's on fire, and so but a number of systems from prominent tech organizations were compromised and emergency response teams from around the world were activated uh to respond. So this is a really serious issue. In fact, that's why the cve for the log for shell vulnerability was given a criticality score of 10 out of 10.

B

I thought that some of the analogies that were published in the wake of this exploit were were kind of entertaining here's one from a security engineer at cardinal health. How I explained logs for shell to my to log for jay to my mother.

B

Imagine there's a specific kind of bolt used in most of the cars and car parts in the world and they just said the bolt needs to be replaced and then glen thorpe from rapid, seven added this, and the presence of that bolt allows anyone to just take over the car, pretty serious stuff right. um Here's another one I thought was was pretty good. Imagine they suddenly discovered that that scotch tape spontaneously explodes you throw out your scotch tape rolls.

B

But then you realize you have tape everywhere, um and this actually, this analogy highlights one of the more insidious aspects of this vulnerability. You might assume that you can simply scan your own systems and look for direct use of a vulnerable version of log4j, but there's much more to it than that. Not only does direct use of log4j puts you at risk, but any but any system that depends on a library or maybe even an external service that uses log4j could also be exploited.

B

So how does a log for shell attack work? This diagram from the swiss government's computer emergency response team illustrates the anatomy of an attack quite nicely.

B

The key to this attack is that a special string is passed to an application via a common mechanism, maybe a request header, maybe a parameter on a request path or maybe via a posted message payload, and that string conforms to a specific syntax that log4j interprets as a request for a lookup substitution.

B

Now, most of the time, those substitutions are completely harmless. For example, log for j supports a time lookup, so you add a message into the log saying that you want the time of this message to be displayed in a particular format. Log for j simply replaces your time request with the actual time when the when the message was logged.

B

But what happens in a log for shell attack is that a different type of substitution is requested, most commonly the attacker requests, a jndi substitution, although other protocols like rmi, have been exploited as well, and when an unpatched version of log4j sees this jndi substitution string in a log message, it interprets that, as a request to look up a java object on a remote server.

B

If that server is controlled by an attacker, then it can serve up malicious java classes that will be executed on the application server with its with its permissions, and when that happens, the attacker can potentially exfiltrate sensitive data from the application or just generally cause a lot of mischief uh on the on the compromise system.

B

So, in short, if number one, the attacker passes in a malicious string through a common web application parameter and number two. If your java application logs that string or potentially passes it to a java library that logs the string and number three, if your log4j instance interprets that malicious string and then uses it to execute a remote request, then your application may be dangerously compromised.

B

So let's turn our attention to some remediation strategies, and we really want to answer two questions here, number one. You know if the internet's on fire, how do we douse the flames right? How do we fix the immediate problem, but number two and equally importantly, how can we prevent fires like this in the future, because this is not the first attack of its kind and it certainly won't be the last.

B

I've been around long enough to remember things like sql injection attacks that have that have been around for about a quarter century now and they operate on exactly the same principles as log for shell attacks. A malicious string is passed through a valid input mechanism and it causes an unintended execution sequence in the application.

B

Javascript injection attacks follow this pattern as well. So how do we stop it number one and number two? How do we prevent it going forward or at the very least, how can we lay a foundation? That's going to make it easier to stop it the next time? Well, at a fundamental level, we have three opportunities to respond to these attacks number one. We can intercept the malicious request when it enters our system that leads us to technologies like web application.

B

Firewall that we'll discuss in more detail shortly number two: we can interrupt the processing of the malicious request in our application servers. We might do that by patching logs for j or perhaps reconfiguring it to to disallow remote lookups and third and finally, we can block the outbound requests that attempt to reach the attacker's server, pretty simple right: we either catch it at the ingress point. We interrupt its internal processing or we stop it at the egress point. Those are our three options.

B

So, let's begin by considering this middle option of blocking the request or blocking the processing of the malicious request internally, I start here because that's where I've seen a lot of focus in the in enterprises in the wake of this of this exploit being published, and it involves activities like asking developers to inventory their applications and identify vulnerabilities or perhaps automating that activity using a variety of scanning technologies.

B

Then once vulnerable systems are identified, there's a process for applying the fixes that might include upgrading to newer log for j libraries or reconfiguring the version of log4j you're already using perhaps even modifying some application code to avoid dangerous features. And then, of course those changes need to be functionally tested, perhaps penetration tested and then ultimately deployed.

B

So I think we can all agree that these can be pretty time-consuming and labor-intensive endeavors. So don't get me wrong. These are really important activities and and they need to be carried out. But if you need rapid response to a zero day exploit like log for shell, it's not the most efficient approach taken simply by itself.

B

So let's look at another strategy and that is blocking the outbound requests from the application to the attacker's server.

B

As you recall, a log for shell attack depends on log4j interpreting a malicious input string as a request to access a remote server. So if we can apply policy at the network egress point, then we can potentially stop these attacks without modifying the underlying systems. So this is a pro.

B

This is an approach that can be both effective and efficient, but the challenge is that in many organizations, there's not a great mechanism for applying policy like this at the egress point, it's more common to be able to apply policy at the ingress point, especially with technologies like api gateways, but there are emerging techniques for establishing policy like this. One of them is through service mesh service.

B

Mesh technologies like istio, especially when combined with a management plane like blue mesh istio, makes it really easy to assume a zero trust security posture when it comes to network egress, and that makes it a great vehicle for blocking outbound malicious requests.

B

Now, in this talk, we're focusing on blue edge security, so we won't dive any deeper into this particular approach, but if you're interested in learning more, let me refer you to the blog, that's linked on this slide. It delves into details and presents a working, a working example. You can also check out the lightning talk that my colleague will mckinley and I will be giving later in this conference, where we'll describe this technique for addressing log for shell in more detail.

B

So so far, we've discussed two approaches to counteracting these attacks. One was to block uh processing of the request internally, which we said could be effective, but might not be the most efficient approach when responding to a zero-day vulnerability.

B

Number two was to block outbound requests to the attacker server, which could be both effective and efficient. If you have the infrastructure in place to apply policy at the network's egress point.

B

Finally, let's discuss a third approach, which is to block requests at the ingress point by leveraging the enterprise edition of blue edge api gateway, specifically we'll look at the web application firewall facility for applying mod security rules directly on board the envoy proxy.

B

So what is waf web application? Firewalls offer a standard mechanism to inspect incoming requests and apply, potentially a suite of mod security rules to ensure there's no malicious content. It can be used to block many kinds of threats. Attacks like sql injection and javascript injection, plus requests that come from blacklisted ip addresses uh and much more. The owasp organization also offers a curated set of popular mod security rules, called the core rule set that you can activate directly within glue edge and with glue edge.

B

You can front an entire suite of applications using these rules without touching one line of the underlying application code. That's the beauty of being able to apply policy at a single ingress point, so we believe that blue edge provides a killer capability to block inbound malicious requests in a way that is both effective and efficient.

B

If you'd like to explore this technique in more detail with a working example, then check out the blog that I've linked on this slide.

B

But, rather than just tell you about using waff with blue edge we'd, much rather show it to you we'll do this by demonstrating how to use glue edge to apply a mod security rule that blocks some common log for shell attack vectors. So what we have today is just a a a basic gke cluster running the the 1.10 version of blue edge enterprise, and so, let's take a look at this blog. This blog that I mentioned earlier, where we walk through an example of uh of blocking a common log for shell attack vectors okay.

B

So what we're going to do here is in our we've already set up some of this ahead of time, so we're going to set up a target service that we're going to uh that we're going to protect uh and it's this is the http http bin service. uh It's pretty common! You very likely seen it before. It basically will just echo uh the the request parameters, uh potentially uh potentially things coming in through payloads, that sort of thing, and just echo back what you uh what you passed in as a request to that service.

B

So uh so what we've already done is we've established a blue edge upstream. That's going to front that uh that service that we're that we're protecting so that's already in place and then finally, we're going to build a routing, a routing policy, uh that's going to route requests from the firewall to this, I'm sorry yeah from the from the gateway to the uh to the underlying service and then and then show the response to us. So you can see this is a very, very basic uh routing rule.

B

um It's it's packaged up in a blue edge virtual service.

B

It simply looks for any request: that's coming in targeted at any domain, and it's going to take that and route that to this upstream that we just that we just created so we've already established the upstream we've already established the routing rules through the virtual service, and uh so now, let's see that our uh that our initial service here potentially has a security problem, and so we're going to illustrate that using this curl command here and so what you can see is we have embedded uh we we're we're, sending a request uh to curl through with curl that's going to um to populate the the user agent header, with this malicious string that potentially goes off and and hits some hit some attacker's server and we'll use that just also passing in a to a normal, uh a normal http bin request, okay, and so, if we, if we run that here, let's uh let's copy this and paste it in and what you can see is in fact this.

B

uh This malicious string was not um was not captured at all. We got back, we got back a 200, ok request and you can see that http bin actually prints that that that malicious string out in the output, so if this were say a java application and it logged the user agent uh log, the user agent header as part of its request processing. This could have potentially triggered an access to a to a malicious server and and bad things could have happened. So how can we?

B

How can we guard against this sort of this sort of exploit? Well, let's take a look at a very simple policy and, as I mentioned before, waff web application firewall is built on a foundation of apache mod security. uh Mod security rules he's been around for years and uh you can see. We won't do a detailed tutorial here, but you can see here's a here's, a secure, here's, a mod security rule and you can see it consists of three parts.

B

One is what what elements of the request do we want to take a look at what uh what rule? What matching rule do we want to apply here on line two and then third, what action do we want to take in response to a match?

B

Okay, so here that we want to check a whole variety of input elements as a part of this request, we're going to look for particular strings here, you can see, there's the uh there's, the gen d string, we're looking for particular strings that have been identified as as being as being associated with this exploit and then. Finally, if we see that we want to deny this request with a a 403 forbidden error.

B

Okay, so that's that's our goal, that's where we're headed and to do that, we're going to modify the glue edge, gateway component that actually fronts all of the applications that we have deployed in this um in this particular system. Okay, in this cluster- and so you can see, here's our here's, our rule, that we just uh that we just took a look at and we're going to uh we're, also going to just print out this message when we get a match that hey. This is a long for shell, malicious, payload.

B

Okay, one thing I want to point out here is: don't don't take this rule and apply it without doing your own due diligence, so there's still even at this date, there's still kind of evolving attack vectors that are being applied via log for shell. So be sure you keep up with the latest in terms of rules that are out there for blocking for blocking this attack.

B

But this is just an example that shows one uh one: uh one potential uh matching string that was deployed in the early days as a rule to uh to block these uh to block these requests.

B

Okay, so now what we're going to do we're going to take this modified gateway component and we're going to uh apply that to our cluster? So let's do that here, okay, and so that all worked, and when we, when we do that now, let's actually take a look and see. Does this in fact block uh block these requests?

B

So, let's start with the one that we already used here, which is actually attacking through a through a request, header and so we'll try that again and as you can see now, instead of that process being requested and the malicious log potentially being recorded, in fact now, that request is being rejected with a 403 error, and you see this- um you know malicious payload message instead, so so that's all good. But let's, uh let's take a look at a couple of other popular attack, vectors as well.

B

Here's one that is actually attacking through through arguments that are passed in on the request path. You can see. We've uh we've added that here as an argument to this uh to this this http bin request.

B

So let's copy that and we'll try that try that out so where that would have where that would have worked before now, you can see we're getting back a 403 forbidden message, so we're also blocking uh attacks through the through the request argument. Finally, let's take a look at requests through the attack payload. So here you can see we're actually posting a body to this request that has embedded within it.

B

A this malicious attack string, so let's copy that and we'll try that attack out- and you can see just like before we're now getting this request as being blocked and we're getting back a forbidden message, which is, which is what we want.

B

So so pretty nice pretty easy by applying a single, a single rule um in a policy that fronts a whole suite of applications, we're able to intercept the request at the at the input and uh and be able to block that without modifying anything in the underlying applications that we're protecting all right so just to wrap up in this talk today, we first surveyed the blue edge security landscape, and then we dove into this particular log for shell exploit and we demonstrated how blue edge can be the foundation for a security solution that not only responds to this problem but potentially lays a foundation for responding to similar future problems as well.

B

So thank you for listening today. We appreciate your time and nick and I will be happy to answer your questions.

B