Cloud Native Computing Foundation Cloud Native Security Day 2019 (San Diego), 3 Dec 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: MLGuard -- Detecting Malicious Web Requests using a Serverless-based Machine L... Abhinav Srivastava

Description

Join us for Kubernetes Forums Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

MLGuard -- Detecting Malicious Web Requests using a Serverless-based Machine Learning System - Abhinav Srivastava, Frame.io

Web Application Firewall (WAF) blocks incoming web requests using a variety of signatures such as SQL injection, Cross-Site Scripting, and Bots. Proactively identifying and blocking bad requests, which avoid exhibiting the known malicious patterns, is both challenging and essential from security operations perspective. In this talk, I will describe a serverless-based end-to-end system called MLGuard that ingests AWS load-balancers log data, creates a machine-learning model (Isolation Forest) with the frequency distribution of cumulative HTTP response code using Amazon SageMaker, invokes the model using the HTTP API to detect unusual requests, and sends alerts to Slack for the security team to block IPs. MLGuard utilizes various Serverless technologies such as Function-as-a-Service, DynamoDb, and API Gateway, and since its deployment a year ago, it has helped block thousands of bad IPs.

A

Good morning to all of you, my name is Sabrina I'm, the head of information security mile, which is a New York City startup, before joining for mio I spend six years in AT&T research, doing the securities research on IOT cloud, security and software-defined networks, and before that I spent five Great years in Georgia Tech on my PhD in cloud security and as the head of security of mio I oversee the entire security program, application security cloud security threat, intelligence security operation is in Incident, Response and, of course, compliance and today I will be talking about ml guard service that my team developed to detect malicious web requests in the cloud using server glass.

A

So just to give you a quick background on what frame dot IO is so we are the world's famous video review and collaboration platform, the allow users to upload media content contents in our cloud and then invite team members and collaborators to actually our team members and clients to collaborate on those videos. So you can think of us as a github for videos or Google Doc for videos.

A

One of the main feature that we provide is a frame, accurate communication. So, for example, if you can go on a specific frame and then type comments, saying that hey change these colors or look at the fonts and so on and and those comments get actually embedded in the video itself.

A

And then you can also annotate the video if you want to be more pictorial in in your comments, and these comments can be seen in real time to your team members and collaborators and they can perform actions, and this saves a lot of time and awards long email threads, that you have during your review process.

A

Another thing that we allow people to to create a secure hub for their film productions. It's a single place, a single place where you can, your teams can control who can see what and when, and it also lets team to easily organize and manage divergence of each assets so which is again very important when you are working with thousands of episodic contents and you want it, you want a place where you can easily manage the and those are videos.

A

And finally, we allow users to securely share this content with external parties right because sometimes you have to go in and get approval to executives or all those people who are not part of the platform and through our platform, you can do that, and you can also customize that how and what they will see and again. This is very important end up in the post-production phase, so I mean these value.

A

Propositions have have enabled us to work with some biggest customers in the media, entertainment industry and also in the technological industries, because these days, everybody's pretty producing content and also this habit, and this this feature have made of a platform a target for attacks. So let's take a quick look that, after getting a 30,000 foot overview, how actually gets translated into actual platform- and this is a very simplified view of our platform. But this is but this this is relevant for the today's talk.

A

When users come to a platform, this and request, and that request comes through the load, balancer and load balancer decides where to send the request- and we here and containerized workload, and we are also moving to kubernetes as well, and we are entirely hosted in in AWS.

A

So all the workloads running inside VPC and we have lot more readable services running behind the scene and security group and so on, and the database actually is used to store the transaction data, and there is a separate path here that I am not showing here, which is not relevant for the today's talk is to how we talk to s3 bucket for uploading and downloading videos.

A

Now, as I said, that is the best platform, so users are sending requests and attacked us also also sending bad requests with some some weeks. We see millions of attacks to a platform. Some some are targeted, some are scanners running in the system or on the net, and when it happens, though, the the best way or one of the ways to secure or block those malicious requests is to use Web Application Firewall AWS has a rough service that we use a lot.

A

We have developed our own custom rules on top of graph, but we also use some knowledge rule services from the marketplace and that help us block WASC, top-ten attacks, attacks coming from bots and and so on, and that gives us say if gives us the first layer of defense and block malicious requests.

A

So when we started running these graphs and started collecting data from the replication law firewall logs and we started seeing what kind of requests are getting in, what what is what kind of requests are getting blocked and he found that graphs are easy to evade and I mean. This is not a news. We all know that right. Signature based systems are susceptible to evasion techniques because they are static, so the ones you craft, the rules, rules you don't change that. Often they are signal and they are signature.

A

Based systems, so so you have a sequel injection signature. They try to mass equal injection. But if you encode your input multiple times, they won't be able to parse it and then that's why it's very hard for four vas to detect zero day attacks and.

A

Once once we started seeing the requests we in our logs that we thought that it should be blocked by valve, they were not getting blocked by valve. So so we started looking into anomaly detection based systems. Again, it's well known that anomaly based systems are dynamic because they they learn over time. How the system is behaving, what the clothes are good, what APIs are supposed to be blog and then they can easily find anomalous requests.

A

They are more context aware. So, for example, if you are running a ruby or elixir based applications and your scanner is sending you PHP requests normally, since valves don't have any application context, they will still that all PHP related requests go through your firewall. But since anomaly detection system knows it's more context aware it can block those required. Can it can find them anomalous and it can? It can alert you that some scanners are sending PHP requests and, and due to this nature, they can easily detect 0-2 exploits once you know that there is enough.

A

Anomaly is seen in the in the request pattern. You can block it whether it's known attack or unknown attacks and with this insight into our own own systems. We, we came up with this idea of developing ml guard. It's a service based machine learning system that basically detects malicious web requests. You we, when we when we were designing ml guard, we really didn't want to create our own infrastructure footprint right.

A

We didn't want to run a lot of V arms or heavy machine learning systems and then having the responsibility of managing it, managing our own security infrastructure and that's why we we decided to use a lot of server as there's technologies as well, and that I will show you this dog.

A

So emily guard has has five stages and as most of machine learnings model or our systems have in the first stage we collect logs logs will help us build models, and once we have the model we deploy the model and then using those models for future. It was we use we using the model for future requests. We decide whether it's a good request or bad request, and then they say no.

A

Normally we notify to two security team that this IP is sending bad requests and it crosses the threshold and all these stages use service technologies in our pipeline.

A

So before we look into an ml guard, let's take a quick overview of what what kind of service technologies we are using. So we are using mainly functions as a service and in it, since we are entirely hosted in AWS, we are using a dove this lambda extensively in our platform. So when the cloud when the when the in the in the beginning cloud providers were offering highest services, they still offer, but but in the beginning it was the only service that is being offered.

A

They are hardware and hypervisor or managed by cloud providers, and virtual machine operating system and app was the customer's responsibility in the platform world. It got changed little bit because now, hardware, hypervisor and virtual machines are being managed by cloud providers and containers and applications were the responsibility of customers. They have to do patch management, they have to make sure that vulnerabilities gets addressed and so on.

A

But if this got pushed to an extreme in the function as a service world, we are now cloud providers are managing hardware, hypervisor virtual machine operating system and even the runtime. So you don't have to you just as a customer. You have to just bring your code and start running it. So if you want to develop Python based application or Java based application, you don't have to worry about their setting of the runtime or setting them the environment.

A

You can just bring your code and start working, I start executing it, and, and that way we don't have to worry about patching our own infrastructure. You can you can transfer this responsibility to the cloud provider that those millions or machines are properly managed and secure, and you just have to worry about the security of your own code to make sure that you don't have any bugs in there and there are couple- and there are several benefits of using servlets right first.

A

So as I said that no servers to manage, so you don't have to worry about Danny about patching or securing those machines. You don't have to worry about the scaling of those service functions as well, because if there is yan is the if at the peak demand, if you want to run thousands of concurrent execution of your service cloud providers, will do that. You don't have to set up automatic scaling group and so on.

A

You had it's, it can help you save cost, because the it's provides fine-grained metering, so you are going to be charged on each invocation of your functions. So you don't have to worry about that. Functions are unlike VMs, where you have to pay when the VMS are running, whether they are processing the workload or not functions. You only pay when they are involved and it's and it's a faster time to market. Because again you don't have to worry about the environment, so we use server server less a lot in our entire infrastructure.

A

We have a transcoding pipeline where we transcode videos using servers and step functions and in security team is using a lot and based on our extensive usage. We came up with six or six different design patterns for service. We even wrote a research paper last year in hot floud that describes all the design patterns and how those patterns count came together in our third intelligent system, but in this stuff, or for this talk, I would to cover two patterns that we are using a lot in ml God.

A

So one is even driven and one other is periodic invocation. They are very easy to understand. Even Durin's functions are when these functions of the service or lambda functions get at get subscribed to events.

A

So, for example, lambda functions can can sit on top of s3 and listen for all the put objects operation and as soon as objects are stored in the industry bucket. This lambda function gets invoked process the event and and and go back to sleep and and and this kind of functions are very useful because you only pay for their invocation, so you don't have to so that saves, cost and also you don't have to do polling it because they are only being involved when the event is at of interest is actually happening.

A

And s3 is one example you can also use. Lambda functions can also be attached to DynamoDB so a table. So if they, if you, whenever you add new entry, these lambda functions can be invoked to process those added roles.

A

Periodic invocation service functions are nothing but think of this, a glorified cron job sort of that we have all used in our in in the past, so they are invoked periodically every minute we are daily, they can be customized and then they get when it gets invoked. It does, is its performance. Is this job and then uses some asynchronous mechanism or sometimes email to is pass the result to the pass, the results to to the function that is calling or to the service?

A

That is calling and that's very useful, for example, if you want to develop it, it's service that that is using a larger base algorithm to transfer data from s3 to glacier these lambda functions can be invoked once in a day, look at the LRU policy or for each uploaded object and this transfer to Glacial since V. As you know, as in our platform, we allow users to upload media contents right, so we have a petabytes of a storage right so for us to save cost.

A

We would like to we have this kind of lambdas running that can transfer data to glacier to for the vault to save cost, and this is also good for for compliance purposes, because dailyvee you want to check whether our infrastructure or services or cloud configurations are compliant for sock or for other compliance framework that we we are part of.

A

So with that information, let's look ml guard again so again, five stages and and collect, build, deploy prediction and notify. The first step is that how do we collect logs right, so ml guard, as I shown you in the first slide that the all our incoming requests hits the load balancers. So we ml guard works off the load, balancer data, it's a very similar to web logs. It has all the fields that you normally find in the web logs and since it is, it captures all the interactions with the customer.

A

We are using that log to train the model and all the logs are provided by AWS, so these logs are actually stored in the in the s3 bucket and we call them raw logs because they are, though they are. That's the log that AWS provides, but then we use even driven lambda functions on top of this s3 bucket, and this lambdas job is to enrich the data that we get from AWS, so enrichment, for example, ELB logs, have IP addresses then, but we enrich the data by adding geo locations in there in it.

A

It has IP addresses of our instances or VMs, but then we attach security groups or other important informations that we have. You know you know in our environment to do that that entry, so that it becomes enraged. It has lot more context and that's going to help us train the model and and and and and find problems when night. When we look at the logs and then this lambda function is toward the enriched data into the into another bucket around the prefix and the s3, and we also pass this data to our to elasticsearch.

A

So we use elastic such as our sim, so it can help us create graphs, visualizations and- and even then we want to when, in some in case of attacks, we can easily go to elastic, search and see what kind of request was sent.

A

Once we have the log, then the next step is to what algorithm should be used to actually build a model. So there are supervisor algorithms in machine learning. There are unsupervised, there's reinforcements right so supervised. We need label data, but it's very hard to get the label data in the in the security was right because you don't know what all the task patterns right. So we resorted to unsupervised learning right because we don't need label data for that, but it does require us to have a huge amount of data set right to train the model.

A

Luckily, for us, we get millions of requests every day, so millions of logs ELB logs that is good and sufficient enough for us to train the model. So we use unsupervised learning to basically basically find the anomalous requests sent by attackers.

A

So it becomes an outlier detection problem right, find the needle in the haystack. So we we use isolation, forest algorithm. It says in outlier detection algorithm whose job is to find, and on this point by another's point from a given distribution by creating in Samba of binary trees right so in a nutshell, without going too much into the details and we can talk offline on that the way isolation forest works is like this: it's very intuitive to understand that it basically finds outlier right.

A

It's fine, it finds so that the point is, is is anomalous so the way it does, it basically creates cuts in the in the in the data set. So if you are or splits so, if you, if in a few splits, if you can isolate a point, means point, is an outlier right.

A

If it's to think about in this point, if you kid take a one cut and the point is off, then it means an outlier, but if you are using 10 15 20 cuts in order to split the data and and now and and isolate a point means that this point is normal right. That's what's shown here that if the red point is in three cuts B, we could isolate it. It means that it's an outlier but the point on the in the center. We have to go through almost 700 cuts.

A

It means that this point is not an outlier normal point and that's the basic idea behind isolation. Forests.

A

Now we have data, we have model. So what features do we need to do? We need to use to train the model and and the idea that we want to come up with that. We don't want to know individual HTTP requests or individual requesters. They are good or bad. We want to know whether IP is good or bad right and- and we can find it out, based on what kind of request is being sent by the denied by that IP. So on the Left.

A

If you see that if IP is sending a lot of good to a lot of 200 requests, 300 requests, which are supposedly good and very few 400 500 requests, which are signs of a bad, it was forbidden or internal server. Error means this IPS is overall good IP right, but on the on the right side, if an IP is sending very few 200 300 requests, but a lot more 400 and 500 Ric was. It means that the type is either is scanning they're trying to find its way and that's where they are getting denied.

A

So many times knew the site is bad and that's when we want to look into what that IP is doing so that we can block it.

A

So the next step is to build train and deploy model, and for this purpose we use AWS sage maker. It's a machine learning as a service that AWS provides. That makes your job lot easier in terms of build and train and deploy model. Once of our algorithm was ready. We in few hours, we could have our system up and running we're using AWS age maker and there we are using again lot of star wallis technologies. So here we are using periodic lambda invocation.

A

So what what what we did once we have once we know we are using isolation. Forests and we are our model- was coded. We containerized it right so and published into the ECR and wrote a periodic lambda function whose job is to wakes up every 24 hours run the container automatically and container talks to the english lb bits that enrich data from s3 retains the model every 24 hours and deploys the final model in the stage maker. There is no manual intervention at all a model every everyday more.

A

This lambda function is training the model automatically and once the model strain it is running in the stage maker and that's when we go to the prediction stage and what happens when you deploy a model in ch maker? It gives you an HTTP endpoint that you can directly invoke, and it can tell you what the request is good or bad.

A

So again, we are using periodic lambda invocation that that invokes every few minutes and looks into the recent request that was sent by by IPS and it gets the model from the database and starts invoking invoking invoking the model with the HTTP request or with the with the collection of HTTP requests based on IPS and started worrying.

A

Whether this is a good IP, bad IP, good, IP, bad IP and stage maker tells us that our model tells us that the results and if the mother, if the predictor lambda finds out that this is a bad I P, we get modified onslaught. Saying that this is the attack going on or this IP sending bad requests look into it and through slack we can again automatic response we can take.

A

So this is a notification to security analyst duties. I took some details off from this. This alert. Those who are interested I can show you offline. So it says that anomalous IP this IP is anomalous and this post a link to our Cabana elasticsearch dashboard. If you click on it, you can directly go to the elastic search and see all the requests sent by that IP in the last few minutes. So you can see that in real time what is happening then we show the IP, but we also add IPS reputation along with it.

A

So if you click on that IP, it can take you out to the reputation website to show you whether this IP is part of Dracula is part of email. Spoofing is from China is from Russia. You can know all those things and they also have a geolocation, but some geo locations are changing. Sometimes, so you have to get it and keep updating your geo I play database and we have a block list and the triage.

A

This means that if you triage means that ok, you looked into it, you you ignored it, but if you say block, we can directly block if you click on this thing, web book gets fired. That invokes few other lambda functions and we directly add this IP to our Web Application Firewall list or block list, and that's when this IP is blocked automatically.

A

So few metrics related to our system. In the last month of October, through our system, we blocked 339 IPs alone that were basically flagged by our system, and then we found out that these IPS were indeed sending bad requests. We so we blocked them. It's not an accurate and 527 unique IPS were reported by another guard. So if you look into the person just like a 65% of IPS that were reported, we actually block them, but that's not an accuracy test, because sometimes Amal guards still detects that this IP is a bad IP.

A

But for a business reason we decide to not to block it right because they are either hitting our rate limiting violation and we do want them to still test it.

A

But we looked into it but still want them to go through, and sometimes Amal guard identifies problems in our web application right that that we, then we go to developers and tell them why we are sending this error code when we were supposed to sending some different error codes, so they use to end user and that's when it's very good for our developers to see that what's going to get a real insight on on how what are the problems are within the application? So they can fix that.

A

So the key takeaway from this talk that it used to be very hard to develop machine learning based systems right. You have to spend so much as days months to build train and deploy models, but with the in the cloud with the server less and with the machine learning service, this becomes very easy. We are using lot more algorithms right now in our pipeline or in our in our layer, defense that are AI and machine learning, driven rather than just signature-based driven so do try it out. I mean it's like developing.

A

Integrating these kind of services in serve with server, less and services are, are the I mean are possible and is the way forward? Thank you.