Cloud Native Computing Foundation CloudNativeSecurityCon 2023 - Seattle, 2 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: From Illuminating to Eliminating Crypto Jacking Techniques in Cloud Native - Mor Weinberger

Description

Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

From Illuminating to Eliminating Crypto Jacking Techniques in Cloud Native - Mor Weinberger, Aqua Security

Ever since cryptomining had emerged as a novel promising digital currency technology, its evil twin cryptojacking has gained popularity and become a major type of attack. Threat actors consider this attack as a low hanging fruit which allows them to easily cash out their attack, since one can easily convert compute power into digital coins. Moreover, defenders often mistakenly perceive this attack as a noisiness rather than an attack that allows to freely run remote code on your server. At first threat actors deployed cryptominers on unpatched servers and targeted browsers. Today attackers focus on the cloud native, including exploiting containers, Kubernetes, CI/CD and SCM platforms. In this Talk, we’ll explore the key concepts and techniques related to the evolvement of cryptomining and also explain on how to proactively protect your environment with open-source tools and approaches that will help you strengthen your security starting from static analysis and up to runtime protection. Below are some of the topics we shell include:

Reviewing of attacks, techniques & exploits.
The main challenges threat actors face and overcome, how they maximize their gain and conceal their attacks
Finally, we will present measures to mitigate and strengthen your environments

A

I'm so hi everyone, I'm Mo, Weinberger and I'm here- to talk about from Illuminating to eliminating crypto jacking techniques. In Cloud native on The Today agenda, we are going to I'm going to present myself. Then we will talk about the crypto mining process. Then we will go over the birth of crypto jacking and why it's all appealing to Bad actors, then we will go deep dive into some of the um crypto jacking evolutions and that Trend and we'll we will end up with detection and mitigation.

A

Okay, so quick, intro, so hi everyone again, I'm more Weinberger I'm, a Staff software engineer in aqua security, I joined Aqua a year ago as part of the acquisition of argon security. Before that I worked for a Microsoft for about four years on the cloud security products and I really like to mix the engineering work with threat, hunting and security researching stuff, and this is basically why I'm here- and this is basically why I'm talking about this topic?

A

Okay, so let's start so, let's talk a little bit about what is the actual crypto mining process. So, in short, it's basically the process of verifying transaction on the blockchain. uh This verification process is actually solving a cryptographic puzzles and this requires compute power and because this activity of verifying transaction is super important for the security of the crypto Network and they those miners are getting reward with C with the transaction fee.

A

um In a type of uh you know, new coins. Okay, so let's see how easy is that to become a crypto Miner? um So all you need is basically four things. So the first thing you need to choose your cryptocurrency coin. There are like multiple coins and each one has its own capabilities and its own features. Then you will need to buy uh bio equipment and, as far as it will be uh strong equipment, you could mine efficiency efficiently.

A

Sorry, um but if you're a bad actor, you can just hijack systems and run your a minor. You will also need to create a crypto wallet, it's very simple and totally open source and free, and you need to choose the mining software and configure it on your device. So it's super easy. You can do all those all those steps in just few minutes.

A

Okay, so, let's quickly quickly review the birth of actually crypto jacking um activity, so it kind of start on 2017s, when corn Hive offered a web client minor code, which basically means if you have, if you are an owner of a popular website, you can take the card, put it in your website and then the visitors will start mining cryptocoin for you for helping with the funds of the site.

A

Then we start seeing that there were some website popular website, especially on the video sites that adopt this approach and add Conway code into their website, but I think is most of them added secretly, which means that they didn't notify the visitors and, of course, the you know, the visitors, the users didn't really like it. It basically utilized their CPU tilde maximum.

A

um Then we start seeing that bad actors that bridge uh website focused on government-based and University affect them with Khan I've called, and we also saw that those Bad actors targeting unpatched servers with some non-vulnerability and again they just affect affect them with crypto mining.

A

um Then we starting that they also targeting unpatched routers and actually affect them with with corn Hive code, which basically means that on every HTTP request that sent into the router, the router, injecting response can have uh add a conif code, and at that point we start seeing like Reaper mining everywhere. This is basically an advertising screen in some parking lot, and did you spot a miner here? It's on the right here. It is. This. Is a nice hash Miner, a client that run on this iot device?

A

Okay, so basically, but why it's so appealing to Bad actors?

A

So there are. There are multiple reasons for that. So the first reason is about the anonymization.

A

So, basically, when you create a crypto wallet, you don't need to provide any identification and to do it to doing the correlation is uh impossible and there are like additional coins like Monero, that add another layer of anonymity and they basically not allows to see their blockchain. So, for example, if I have a crypto wallet, I'm a Bad actors, I still crypto coins, I am a hijacking systems, and now the everyone um detect my crypto wallet as suspicious.

A

So once I just transfer the funds into another wallet, it couldn't be addressable because the transaction on the blockchain are not public.

A

It's also very easy to fast cash out once I have the funds on my crypto wallet, I'm good, to convert it to other coin use exchange to cash out, and we will see that it's super easy to do it in scale. It's also suitable for a newbies and script kiddies. There is no like deep knowledge that you need to have on on crypto or currency Network. You can just run one line of code and you are good to to start in the crypto mining and, unfortunately, um organization consider discrete risk as a nuisance.

A

Okay, so enough with that, let's start seeing some crypto jacking techniques that we saw in our research team called Nautilus in aqua security and also when connecting with organization.

A

So the first thing we will focus on is uh is kubernetes environment.

A

So on the left, you could see the master node, which contains the API server and other stuff on the right is yet another node, okay and we will go over on the popular attack vectors that bad actors, specifically crypto jackers, are targeting and looking for so the first attack Vector is basically looking for a vulnerable application, which means that in this scenario we have engine Linux that running on a pod um it exposed to the internet, and this engine Linux has some RS RC vulnerability and the attacker doesn't at that point doesn't really care if it's run on kubernetes or not.

A

Okay, just scan the worldwide APS looking for a for this non-vulnerability since it exploitable. So it's very easy to retrieve the exploit code and it just execute it, and now we have a foothold in the pod.

A

So at that point we start seeing in the last few years that those actors are start being familiar with kubernetes environment and they are trying to look for a lateral movement for a privilege elevation.

A

um They are checking. If, though, if this pod running any privileged container or any configuration like Mount, was that allows him to to control the whole node, and this is basically the first attack Vector that I review here.

A

Second, one is the misconfiguration misconfiguration on the docker Daemon, so Dockers Dockers was around a little bit before communities became so popular and we, we still see customers that and organizations that are using Dockers, and this one is about a misconfiguration of the docker demons, which basically means that the docker demon Port is exposed to the internet, and there is no any authentication or access level control that is required.

A

And again, this bad Act was basically scanned worldwide. Looking for this, uh this specific port and they're, just starting to communicate with the API what they are, what they actually able to do is to list all the images list, the running containers and getting shell on those containers and deploy new containers with crypto mining.

A

um Third attack vectors is about poisoning the public registries, for example. Dokarab is the most famous registry and we see that actors doing something that call this technique. This technique technique, sorry called type of squatting and basically, what they are trying to do is to push malicious container. For example, this on the right.

A

We have the valid and popular tensorflow Docker image, it's very pop, it's very popular in machine learning, foymock and this actor published a Docker image that called tensorflow, which, as you guess, contains a crypto Miner and they are basically depends on the on the fact that developer will misspell the container image.

A

um As you can see, the this image specifically has more than a 1K pools, so this technique is is quite work.

A

um Okay, the four attack Vector is about a misconfiguration of kubernetes dashboard, so kubernetes dashboard is a easy way to easy interface on your qualities, cluster and what you actually capable of is to list all the running pods list secret and even get a shell on on those container all just from your browser.

A

Okay, this is actually uh an exposed, Docker dashboard in the while and once and once you and- and this basically allows you to have full control on your cluster.

A

um Another attack Vector is about accessing the kubernetes API when it misconfigured again, the API, the kubernetes API Port is exposed and it doesn't require any authentication or access level control and the recommendence API is actually responsible to communicate it with the node and once you able to control it, you basically have a cluster level control um and the last one is about a misconfiguration of the kublet. So the kublet is stored on every one of the node and irresponsible to communicate it with the kubernetes API and once you misconfigure it.

A

So the bad actor has a node level control here and basically he allows to to execute command on the running containers and deploy new containers, and- and at that point we also saw that those Bad actors are familiar with uh with the kubernetes features itself and they are leveraging them in order to deploy the the mining containers in the scale means they are using a demon's deployment and and replica said, in order to deploy the their stuff on every on every node in the on every pod.

A

Okay, just second.

A

Okay, so basically those attack vectors are being around for like many years, but we still see that crypto jacket are upgrading their techniques and doing their optimization.

A

One of those optimization is about changing the huge pages which basically allows the actors to increase the memory page and basically, those actors uh one once they are have a foothold on the containers if they will just run the minor, as is it will not be that efficient for them, so they are basically looking for how many cores and basically the architecture of the environment, and they set the huge patients to to be optimal and by the documentation of a few of those open source miners.

A

This could this could help with increased uh efficiency by 50 percent. We also saw that those Bad actors are aware that there are more Bad actors on the environment. We also sit on our honeypots that are like multiple actors that basically battering each others, because no one of them wants to share the resources right. So we see them trying to disable and kill each one of those, um and we also see that they are adopting techniques of wood keys and fireless and they're also trying to disable and remove the Cloud security agent.

A

Those two examples are one of alien, which is the Alibaba Cloud agent and the bottom one is the gcloud agent.

A

But what do you think? Okay, we saw some techniques on the kubernetes style on the runtime side. What do you think are those crypto jacket, shifting left with the ecosystem? Are they aware of of the new technologies that the organizations are adapting around the around the left side?

A

So, of course, yes, this is basically a campaign that we discover a year ago of Bad actors that are trying to abuse the cicd32. So basically, they are not needed to find any any unpatched servers or music, sorry misconfiguration.

A

um They are also. They only need to craft some um some, some techniques that they can do it in scale and basically, the the this method is unlimited. So what those Bad actors are basically done here is creating a GitHub repo connected into cicd platform. Okay, use the free tier, of course, and on every change of this repo, it's basically trigger a build job, a build process, all right, and basically those actors understand that the build step could be hijacked. The results of it could be hijacked in the favor of crypto mining.

A

Okay. So what basically happened here? How they made it in scale? So, on the build step, they said that first they will clone the repo. Then they will choose some some random file and Mains and made some random change. Then they will push this change inside the repository and what will happen?

A

Another CI job will trigger, which basically means at that point, that they have infinitive Loop, because every build will trigger another build and after they trigger another, build, they are pulling a custom-made Miner from another repository, and they run this Miner until the maximum time of the CIA offering this table we found on one of those. Bad actor repositories is basically elaborate. What are the three tiers on those cicd and Sr services?

A

um And we also saw that those badactors are learning and they are aware of every limitation of of every ECI CD platform. This one targeted, GitHub action. As you can see it, it set the maximum parallel jobs to 20, which is the limit which is the maximum allows. They also said that every job will run no matter if a previous job will failed and they also use interesting evasion.

A

Techniques here, which The Miner, is an elf binary which basically run on the Linux, but they run it on a Windows box and the use and the able to execute dclf by the WSL feature of of windows, which is the windows subsystem for Linux, and we believe they they doing so for the for running the Yale of binary in kind of sandbox mode.

A

um Another eventual technique that we we saw here is in order to make the network detection a little bit higher harder. Sorry, so what they done here. They are using root, Keys techniques that erupt by npm package, so they basically just pull an npm package which is very regular and popular. You know um things to do with as part of the as part of your build, but this one is wrapping a wood root.

A

Keys CLI that allows the Bad actors to hide is minor from the process list and, as you can see on the right, they managed to round multiple GitHub action jobs successfully and each jobs run into the maximum of uh of uh of the allow of GitHub action like until it exceeds the maximum time.

A

Okay- and this one is kind of funny uh shortly after I published the write-up about how many actors abuse those cicd features, I got a message on LinkedIn that says: hi, Mo, I'm, a crypto, Miner and I think we share the same interest regarding the field and I think to myself that shouldn't be the case. The key point right.

A

Okay, but um what about supply chain techniques? Are those Bad actors adopting some supply chain techniques?

A

So, as you guess all the answers here are yes, um so yeah there are shifting left and they are aware of the risk on the software supply chain and just to align here. So basically, the software supply chain is start with developers that write code manage it in source code management, then build it ship the package it into artifacts and then deploy it in into the runtime and the thing about it that every one of those steps that I just mentioned um using open source third-party dependencies and those Bad actors are understand.

A

One single point of value here um and they target exactly that. So the first attack Vector that we saw is about a technique that call a type of squatting, and this one was discovered by our friends on check marks. We saw they saw that these Bad actors push multiple open source package into a pipeline into the public pipeline um and it contains, as you guess, a minor and they try to mimic the popular request package on a Pi Pi and because because Pi Pi just requests the GitHub repository that correlate to the the package.

A

So this bad actor just put the valid request one and it gets the exact amount of stars and forks and it was able to really make a good view of of this package. um Another another attack. Vector is around something that call dependency confusion.

A

It happened with pytorch just like a few weeks ago and basically, what happened here is that as part of the pi torch build processing, it's a load um is owns dependencies and some of them were internal dependencies and and the way that and the way that, when you're trying to load dependencies on a python, so you're first going to the public by index, and if we can find it, it will get into your internal repository and because python didn't preserve this specific library, library on the Pi Pi public index.

A

So this Bad actors founded he pushed a new package into the public Pi Pi, with malicious Cod and pytorch, pulls it as part of the build, and he totally got a compromised this version and if you think to yourself, okay, we saw a type of spotting techniques. We also see depends the confusion techniques. It could be like some security solution around that that could help with defense.

A

The thing about open source dependencies is that there are multiple popular packages that are maintained by just single maintenance or maybe a few maintainers, and we are not really aware of who stand behind them and what is the security measurement that he takes and how secure and how kills? How secure is he? So we see here a bad example of uh parser that happens a year, maybe two years ago, which is very popular UI library on the npm and this one got compromised by a leaked token of of the owner and it got breached badly.

A

This library was used by the most big Tech and this bad actor had crypto mining, but also passport, still stealing a code, and he targeted both windows and Linux, and basically trying to Target both the developers, the build system and runtime, but you could think to yourself: okay, like what is the actual risk here.

A

um So the obvious one is about financial loss and the service degradation either it either. It runs on your AWS account or like cloud account or on your build. It will increase your uh charge it. You will also face the service degradation, but the most important here is about those Bad actors. Has the ability to run um I'll be the record on your environment?

A

If it's on your build, they can basically steal your secrets and on your around the environment, they will start, they will try to elevate permission and lateral movement to other high assets like your DB and other sensitive information.

A

So what we can actually do about it.

A

Okay, so um so, if you don't have any security measurement or if you have some so my suggestion here is first follow the compliance and best practices. There are like multiple organizations out there. This is just two of them with uh security expert that invest their time and their expertise in order to create standards and best practices specifically for each popular technology, and the second thing about it is you need to understand your architecture from your Cloud into your formula code into your Cloud understand what what are the technologies that you are using?

A

What are the integration parts and basically follow those best boxes and guidelines, and there are many like open source tools out there, that that could help. You run those around those guidelines and checks. This is just two of them.

A

The first one is trivi that basically can help you run on your kubernetes cluster and even on your way, yaml before it gets deployed uh security posture management to find those means, configuration that I just present you and the others um and chain bench with which is focused on your software supply chain and they're running the same approaches approaches, but on your left side and those two can also provide you a compliance report, either it's a CIS or other, and then you could see your security departure management based on those compliance report.

A

If you got to here so you can also scan your code. Your infrastructure is called and dependencies for vulnerabilities for misconfiguration, and you can also do it with trivi, and you can also run a static code analysis. Some copy is a great tool for that with more than 1 000 rules that supports most of the popular languages and it's able to detect a risky code even before it gets merged into your source code like, for example, it could detect xss and injection injection risk within your code.

A

But this is not enough right. Trying to reduce your risk and fix misconfiguration.

A

Most of the time is not enough, because, basically you have some vulnerabilities and other misconfiguration that you didn't have the time to to address, and there are always like a new uh zero days and other vulnerabilities. So you need we need to get one layer deeper. We need to look for a abnormal and suspicious activity again on your cloud account to look for any increase in your resource creation, on your abnormally identity activity, um um and also you need to have some random protection.

A

There are like great tools out there. I will just mention. One of them, which is a Tracy, uh is, is based on the ebpf Technologies, which means it runs on your kernel level, and it allows to detect root, keys and files attacks.

A

And, if you don't trust your third party libraries, you can use guard dog by datadog that has the capability to run a static code analysis and detect malicious code like crypto mining, on your Pi Pi lovers.

A

um So this is it basically thank you. Everyone and uh I would appreciate if you could scan the QR code and provide a feedback thanks.

A

I, don't think I have enough time for question but feel free to connect me in person thanks.