GitHub GitHub Universe 2022 - Security, 21 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Securing our secrets, together - Universe 2022

Description

Presented by: Mariam Sulakian

Secrets leak every day. Hundreds of them. These credentials, which can be left anywhere software is built, expose individuals and entire organizations to malicious attacks. To tackle the critical security threat proactively, it’ll take an industry movement. This session walks through how developers, security teams, and credential issuers can help secure the world's code and eliminate credential based breaches. Together.

As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub

Thanks!

Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github

About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com

A

A

Have you ever told someone a secret and before you know it, it feels like everyone's in the know.

A

Well, secrets and software aren't much different one Lake, and now your secret is fair game, I'm, Miriam and I'm the product manager for github's secret scanning and today we'll dive into how, as an industry, we can secure the world software by eliminating credential based breaches, we'll get into the scope of the problem and why you should care you'll, probably wonder how you can efficiently detect your Elite secrets and how you could take action on those detections and once you realize how much work remediation takes you'll wonder how you can prevent those secret leaks in the first place, whether you're, a developer organization or service provider, you'll come out with clear action items on how to secure yourself and become a part of the solution.

A

On that note, credential leaks are human error, so it's hard to place, blame and we shouldn't, but how do Secrets get leaked? Well, let's take a step back and understand what a secret even is secret is like a password that lets. A third party know that you're the one requesting access to their service, it can be an API key, an oauth token digital certificate or another form of credential and secrets can get leaked on accident.

A

Imagine if I'm, a developer and I use my credentials for a quick test to debug I'm doing my work developing working fast, because we love to develop fast I commit push. My code and I forget to take those credentials back out. Secrets can get left in git commit history. Suppose an admin is remediating leaks and they remove the secret from the main branch, but they don't scrub the entire git history. That's like.

A

If you have a really embarrassing text message conversation and you delete the last text to make that memory go away, but you leave the rest of your conversation there and that doesn't really solve the problem. Does it? Finally, Secrets can get leaked intentionally? Maybe students or novice developers leave secrets in their code, unaware of a leak's potential impact? Okay, so we get that Secrets get leaked, but if this happens once in a blue moon, do we really care well? Secret leaks happen.

A

A lot- and they happen often so how often do Secrets leak to set some context. Github has over 73 million developers and over 2.6 billion contributions per year and at GitHub we scan these billions of commits to detect secrets and public repositories. Every month we detect close to a hundred thousand leaked tokens, and every day we find over 100 of our own access tokens exposed in public GitHub repos, and you can expect similar numbers for other large service providers. As you can see, the problem of leak Secrets only continues to grow.

A

It's not a problem that will go away and there are a lot of Secrets leaked in code, but here's the thing Secrets, they don't just leak in code Secrets- can leak anywhere you're building software from scanning billions of commits we've learned that, based on leaked GitHub tokens that Secrets most commonly leak in code content about 84. They also leak in commit metadata, and we see about 15 percent of leaks there.

A

That might seem surprising, but it's actually an understandable mistake, assume I'm a developer setting up my IDE on my machine and I set up a new project. I might be prompted for my git author name and email, but I accidentally put my username and password because of muscle memory, so every time I commit now I'm saving my credentials as commit metadata. Finally, we detect about one percent of secrets in other content types, think pull request, comments or issue descriptions.

A

Now what percent seems small in comparison, but it's still something and a single leaked token can have devastating consequences. So we want to treat Secrets leaked anywhere with urgency and scrutiny. All right, you might be wondering, do I really care if it's just one secret leak, one secret is all it takes, and the consequences of those leaks are visible throughout high profile security breaches like solar winds at a very high level. In the solarwinds example, an exposed credential gave hackers access to the solarwind servers.

A

Hackers then use that access to inject malware into customers, software updates- and this gave the hackers access to customer data. Here we have a full supply chain breach and the solarwinds incident impacted an estimated 18 000 customers, hackers, with that one credential now had access to all this data that they could alter. However, they wanted- and we still don't, really know the full extent of what information those hackers could have stolen. Like government data or employee information. Could you have been one of those people?

A

There are many many more examples of credential leaks, leading to compromised products, organizations and even government entities. So do we just stop using third-party services, not quite that'd, be near impossible. Do we provide more training to developers to not leak credentials again, leaks are just human error. Even if we have the right knowledge now. Does your organization have any leaked secrets? Well, you can find out with good and efficient secret detection, so you're working on a project, and you use your GitHub token to authenticate yourself.

A

Then you accidentally paste that secret into your code and push your code to a public repo without a second thought, so that Secrets now leaked in that public repo. Now what since you're working on GitHub you lucked out for public repas, we scan all code for our GitHub patterns and our partners patterns by default and for free. Our partners include AWS, Google and Azure, and usually just a few seconds either we or our partners take action. Remember in public repos we detect about a hundred GitHub tokens a day.

A

We revoke these tokens at once and alert our users about those revocations. We also scan private repos for GitHub Advanced security customers and over the past two years, we've seen an exponential growth in the number of repos, where secret scanning is enabled. This trend correlates with the rise in security breaches over the past few years and on average we find about 70 Secrets per customer when they first enable secret scanning on a private repo. We scan for Secrets every time a user pushes code into a public repo.

A

So when the user does push their commits, we search those commits for secrets that match our specific patterns for public repos. We automatically scan all code for our partner patterns, including AWS and slack, and our partners then choose to revoke and or notify. Users of those leaked credentials scans usually take less than a second, so we can very quickly catch leaks as they occur. But what, if we didn't do this? Well, at GitHub we love to experiment.

A

We wanted to see what would happen if we did leak a token, so we leaked an AWS access key and secret. We got an email from AWS very quickly after to let us know that our credential was leaked and the next steps to take for our case study, we decided to ignore the guidance we were transparent with AWS that we had committed the token intentionally. We were trying to see what would happen in the wild. We thought it'd take a few hours at minimum for anything to happen, so we went grabbed.

A

Some quick coffee came back and within an hour we had 17 attempts to use the credential from random users or Bots and all those emails we received not spam, their notifications of actual use of our leaked credential. Of course we monitored our credentials and just for this one scenario, here's the impact with the downstream charges about 81 dollars from our little experiment that racked up in under an hour. So imagine if you leave your secret in code for an entire day or a week or even longer, you can easily get charged.

A

Thousands of dollars a day again in this case we were transparent with AWS that we did leak the token intentionally and our account was flagged. But for that intentionally leaked secret. This was our impact, ensure every leaked secret is different and we might not always see the same behavioral patterns, but this case study shows a very real example of what could happen: okay, detections they're great in theory, but in reality there can be a really really big problem with detections false positives. Well, a GitHub and I hope everywhere else.

A

We want to strengthen developer trust and security, and we want to equip security teams to easily tackle the most critical vulnerabilities. That means increasing signal, so every time we send an alert, a user will trust that it's important and coming from a real security threat. That means reducing noise from false positives. So let's dig deeper into this for a GitHub Advanced security patterns. We only surface detections for patterns with below a 10 percent. False positive rate I mentioned earlier that when customers first enable secret scanning on their private repos, we find about 70 historical alerts.

A

So if we assume a 10 false positive rate, that's at least 63 or more real Secrets, not bad, but why do we care so much though? Well many tokens with high false positive rates have unstructured patterns. Think of passwords, you can still scan for those but secrets with less structure mean they're, less identifiable, which means they're harder to accurately detect and secrets that are harder to detect means that security teams in charge of remediating those Secrets now have potentially hundreds, if not hundreds of thousands or even millions of tokens to review.

A

We also notify commit authors of their Elite secrets, so those developers are now also inundated with these hundreds of false positive alerts. High false positives mean low Precision. This leads to alarm, fatigue and low adoption or use of any tool, and all this to say that false positives are a big annoying problem, but we're working on it. We can help the false positive Problem by an industry-wide adoption of token formats that we can easily identify for defense in depth.

A

We can use these patterns to develop products that detect leaks with high confidence and automatically revoke the compromise credential. For our own GitHub tokens, we added a prefix, and this helps us identify that the token does belong to GitHub. We also added in a 32-bit checksum to validate the token. Without hitting our database, we chose our version of a checksum to strike a perfect balance between maintaining entropy levels and establishing confidence in the checksum.

A

We work with Partners too, like digitalocean to Mint identifiable tokens like GitHub digitalocean has a prefix and they have a whole lot of entropy or Randomness to Signal the findings uniqueness and for organizations who have specific patterns they want to scan for GitHub Advanced Security offers our customers easy tools to Define their own patterns and to dry run those patterns before they create alerts and on our mission to reduce false positives. You can dry around your patterns before you create alerts to reduce noise over 50 percent.

A

More customers use custom patterns, since we release dry runs, so good patterns mean fewer false positives, and this means that security teams have higher confidence in the alerts that we do service, which makes it easier. When you are remediating lakes to avoid any damage, you should work with. The right team members to ensure sure the secret is properly rotated and redistributed. You can find open source examples of how to rotate tokens across GitHub, and this is just pre-work once you've done.

A

That GitHub recommends that you revoke the token to avoid any damage like if an unauthorized user gets a hold of your token revoking. A secret is the only way you can prevent further damage by unauthorized users. Steps before you revoke might differ in public and private repos, but no matter where the Secret's been leaked, you need to make sure that that secret is revoked so revoking. The secret takes care of further damages from a malicious actor, but you should also try to prevent other developers from adopting poor development habits.

A

That's where rewriting history can be especially impactful and once you've made it through. Your main checklist there's still more work to do. Even if your leak was in a private repository, you still need to take action. Private repas often give a false sense of security. Just because a repo is private doesn't mean that information can't leak, also check your API activity logs to review your level of exposure. Did you revoke the secret in time so that no one else could access it?

A

Or do you need to take other steps in case there was a breach check your repos access permissions make sure that nothing has changed, make sure your contributors know how to store credentials safely, like using a Secret's, vault and Loop in your operations and appsec teams for guidance on next steps, and with all this in mind, if you're using custom patterns to scan for your orgs specific Secrets dig into those Dry Run results and hone your patterns as much as you can to reduce the time that your security team needs to take to triage these false positives.

A

So with all that said, remediation takes a lot of work, but here's the thing we wouldn't need to remediate if we already were just preventing leaks in the first place today, most detection tools, search repos for elite credentials after a contributor has pushed this, requires appsec teams and developers to work retroactively to remediate tokens and to revoke those credentials before a hacker can exploit them, but we can be proactive about secret leaks to prevent them from the start to review good identifiable patterns help us detect tokens with high accuracy.

A

If we can identify the tokens, then we can also prevent those tokens from ever getting into the code base in the first place, with push protection in April GitHub released a push protection feature for our GitHub Advanced security customers using secret scanning now, contributors and organizations with Advanced security receive notice of any identified Secrets before they push their code into their orgs repos and as a push protection secret scanning will scan high confidence tokens. Those with really really low false positive rates and block a remote push.

A

If secrets are detected, this flag Secrets before they're exposed. So how might that work? Well, let's take a look. Here's a file where I'm adding a GitHub token into a GitHub action file, I saved the file I commit my secret I, try to push and I'm blocked from pushing that secret as a contributor to my orgs, repo I know where my secret was found and in which commit I can also view resources for my orc to help me out or get a step-by-step in the GitHub docs, but sometimes that doesn't always work.

A

We know that developers and security teams need to work together if I'm a developer and the detected secret does happen to be a false positive, which should be rare or a test case or something else, I have the option to unblock Myself by bypassing the protection. Of course, this is for the rare circumstance, but we want to make sure that we are only blocking our developers and slowing them down for real threats and, of course we want to prevent leaks anywhere.

A

Software is built, whether that's through an IDE or the GitHub web editor, and these protections work especially well for patterns. We know have low false positive rates. These are our high confidence patterns or our highly identifiable patterns. Why don't we just add every single pattern into our push protection experience that are safe than sorry right. Well, a lot of customers have this question. This is like using a really big hammer to solve. The problem. Imagine there's a token with a 20 false positive rate.

A

So that means, when I'm writing code 20 of the time that I hit a block. For this token type, it's going to be a false positive rate, and this leads to developer frustration, loss of trust in the product, and that means that developers are more likely to bypass the alerts, the protection and then what's the point, our protective step loses purpose and you might think okay I, get it only patterns with low false positive rates and we're off to a good start with over 50 percent of our push protection patterns being highly identifiable.

A

This means that we scan for these pattern types and we block them with push protection, but we have many many more patterns that we scan for in public repos. A number of those patterns have false positive rates over 10 percent. Compare this to our highly identifiable patterns that a false positive rates below one percent. This means that we have a lot more work to do to Mint good tokens as an industry. Overall, the good news is that our new feature has actually encouraged some of our partners to update their patterns.

A

So, to recap, if you mint tokens that are highly identifiable, you can lower the false positives for those secrets and lowered. False positives means that we can detect secrets with more confidence and high confidence, and detections mean that we can protect those secrets from ever leaking in the first place. So with that in mind, what can you do now as a developer? Remember, security is a shared responsibility, so every person working on a project should consider security rather than leaving that to your security, QA auditing team. To take on that responsibility alone.

A

Pay attention to your alerts when you do get a notification about it, detection, take action and take action quickly, remediate the token revoke the token rotate the token, if needed, or reach out to someone on your security team. If you don't have a security team reach out to us at GitHub, we're always here looking out for you. Finally, good hygiene, it goes a long way what that means is using two-factor authentication to start regularly updating your password so that you're not using the same password for multiple sources.

A

I know I'm, definitely guilty of this and use a vault for your secrets. Vault is an identity based Secrets management system. It never stores a key in a persistent location and, lastly, minimally scope permissions for your secrets. For example, if you want to store a secret at the refill level, then give that secret repo level and not org level permissions as an organization administrator, detect your secrets as a first step, make sure you're actually scanning your code for leaked Secrets, an audit contributors accidentally leak credentials.

A

We know that it's human error make sure you're monitoring your alerts for potential leaks, found in your code or your non-code and remediate. You might think that your org is safe if you're using a private repo, but if one of your contributors pushes a secret to a public open source project, a hacker can use that secret to get into your private repos. This has happened, and so Begins the breach cycle, remember remediate and revoke. Finally prevent exposure with products like push protection. You can proactively prevent secrets from hitting your code base.

A

This helps you stay clean, avoid messy cleanups and reduce likelihoods of breaches due to leaked Secrets. Preventative measures are the best way to protect yourself. Finally, as a service provider, your secrets they're probably already leaked on GitHub. So what can you do? Make sure your secrets are identifiable so that they're, easier to scan for and to take action on? You can work with secret scanning for free to scan for your secrets that are leaked in public repos and prevent those secrets from ever leaking with push protection.

A

Just like that, whether you're developing code managing organizations or minting tokens, we can secure the world software together. Thank you again for tuning in, if you'd like to chat some more or just a message away,.