GitHub GitHub Universe 2020: Developer - Secure Development, 9 Dec 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: So you committed a secret... - GitHub Universe 2020

Description

Presented by Sasha Rosenbaum, Senior Product Manager, GitHub

Every month, GitHub detects about a million potential secrets in public repos, and one in every seven private repos has a potential committed credential. In this talk, we'll show you what you should do when you inadvertently commit a credential; how to doing so in the future by externalizing your secrets; and how to use GitHub to scan your projects to make sure no secrets make it into your source code again.

For more from GitHub Universe 2020, visit https://githubuniverse.com

As always, feel free to leave us a comment below and don't forget to subscribe: http://bit.ly/subgithub

Thanks!

Connect with us.
Facebook: http://fb.com/github
Twitter: http://twitter.com/github
LinkedIn: http://linkedin.com/company/github

About GitHub
GitHub is the best place to share code with friends, co-workers, classmates, and complete strangers. Millions of people use GitHub to build amazing things together. For more info, go to http://github.com

A

Thank you so much kath and brian. I am super excited to be here, and hopefully uh we get to talk about something you haven't seen before, and I give you some details about very cool features that kind of has to offer. um So without further ado, I'm gonna jump into the presentation I do want to mention like has mentioned that my teammates are right now in the discussion board. So if you have questions please head over there and ask the questions um and then you're going to be able to um see what's happening so on.

A

Technology technology is complicated. Let me start by really quickly introducing myself. I started my career as a developer 15 years ago. I can't believe it's been so long either and then I kind of transitioned into ops and devops infrastructure as code and things like that.

A

I was an architect for on the microsoft for microsoft for a number of years and right now, I'm a product manager uh for github, and you can find me on twitter and on github and by the same name, uh divine ops, so please feel free to reach out even after this conversation.

A

um So, oh wow, something is weird about the next slide, but, okay, I so, let's get started about I talking about the state of security today. My favorite analogy for this is saying that you've just built a house, so imagine that you just built the mansion of your dreams and you've installed all the most expensive furniture. Some of the best appliances features fixtures. Everything is just completely amazing, but you just didn't install any windows or doors you're. Just like okay.

A

Well, you know the important part is to get all the features in and to get all the furniture and the windows and doors can wait right. This is kind of what we think about security today, as developers. We just keep continue to say that, like okay, we're just gonna work on the features and release them into our product, and if we have time left for security in the end of it, then we're gonna address that. So, if you've seen github conferences before you might have seen this okay.

A

Okay, I can try to switch mics if you like yeah. It sounds like uh all right: okay, lots of lots of issues all right, so I started off not so great. Today. It's fine, hopefully you're still with me. So um if you've seen sorry.

A

A

All right, can you hear me now, hopefully, so all right so yeah.

A

Sorry go ahead.

A

So sasha it sounds like you're going live now all right, let's try again so um I'm sasha rosenbaum, and this is a session on what to do if you've committed a secret. So, let's go on to my next slide and I'm just going to skip the intro this time and we're going to talk about this state of security today.

A

So again, my favorite analogy for this is imagine that you just built your most amazing favorite mansion and you have just installed all the most expensive furniture and all the most expensive appliances, and you just forgot to install doors. So this is sort of how we treat um security today we just kind of say: okay, we have time for this. We're just gonna first deliver on all of the features and then we're going to basically add some locks on the doors and some doors to our mansion right. Because features is what's important.

A

If we look at the chart of how we deliver code, then we actually see that the more lines of code we write the more security threats we actually introduce, and so we don't learn from our mistakes. We just basically continue committing bugs and moving on and just like committing secrets, committing uh security vulnerabilities into our main code base, and despite that, as developers, we often treat it this way. So we just basically say: okay, it works in dev. So it's security problem now right.

A

So I have developed my code, I committed all my stuff, and now I can just get and uh ship it to security, and then security is gonna fix it all for me, but we know that that's not possible, because there's so many more developers than security professionals.

A

Basically, we have about 45 million estimated developers and only 70 000 estimated security researchers in the world, so our security researchers just can't fix all of our issues for us.

A

So today I'm going to basically talk about how github can help you fix your security issues for you. um So we have a large number of features in on github advanced security that can help you integrate security into your developer experience. So it's not extraneous to what you do, but rather it's shifted all the way left and integrated into your development lifecycle.

A

So basically, we help you secure multiple parts of your code, so we help you secure the supply chain, the code, the development lifecycle and, of course, your platform. But today I'm going to talk specifically about the code part and even more specifically about the secrets part. So why is um secret? Why are secrets important? Why is securing your secrets?

A

Important, well we're looking here at a chart that comes from a report uh in 2020 by verizon verizon data breach investigation report and basically these numbers don't add up to 100, because some of them overlap, but not all of them overlap. So if we look at this, we see that 37 of all breaches actually use stolen credentials and in this particular chart this is not including fishing, so phishing is classified under social engineering and it's in a different bucket here. So this is just stolen credentials like something that's picked up from your test servers.

A

Something that's picked up from your repository, something that's picked up during brute force attacks, so we can see that almost 50 percent of all the breaches in the world actually include stolen credentials and yet we're pretty bad about. Not committing these credentials or these secrets into source control as revised stories and this where secret scanning actually comes in, so um we have different uh sort of different oriented features for public and private repo. So I'm going to start with public repos for public reapers.

A

We actually have secret scanning for a number of years already and we are doing it at massive scale. We're scanning about 150 million uh pushes a week. So basically every push into a public repo basically gets scanned by github to identify secrets and many people don't know that it's happening, because just because it's transparent to you so um on it just just to realize the scale of problem. This is just github api keys committed to github public readers.

A

So it's uh we can't share data from our from other providers, but we can track the data for ourselves. So we know that there's hundreds of github api keys committed to github public repos every week, there's rain this chart. It ranges somewhere between 200 and 900 for every week over the past year. So this is actually pretty bad. um Now github works with about 30, uh actually more than 30 different uh partners to identify top used secrets and to be able to identify them in the repositories and then handle that.

A

So how do we actually handle that? Well, if you, as a user check in your code and that code actually contains the secret, so we actually run different, regular expressions from known secret scouting partners to identify if something you committed actually matches one of these secrets. So basically, we then send the matches to a partner, verify endpoint, and then the partner can verify if it's a real secret or if it's not and then, depending on what the partner wants to do with this. Sometimes they revoke the secret, and sometimes they also email, the user right.

A

So basically we have this process in which I completely transparent to the user. In some cases, your secret actually got it gets revoked by the partner, and so your secret is no longer in the repository. So if you say committed an aws uh secret into your repository, then, uh instead of someone stealing your credentials, we're actually probably gonna revoke it for you. So um let's show a real quick demo of this, so this one is actually playing from the video because it's a little bit easier.

A

um But basically I am going to uh commit this secret into this very, very simple public repository I so I'm just uh typing in a token on right and then I'm just going to commit it directly into the main branch of this repository and then it just takes about a couple of seconds, but almost immediately I get an email. So this secret was a doppler token um and the doppler in this case they automatically roll the secret for me, so they actually rolled the secret.

A

For me, I didn't have to do anything the moment they identified that I committed my secret into a public repo. They immediately handled that for me and again keeping my code more secure.

A

So what do we do for private repos? Well, private repos are a little bit different, and so this feature is right now in beta and very very close to going ga. But um basically we find the secrets uh the moment they are pushed to github. There are secrets from the same providers that we talk about in terms of public secrets, um and then we are doing a number of other things that we're going to talk about um in this presentation to do uh secret handling.

A

So um before we continue talking. Let's just uh let me show a quick demo of this and then hopefully this works, so all right all right. So hopefully let me know if you can see this okay, um so basically this is a repository that I have that has uh already actually a secret in it, but I'm going to commit a couple more just to blow it out. um So I'm gonna go to my into my script again, some very simple code in here. So I'm going to just add.

A

Oh that was not what was supposed to be in my clipboard, but um typing is hard. So all right, I've added a couple things again, I'm committing straight to my main branch, because I live dangerously.

A

Don't do this at home, so uh we're going to go into my security tab and almost immediately, you see- that's even been less than a second, but I can see that instead of one security alert, I now have three security alert and I can see that I've identified a number of different keys in this repository. So I have a stripe key and a google api key and a mail gun api key.

A

As we mentioned, we've worked with different part partners on different, regular expressions in terms of what you can see right and then I'm going to just go and um choose one of the secrets and say: okay, I want to handle this. So in this case I have a number of options in terms of how I'm handling this, so I can say it's revoked.

A

I can see it's say it's used in text or tests or if it is a false positives and stuff like that, so I can go into revoked right and once I did that it actually shows up under closed right, so I can actually go and then um you know choose to unmark it as uh revoked and stuff like that. If I made a mistake, but also I can see this nice added trace right, so I can see exactly who marked this as revoked.

A

So if I needed some auditing trail in terms of what actually happened here, I can I go into this repo and review this all right. So let me stop sharing my screen, and hopefully I can go into my next slide. So, um yes, for, even though this feature is in beta, uh we already do about two million pushes uh scans a week, so we scan massive amount of secrets already um and for us, the most important part for us was actually how to figure out how to maintain a reasonable signal to noise ratio.

A

So we didn't want to throw a bunch of alerts out at you in terms of uh secrets that were going to prove to be false positives. So actually what we have right now is if it goes to the next slide. uh Oh so um actually, my next slide is about like potential false positives and the fact that it's not a simple problem. So an example I want to give you is gold card list so for go card list.

A

The secret part pattern actually look like this, so it's live dash and then some strings, some numbers uh with some dash, so pretty specific string, and so we didn't think we would see a lot of false positives for that. But actually we saw a ton right when we first ran the scan for the secret pattern.

A

There was a lot a lot of different urls and if you look at this, you can notice that these are all urls right, so certain um websites build urls for, say, blog posts in this particular fashion, so they all look like this secret right. So the solution to this particular problem was to uh get dictionary pass processing. So we can uh then run these the potentially identified secrets through a dictionary, and if we find dictionary words, then it's probably not a secrets, because secrets are usually random strings right.

A

We also identified a number of other different uh potential, false positives, for instance, some images that are encoded in base 64 look exactly like secret patterns from certain providers, so it's actually not a trivial problem to solve.

A

But right now, if we look at our own data for the past months, we see that 61 of all secrets that we've identified through secret scanning in private repositories has been revoked and only 26 percent of them uh was marked as false positive, and this is a process that we are continuing to improve, so we're actually going to continue to reduce the number of false positives.

A

But again, this is a pretty good result uh for us to be able to identify secrets and for you to actually have to worry about this right, because you do not want to get into a position where a secret is actually leaked.

A

All right, so we've talked about. um You know not checking in secrets, but what should you actually do, instead of checking secrets into your repositories? Well, I'm gonna talk really briefly about that, and the biggest answer to that is to use a secret store. Now, there's a lot of different secret stores out there on the market, I'm not obviously going to talk about them all, there's different types and sizes and different features for all of them.

A

I'm going to briefly just talk about our own secret store. This is actions encrypted secrets, so these are secrets that you can use in your repositories that you can then leverage in the actions workflows right.

A

So let me just quickly show a demo of this, so I'm going to share my screen again and then I'm going to go back into my repository and I'm going to go back into settings um and then roll down into secrets, and you will see that I don't see any particular secrets in this repository, but I do see that there's organizational secrets that are already in here.

A

So if I wanted to, I can leverage the organizational secrets, um but I can also add a new repository secret and I can say: okay, this is my secret name and then I can say hi. This is my super secret.

A

A

Writing is really hard. Okay, another matters in this particular case. So I just checked in the secret and again I can leverage that in my actions workflow, so I could potentially just go into my actions. Tab and I can go set up a workflow and then I can leverage the secret in my yaml file. uh So in this case I'm just going to copy paste the whole thing, because typing yammo is not a particular um good exercise and I could never do it without making a mistake.

A

But basically I can leverage this syntax to be able to leverage my secret in my actions workflow. So I didn't have to expose my secret by the way, don't print secrets to logs. We do catch that. So if you actually go and uh run this, it's going to catch it, but we can't catch them all. So please don't do this. Don't live dangerously, don't print your secrets into your logs. Okay, I'm gonna go back into my slides.

A

All right, so we basically have a lot of different secret vault providers, I'm just mentioning the major clouds and I'm mentioning hashicorp, because these are closely integrated with your infrastructure as code, so you can choose to leverage the same um exact. uh You know the same secret, key vault or secret manager uh from your cloud provider. So it is closely integrated with your workflow and it is easier to work with um again there's different types, but the important part part is that you retrieve your secrets at runtime.

A

Instead of committing them into code or even into configuration, even configuration isn't safe, so most obligations actually get breached through some configuration problems or code problems. So be careful about that all right. So let's talk briefly about what our roadmap is for this.

A

If it goes to the roadmap slide, okay, so um we do want to add an api for secret scanning finding. So this these are the things that are coming out shortly, uh so we will have an api, so you don't have to go to ui. To handle your secrets, um you will be able to do this through the github api.

A

um We are going to work on an updated permission system, uh so basically, there is admin level permissions versus developer level, permissions in terms of who can handle what um and where um we are going to work on more false positive reductions um and we are going to uh work on the org level view for detected secrets um again right now.

A

You have to manage secrets in every particular repository, which may be a little bit harsh if you are looking at a large enterprise, uh but then again, sometimes it doesn't make sense to handle this the secret pattern at the end price level, so we basically are working on figuring out what this experience really should look like um and then some other things that are more inspirational that are coming um is one: is custom patterns, so you're going to be able to define custom, regular expressions or patterns that you're looking for.

A

So, if you are generating your own secrets and you're, not a secret provider, you can still scan your repositories for secrets and you're going to be able to do pre-receive hook scanning so once you've committed your secret once it's committed and in pull requests, it's already been exposed. So it doesn't really help us to. uh You know, block a pull request from going in, uh but it would help us if we could prevent the progress, the committee from even going into your repository now.

A

This is kind of hard to do, because we can't stop commits for potential false positives. So we can only do this if we're super sure that a secret has been identified um and then again organizational white triage. So we're going to be able to uh say revoke a particular secret pattern at a repository level. So I do want to wish you all to stay safe both in terms of any you know, any secrets that you are working with and in terms of the global pandemic.

A

Hopefully github can help you secure your code and make it as easy as possible for you and um thank you so much like. I say you can find me in multiple places by the name, divine ops and I'm going to be in discussions for the next half an hour.