Cloud Native Computing Foundation CloudNativeSecurityCon 2023 - Seattle, 3 Feb 2023

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Sharing Security Secrets: How to Encourage Security Advocates - Cailyn Edwards, Shopify

Description

Don't miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe in Amsterdam, The Netherlands from 18 - 21 April, 2023. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Sharing Security Secrets: How to Encourage Security Advocates - Cailyn Edwards, Shopify

The cloud can be a big scary place, and with malicious actors around every corner it’s important that security teams have the power they need to keep data safe, and services available. Although it can feel like we are alone in our mission, and sometimes security practices are seen as burdensome - it doesn’t have to be that way! If we can take the time and make the effort to share our security secrets, introduce teams to Alice and Bob and encourage a healthy amount of suspicion; we can create a company-wide culture that cares about security. We can’t be everywhere at once - so having others looking out for security risks in their work is invaluable. In this talk Cailyn and Ann will talk about their successes in educating non-security teams, and enlisting security advocates across Shopify. They will dive into some of the methods that were well received and talk about efforts that were not as successful. Cailyn will also talk about the new security review strategy that her team launched this year. You will walk away from this session with information and ideas on how to start a security advocates program in your organization.

A

So my name is Caitlin. uh Thank you so much for sticking around today. I know this is the last talk and I'm all that's between you and probably tasty food and beverage. So I really appreciate you all being here. My talk today is sharing security Secrets how to encourage security advocates.

A

um This is just a rough summary of what I'm going to go over we'll start with the basics, the what the who the why and then we'll dig deeper into the how specifically I'm going to look at techniques that managers can do and techniques that individual contributors can do we'll finish with a couple examples from Shopify as well as tag security and Sig security.

A

So we'll start off with the easy bit about me. My name is Kalyn. As I said, I've been at Shopify for about five years on a variety of teams, but currently I am on infrastructure security. Doing a super awesome placement on our kubernetes foundations team, which I could spend this whole talk talking about. So if you want to hear more about it, please hit me up later. This slide illustrates what I do when I am not doing.

A

Computer programming things which is hang out with my fur, children for skit and Luna, go running with my husband and in the bottom corner. You can see our dream house that we're building, which takes up most of my fantasizing currently.

A

So, let's start off with what our security Advocates, also known as security champions, I like to think of them as various as little birds or as the neighborhood watch. So these are folks that are spread across your company, not necessarily on a security team that are thinking about security. You can see the definition here. Cyber security Advocates attempt to reduce exposure to cyber attacks by promoting security, best practices and encouraging security adoption.

A

So you can imagine, not only are they influencing the projects that they're working on to make secure decisions, but they're also able to pull in security teams when they think that it would be reasonable, which is nice.

A

So, more importantly, who are slash should be security advocates, spoiler everyone, anyone all of you, please be a security Advocate. It's awesome. um It's really important to note that they don't have to be cyber Security Professionals. Those are going to be the natural choices, obviously we're all here.

A

We care about cyber security, that's something that we're already doing, but um it's even better if you can find folks that are in other areas of expertise that can bring unique context that we don't necessarily have and then apply security mindset to that we can't do it all so, hitting more on that is we can't know every single thing that's happening at our companies, particularly in a large company, but also in a small company, because, if you're in a small company likely your trust orc is not that big.

A

In fact, there might just be one person who's responsible for securing the company, so it's really important to have people looking out for you and wanting to pull you into things where they think that your input would be valuable and security. Advocates are a great way to do that.

A

So why do we need security? Advocates I've touched on it a little bit, but I'll go into some more of the details.

A

um We've all seen these headlines- hopefully not too many of us- have had to deal with responding to these headphones Headlines by being at the companies, um but security is part of our everyday experience. I think like most people who don't know anything about computer programming, know about hackers. They know about compromises. This is what the media sees, and this is where crime is happening. This is where more and more sensitive data financials health information is all going online and this is what we need to protect.

A

So as security Engineers, we are small fish in a big pond. That means we have less influence and sway in a larger organization. It's I, don't know a company that has a larger security team than General Workforce, because due to necessity, you need to have more Engineers to build the product, but it means it's really tough for us to secure everything. Like I said, we can't be everywhere at once. So how as little fish, do we make sure that that whole Pond is secure?

A

We chat up those big fish and we make sure that they come and they see us and they tell us and they talk to us and we make sure that people care about security and want to ship secure products.

A

At one example that highlights this really well is uh Conway's law, I'm, not sure, if you're familiar with it, but it states that an organization's structure will be reflected in the software they build. So if you have one big flat structure, it's likely that your company's software will be kind of Monolithic and one Big Blob, and if you have a really separated culture, then you're going to end up with something that's more serviced and split. And so how can we look at that from a security perspective?

A

If you have your entire org in this tiny little bit thinking about security, it's very likely that your software is going to have a tiny little bit of security coverage.

A

But if we add security into the organization at different levels, each piece of software is likely to have a security concept or be thinking about security, which I think is exactly why a security Advocate Program is an awesome idea in the cloud native security white paper.

A

There was this quote which highlights exactly what I'm trying to get at in this talk, which is by integrating security as early as possible throughout the development life cycle, or even earlier with interactive developer training security organizations can enable preventive security rather than reactive, security, and so it doesn't have to just be security organizations. It can be within an organization all the time. Our developer teams hit us at the point of release and they say hey.

A

Can you just give us the check mark so that we can release this project project, or so we can move tears and all of a sudden. We have to say so sorry, but you need to fix this and this and this and this and now we're impacting that team's goals and all that they're seeing the security team, as is a burden and a block to them meeting their cycles and addressing the pressure they have from above.

A

um So now that we've talked a little bit about what security Advocates are and what the goal is we're going to talk about how we can Implement them and I just want to note that there are a whole bunch of really fantastic formal education programs for creating security, Advocates or Champions, and those programs are focused around educating and having a formal role. This. This talk aims for the stage before that, so educating and garnering interests so that people want to participate in programs like that and just having a general security culture at your company.

A

So what can managers do? Disclaimer I am not a manager nor have I ever been a manager, but I've harvested some really good tips from The Amazing managers that I work with primarily Ann Wallace first off is Be A Champion yourself.

A

We need Champions at the top level of companies, because if the company is trying to sell security as a goal- and there aren't security advocates in senior leadership, you're going to have to justify that business cause, and it's really hard to explain how security makes a company money um I mean we don't we save companies money and when security is going really well, you don't know what's happening because you're not ending up in the headlines, It's Just Happening seamlessly.

A

So if you have folks in senior leadership positions a they have the knowledge to sell security as a business expense and be or as a business argument and B, it gets built into the company-wide goals, we're all in a really tough time right now.

A

For software, it's not, you know, just throw money at everything anymore, and so it's really important that teams are working toward business goals and we need educated folks to argue for security, so that that can happen, and this just goes a little bit deeper into that, so ensuring that not only just at executive leadership, but in management as well, that your okrs and your team goals and your cycle goals have security as part of them.

A

It sucks to be a security team and not feel like anyone is valuing the work that you're doing instead you're looking at. How can we increase users, increase income um and I? Don't have the skills to tell people why we really really need to invest in security, so having senior leadership and management? Do that is very. Very important.

A

Next is uh the more fun part which is support and sponsor events like hackathons or um learn to hack. Any kind of interactive events that you can you can sponsor or host will help spread security awareness in a really fun and approachable way. A few ways that we do this at Shopify are a learn to hack Workshop. So this exists in our in our repository and anyone can do it at any time. It's super super fun.

A

We've got like a really vulnerable web page and we walk you through finding different exploits and during any kind of security awareness periods of time. We will run in person workshops where we'll actually have a whole bunch of security. Nerds help you do this and be really excited that you're taking the time to learn and then a big thing.

A

Every October we do cyber security awareness month, I'm an infrastructure security engineer, I do kubernetes, that's what I like to talk about, but at this point in time our whole trust org comes together to share all sorts of security information and we go well beyond you know: don't click on this phishing email we go into how their services are protected and how they can involve Security in their day-to-day workflows.

A

So what can ICS do? This is much more my area of expertise, and this is something that I've been doing a lot of since I started on a security team at Shopify.

A

um First off be a security Advocate yourself do what I'm doing stand up, tell people about security? Why it's awesome why we should educate people just do it just talk about it get out there next is consult, don't dictate. So this I mentioned it a little bit earlier, but I've been a part of a few projects that have involved going out to various teams and having them actually much like the talk that just happened before this.

A

Having them run, do a patch or having them deploy some kind of security measures that we have decided are mandatory and often by the time, I get in the room with them. It's not that they don't want to secure their product. It's not that they disagree with me, but I'm actively, making it harder for them to achieve their goals.

A

They have to pull people off of other work to do the security work and they don't want to, because no one on the team has invested, their leadership isn't invested and I'm just this person telling them they have to do a thing.

A

um So we all know when we pair program you totally change the way that you're approaching a problem all of a sudden you're writing the best code you ever have, and you can't type it all, of course, because it's terribly stressful, but you when you work together, you're going to both bring your context to make something better and you're, also going to do good work, so I think trying to create a security.

A

Advocate Program means that in the design phase you're helping people learn about what decisions they can make to make the product more secure and then, when it comes to that, oh we have to do the security review. It's just a check mark because they followed the steps educate so again, I'm doing that right now and uh everyone I've done it at Shopify in a variety of ways: um I love! This comic I wish I wish.

A

I could read it out, it's very funny, um but basically you know I I, remember when I joined security, everyone was like oh yeah and like Alice and Bob and Eve. Who are these people and why are they so popular in security? But no one explained it to me because everyone took it for granted that you know who Alice and Bob are you're not in a security room, and you don't know who Alice and Bob are people don't know and people don't know a whole bunch of the things that we think are really basic.

A

So I think that it's important to educate and educate really openly and welcome lace start with the basics. Don't ever hesitate to to motivate people to ask questions. Explain everything as much as you can so that people aren't feeling intimidated by security.

A

um So when I've crafted a few uh educational programs at Shopify and I'll talk about some of the focus points, I have when I'm creating content. So the first one is to talk about security incidents, both internal and external. We have done at Shopify a few sessions called um anatomy of a hack, and we take this opportunity. So a recent one was after the Uber hack, we didn't. We saw a whole bunch of people in our slack channels being like, oh man. This is crazy. This happened at Uber. Look at this Hacker News article.

A

Oh, my goodness, security. It's awesome are we doing security at all and then our security team was like. Yes, in fact, we do a lot of security and we'd love to tell you about it, and so we hosted these sessions myself and a few of my colleagues were on a panel and we started by explaining the incident again demystifying it going to all the details of what happened, why it happened, how Uber responded and then we did our best to explain how a similar attack might happen at Shopify. What preventative measures we have?

A

What would we do at various stages of the attack and people were really really really interested in it? We finished with it ama, where folks were able to ask a whole bunch of questions about the way that we do security at Shopify, and it just got people engaged in talking about it.

A

The next one is how to threat model. uh Recently our team has grown and we've ended up with a whole bunch of more traditional background security. Folks that introduced a lot of threat. Modeling I had never done a threat model before and now I've done. Countless and I think that this is. This is now something I do in every security education program I run at Shopify, because it allows people immediately to apply the security principles to their own work.

A

Not only that, but it's a it's a way of thinking that is already ingrained in developers like look at your system, find points of failure. That's something we all do every day when we're writing in code and dealing with software and the threat model is just an awesome way to get people doing things right away.

A

Next is I'm like really hammering this nail in as someone who joined software previously, being a farmer cover the basics, don't assume any knowledge? Definitely, if you're doing your third or fourth educational talk, you don't want to start it off with like okay. This is what security means so create recorded, accessible content that folks can go and look at before attending your sessions. Send it in the email put it in the event.

A

Have it as part of your onboarding, so people can go and know the security basics and not feel intimidated when they're attending anatomy of a hack and getting into some of the deeper Concepts explain the Security Org so again, I said: I have worked in a whole bunch of different places at Shopify and I used to just work on our core product. Doing um back-end, rails and I didn't know that security existed.

A

I didn't know that we had a Security, Org I, certainly didn't know what they did and I find that a lot when I talk to people like oh wow security. What do you do over there? How many people are you? Do you guys even write code? So I think explaining the Security. Org is just a great way to have people know just I mean not always Shopify is really lucky, but how much access they have to Security Experts and the last one is to go over security tools.

A

So at Shopify we have this awesome internal, just-in-time access, control that allows people to get permits to access resources. It's really great. It works in a variety of ways, and it's one of the tools that we own that pretty much anyone at Shopify can make use of, and so every time I talk about security I tell people about this tool. It's called cloudu, it's awesome! This is how you could use it at your company or at on your team to control access to something that might be a sensitive resource and I.

A

Think we all have that. It's a great opportunity to say Hey. You know that little thing that pops up on GitHub that's actually a tool we created, that's checking for x, y and z, okay, now the examples part um so recently, I again disclaimer I wasn't directly involved in this project. um Some really amazing people on my team were, but when I proposed this talk, um I immediately thought of this project, the goal we did uh our annual planning and decided that we just wanted to be involved in every project.

A

We wanted infrastructure security to be everywhere. We can't do that. There's not that many of us and there are a lot of different projects and services being created all the time at Shopify. So the way that we decided to handle it was to try and move up our involvement in the development or project process, so it at Shopify. We call it get done, but it's just our regular project workflow. It starts with a proposal moves to a prototype.

A

We trash the Prototype, hopefully and then build the product and release the product, and previously we were called in only at the release stage to do a Security review or again. If they wanted to move tiers, we would get called in to do a Security review now with this new project, we're involved as early as the Prototype, because it's automated and built into GitHub.

A

So it's possible that they're already getting feedback on decisions that they're making in their earliest, probably sloppiest code, which hopefully influences the decisions they made in the build, but at the very least, we're being pulled in to the build before they've before they've had to request the security review. So to me, that's like really really huge and awesome.

A

Some of the goals of this project were a shorter lead time for Security reviews. We were pretty good about getting to Security reviews. We have a team meeting every week. We would triage any new reviews and usually someone would tackle them within the week, but after they would do the review, it would result in some actionable items and then we would have to re-review so I think the average was about two weeks which, if you're a service owner trying to release sucks.

A

So we wanted to reduce that as much as possible and then tied into that is we wanted fewer tier move blocking change requests, so we wanted to try and have all of those big blockers sorted out right away as soon as they make a decision in their code. That would be a blocker, something should alert them. That they've made a dangerous decision, uh 70 faster team onboarding to review process um I, think we say three to six months at the moment.

A

It would take someone to feel comfortable running a Security review on their own, um so we're hoping that really the tools doing almost everything and that people could join and do it within a month or two and lastly, is infinite. Infinitely, better relationships with service owners, so much of the rejection of security comes from developer, toil and not adequately.

A

Considering um a developer, workflow I think Shane touched on this in his keynote, but when we put in blockers for folks in something that they do every day, not only are every time they hit that blocker are they going to think uh Hey security they suck, but they're also going to think how can I get around this blocker?

A

Is there something I can do they're going to try and look for ways to get off that paved Road, and that is not the kind of behavior and thoughts we want around security, some wins already, so this project is still in process and I. Haven't really explained what they did at all, which I should do. Basically, it used to be a really well documented list and you would go through the checklist and you would do a bunch of things and make sure that all of our security baselines were being met.

A

Now all of those checks are automated and it produces a report, and that report is sent directly to the service owners and any vulnerability that is possible to surface in. That report. Report is extensively documented in our docs so that they can go and start to fix those issues right away. So the first one is that now security concerns are surfaced in the GitHub UI.

A

It's promoting service. Every time they make promoting awareness every time they make a change to their code, they're, seeing that directly in the CI, which is awesome and it's stopping vulnerabilities for making it into production, which is extra, awesome and then the next one is what I just mentioned.

A

There's a report generated for service owners, so the review results in that list of vulnerabilities and they can address them right away and still wait instead of waiting for us to come to them and tell them how to address them and the last one is that issues are open from scan results. So this just fits a little bit better into a standard developer. Workflow in my opinion, and means that, instead of a comment on the review they're seeing the issues in the same way that they're triaging all of the work for their project.

A

um And so the next one is security self-assessments, which have also been talked about a few times uh today and I think are also a really great way to build security, advocacy, um both tag, security and recently kubernetes. Six security have the self-assessment program, and this line is taken from TAG security. The self-assessment is the initial document for projects to begin thinking about the security of a project determining gaps in their security and preparing any security documentation for their users.

A

It's a little bit less formed so far in six security, but basically we're doing it on a volunteer basis. Any sub-project of kubernetes that wants to do a self-assessment can come to us.

A

Hopefully they have their data model, but if they don't we'll work with them to create or sorry their data flow diagram and from that we'll create a threat model and we'll talk them through the various vulnerabilities that exist and then work with them to create actionable issues that again will exist on the repo and hopefully get folks to go and pick up those issues and work on those vulnerabilities and similar to tag security.

A

It results in a really nice documented security summary for those projects and Sig security is planning on running a workshop to do threat models at kubecon and eventually, potentially as part of the contributor Summit, but if there's interest, maybe also as part of the mainstream at coupon we'd like to do uh like walking sub projects through this process in a way that people could observe and participate.

A

So the summary my takeaways for this talk how to encourage and build a security, Advocate culture so first off attend this talk, start looking at how other companies do it just think about? What's out there start advocating yourself tell people about the cool things you're doing really, if you are a manager, start advocating for security and get other managers to understand why security is important. If you're, not a manager, then finds the managers get them to support you.

A

um Let them know how important this is: don't gatekeep! So that again, is me really dwelling on, like don't assume anyone has any knowledge, don't make security, Mystic and secretive pull everyone in it's awesome and they should want to help us make security education relevant for your audience, create opportunities for Hands-On learning. That's talking about those like learn to hack workshops, capture, the flags, any of that stop collaborate and listen talk to teams as soon as you're going to work on a security product find out who it involves.

A

We have customers as security teams, that's everyone that we work with and all developers and they shouldn't be finding out about the tools we're creating to secure them after we've already developed them. It's the exact same problem in Reverse. We need to know exactly how the tooling we're creating is going to impact the people that are going to use it or have it built into their workflow and then get security goals at the top level. Okrs company Mission guiding themes I want to see the word security when I'm.

A

Looking at what my CEO is telling the entire company, we need to focus on which we actually did this year and I'm very pumped for those the first time since I've been around that it's been said like we need to be secure.

A

Thank you so much for coming. um You can find me on Twitter on GitHub in the kubernetes slack as Kalin codes on Instagram. If that's your jam and you want to see lots of dog pictures food um feedback, please, this is my first ever talk outside of Shopify I'm sure you can't tell how nervous I am um so, please scan the QR code. Let me know how I could do better next time and what I should keep doing, and finally, q and e um the E for experiences.

A

If you have something that you do at your company, that has been really effective, I want to know about it. You can tell me now- or you can come tell me after or you can contact me in any of the ways I listed below.

A

Does anyone have any questions.

A

A

Yeah, so uh the question was: how do we get involved in the Prototype and design phase in a distributed company, uh because that can be really hard? There's one advantage is that Shopify has a central area where we, um where we do our project proposals, so those are accessible to anyone and we can see those. So we can. We can use the API from that to get notifications, but, more importantly, having come from um from being on a whole bunch of teams.

A

I was able to build up a really big network of folks at Shopify, and my special interest is network security, so I am often reaching out to the folks that I know from my time working on networking teams, saying hey: what are you doing? Are you doing? This? Can I see a road map per chance? I would love to see what you're working on and that behavior doesn't just benefit me like. It recently has resulted in a in a cross-team project. That's really valuable to both teams and now I.

A

Don't need to reach out to that person they're reaching out to me so creating that culture again of of sharing and talking about it. It make it friends anyone else.

B

Are not obviously so when you're.

B

How do you justify education.

A

That have much more immediate metrics yeah, that's a fair question, so the question was: how do you justify security education, basically as a as a business goal when there's so many more important goals within security alone, as well as the rest of the company's schools?

A

I, don't I haven't done too much of this, like Shopify I've been really lucky, they've been really receptive. There are a whole bunch, particularly because we're distributed there are a whole bunch of opportunities to educate um and and I I like doing that so I've taken that opportunity, but one thing that I have not tried yet but I was thinking would be really valuable to get.

A

The momentum is create a special interest group and have again have those people that are not on security, so you're not pulling security folks away from security projects full-time, but you have these sessions, these special interest groups. Maybe it's just an hour where you meet and you chat about what projects are you working on? Do you have security concerns?

A

I would assume so because you showed up to a security special interest group like how can we give you the tools to um to move forward with that project and then, hopefully, you're able to approach education, not from the perspective of I'm, a security, nerd, I love security I want everyone else to be a security, nerd and love security, but you're approaching it from I've talked to so and so, and so and so and they're working on these projects, and they don't know how to address the security concerns and in the past, We've Ended up at release date delaying their project by two weeks because of a Security review having to make fundamental changes that would have been like would have not meaningfully added to the project time had they been earlier and again, that's like I think we all who work in security know about the Hard Sell of like it's not going to be a direct, easy correlation.

A

But hopefully, if your team is already if your company is already investing in security, they'll follow that that thought process, but I so far have not had to convince a pointy-haired boss to. Let me educate people about security.

B

To figure out like.

A

Yeah, so the question was: do we have any metrics for the impact of these kinds of programs so that after doing them, we can argue for continuing to do them summarizing questions? That's very hard! uh No is the first thing: I, don't think we have any clear metrics.

A

What I can think of offhand is one one of the educational formats we have at Shopify are called coffee and tea sessions, so they are, anyone can volunteer to do them and you basically have an hour to talk about something that you're very interested in and usually they're like involve a technical demo or something, and with that you can see like how many people signed up how many people actually attended, and then we also have feedback for those sessions, so that could be.

A

One way is like if, if 70 people at the company want to take an hour of their day to listen to someone talk about security to me, that's indic, indicating that there's like there's a need for that, and second I think that I'm not on the security review project, but I do think that they're actively measuring the difference in time.

A

From when a Security review is started to when the project is able to be released, and that's going to result in in metrics that one's a little bit a little bit less education focused because obviously it's it's automated tooling, like the education, is part of them just doing the security work. But to me it's the same thing. It's like we could take even more time off of that if they don't have to read a list of security concerns, and they just know how to uh how to secure their services.

A

All right well, thank you so much and I'm so glad to have talked to a whole bunch of you here this week. This conference was awesome, enjoy the rest of your night.

A