GitHub Demo Days, 16 Dec 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Mercado Libre's SAST Team and CodeQL #DemoDays

Description

Mercado Libre (NASDAQ: MELI) is Latin America’s leading e-commerce platform, and LATAM's most valuable company by market capitalization.

In this edition of Demo Days, we join MELI's Static Application Security Testing (SAST) team. We’ll learn about their unique approach to application security, and how they are using GitHub CodeQL to secure the code that powers a platform with over 132M active users.

Valeria: https://www.linkedin.com/in/valeria-silvestri/
Camila: linkedin.com/in/camila-seoane-053361227
Natalia: https://www.linkedin.com/in/natalia-lucia-pesaresi-b835a1183/
Lucia: https://www.linkedin.com/in/lucia-ines-romero-4a099114a/

https://codeql.github.com/

A

Hello, hello: everyone welcome back to demo days thanks. Everyone for joining us today today is going to be particularly exciting, as instead of us talking about how we do things at github we're going to explore application security from the perspective of one of the largest e-commerce platforms in latin america.

A

I'd like to welcome my co-hosts for today, joseph and andrea, who will be introducing our guest from mercado libre, welcome.

B

To devil days, I'm joseph and today with us, is mercado libre the leading e-commerce and fintech platform of latin america with over 132 million users wow. Today we are going to learn about their approach to application security.

C

That's right, joseph thank you so much everyone for joining us and we are super thrilled to be joined by this power team of engineers, ready to dig in they're going to share their experience with us. How they've been utilizing cookie well and over all their experience with star application um security testing. So without further ado, let's get started. Welcome, valeria, natalia lucia camilla. Welcome. Welcome, thank you for being here with us today joseph go right in.

B

First question is to lucia: can you tell us about your role at mercado libre.

D

Okay, our team, our main goal- is to prevent vulnerabilities, reaching production so mainly because it's so much easier and also cheaper to fix any bag, even security ones, while coding, while developing, rather than several months later, when it's already productive.

D

So I think study analysis help us to find some vulnerabilities that we wouldn't be able to find so easily, maybe like doing some penetration testing or so so. This is our main goal here in mecca river, as the sus team.

C

Awesome, thank you lucia valeria. Can you tell us a little bit about how mercado libre implements the sas program, which you folks are the team you're? The first teams.

E

Yeah sure, as luciasl, our main goal is to catch vulnerabilities before they get to production. So the first place where we want to be is scanning pull requests right, uh so we scan for requests looking for mercado libre's, most common vulnerabilities, and when we find some, we block the pull request. That is, uh you can't merge until you fix it, and a blocking pull request is a big deal. You know um with great power comes great responsibilities and static.

E

Analysis comes with false positives, that's how it is so um developers they can report these false positives, and this will stop blocking the pull request and then later we'll review these reports and use the feedback to improve our queries.

E

That's our main goal: that's the the first thing we implemented, but then we said we can't stop here and so from a security perspective. um You don't want to know only the new vulnerabilities. We you want to know all of them, so we started scanning the whole code base and looking for legacy vulnerabilities and, on the other hand, developers started asking us for a way to run this scan on their own computers before pushing a change to a repository.

E

So we developed also a command line tool that they could run in their local environment. um It would output the results in a serif format, so they can see them in visual studio and I think that's a good sum up of what we're doing in soft.

C

That's amazing, thank you so much for sharing and also thank you for showing love to the legacy code basis. That's something that not all teams take care of, and you know it's important to get over that technical debt, especially when it comes to security. So natalia question for you um because of course, maybe you used another tool in the past. uh I'd love to know if you or your team have used any other sas tools before.

F

Yeah sure, actually, we complement, call ql with other tools. One of them running now nowadays is based on regular expressions and dynamic validators to search for a hardcore credentials in the goal. We also are using a semrep as an alternative.

F

We doesn't need to build the cold ways, because we have some c plus plus applications that need an environment that is hard to replicate, so it was easier for us to put them in a github action and something else we want to to tell us to tell you is that in 2018 we built our own sas tool and that tool analyzed only growing and go applications, and what one day someone said.

F

Well, we shouldn't use a groovy anymore, try to migrate all that applications, and so we need something to escalate and continue doing just, and so we start using a call qn.

B

The next question is for lucia. We at github have codeql. We would like to say that it is our industry-leading semantic code analysis engine. How did you hear about code2l.

D

Okay, the first time we heard about codeql was back in 2019. I think we learned about microsoft experience and we thought you know yeah, that's exactly what we need, but later in 2020 last year, when assembly joined github, we are here in mecca olivia. We got github enterprise. So that's when our journey with ql started.

C

Yeah such a great partnership there we love that team um that horse went through so all right, well wondering about. um You know the beginning, learning new things: um natalia camilla, I love. If you could share a little bit about how you personally got started, writing queries.

C

Was it difficult um just a little bit of sort of your journey? Ramping up it'd be great if we could share with our audience.

F

Yes, it was a lot of trial and error. Some some steps were easier than other, but we just first. We searched the cochlear repository for examples because we didn't know about the language and there are lots of queries in all the language supported, so it was held for for us. Then we had training sessions with the github team to solve a specific problems we were having with with some things and when we were ready, we wrote our own documentation and our workshops for our mates, so camilla took one of those workshops.

G

Yes, in my case, I participated in the internal cultural training that the girl did where they took the github training as a basis. The documentation that they were peering together was very, very useful for me, where I had step by step on how to fail learning in order to get to the point of making my own query and improving the existing queries, it was very helpful to do a research on the queries that they already had implemented. There was also a lot of trial and error. Yes,.

B

This is so impressive. I mean I love how this evolves from the start, to what what is now and question to camilla is how many members of your team are writing code ql at the moment.

G

Yes, uh the four, the four of user, the qr developers maintain were the ones who write to queries to scan and block pull requests. We need to achieve a high quality standard. We also tell qr to the rest of the application security team. Some of them write a quick queries to assist them in pen test.

B

Thank you so much for sharing this and valeria. How are you using code12.

E

Okay, um camila said our main goal is usually to find vulnerabilities.

E

So uh when you want to write a new query, the first thing we do is to decide like a generic pattern that describes this vulnerability, and then we start searching on the mercado libres code to find the many variants in this where vulnerability could appear, because developers can get really creative, so yeah. We need to to know all these different ways of doing the same thing and then, of course, we also need to know how to fix this vulnerability and again the many ways of fixing it.

E

So once we had all these these examples and we start writing the actual query and we try to support at least most of the bad cases and the good cases right and the whole thing about this process is that um once you're done you're aware of most of the false positives and false negatives, you can get right because you first had the examples.

E

um The the other use for call qa here at marco liure is usually related to interesting for code exploration in the recon stage. uh Culturally is very useful to list all the end points and map the attack surface. So that's the other guys.

C

That's so interesting. Thank you so much for sharing that I love I mentioned that developers are very creative, yeah, of course, and then it takes you know. Obviously it takes a great team of um super smart folks like yourselves to stay on top of everything that comes in to making sure that code is secured. So that's amazing, um thank you for sharing that another amazing component to cook ql itself. Of course, it's equal system, the fact that the queries are open source and you touch a little bit about how you started.

C

Writing your own queries and how you started actually taking a look at the beginning. What existed, but I wondered now um since you've been using coke ql. Are you using the community queries at all, and I love it if lucia, if you could dig in a little bit on that, that'd be great.

D

Yeah sure so the open source queries are extremely useful, uh mainly for inspiration and also acid samples. So we could understand the pattern that we are we want to model, although because of mercury, private libraries, we were unable to use them as they come.

D

So we have to model those those private libraries we have a bunch of them, and so we have to model that and then extend these open source queries, but we even model quite a different pattern for uh some queries due to our internal architecture and as bali said before as the way things are done here in belka libre, so also because of the way we're using coql.

D

As wally said before in our continuous integration pipeline and block input requests, we needed the queries to be like extremely accurate, so we pay close attention to uh reducing positives modeling, a significant number of sanitizers and sanitized cigars.

D

They have for those in in the medium sanitizer guys are like in transitions like in simple words, functions that make your code safe so and these sanitizers, and also this quite of unique way. We model some vulnerable patterns are the new contributions that we made to the calculus repository, so two server side request. Forgery. Queries are now in the experimental folder for both both go and javascript.

C

Thank you so much. I was going to ask you about that. Actually, next, um obviously, because the nature of the queries being open source and the fact that you're now contributing back, um why it's so important, you know to look a little bit beyond consumption and start seeing how you can contribute, for, I guess the greater good of the ecosystem joseph. Do you got any thoughts on that?

C

I know this is something that we talk about a lot internally, about the importance of contributing to open source, um how you know, maybe going beyond finding your solution and seeing how you can help others find their solutions right.

B

Of course, open source is all about giving back to the community, and here at github we have. We have a community of people contributing back to cyber security through code 12 queries. One of these are with us today and thank you so much again for being with us, but we also have um a huge amount of people that love cyber security and, of course, they'll have to give back to us their knowledge through codifying this knowledge in code ql.

B

um We of course love to have more contributions in the future and, at the end of the day, open source is about people contributing together coming together and um help each other.

C

You said it: that's that's the beauty of it. Yes for sure, so I'm definitely a huge advocate for that, and I know this team is being as they're contributing back to the ecosystem, so lucia you mentioned. There are two uh now queries that are available that you contributed back, uh maybe at the end of the show, we can link that in our notes.

C

um Aj in the production booth can link that to the notes, so that folks, who are watching, can take a look and then just kind of explore and see what what you give them back and then maybe you can give people ideas on how they can actually start getting their queries. Ready to, you know, go back into the contribution, so that's awesome.

C

Thank you so much. I feel like we kind of flew to our questions, because that's what happens when you have a power team of folks that know what they're doing you're very efficient. So that's amazing. um I I'd like to take a moment to ask a question.

C

um Well, we'll go back in touching into open source a little bit later, but then I'm gonna pass it on to joseph now, and we have a question for everybody joseph. If you wouldn't mind asking and then maybe after that, we can take a look through the chat and see if there are any questions for either joseph myself or the team. um Anything that we can. You know, help the folks who are watching so that they can get started in their journey with coql or anything that they can learn from this amazing sas team.

C

So go ahead joseph I love this question. This one's coming up is my favorite.

B

So this question is coming up a lot. We would like to ask what advice and this question is for all of you. What advice will you give to someone who is looking to start into cyber security to get into cyber security or may already be in a field, but would like to explore contributing to open source.

G

Yes, a new cyber security as a student, there is a much security content in most of the university, but you need a major in computer science, software engineer or others related to it. Well,.

F

If you are a qrs developer, I can say you would explore some blogs there in internet and study or search security standards. It's very helpful and maybe attend conferences and then also it's very important to show in the community and give something back for the past.

D

So, for me, one of the most important things to consider is to contribute with a solution to a problem that has no solution at the time. For sure you could code. I don't know a jason double, ah it doesn't say some web token library that is better than any library made before, but it's kind of odd.

D

So if this is your first time contributing to open source, keep it simple, so uh I guess the most organic way to contribute is you know you have a problem, maybe at work or maybe while you're studying uh whatever, and there is no solution, at least in the open source community.

D

So there you have, you could build that up and then maybe make a request uh if this is an addition to an existing project but and a big fact here, if it's not if you're, considering like uh creating a new framework or library or whatever, you have to be aware of the responsibilities that come with that project. You know no security team wants yet another of an open source library in the wild.

D

So no security upgrades- and I don't know- maybe then a mysterious developer- comes to save the day and take care of the orphan library. No, I think that we all know how that ends. So maybe a good way of contributing for the first time is to add some functionality to an existing project. You know, maybe, when you are already using it.

E

Yeah, I I think contributing to open source is a lot about giving back to the community. um I wanted to talk about our experience and contributing to cultural and, with the server side requests for free inquiries. um We we have our queries that they were um running. We we knew they worked because we use it here in america, but and what I want to to tell us advice is: it will take time to contribute.

E

um You have to adapt that query to the open source repository that may have a different um quality standards and they may ask a form for changes or corrections or whatever, and so that takes time and effort, but I think it's it's worth it because in the end, you end up with a better query for yourself too, um so that it's it's a it's a great experience and we hope we can find another query to give back to.

C

Thank you, that's. This is some of the best advice I've heard lately, like I'm taking notes myself. I hope the folks who are joining us are also taking notes. Thank you so much for being so open to sharing candidly um advice like this. It's free for you today watching demo days and security, but it does not come easy and I know the four of you are speaking from experience, which is so incredibly valuable. So thank you so much. um I agree with lucas kundari.

C

We have lucas on the chat, saying you are impressive professionals, because, yes, of course you are thank you thank you. um So we have a question um from folks who have been taking a look, and I think maybe valeria you can help us with this one, um and this is a good one. So how do you go about choosing what query to build? First.

E

Okay, yeah good question um and first you you need to have a problem. So usually in a company like marco libre, you have a different resources that report vulnerabilities, maybe a backbone t program. Maybe an internal issue tracker I don't know, but you have to take a a lot of metrics um check what what's the the worst one for you right for your context, and um you want to choose a vulnerability and um a language, because it is an essential query for javascript or golang or shawa or whatever.

E

So both of those and of course you want the vulnerability that can be detected with call ql. Not all of them are suitable for that, and um you need to know it very well. You need to have lots of examples and know how the vulnerability works and how the fixes work.

E

So um it's usually about a information which query which vulnerability do you have more information about and um which ones are maybe the most critical for you or them like you have a lot of them right uh would say sorry, I would suggest a going for a quick win first and then a adding effort to two dollars.

F

I can add that it's important for the query: don't have lots of false positives, because if not, the devs will want to kill us. So it's something important.

C

Of course, when you're doing your job so well that there's gonna be right, you're gonna run the risk of having those false positives and I'm assuming. Obviously that comes from experience and understanding the patterns. And then you know you just get better at it right, but I can see how folks will get really. You know a little bit on edge um anything that kind of stops functionality. So that's awesome, though, thank you so much for sharing. um I appreciate that advice. I think uh there is another question about uh limitations.

C

Joseph would you mind taking that on and then, if any of you, ladies, I think maybe lucia, would you mind taking that one all right go ahead? Joseph.

B

So there's a question from santiago rodriguez, which is: did you find any limitations in terms of performance? Given the mercado libre scale,.

D

Okay, I I think I have like different things to to answer to that. First of all, I think that out the scale of maca, liver and also what I said before being in the continuous integration pipeline and blocking the request, uh we had to be really careful with the queries we made um because yeah we don't want to be blocking like six uh thousands yeah, six thousand of repositories at once, so be very careful about that uh about performance, not that much right now we are using the lg tm server.

D

um We didn't have much problems with performance, maybe another problems, but not with performance and nowadays we're trying the advanced security that it's on github and we have to figure out some things on that. But what I think the most important thing to say about this, uh this question, uh let me think yeah is also what valeria said before manny just said. Everything that I want to say is these things about. uh We have to do a lot of tests and a lot of examples about these queries and all these um coding examples.

D

But mercury is really big, so everything is changing us in the technology world. Of course, but everything is everything is changing and you have like a million ways to do anything, so sometimes we are behind with validators with sanitizers and cigars. Sometimes we are behind even with new libraries that could be seeing for our queries, so we got new, false negatives that we didn't know before. So I think that it's the most like challenging about this big company, uh michael oliver. I don't know if that answered the question, but it's what I thought. ah Sorry.

E

Yeah, I have uh something as well and not related to performance but yeah to the scale. uh Since we have so many repositories and so many developers, we get a lot of false positive reports that we have to keep up and we review them once every two weeks and it's not enough.

E

So we want to keep up with that feedback, but yeah it's a lot. So maybe that's another scale issue.

C

Yeah, thank you so much. I think that definitely covers the question. So in a big thanks to santiago rodriguez for asking the question, I appreciate it. It's a tough one and then I I really appreciate all of you being here today. I mean I feel, like I learned a lot about the process of a team such as yours. I am super impressed. The marketplace is so huge, 132 million folks.

C

So for those of you who are watching and think that maybe wow yeah, if I break something um you know my team is gonna really shout at me or something I mean, imagine having 132 million users and able to to use your platform so wow wow. um So thank you. Thank you. So much for sharing that joseph. I don't know if you have anything to add when it comes um about scale, maybe we can add that and while we take a look, I think there's there might be another question coming in.

B

um The scale is impressive: like andra said, 132 million users have an attack surface. That is huge. Not so many companies in the world have to process so much data and, at the same time, have a production that can support real-time cyber security at this scale.

B

So um a sas team that is working with state-of-the-art tools like yourselves, have to have another challenge as well, which is not just only to keep up with the latest trends of cyber security every day, but also to adapt your tooling to match these trends and as software evolves exponentially, and cyber security evolves linearly. You have to everyday close that gap. So for me this is very impressive and I really want to know even more on top of what I've learned today.

C

A hundred percent, so I think we we have a question on the chat and maybe if we can get to the questions um you know for sure, maybe we can try and touch on the notes after but uh joseph, if you don't mind, taking a look and see, if maybe that's the question that you might answer yeah and the meanwhile again, I'm super excited that all of you joined us today. uh For those of you watching um this fantastic team is on linkedin, so find them adam.

C

I think once we publish this video, we'll add their direct uh urls into the notes so that you can learn from them follow them because um they have definitely a lot to teach us. um What.

B

Do you think you're saying thank you from ankit thanks so much for asking ankit? um The question is: does code12 help identify causes of most common zero-day attacks identified in past and possibly and possibilities to removing them before they could happen?.

D

All right we want to okay, I will answer.

C

Oh, it's totally. Thank you.

D

Yeah, so I think that the most important thing to to know about cocul, you have to know what you're modeling you have to know the pattern you want to model so for past zero days, yeah for sure, but new vulnerabilities that came across at just not possible because that's the definition of saturday.

G

D

No, not not for new ones, um but maybe for we have some other two for open source libraries or open source code where we kind of try to respond the most quickly we can to surveys and serve these attacks. I don't know if any of the girls want to add up something.

D

So I don't know if that answered the question. It's what I understood.

B

It definitely answers the question, and I want to add, on top of that, aren't you that, of course, the nature of zero day vulnerabilities is that they are problems that are seen for the very first time, um like our amazing team here said uh before, uh they are also looking for variant of those zero-day vulnerabilities, which is something that cultural can definitely help with like, for example, it can be a new zero day, something that we haven't found before, but it's very similar to a previous run and code.

B

Ql can definitely pick this up before becoming a new uh problem in a range of databases.

C

Absolutely thank you so much. Thank you uh both for adding that. I think that that definitely answered the question. um Are we done? Is it over and away? We, um I think, we're coming to the end of our time. So um joseph, would you just give us a brief overview of what we learned today and yeah. Thank you like this has been we've.

B

Learned so much absolutely helpful and we can make a little summary of what we've learned so today, throughout this through this very informative session, with the sas team of mercado libre, we learn how they leverage the power of code2l towards secure software for those joining our conversation. Late codeql has two main goals. First, is to explore code towards finding vulnerabilities using queries and, second, to make these queries.

B

Generic to find variants of vulnerabilities using codeql security professionals are able to codify their knowledge in an expressive query language, exactly like lucia, valeria, natalia and camila, do to describe what to find and not how to find it.

B

If you are curious to find out more about culture check the links below over to you, andrea.

C

Yes, thank you so much. I just want to take another moment to thank each and every one of you camilla, natalia valeria, lucia. It's been an absolute pleasure to learn from you today. Thank you so much for everything you do in the open source ecosystem as well, and always looking to see how you can contribute back.

C

That's super inspirational um and then, of course, I have to share the melody team is amazing. Their engineering team is amazing and they have a wonderful blog that all of you can access right. Now, it's free, uh it's gonna, be on medium and there is a ton of information there beyond security, so you can learn about how they leverage our technologies to run this very, very impressive platform and keeping their users available and ready to you know, hey buy or do all the things that they do on the platform.

C

So again, thank you so very much to the melee team. It's been an absolute pleasure and an honor to have you with us today, uh folks who are watching, don't forget to follow them on linkedin, so you can stay in the loop of their amazing work and you can also follow joseph joseph I'll give out your handles at some point, uh but thank you so much team. Really it's been. It's been an absolute delight. um We appreciate you being here with us today.

C

Okay, thank you. Well, I think this is it aj, I think we're wrapping it up so joseph any part in wars.

B

Just want to say a big thank you as well, and to wish to everybody to enjoy this weekend.

F

Thank you. Thank you.

C

Thank you. Thank you, mali. Thank you, mellie team. Thank you so much. uh We need to have you again. I love this. So thank you all so much and thank you. Everyone watching at home and we'll see you next time. I want to say the next demolish is going to be next year. What maybe not aj correct me if I'm wrong uh 2022 here we come so thanks again. Everyone thank you for joining. Thank you, joseph you're, a fabulous co-host. Thank.

B