Cloud Native Computing Foundation Cloud Native Security Day EU 2021, 14 May 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Secure Code Development and Lessons Learned from etcd Security Audit - Sahdev Zala & Hitoshi Mitake

Description

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon North America 2021 in Los Angeles, CA from October 12-15. Learn more at https://kubecon.io The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.

Secure Code Development and Lessons Learned from etcd Security Audit - Sahdev Zala, IBM & Hitoshi Mitake, Indeed

When it comes to the importance of writing secure code, it gets a unanimous vote. This is even more important for an open code. Checking the security of your code needs manual steps as well use of automated tools. As project maintainers for the etcd project, we recently led a third party security audit of etcd code. In this talk, we will share our experience of what are the common areas in code that get overlooked and pose a security risk from general weaknesses to critical threats. We will also provide a walk-through of security vulnerabilities that were reported from the audit work.

A

um Hello, everyone welcome to this talk, this recording uh for secure code development and lessons learned from hd security edit. My name is sadio zala. I am a senior software engineer at ibm and in context of this stock I am one of the fcd project maintainer and I have hitoshi mitake with me. uh He is a site reliability engineer at indeed, and he is also one of the project manager for the lcd.

A

All right so in this talk we basically will uh briefly cover. You know a couple of the best practices like identifying the you know areas where you should be paying more attention. As far as security of your project, your software is concerned, uh we'll talk about code analysis uh and you know, then we will have uh mitaki uh talk about uh that's the security model. We'll talk about uh some of the vulnerabilities uh examples that uh you know we.

A

They are real life, examples that we run uh while uh doing the cd security audit and then we'll talk about. uh You know working with github uh to publish series uh and then we'll conclude the talk. uh We will not go into the you know. The full software development life cycle and security will pretty much. You know. Focus on the coding aspect uh in other things are out of scope for this talk, so all right. um So there are a couple of things that we really want to emphasize here.

A

You know, as far as these uh secure coding practices are concerned, the first one is, you know, identify uh you know, high securities areas right that you want to pay a special attention um and you know how do you do that? Well refer to your project architecture, right, uh your software architecture, that will help you to identify those areas where you know that it needs a special attention and we'll talk more about those checklists in the next slide.

A

Then we'll talk about you know the most important part right: the code assessment. uh There are like two main methods in automated code assessment using the automated tools and doing the code review manually.

A

You want to make sure that you define the role of the assessment team up front right, so that will help you as you go forward with the process, and you also want to make sure that you plan on how do you how you will be addressing the findings right in short term, like you know as soon as possible, and then have it uh plan for the longer term, something that you can.

A

uh You know apply so that in future you can prevent any any any of these issues that you ran into it as part of your code review, and then you know, depending on your project, depending on your software, uh you may want to plan uh on on. You know publishing the cves right. You may not need you may um so you decided, but cbe is uh they help you?

A

uh You know typically uh to disclose uh any security flows with your software uh publicly right. It helps the consumer of your software, your project, to stay relevant um all right, so you know the things will talk. They apply to. You know pretty much uh any development environments. You know any programming language that you are using, but uh for this talk we will use uh you know golang and the lcd project as an example.

A

Okay, all right, so I mentioned earlier that it's very important to identify the the areas that you think are you know critical crucial for your uh project security right. So here is list of uh some of some of these um areas right. uh You can have more uh than what I'm gonna mention here, but these are some things that you definitely want to keep in mind. While you, uh you know, worry about the security of your your project, your software, your application.

A

So, as you know, uh the first steps, for you know security or software security is, you know it starts with authentication right so uh for external users for internal users, you know, regardless make sure uh you have a a good authentication method uh in place right and then you know you are doing a lot of things like you know not just accepting any password right. I have some validations there password length, and you know that kind of things and the the follow-up step is uh authorizations right.

A

So once user get authenticated, you're just making sure that they only have access where they are authorized right. So you know using like a role based access control, um tls certificates. You want to make sure that you have a encrypted communication right and you don't allow the monkey in the middle interface uh input or data validations. You know when it did all uh input parameters right. um You know data type length range, whatever you can think of. uh Do the proper validation right uh file permissions. It's another important area.

A

uh You know make sure that, while creating new files, while working with any existing files directories, you are checking and handling the permissions. You know and pay special attention to any third party libraries or tools that you are using for file management.

A

We will actually show you an example later in this talk, error handling, uh that's another important idea for the overall health, the security of the project or software. So make sure that you know accidentally you're not leaking any information that can help. You know. Hacks attackers write, for example, the password um and then locking right uh anything that you think uh can helpful towards uh identifying any security issues. uh You know uh alerts right, you, you should be logging it.

A

You know that can help you with the auditing purpose later on, like where the request is coming from. uh Who is sending the request right data exposure to make sure that you are not exposing any sensitive data? uh So keep that in mind- and you know the last but not least is uh if you are using a third party tools, uh then make sure uh you know you you, you use their latest release right and you keep buying any cve, advisories. Okay, uh all right, um so code.

A

Analysis, uh as I mentioned earlier, uh that that is critical uh for the uh you know, overall, security of uh your code right uh just making sure that uh you, you know things up front while your development or you know after your codes in place right, make sure you you do the assessment right um and there are two main ways: um either it's going to be an automated way using the tools or you can do the manual code review.

A

So when you talk about automated tools, you know you have. uh You know either going to use the static tools right that can prevent. uh You know uh you know mistakes that can you know, if not directly, uh maybe indirectly impact your security uh off code right. So, for example, like variable shadowing, you know the unreachable code right, so um I'm sure you know most programming languages. They have uh tools uh specific the language right for go. You have actually a whole list of tools available.

A

I have provided link here how you can take a look uh to see the whole list, but, like you know, those like go wait, static check. You should be using and then uh dynamic tools right, so like fuzzer right uh that can help you find uh bugs by you know they by automatically injecting uh randomly uh injecting uh data and- and you know, uh find uh you know different kinds of uh bugs based on their the injection data injection.

A

uh The second thing is manual code review. uh You know, tools are great. They can take a look at the code automatically. You know and and point out to the possible issues, but definitely a human needs to verify those issues right and make sure they are the real issues, and you know also uh the development team. They should be.

A

You know doing the thorough code review of all the areas we talked about right and make sure that it looks good to them right um and keep in mind that there's no alternative to you know manual code review right. That is critically important, um all right. So you know with that. uh You know before we talk more about a cd security model before we talk about uh some of the examples of the vulnerabilities that we ran into uh as part of the audit right and, and you know how we address them.

A

Let me just briefly mention about lcd. You might already know it right that hd is an open source distributed qle store uh it, you know, provides you consistency and high availability and it is used to store critical data of a distributed system. So, for example, kubernetes store all its cluster data in hcd.

A

The diagram here uh it just shows a single node hd cluster, uh as you can see, has to use raft, consistence algorithm uh to maintain the replicated states and the data are first persistent, white, uh storing disk, uh typically in productions. uh You would use uh three to five node cluster.

A

uh The lcd is a it's a you know it's a cncf project and you can learn more about it. On etsy.io and the hd community, they love new contributors. You know that helps grow. The growth of the project, the you know, makes the project better. So you know, um if you're interested, please contribute.

A

uh You can find more details on contribution guidelines and you know uh other things on the development environments and whatnot on the gita repo.

A

um All right with that, uh I will hand it over to mitaki to talk about its city security model, and you know some of the examples so medaki to you.

B

Okay, let me introduce the security model, opacity, it's the clients and it's. These servers communicate with jrpc over tcp. It's also possible to use grpc proxy for multiplexing at the clients. Etsy servers use a specialized http protocol named after http, and the tcp connections used for jrpc and http protocols are encrypted and authenticated with trls. If our cluster follows a recommended configuration, we also have a component named gateway. This is a component which only lays tcp connections between clients and servers.

B

This is a component which only helps discover your bentley cluster visible next.

B

And if a user needs more fine-grained access, control keys, the 3d provides artwork-based authentication and authorization as an optional feature. With this feature, we can grant permission to read and or write about specific gear range to users from the cluster use. This feature servers have two options for authenticating and authorizing users. One is using authenticate, jrpc method. In this case, clients invoke the method during initializing the client plot object.

B

Information of authenticated user is stored in a job token returned by the server. Another option is using commonly period of trs certificate. In this case, no password-based authentication is required. This mechanism also supports limiting uh special administrative observations. Like membership change, the special user root is the only user which is allowed to execute such applications.

B

Let me introduce the security audit project. It's liquidity performed security audit last year. Cnc supported this third-party audit and trailer bit helped this project. You can download the full report of the audit from the url. We will introduce some findings from the report.

B

Critical things.

B

The first example is an issue related to logs a cd has a problem of inaudible related to failed authentication attempts for users, which can only be used with common name based authentication, as I mentioned earlier, silly supports multiple user authentication mechanisms for our work, username, password-based, upload and common name of the trs certificate approach, especially for the other one. It's only supposed to create users which only support commonly based authentication, which doesn't support username, plus password-based approach.

B

When a client applies authentication with password for such a user, a cd log isn't useful for understanding that the client gave a wrong password, or user gave a password to no password user. It is harmful for auditing failed attempts for from the box.

B

We fixed the issue by adding the new record for representing such a failed attempt and make the log clearer.

B

Next, I'm not sure the next example is related to gateway gateway. There is tcp connection between sv client and servo. The component doesn't validate acceptance of trs connection to its endpoint. It only checks tcp itability during initialization, so if the pain, if the endpoints are misconfigured and appoint a malicious server accidentally, they can receive data sent by the clients. If the clients don't use tls, probably properly gateway, is a component which only helps discovery of the cluster and doesn't tell me a tals connection.

B

Using theories is the responsibility of the development. We fixed the ambiguity of this program in the documentation.

B

Sure the next example is related to metric total number of database key is compounded. Battery was never changed. This is because the code of conduction will work the statement of increment the variable for the metric.

B

The issue can result with understanding about the resource usage at the time of auditing. The issue was already fixed in the master blanche, but the fix wasn't but ported to the stable bbc's.

B

Okay, next and I'd like to handle it sure.

A

uh You want me to talk on this mitagi yeah. Okay, thank you, hey! uh Thank you! So much again, uh so all right, a couple more examples. Right, uh mitaki did explained a few of them I'll talk about couple more examples here.

A

uh So very uh you know a small snippet here um which is basically doing nothing but creating a new directory by calling make their all. Do you see any security issue in this snippet?

A

Well, I wish we were talking in person and I could see the raised hands. uh You know besides mitaki smiling, uh but uh we're recording. So I will go ahead and show what's wrong here right well, so the problem here is: if the provided directory path is already existing right, it's it's an existing directory, then the maker all does nothing and it returns new.

A

So then the flow continues right with the nail. You are controlling the flow, that's what the project was doing and uh that's not something we were expecting right uh because what happens if there is an existing directory with, uh you know, 777 uh permission mode right.

A

Somebody could have created that directory upfront with extra permissions there and you know, with some malicious desire- and you know- could hurt uh the the project uh you know running ncd, eventually right so the fix was. You know to make sure that if the directory exists, then uh it has the desired permission. If not, then you know we will consider that as an error case right.

A

um So that's how it was fixed right. uh It was pretty quick fix, small fix, but could have uh you know, security related uh problem, if not handled um so another snippet here, uh another small snippet, you know we. We try to put uh some of the smaller uh issues here, uh so data validation rights. uh Remember I mentioned earlier the input. Data validation uh is very important and you want to basically validate. You know everything right.

A

If you don't, then that could hurt the running project right, so the parts compaction retention function. uh Do you see what's wrong here? Any security issues? Well again, I wish we had to raise ends, but let me go ahead here. So the problem here, as you can see the string conversion functions right. uh The written value can be negative and uh that's not an error case.

A

Okay, you can write a negative value and that's not my error case and, as you could see the project you know we weren't handling the issue right and we were just checking the error and if that's me, we will just go ahead and you know continue the flow.

A

uh The secret issue here was: if someone uh you know who has that access right, that that role, who could uh misconfigure the art of competition compaction retention uh by setting it to negative- or you know, maybe accidentally right, but if it's set to negative value, then you know that could impact uh the absolute. uh You know the process right.

A

It may not work properly as expected right because that would be. That could be like a forever compacting um fill up the disk space and whatnot.

A

So the solution again here was simple. You know just not to accept any negative value and you know handle it properly.

A

The other thing I would mention here is the documentation right. We need to make sure.

A

That you have a proper documentation uh for the for the you know for the for the soft right in the code.

A

uh You may not see that, as an you know, direct impacting direct impact on your security uh for the project. But but you know three of the report issues that we had in uh city, secure audits. They were related to the documentations and and naming of functions.

A

So you know we had a couple functions. They were uh having a misleading name um again that happens accidentally and you know something that were not caught as part of uh in a good review. While you know those pull requests right um and then we had uh a couple of uh you know: misleading uh documentation, right description, so you know that that, as I said, you know that that ended uh up as in three reported issues by the auditing team.

A

uh So you know the lack of documentations or not enough coverage right or poor naming of functions. They can create confusions right during the code review process and that can impact productivity and it can also mislead users. So uh you want to keep in mind that you know you have a good documentations right.

A

The other thing I want to cover is, uh you know before I hand it over to metaki again is uh this is something we learned as part of uh the auditing process. Right uh as, as I mentioned earlier, the cves are important and uh you know they can uh basically help you to to. You know, disclose uh the security flaws issues publicly uh and so that the users of project software uh stay relevant.

A

uh The process can be confusing, uh but we learned during this process. You know the security audit process that we can use the relatively new features, which is integrated part of github to request and publish cvs and advisories right. So we just tried that, and you know we loved it. That was easy to use, uh and very you know this can be very handy for you if your project is hosted on github right and you want to publish advisories- and you know cves right, because now everything you're doing is is on github.

A

You know you're not going to like a military websites, or you know anywhere else, uh to work on cvs right, the publishing uh other stuff. So if you uh have an admit admin access to your github repo right, then you you will actually see under the security tab and option to you know to draft security advisory.

A

Excuse me and uh the you know the other great thing here is: you can actually privately discuss uh and- and you know, the issues and fix issues with your code changes and it's all private, and then you basically publish you know after having reviews uh in your internal teams right uh a team of maintainers, for example right or interested parties, and then you, basically you know, uh push them uh those changes to your repository.

A

um It also has a really good template. uh You know it shows what kind of uh vulnerability is it right, the one that you are interested in publishing? uh Who is impacted? You know any any problem.

A

That has been patched what version should the user of software project? uh You know they should upgrade to right and and if there's any work around so it it. The template allows you to put like a good detail that can help uh users understand the cv, so we want to mention because, as I said, this is something we learned as as part of the security audit process. Okay um with that uh mitaki would you would you like to conclude the talk.

B

uh Let me conclude this talk. uh Yeah writing security is challenging but possible uh for the purpose of the code review and various tests and uh numbers two starting static, analyzer and further very helpful, and also, although it costs time and budget. Somebody audit is helpful for checking the status of the project and also making documentation and walking is difficult. uh It's not trivial at all, and creating and publishing uh cds with github is easy. uh Let's utilize, the future for managing security issues and also less popular features, can use also problems.

B

In this case of security entity security. Audit, multiple issues were found in gateway features which isn't used heavily. We need to be careful about such features and think before on the new features for making the entire project secure.

A

Let me this is the last slide we have so uh uh we want to thank you. uh You know uh many folks here all the contributors of that city, all the mentors right, uh especially wanna, thanks uh siang li yoli, jingy and brandon philippe they are maintainers and they were, you know integrated part of uh you know, working with uh the security issues that that we resolved that came out of the lcd security audit. uh We wanna thank cncf for sponsoring the uh the audits are going to thanks trail of bits for conducting the audits.

A

uh They have great materials. You know blog post right. That can be like a really big good learning resource for security, uh understanding, the security of the project. uh We want to thank owasp. We have a lot of good practices and other materials, so we did refer them those materials. We definitely thank our employer, um ibm and indeed- uh and thank you all of you who are watching this video all right well, thank you. So much with that, I think uh we will end the thought mitaki. That sounds good.