Hyperledger Foundation Hyperledger Global Forum 2022, 6 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Securing Hyperledger Besu on Ethereum Mainnet - Jyoti Ponnapalli, DTCC

Description

Securing Hyperledger Besu on Ethereum Mainnet - Jyoti Ponnapalli, DTCC

Jyoti will be presenting the key risk and security considerations for deploying Hyperledger Besu on Ethereum Mainnet. She will be presenting the research conducted on security assessment of the platform, configuration guidelines and findings that contribute towards secure deployment.

Sessions Details & Slides (if available): https://sched.co/14H4i
Access All Sessions: https://events.linuxfoundation.org/hyperledger-global-forum/program/schedule/
----
Hyperledger – Open Source Blockchain Technologies

Hyperledger is an open source community focused on developing a suite of stable frameworks, tools and libraries for enterprise-grade blockchain deployments.
It serves as a neutral home for various distributed ledger frameworks including Hyperledger Besu, Fabric, Sawtooth, Indy, as well as tools like Hyperledger Avalon, Cactus and libraries like Hyperledger Aries, Ursa. Learn more about Hyperledger projects: https://www.hyperledger.org/use

Discord: https://discord.gg/rVCn4j7UFd
Case Studies: https://www.hyperledger.org/learn/case-studies
Training & Certification: https://www.hyperledger.org/learn/training
Tutorials: https://www.hyperledger.org/use/tutorials
Webinars: https://www.hyperledger.org/learn/webinars
Events: https://www.hyperledger.org/events
Vendor Directory: https://www.hyperledger.org/use/vendor-directory

Subscribe to the Hyperledger Newsletter: https://www.hyperledger.org/newsletter

Follow-us on Twitter @Hyperledger

Learn about Hyperledger Membership: https://www.hyperledger.org/about/join

#Hyperledger #Blockchain
License
Creative Commons Attribution license (reuse allowed)

A

Good afternoon, everyone um thank you for joining. My name is Jyoti panapali and I'm. A director in the blockchain Technologies lead at dtcc. For those of you that don't know, dtcc is a post-trade market infrastructure services provider for Global, Financial, Services industry. We have about 21 global locations, dtcc along with its subsidiaries, centralizes automates and manages risk for the capital markets, and just to give you an idea about the scale in 2021 we processed Securities transactions in the amount of 2.3, quadrillion US dollars.

A

Now in today's agenda, I'm going to talk through a little bit about the complexities of blockchain security uh when we started uh implementing DLT some of the challenges that we met, dtcc's thought leadership in the security landscape and the leading that into the DLT security framework initiative that we put out the risks, assessments of hyperledger, Bay, Zoo and then I will also discuss some of the security assessment findings, as well as the configuration best practices that we uh that we have put together.

A

That will lead further into the smart contract, vulnerabilities, best practices and certification specs.

A

So the nature of DLT itself is a huge complexity when it comes to blockchain the concept of decentralization and distributed nodes. The fact that a domain does not centralize and maintain all the nodes within the borders or the boundaries of the corporate domain um that led into emerging risks, so a significant lack of understanding of the threat landscape, as well as the risks associated with it add on to further complex the security posture for dld applications.

A

Now, a lack of guidance towards secure implementations is another complexity, because we all know that we, you know, consider cyber security Frameworks. They do assist with traditional ID security assessments significantly, but then there is quite a bit of a gap between dld security and traditional I.T security. Developer. Community is not equipped with the correct tools and techniques that they would require to be able to do secure coding, secure coding, best practices, secure deployment as well as devsecops.

A

Of course, we all understand that patching and upgradability on immutable ledgers is is another complexity, added on with continuous monitoring capabilities, so blockchain networks today, both public permissionless, as well as private permissioned, are running business, workflows that involve transactions and custody of value in the form of digital assets.

A

So cyber security attributes such as confidentiality, privacy, Integrity of data, do take the center stage for blockchain, but at the same time, compromise or hack of any of these attributes can result in high value business impact that could be loss of trade, loss of funds, loss of trust and credibility for the entity. Basically, a high value impact for an organization.

A

um So, as I mentioned earluded earlier that dtcc's SK at scale the economies of scale, the the requirement for them to be able to process trillions of transaction um is, is that is? Is there a business model?

A

So risk management is at the core of what we do so corresponding to that the TCC put out uh white paper in 2020 more like a call to action to the financial services industry, inviting all financial industry stakeholders to collaborate and contribute to the best practices that will eventually develop into an agreed upon standard by the financial services industry and, as I said, our role is to be able to effectively manage the risk and identify the appropriate countermeasures.

A

So for that purpose we identified some of the security Baseline security Assurance standards, and we also proposed that this be achieved via some sort of Industry Consortium.

A

Security must be considered at all stages of DLT lifecycle, which includes design development deployment, post-deployment governance and compliance. So the chart that you see up here illustrates 15 categories of security considerations that was gathered across 20, plus research, artifacts and Publications that was conducted over multiple organizations.

A

um The size of the block that you see is indicative of the fact that more count of organizations mentioned that category as a security consideration for them. One of the things we noticed was you know, as the pace of um you know, development and love using of DLT and blockchains increased.

A

We also realized that internally, we were accelerating our efforts, but at the same time we were also challenged with appropriate guidance towards securing these deployments, and when we were putting our thoughts together, I mean if we were internally accelerating. We assumed that other Financial Services organization were also in similar position as us. Hence the white paper. Hence this research. Now this actually helped us justify the fact that it's, it was not just as Solo in the industry, there were other organizations that were thinking through these unique drivers.

A

So, having um had this this as a support, we decided that the industry needs a DLT security framework and the vision around it was we put together a few sets of goals for being able to develop. That one was that it should assist Financial Services organizations to evaluate the risks across an individual firm's, secure assessments. We are best practices and tools; it should be able to assist with third-party management Incident Management.

A

This must address key aspects when it comes to. uh You know: Key Management life cycle, especially with respect to Creation um creation, maintenance and Storage storage, as well as disposal of sensitive information.

A

This framework must also be able to provide security, guidance and practice best practices when it comes to standard access, controls, authentication methods primarily be able to bridge the security gap between traditional I.T, uh as well as DLT specific security.

A

The first layer that you see up here are the alliances and the consortiums that we reached out to to collaborate as part of our planned efforts. After we wrote put out our white paper, which was a call to action, and we created working groups, DLT specific security working groups or we partnered with working groups that were specific, focused on building security, artifacts and research articles for security. The second layer that you see are the focus areas, and these are the focus areas that we created more like under the umbrella of DLT security framework.

A

We created sub-working groups that focused on this specific domains where we could put further together the the skilled resources, the tools required. The resources required to be able to conduct research and develop security, artifacts specific to that Focus area. The third layer that you see are the platforms that we directly participated to deliver Publications in. So what you see up here is hyperledger foundation.

A

We have published close to three artifacts that cater to security controls in hyperledger we participated with Enterprise ethereum Alliance to uh to publish a security, smart contracts, audit specification certification levels as well as under the CSA umbrella. Again we participated to do to produce an architecture security report as well as a security controls checklist for R3 quarter.

A

And as I mentioned, these are some of the security research Publications that we published since we started our initiative in um September of 2020. So um that's a lot of work. That's been put out so, if you're interested in Reading those articles, please feel free to go to CSA, Cloud security, Alliance website and they're all in public domain they're, not behind the paywall, so anybody can download registered at CSA and download those, but the ethra security level. Spec is under ethereum Enterprise, ethereum Alliance and that's also on a public domain.

A

Two of the research artifacts are work in progress at the moment, so they will be published either in q1 or Q2 of 2023..

A

Now I cannot move past. Having announced our initiatives without thanking our contributor Community, this chart specifically shows our contributor growth and retention under CSA, but we have similar stats uh under other alliances as well.

A

Now I will dive into the hyperledger Bezos security assessment that we conducted Okay. So we didn't want to reinvent the wheel. That was not the intent so.

A

Our efforts were centered around identifying the gap between traditional I.T and DLT specific security, so for any security assessments we conducted, we still leveraged the nist cyber security framework as our basis like something that we use. So we first identified the platform whether it is hyperledger bezo, hyperledger fabric I mean I'm I'm. Trying to um you know, walk through the process that we implemented um and we would identify the architectural risks by applying a threat modeling on an assumed use case that be fits the platform under consideration.

A

And then, while you know we correspondingly, we implemented, uh you know the threat, modeling and then developed an architecture security report, along with that, we also developed a security controls checklist that directly ties into the functional domain um as specified in the list. Now the risk identification process comprises of three different layers.

A

First, we um the first layer we we took, as was DLT the second layer that you see is hyperledger Bay Zoo. The third layer is Enterprise ethereum Alliance. Now you may ask they're all you know blockchains or DLT. The reason we split them into layers is commonalities of a DLT platform or a blockchain. Our intent was to be able to abstract that commonalities across multiple platforms and then be able to roll that up into a company's policy and a control standard.

A

If we don't uh identify those commonalities, then you know for any DLT project that gets rolled out for any organization. It'll be a challenge to develop policy and control standards. Every uh DLT platform- you know another complexity, I forgot to mention- was that every DLT platform has its own technology stack, which means that you probably will you know if you didn't have those commonalities? You might end up developing a separate exclusive policy guideline under control standard guideline for each platform, which is a redundant process. So that's why we identified.

A

Let's see if we can abstract out any commonalities and then put them together as a policy guideline for an organization and then, of course, hyper legit bezo, which is again. You know customized apps threat, modeling on that apps that were deployed an Enterprise, ethereum Alliance, given that it's a public, permissionless environment, there might be risks that we have to identify which the organization must be willing to accept, if not be able to mitigate our approach.

A

Based on once we identified the risk areas of focus is one co-create with the security teams and the reason we say that it's it's very important to Loop in all the skilled resources from different squats, and that brings us the shared best practices and experience for implementing a specific technology, and then once we had the squats together, we identify the threat model, risks at a component and an Enterprise level. Please bear the reason. I say: Enterprise level is eventually all of these have to be able to roll up into Enterprise risk management framework.

A

Currently it's like down. You know at the component level, which is again, you know, could end up being written and processed down the line. We record risks to nist fiber security framework, which is again, you know the basis be able to quantify those risks, recommend short-term long-term risk acceptance, plans and prioritize the remediation planning.

A

Now our entire effort, I'm jumping the gun a little bit and showing this ahead of time. um Our entire effort resulted in quantifying the risk we identified into specific functions. Now we grouped these by cyber security function area. So, as I mentioned earlier, we you know we Loop in the squads or the squads, help us build the mitigation steps. Now, please bear that. This is a point in time slide and what we identified here was when we were conducting assessments specific to the applications we deployed and the threat surface corresponding to those.

A

Now you see the smart contracts, you know shows a 67 and the secure configuration of bezoo shows like a three percent uh risk, but this is a moving Target at any given point of time that 67 on Smart contracts could be down to 20 or up 80. So uh please bear this is a point in time slide, and this is, um you know, not universally applicable.

A

Now, once uh now, I'll, you know walk through the hyperledger Basu security assessment finding. So our initial assessment of the known risks in the ecosystem resulted in bezo being a well-defined and implemented Java client, which was good news uh prior to assessment prior to our assessment, even of course not. The first ones tavora, which is a third party conducted security, audit on Bay Zoo and identified primarily two vulnerabilities, and, to my knowledge, both have been mitigated ever since in the iterative versions.

A

um Also, what we found was that hyperledger Bay Zoo had a development and the life cycle management, which was found to be quite robust, with appropriate defect, management and security. Again, that's the reason why we were only able to quantify three percent of the risks to um towards secure configurations.

A

ah Major dependencies were related to each signer and the hosting environment, while the operational dependencies identified were, of course, on the cloud environment a little bit on the ethereum.

A

Some of the critical findings um we identified were, as I said, you know, each signer is a major dependency and the e-signer source code is simply checked into a repository, and we recommend that downloading each signer must be done via verification of appropriate process, and so is bezosource code ensure that you are downloading the source code from an authoritative source.

A

um Each signer version should be updated to the most recent version, as always a recommendation, but even if you didn't, for example, you, let's just say you implemented- ensure it's not the version 20.10.0 which, because it was identified to have a log, 4J vulnerability, um basic configuration, critical findings that it turns off SSL by default, so insure give it a check.

A

um Keys should be managed according to the best practices, so Key Management best practices and control standards are at different levels of um you know, maturity and processes depending upon the size of the organization and the industry. Given that we are a financial services industry and we are heavily regulated for us, it's a really critical requirement. So we always follow our standard best practices, um as advised by our industry standards.

A

Some of the moderate impact findings was um always start each signer, as described in the initialization documentation, because initialization is a critical first step.

A

The second one with respect to bezo configuration is today bayesu configuration uses a mix of config file, as well as the command file, it's known, to have resulted in several error-prone patterns. So what we recommend is preferably use configuration file. One is it gives you maximum auditability and second thing is for reproducibility of your errors.

A

um Again, each signer could be isolated within the container. It's not a mandatory requirement currently. So if you were spinning up a node and you would you know um putting them together, if the best practice recommendation is always separate out the the uh the VMS, because the events and the event listeners with respect to each signer could process sometimes um untrusted data.

A

Some of the minor impact findings was again at the OS level, the each signer you limit. It's a configuration setting, preferably set the U limits, um 60 25, or something that's what the recommendation is and the default Bay Zoo Network in the config file currently sits nil, like it's empty, of course, because Bay Zoo can be configured in multiple different ways, but we always recommend set a value to it like. If you are, you know, using mainnet, then set it as main net instead of leaving it as empty.

A

Following uh following our critical findings, we also put together some of the configuration best practices. So now, I'll read out for with respect to hyperledger base you the tooling ethereum, VMS and Cloud.

A

um The first one is the hyperledger Bezos. So, as I said, unlike its signer, bezo is configured to be downloaded in authoritative source. So the repository is, you know, for sure it's an authoritative repository, but by default the configuration turns off TLS.

A

So when you're downloading, um if you do, are not using secure network communications, then the corresponding hash to verify the Integrity of the downloaded payload is not available, and this could potentially result in a network level attacker that can intercept and Supply their own bazel source code. To you so preferably use some sort of a secure, Communication channel like TLS and verify the Integrity of the downloaded payload, currently bezo releases their software every two to three weeks, and actually they are pretty good at every two week cycle, so ensure you're tracking.

A

The updates, in accordance with the force scans, which is a standard process, preferably use a scripted build process as a best practice and load balancing. These are like common best practice tips when it comes to tooling. Each signer is a major dependency. Ensure monitoring and best practices is usually using a you know. Official Docker file, instead of downloading the release from a pre-docker, build script, Cloud best practices. Every organization is vested in their own cloud service provider, whether you're using AWS as your or Google Cloud.

A

We recommend, following the best practices of the cloud service providers, documentation that they provide and ensure it meets the security and compliance objectives of your company and the industry, and these are standard. Some of the configuration best practices also applicable for DLT I mean they are VM best practices, but also applicable for dld's. Cellular components must be further privileged separated, um removes the complexity for um a compromise, and um another fact is ethereum.

A

Clients generate huge amounts of State, uh you know so starting a clean node is very slow, so it would be good to to maintain snapshots of the each client state. That is because, if you know if there is an introduction of some some sort of a corruption, then the snapshots recovery could be a pretty costly affair.

A

The underlying operating system must be always kept up to date in regards to security patches, again standard best practice and configuration best practices when it comes to ethereum has ensured the boot notes are carefully audited.

A

um Several well-known entities maintain boot notes, basic health and safety monitoring of each node's view, not just the nodes that you are spinning up, but also having an understanding of the state of the network, and the health of the network is also important, um use, use logging and log management geographically diverse nodes as much as possible um check on API external API usage, always a good idea to keep I said. You know, watch on the state of the ethereum's health and network state.

A

Now, coming back to the glaring 67 risk assessment on Smart contracts- I'll repeat again: it's a point in time, quantifying um Factor, so the smart contracts. You know that today um are a superpower, the power, the decentralized applications on ethereum they've, had security issues, um and there was no good way up until recently to be precise up until August 22nd.

A

There was no good way to understand if a particular smart contract had any sort of security, audit or security verification done, but on August 22nd Enterprise ethereum alliances um published a smart contract security audit specification to ensure consistency when it comes to Smart contract security. So the publication is called um each trust. Security levels, specification version, one: it's developed by the eth trust security levels, working group. It's a new specification that aims at um certifying a smart contract through a full security audit and it provides you with three different levels of security certification.

A

So, depending upon your organization or the size of the organization or the needs of security audit, you could pick and choose which level of certification is more apt um and it provides a you know, a stronger assurance that a smart contract does not have specific security vulnerabilities.

A

The spec also provides some basic smart contract best practices. um As it says, new vulnerabilities are discovered from time to time. So, even if you think you have conducted a security audit on your smart contract deployed it, it's not the end of the story. You have to constantly keep checking for new vulnerabilities and ensure that those vulnerabilities identified will not impact the functions of the smart contracts that you've deployed, um uh keep checking for the current version, as well as the previous version use latest compiler for solidity.

A

um You know, follow the ERC standards um as well as, if you identify or come across a vulnerability that hasn't been reported, please disclose them in a responsible manner.

A

The smart contract spec also provides you recommendations on making external calls normally exceptions in the sub calls bubble. Lab, like solidity, gives a quite a few low level. um Low level call functions like delegate call static. Call these call functions behave differently as in they return a Boolean value and indicating whether a call completed successfully or not. So if you are not checking for that Boolean value and if the call fails, then it could potentially lead to an unexpected behavior of the caller contract use check effects, interactions pattern.

A

It ensures that validations of the requests and changes to the state variables of the contract are performed before any interactions take place. This is important because the scope for re-entrancy attacks is reduced significantly. If you're doing these checks do not use a delegate call instruction. It's an external call contract to manipulate the state of the contract, ensure you know your tested code is going through these reviews. It comes to external call contracts.

A

There are a number of known security bugs in various versions. In solidity, compiler some bugs were introduced in known versions. Some were always assumed to have existed previously. The slide you see up here is, you know, compiler bugs that the audit must check for security level s, but then there are recommendations for the other security certification levels. Also, which is M and Q, so always a good idea, especially for the developers in the room, ensure that you're.

A

You know you're going through this pack, because it gives you a good knowledge and information on secure coding, best practices. uh Now I will walk through the security levels, certification requirements. This is the the first or you know the the lowest cater. uh And if you go through this certification and if you have conducted an audit security audit successfully, you could potentially claim the badge and publicly showcase that you do comply with each trust. Requirements and interest.

A

Certification is like, like I, said, isn't available in three different levels: uh each security level s is intended to allow for an unguided, automated tool to analyze the the source code and determine if it meets the requirements and some of the standard uh you know, recommendations are like no transaction dot. Origin.

A

um Don't use no self-destruct. That actually reminds me of his story. A few days ago, I think on Twitter I saw um there was a post saying, am I going to be arrested and it was coming from a developer and he had accidentally called a self-destruct on one of the smart contracts and the naivet of the developer didn't realize and um well then there was, you know, a trickle of more uh annoyed and in more messages from other people saying oh, my God could some very, very basic knowledge.

A

But then again you know it's a developer's fault and error.

A

um Each trust, certification, security level m means that the tested code has been successfully reviewed by a human auditor. Now this is not. This is a little bit about the automated tool and the team is doing a manual analysis and important security issues have been addressed. This level um includes a number of uh you know for this level. It's assumed that you do meet the lower level.

A

S has already been met when it's, the audit has been done by an automated tool, basic requirements and now you're coming up to the second level, which is level m, and it also has um the following set of recommendations like um no one. No unnecessary Unicode controls, declare storage explicitly, that's very, very important. Storage variables must be explicitly declared and then sources of Randomness. That's another recommendation.

A

um Security level Q- this is the last, and you know the in in the sequence of the orders. This is the highest level uh verification.

A

um This means that you have verified security um level s and the manual manual audit at level m M, and this uh certification at this level also means that the intended functionality of the code is sufficiently well documented and its functional correctness is being verified and the code and the documentation has been thoroughly reviewed. This also has some good practices on you know, code linking and managing gas use increases in the state changes and also mentions no private data.

A

Private data means different for different jurisdictions, so check what your jurisdictional private data definition is, if you're in Europe gdpr means different and if you're in California this privacy policies are different now I want to conclude my talk today with a few things to think about, and one of the main things is, security should never be an afterthought, which is how it is for most of the organizations Unfortunately.

A

Today, new security vulnerabilities are getting introduced every minute every day, so it's always a good practice to to to want to bring bring a shift in your organization's. Thinking and approach um bring a shift in your perspective and collaboration ensure you're bringing in the squats together.

A

If you have security teams, but if you're a small organization- and you know there are not- you know- you're lacking of funds to have like a dedicated security practitioner, then you know inculcate the thinking security, best practices into your development teams, but that thought process has to be initiated in the design and development process itself rather than developing and deploying something in production because what's happening with public permissionless networks is they're like they're live and they're out there and once you have deployed and if you have deployed bad code and it sits like bad quotes, it's on public.

A

uh You know permissionless networks, so always a good practice to change that perspective. On thinking bring about consistency in your you know, conducting of your Assessments in the governance- and you know see if you can grow the security posture and knowledge and educate education is really really critical um and develop a security strategy program. If, if, if possible and encourage the security, um you know security, education in your organization, talk to your peers, see if you can get some someone excited about thinking through and you know bring them into the security thinking.

A

All right, thank you for your time today. I think! That's that's about my time. If you have any um questions, I'm happy to take them now, but I think I'm up my time um and if you have any questions in regards to the alliances and the collaborations that we have made and interested in participating in any of them feel free to stop me and reach me and ask me questions.