Hyperledger Foundation Hyperledger Global Forum 2021, 11 Jun 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Hyperledger Umbra - Scaling Experiments - Shivay Lamba, Layer5

Description

Hyperledger Umbra - Scaling Experiments - Shivay Lamba, Layer5

For the past two years, the Hyperledger Umbra lab has been worked on by mentees to develop the capability of running the unmodified Hyperledger Fabric docker images under a simulated network environment for the purposes of doing experiments on running Fabric networks in a controlled way. At the end of last summer, Hyperledger Umbra achieved the initial goal of running Fabric networks under simulation. Hyperledger Umbra is useful for doing network-level security fuzzing as well as network scaling experiments. Hyperledger Fabric networks are designed to scale up fairly easily, however nobody has actually tried scaling Hyperledger Fabric to many hundreds and thousands of nodes and done analysis on how it affects the characteristics of the distributed system such as time to reach consensus, time to run an election for block creation, etc.
Sessions Details & Slides (if available)- https://sched.co/j3ft

----
Hyperledger – Open Source Blockchain Technologies

Hyperledger is an open source community focused on developing a suite of stable frameworks, tools and libraries for enterprise-grade blockchain deployments.
It serves as a neutral home for various distributed ledger frameworks including Hyperledger Besu, Fabric, Sawtooth, Indy, as well as tools like Hyperledger Avalon, Cactus and libraries like Hyperledger Aries, Ursa. Learn more about Hyperledger projects: https://www.hyperledger.org/use

Case Studies: https://www.hyperledger.org/learn/case-studies
Training & Certification: https://www.hyperledger.org/learn/training
Tutorials: https://www.hyperledger.org/use/tutorials
Webinars: https://www.hyperledger.org/learn/webinars
Events: https://www.hyperledger.org/events
Vendor Directory: https://www.hyperledger.org/use/vendor-directory

Subscribe to the Hyperledger Newsletter: https://www.hyperledger.org/newsletter

Follow-us on Twitter @Hyperledger

Learn about Hyperledger Membership: https://www.hyperledger.org/about/join

#Hyperledger #HyperledgerGlobalForum2021 #Blockchain

A

All right, uh hello, everyone, I'm shivai and I'll just quickly, just crosstick. If everything is fine, all right, I think we're good to go it's uh time, so welcome everyone to hyperledger uh global forum. This time, it's a virtual experience. Unfortunately, because of the global pandemic we have not been able to. You know, have uh the live in person event uh and I'm really looking forward to next year and hopefully be able to actually meet everyone uh at the hyperledger uh global forum.

A

So for everyone who are joining in for the first time in this virtual experience, uh definitely do check out what exactly are hyperledgers. um If you are interested in blockchains in uh blockchain technology or related to solidity or any of those uh things. Definitely hyperledger is one of the most uh brilliant topics to have actually have a look. Today uh we are seeing blockchain and the use of five ledger almost everywhere, uh including you know, crypto and uh all of different other.

A

But of course, like blockchain is not just limited to crypto and uh it's been used in a lot of different technologies, and today we are going to be speaking about one such topic, and that is regarding the hyperlaser umbra, and uh we are going to be specifically talking about the scaling, experiments and how um you know scaling of experiments within hyperlaser umbra is taking place. uh So just a very quick introduction about myself as well.

A

I'm shivai I'm currently a production engineer and an open source engineer at layer 5, which is a part of the cncf landscape and I'm also a linux foundation, mentorship uh mentor so like an lfx mentor and have been mentoring. You know a couple of different projects, including mystery, that is part of the cncf, and also uh the solar foundation. Projects that are part of the linux foundation and I've also been deeply involved with a lot of different blockchain related projects.

A

uh I've been contributing to a lot of different open source, hybrid programs and of, of course, I've also been uh an intern at blockslab that is uh and blockchain based uh system. So uh you can definitely connect with me if you have any uh questions regarding blockchain or specifically regarding hyper ledger umbra now to give us sort of a quick background regarding uh hyperledger umbra, uh so hyperledge umbra started off as this internship project.

A

So, as I have also mentioned that I have been a nurse foundation, mentorship mentor, uh so a lot of different hyperledger uh programs or like sort of internship opportunities come under the lfx mentorship program and umbra actually started off as one of those mentor programs back in 2018.

A

uh Where, uh unfortunately, due to some uh issues with resources, uh it couldn't be continued. But it was officially continued in 2019 and it was also uh like carried forward in 2020 as well, and uh it was originated uh in 2019 as a program.

A

And essentially it's like an emulation program for different types of hyperlithic blockchains and, as I said, it's already an ongoing project that does help to actually provide a really great research tool to understand how fabrics how hyperlid hyperlithic blockchains actually work and also it helps to understand how they can be used in terms of let's say, providing any kind of consensus based algorithms, and also understanding the scalability of different types of hyperledger blockchain networks.

A

So it's sort of like a simulation platform uh to provide the comparison between different types of uh you know: hyperledger blockchain networks or platforms and see how they scale up against each other. So um if you are into you know, comparing and doing research regarding different types of hyperlizard, blockchains I'll, definitely recommend using hyperledger umrah.

A

So in a nutshell, it's sort of an emulation platform for different types of blockchains and it has support for python 3.8. Above now, the specifically, if we talk about how the networking is actually taking place, uh we are essentially using mininet and a lot of different docker containers.

A

So, whether it's like uh you're using let's say any kind of a blockchain network, it will be in a dockerized container and then we're using uh grpc and protocol buffers and, of course, things like vagrant, virtualbox, uh influx dbs database and grafana for monitoring uh the scalability and all so uh it sort of combines all of this together under one package. So you can definitely look and have a look at the lab.

A

So essentially, uh you can check out uh the hyperledger labs and inside of the labs you'll find the umbra as well to check out and uh try to you know, sort of perform the experiments on any kind of a blockchain that you can have. So um as we move in the presentation we'll be also speaking about the different types of uh experiments that you can actually do and uh we'll be going over a few of those.

A

uh You know in some of the experiments that we'll be discussing so in a nutshell, essentially, umbra come encompasses, or it sort of you know has all these different uh features that are there. Inside of you know, when we talk about any kind of a umbra related experiment that we are going to be conducting now under the hood. What like you know, what's powering or what's ongoing inside of umrah. Is that of course, like it's been written in python, 3.8 and essentially uh the topology that we define is a graph.

A

So if you're not aware of what exactly is a graph, so a graph is a data structure that has nodes edges so essentially within uh when we describe the entire network and we describe the entire experiment.

A

The topology of the umbra is essentially a graph with its nodes edges and each component within uh the umbra is a micro service. So again, if you are not aware of what exactly is a micro service, a micro service is essentially, uh like you know, an upgrade over the monolith architectures, where each service is its own, unique entity.

A

So whether it's like, let's say a microservice uh based on the blockchain network, uh we're going to be having uh different types of microservices and each one of the components is a unique mic service within the umbra landscape and uh the dr pc interfaces and protocol buffers are used uh for each one of these components and uh based on these, like let's say whenever the events are scheduled, uh we are having like, let's say, repetitions and we are having the intervals being defined.

A

So the most com, like the most important aspect, to sort of look at when we are looking at under the hood whenever we are conducting any kind of an amra based, uh like you know, let's say we're, conducting a number based uh experiment. So the most important thing to sort of keep in mind is that whenever, like uh the most important thing to sort of keep in mind, is that because we are looking at and hype like, let's say an example is like, let's say we're talking about an hybridizer fabric right.

A

So, in order to understand how the model works, uh we are defining the topology and, uh of course, when the topology sort of comes over, we are defining uh it in terms of a graph and where each of these components is having a unique service and they have a unique function. So that is what is sort of under the hood for uh fabric umrah.

A

Now underst like to understand how does the umbra architecture sort of work, so it's sort of defined under like these six different categories and uh we'll just go over one of them, one by one, each on each one of them. So the first one that we have is the umbra design. So, of course, like the design itself is not a component, but essentially it's an api that you could say that allows users to, let's say compose experiments for umrah right.

A

So if we talk about an experiment within umrah design, since it's an api, so it's designed so that we can have like, let's say a main instance of any kind of a blockchain project as a topology. So it could be a fabric topology. It could be an iroh typology and, like let's see, all the different events that are associated with regards to that topology. uh Then we have an umbra scenario.

A

So, unlike an umbra design, uh it is a proper component that sort of con sort of consumes the mininet apis that we had described earlier to interact with the topologies that we have defined. So it could be, let's say on containers, it could be switches right and then we have the umbra broker. That sort of consists of um it. You could say like it's one of the core components of umbra and it helps relaying between the messages between different types of components.

A

If, let's say it's like, um we have multiple components in a uh defined in a topology. uh We can start with the coordination of the messages between, let's say uh the monitoring and uh to extract, let's say measurements uh or information about an environment. Or let's say, if you want to call like a specific sdk that has been defined in the blockchain event, uh then we have the umbra monitor that is sort of defining, let's say the element uh that monitors whatever environment.

A

That is there and, of course, all the different topological nodes that are out there and it extracts different kind of metrics from them and passes them to. You know find out uh like different kind of metrics and then, of course, we can influx them into a database like using, like, let's say, influx db and then of course, based on these metrics that we're getting. We can showcase them using visuals using rfana. And finally, we also do have like umbra agent and umbra cli.

A

So, like let's say, umbra agent is specifically designed to understand about any kind of a topology to know if there are any kind of anomalies that are happening within site within the network. So let's say if there has been some kind of high usage in cpu uh there's. You know some kind of issue with the traffic management. uh Then, ultimately, the umrah cli is the component that helps to actually automate all the different experiments of umbra. So this is what we'll be talking about more in terms of when we describe how we are scaling up.

A

The experiments, so umrah cl is what you're, using to you know, automate the experiments and do the installation right and make sure that all the different components within umrah are have been properly installed. So this is one of the most important ones that if you're, uh you know trying to create an experiment of your own. So I'll definitely recommend to you know, go through a lot regarding the umrah cli and go through documentations for amra cli to understand that and then specifically, uh if we want to, let's say, install umrah on ubuntu right.

A

So these just sort of showcase the steps. So, of course, the first step that we have is essentially to you know only install git and the make packages because we are using make files. So we are just installing by using sudo apt, install git and make to only install uh the git and the make packages and, uh if like let's say because uh we are using python so to install all the necessary python files, those will be provided within the git.

A

So uh if you go to uh the official github repository that is github hyperlabs umbra you'll install when when you actually do get clone within your local environment, it will uh automatically install all the different python packages, because those have been already mentioned in the readme file and in the requirements.txt file. And then what we have to do is that, because we'll have to install so uh you can use the sudo make install and then uh we'll be also installing a vagrant virtual machine and uh using like a virtual box.

A

So you can install like vagrant uh lib word and also like.

A

Let's see you can also use virtualbox as well, so we can uh use both of them like either one of them, so either the lip board or like one of the options that is virtualbox and once this is done, then of course we need to install uh some kind of uh blockchain and fabric based project so like currently fabric currently umbra supports both fabric and iroha, so you can install make install, uh make install fabric or, let's say, make install if you want to test both of them and uh currently like umbra, also does support fabric 2.0, so we'll be talking about uh those as well, uh so you can do either of those two right and once you have installed them, that is when you can get started with.

A

uh You know the rest of the installation. So these are the main steps to install and, of course, do make sure that you do have ubuntu 20.04. uh That is the latest version. Of course also. You need to have python 3.8 installed in your system, so those are the two main requirements prerequisites that you need to have before you get on and started installing with umrah.

A

Now, uh once we have, you know, defined and like this is where now we are sort of you know going deeper into uh understanding how to actually install umbra and also understand how you know the entire workflow works so before we actually define how the experiment works. I would just want to, uh like you know, share briefly about how does actually you know the entire process? Is you know, starting?

A

So of course, as you designed that you know, the umbra design comes into the picture right so under design as it sort of allows you to compose experiments for umrah. That is sort of our starting point that we start with. So um design is what we have. Then we define all our topologies events right and finally, then we go on to defining our amra cli that essentially again, as we have described earlier, that amra cli is.

A

You know the component that helps start off the experiments in umrah right and it helps us with all the different steps, whether it's the installation initiation of the components right so after we have defined our after, we have defined our topologies. Our events, we start off with the umrah cli that help us to further call on. Let's say the umrah broker um are things like the umbra scenario. Right and now we define different types of umbra scenarios. That has uh you know like. Let's say all of these.

A

So essentially, since we have spoken about how umbra scenario is being used to uh define or like say, interact between uh different instances, you know because there will be multiple servers and these multiple servers are essentially called as environments, and we will be having multiple such kind of environments right and how uh like how it is being interacting with, let's say, the monitoring, because each one of these different environments, we are monitoring these environments to see how you know uh like different metrics are being shown so we'll be having you can so consider like.

A

Let's say we initially have um one broker now this broker divides into different types of scenarios right and each one of them are being monitored with their own umbra monitoring right or we are monitoring these, uh uh like you know one by one and of course, once this is done, we have within inside of the umbra scenario.

A

We have different nodes and our network which are being monitored, and finally, the monitoring sends them across to the database that we use uh essentially use influx db, and finally, we also visualize them so based on this, like uh once, we have installed uh our uh demonstration like. We have basically installed our local system. Now we can execute right. We can first once we have installed, we can start executing it.

A

So essentially, amra cli will handle all the different uh life cycles of the experiment right and uh when we are designing the experiment, environments, sort of define, you know, which particular components are sort of going to be, you know, be used right or which one which one these components are actually going to be executed, and once we have uh like, whenever we have defined the like, we have a designed experiment. We sort of save it uh and load it using the amra, cli and amra.

A

Cli is also responsible to execute the environments and all the different components that we have and, of course it will also help us to actually trigger uh the execution of the designed experiment right and uh the broker will then actually help us to actually perform the interactions with the other, uh with the actual, with the other uh components that actually, uh you know, come into actually running the experiment, so those will actually be part of um the overall experiment that you know, sort of comes up, so amra broker will be responsible for that now.

A

If we specifically talk about, uh you know actually designing uh the experiment, so that sort of is you know? What exactly is how should we actually start designing? So uh essentially we have a certain uh you know um design, because those help us actually designing the apis, so that helps us to define the different topologies and events and uh the topologies that can be used.

A

I either can be, you know like different type of ones that are supported currently by umbra, that includes fabric topology, arrowhead topology or like indie topology, and then that helps us to you know, because these topologies are sort of the groundwork like that sort of uh really the groundwork.

A

They help us to create the abstractions for each one of these blockchain networks that we are defining within umrah and that helps us to also, you know, define all the different artifacts that are desired to run, and these are controlled by our different components that we have within umbra, and uh the events are essentially all the different interactions that we are going to be having inside of umrah, and these will be scheduled and, of course like these will be having uh the different parameters, like you know, uh repetition of from when till when it will, you know from where it will start and when it will start and whether there are intervals within uh designing of the experiment.

A

And uh finally, of course, like you know so this after we have defined all of this, you can actually run and uh run experiments right, and you can understand how these experiments are working. But, of course that sort of brings us to our main topic, and that is how, uh like you know, we can scale up these uh these experiments.

A

So, as the slide says that you know, hyperledger umbra is really useful for doing network level, security fuzzing and also, like you know, doing, network scaling experiments, and we have this sort of seen how you can actually define one of these experiments, but the need is to be able to actually you know, set up, execute and gather data and report.

A

You know repeating of the results so that we can actually scale up the experiments at different scales right and see how, if we scale up these experiments, how it actually affects the characteristics of any kind of a distributed system to you know, let's say whether it's related to let's say creation of uh the blockchain or let's say it's, you know uh understand how the scalability is actually performing or how much time is it actually taking to.

A

You know reach consensus, so the important aspect to understand is, if uh you know, because again, the idea is that we can scale up these fabric networks to let's say hundreds and thousands of different nodes. So we need to see that whether you know this uh the scaling like if we do go ahead and scale up these networks, because it's definitely possible to do so because there's something that is supported for fabric-based networks as well.

A

But how would be the effect on when we actually, you know, scale up these different experiments and like what exact uh outcome would actually affect the characteristics of the dispute systems that we are running and again, the need is to be able to. You know, develop a mechanism that can actually monitor and activate uh the different.

A

uh You know the scaling and then, of course, to be able to run umrah on fabric two point plus and uh to be able to actually scale it to multiple servers right and uh then also, my like, you know, automatically create monitoring functionalities for the scaling up right.

A

So if we are scaling up it to multiple servers, we should also be have the ability to actually monitor and scale up the monitoring as well so automating the monitoring functionalities and not make it something that you know uh based on once, if you're able to scale up our experiments, uh we should also be able to automatically scale up our monitoring as well, and so this was sort of taken up as one of the experiment projects as well, and uh this was sort of the need and of course um we like, of course like what happened was that some of the main deliverables that were part of the project included, like you know, of course, having the scalability functionalities.

A

That includes, like you, know the environments, events uh to be able to scale up those as well and, of course, having let's say, automation, functionalities that include, like you know, having being able to make the make files vagrant um cli to actually automatically have given the automated uh the automatic functionalities built in and then, of course, having uh giving the network fuzzing capabilities to hyperledger and, of course, being able to use umbra for scaling it up to multiple servers.

A

So whether it's like you know for testing for setups or, let's just thinking with other fusing functionalities, so uh as you can see like you know, these are some of the major uh like deliverables that were met and that were part of uh the overall uh high pleasure. um You know um like specifically when we talk about the scaling of different types of uh experiments right and essentially, the idea is that we are using it for monitoring and seeing how it handles for in case of a distributed system.

A

So these are some of the most important aspects to you know. Consider when it comes to uh doing like you know the scaling up. So apart from that, uh some of the results that actually took place, you know uh because the learning objectives were to be able to understand how the hyperledger uh um like how it is. You know, essentially it's like sort of also understanding right that how we can propose effective experiments to see how the performance varies with the scale of the network.

A

So essentially, that is also one of the most important uh things to you know understand that: how can we effectively look at different types of experiments and uh understand how scaling up of these experiments help with the performance as well, and then by understanding this? We get automatically understanding of how distributed network applications work and how we can study them inside of hyper uh ledger based networks, and essentially once this was actually completed.

A

uh The further the further plans to you know sort of look at is that we can create more and more experiments that focus on distributed leather technologies, and that will also help us. You know enhance uh the experimentation of hybridized umbra itself to also have feature sets that you know can uh better enhance the results uh by sort of using these experiments as a feedback to directly be used within uh the hyperledger.

A

um So, of course, this sort of helps to uh understand how hyperledger umbra will react and behave in those cases of scaled up, high pleasure based environments and this sort of helps you very uh give a real world. Experience of you know how uh those kind of scale-up environments actually can help, so um that sort of uh brings an end to you know the presentation, and I would love to also take up any kind of questions that you might be having uh and, of course, you can also connect with me on uh these.

A

uh You know social platforms on twitter and on linkedin, uh but of course, the main idea behind all these different uh platforms, like that, we have sort of created why this uh intern experiment was gathered in the first place was to be able to.

A

You know, understand how the scaling uh works in terms of experiments when we are writing any kind of we are defining any kind of a experiment within hyperledger um how scaling actually affects it and how it can be used to further understand umrah in terms of when we are running it on hyperledger based networks, so um the overall end experiment, it's still not like you know we could say 100 complete, but it does give a clarity on. You know how uh that sort of works out, and uh definitely it's a really wonderful experiment.

A

If you are interested in, you know simulation and seeing how different kind of networks, and if you want to look at their monitoring, if you want to look at how they're functioning so I'll, definitely recommend at looking at hybrids or umbra and actually running one of the labs as well. uh So we'll take up some of the questions that you might be having.

A

So one of the questions that I can see uh by uh again, uh don't please don't mind if I uh take the question like you know the name as wrong, but the question is by pascal and he's asking: how does umbra interact with docker and can you deploy nodes as docker containers? So essentially uh like that's? First of all, a really great uh you know question and essentially, as I've described, that all of these different so I'll probably go back.

A

You know to uh describing how, uh in a nutshell, how basically umbra is being used in blockchain right. So essentially, what we can consider is that all of the different, uh like we described about the umbra architecture right so essentially all of these different nodes that we have within. Like let's say we define a fabric.

A

We define a blockchain based network right, so it could be. Let's say uh we described about two of them right that are currently supported by uh by uh by umrah, and those include basically, both uh you know uh like everyone or let's say we have you, know fabric or iraq projects right. So these uh you could say these blockchain projects themselves do come under, like you know, inside of these containerized uh apps right, so these themselves are under the container, so you can definitely.

A

I I'll definitely agree that you know um that these uh nodes, that we get from these, uh like these blockchain based networks, that we are locally installing uh these are containerized and, of course, all the different.

A

um You know architecture like within the architecture when we were talking about the different components- and uh you know like all these different components are like deployable as docker containers, so that's definitely possible, and essentially these uh whenever we are experimenting with any kind of uh you know, blockchain based uh competence, whether it's like you know fabric or it's ivory, so you can install those dockerized containers, and that is how umbra is actually you know interacting with them.

A

So I hope, like that, uh answers the question and essentially again, the thing is that you would also need you know docker installed in your system, uh so you can refer to docker nodes to understand how and because what happens? Is that you'll be having these dockerized containers and you'll, be? You know, uh going getting them through uh docker hub, and these are already available on docker hub as well. So you can uh have a look at docker and look at all the different supports that we already have.

A

So I hope, like that, answers the question, uh but if we also want to probably just describe right that, how essentially uh to further sort of point out to the question of uh you know how it is actually functioning, uh the thing is that, like let's say you start off by creating your first ever, you know first ever uh based like let's say you you're running the experiment for the first time, uh essentially, whenever you're either trying it out for fabric based or, let's say, you're trying it out for any other kind of uh you know, blockchain based networks, you'll be in uh you know, downloading the docker container and you'll be interacting with that container.

A

So I hope that answers the question um like I don't see any other questions so just to sort of summarize uh if there are no additional questions, this sort of summarize what we have discussed is essentially the main point of you know. The experiment or scaling of the experiments is to be able to use.

A

uh You know understand how uh the hyperledger umbra is being used to understand how these hyperledger based networks, work and function and being able to actually use them right uh to understand how, if we scale up these systems right, if we scale up these experiments that we are able to design and use within uh the hybrid zero that helps us to actually start off by defining the topologies the events and then, finally, then you know using them to understand how uh we can understand uh right that monitoring them on a database and then uh using refiner to uh you know, understand how they are sort of scaling up.

A

uh We are using the scaling up to understand how will it actually work in a distributed system, and that, essentially, is you know uh what is being used uh anyways. I think uh that sort of is uh you know uh the overall experiment, and uh thank you so much for connecting, and, of course, uh you can connect with me on all these different social platforms uh on twitter and on linkedin and I'll be more than happy to take up any questions uh during this hyperledger forum uh on umbra or hype related to hyperledger.

A

Thank you so much.