Hyperledger Foundation Hyperledger Global Forum 2022, 6 Oct 2022

Previous Meeting

⏯

youtube image

►

From YouTube: Why You Shouldn’t Just Trust Your Blockchain – And Apply Critical System Design - Imre Kocsis

Description

Why You Shouldn’t Just Trust Your Blockchain – And Apply Critical System Design - Imre Kocsis, Budapest University of Technology and Economics

The elevator pitch for distributed ledgers is very powerful: they provide highly fault- and attack-tolerant databases with very high integrity. In reality, they don't. But they may, and with high confidence, if the lessons and techniques of critical system design are applied - there's just still no free lunch. Presented by Imre Kocsis, a researcher at the Fault-Tolerant Systems Research Group at the Budapest University of Technology and Economics, the talk showcases how various internal faults may propagate through a Hyperledger Fabric network and have unacceptable consequences at the application level - and how error propagations can be reasoned about (tooling showcased) and even caught in their tracks by applying classic fault tolerance techniques (such as diversification of chaincode implementations, rollbacks and the like). Chaincode faults - i.e., smart contract bugs - receive special attention in the talk. Copying the example of swcregistry.io for Solidity, we began to assemble a registry for Fabric chaincode weaknesses and created tooling for code weakness detection - and injection, to be able to assess the effects of various types of programming errors.

Sessions Details & Slides (if available): https://sched.co/14H6P
Access All Sessions: https://events.linuxfoundation.org/hyperledger-global-forum/program/schedule/
----
Hyperledger – Open Source Blockchain Technologies

Hyperledger is an open source community focused on developing a suite of stable frameworks, tools and libraries for enterprise-grade blockchain deployments.
It serves as a neutral home for various distributed ledger frameworks including Hyperledger Besu, Fabric, Sawtooth, Indy, as well as tools like Hyperledger Avalon, Cactus and libraries like Hyperledger Aries, Ursa. Learn more about Hyperledger projects: https://www.hyperledger.org/use

Discord: https://discord.gg/rVCn4j7UFd
Case Studies: https://www.hyperledger.org/learn/case-studies
Training & Certification: https://www.hyperledger.org/learn/training
Tutorials: https://www.hyperledger.org/use/tutorials
Webinars: https://www.hyperledger.org/learn/webinars
Events: https://www.hyperledger.org/events
Vendor Directory: https://www.hyperledger.org/use/vendor-directory

Subscribe to the Hyperledger Newsletter: https://www.hyperledger.org/newsletter

Follow-us on Twitter @Hyperledger

Learn about Hyperledger Membership: https://www.hyperledger.org/about/join

#Hyperledger #Blockchain
License
Creative Commons Attribution license (reuse allowed)

A

Welcome my presentation is titled, why you shouldn't just trust your blockchain and apply critical system, design and I hope you will like it. So what I would like to begin is to hammer a pointer home that I'm sure most of us are already aware of. Is that a blockchain by definition gives us guarantee on one single aspect, quoting the hyperledger fabric paper, a blockchain can be defined as an immutable Ledger for recording transactions maintained within a distributed network of mutually untrusting peers. This is a strong statement, but there's also ellipse in this statement.

A

It says immutable but doesn't say anything about the transaction being the right transactions, meaning in the practice that my smart contracts compute. Actually what I want them to compute, it doesn't say anything about high availability, the necessary throughput and latency and so on and so on. All it promises is essentially a high integrity database.

A

What will this presentation be about? My beginning point is opening point: is that distributed? Ledger technology is not an extra functional Panacea. If you like, it doesn't solve any and all problems out of the box. We will see that if we are careful, many problems will become solvable, um but the reasoning, an actual guarantees is possible and this is becoming quite necessary in the system. Engineering context and smart contracts are especially problematic from the point of view of further extra functional guarantees.

A

I will proceed to lay down our first, the blocks of a Dependable chain code, lifecycle model- this is something we are working on actively I will try to draw your uh I will try to draw a Limelight to the fact that some tricks from the old engineering toolbox will be usable, especially on fabric and I, will show some results in advancing fabric chain code, static analysis. These are all research efforts that we are working on at the critical systems.

A

Research group at my University, some knowledge of hyperledger fabric will be beneficial here, but really not that deep. The reason for having the picture of the nail and the hammer is actually two-fold here. What we are seeing in academic research is more and more people enamored by dlts are trying to apply dlts to domains which these systems really weren't originally intended for 0.1.2.

A

Coming from a research group that used to be called fault, tolerant systems, research group, it's only natural that we try to apply those techniques that have been around in Fault tolerance systems for decades, hopefully usefully and successfully. So we distributed ledgers.

A

Quite a few exciting possibilities can be read in various papers and surveys and meta surveys and for my sins, I have to read. These people are trying to do: Federated learning, robotics data marketplaces, grid management as in electric grid management, industrial iot, smart cities, digital streams, Railway Railway operations and so on, and so on, and so on, based on a distributed Ledger, because these are immutable. Data stores, multi-party consensus provide security for them, so it's it should be much better than a single centralized entity, not so fast.

A

Let's play with one of my Wilder ideas, if you like is.

B

A

From Siemens around, they are a world sponsor nobody and it's good this way. Actually, so, let's perform the following foot experiment: we have our classic Railway infrastructure and that's been working quite well for quite some time granted a bit pricey, but still these things are appearing and they are supposed to be self-driving when they we really become self-driving and weather is the flow of the software or the driver if before an accident, the self-driving software switches off say five seconds before it.

A

That's again another matter, but these things are coming and various people are having various wild ideas. I had one of those too that in order to coordinate those two, maybe we should do some movement Authority registry when there's a unguarded Crossing. When can this little thing cross and because the artists are so good and immutable and have an audit, Trail and so on and so on and so on? Let's do it on a DLT basis. Okay, let's try.

A

When you give this task to a system engineer first, they will have some Choice words and second, they will begin to have a look at the structure of the system and any system engineer worth his or in the case of my wife, her sold we'll begin to look at the various internal error modes and system level. Failure modes of the thing we are having in our hands. The internal error hormones simply mean that we have components in this stuff.

A

What are the vl's ways it can fail if you are familiar with hyperledger fabric, you know that endorsements endorsements, that you expect it to go through May Fail or they can be too slow. You run the thing in a cloud and you have performance interferences or too many organizations may be available due to network problems or too many organizations shall I say it may be compromised because that's not they're fully out of question either or you may have messed up your chain code in a way that it gives back some garbage.

A

Let's call that course data error or in a way that it gives back to you some subtle error in the computed results that should be written into The Ledger. Let's call that subtle data error or your chain code makes unnecessary range queries. A whole bunch of stuff can happen there going forward to ordering your endorsements package.

A

The endorsements may be late to get into a block again, for instance, because the ordering service is simply overloaded or someone is doing some rather malicious, Constant front running of specifically or transactions or the ordering service may have been compromised at the end nbcc stuff. We are still not out of the voter because unexpected conflicts may happen. Because do you really know what other chain codes are working on that channel and what other transactions are trying to get into The Ledger?

A

The end result is that you can have updates that missing in contrary to your to your original plans updates. Maybe late updates may be Superfluous on the key value level and updates, maybe even erroneous, even in a detectable or undetectable hard to detect way.

A

So, let's see what happens in our fourth experiment, if we have our unexpected conflicts, because there are other transactions that are trying to get onto The Ledger too.

A

uh The message that I will try to put on The Ledger is that the train will be in the intersection in a specific five minute time interval don't cross, because this is a time when you shouldn't cross.

A

Everything is okay, so the transaction is proceeding in The Logical architecture of fabric. Here it becomes late at the best case, because the client will see that this didn't go through and it tries again, but it will be, it will get on The Ledger later than you originally expected. We will have late updates and the end result is is that we either have no problem because uh well they put it there in the Witching hours and I will go to that Crossing.

A

Only eight o'clock in the morning or this may result in a hazardous situation, because if that don't cross comes later, then I'm still only approaching the crossing. That will be a hazardous situation, so you will have to create a structure structured argument on either avoiding or mitigating this during your design phase.

A

Let's see another example, let's say that we did some subtle chain code data error, I believe there are quite a few computer Engineers I.T people here and I'm sure that if you are one of us, then you have seen and possibly even yourself did some Daytime conversion errors.

A

So if we are dealing with times and dates, then well stuff happens should be codeine testing, but it's not always codeine testing. So let's just say that I do some dumb stuff with the conversion and what gets put on The Ledger refers to actually a Time slow, two hours after then it should actually we go through. We propagate our error through the system here and there's a subtle data error and it goes through manifests in an erroneous update at the service level and what it can cause is again no problem.

A

If I'm not interested in that interval actually or it can cause a super closely denied crossing, the crossing will become unavailable if you like, because I want to come two hours later and I see that I shouldn't cross, but actually the train is already gone or it again can lead to a hazardous situation.

A

So what I try to get with this example is the following. First, that example was certainly a very dramatized use case if you like, but not overly. There are many academic proposals for industrial iot grid control, robots infrastructure over actually. At the end, it may not be about safety, because that's something that that we only touch very carefully but reliability, availability, timeliness and a whole bunch of other system level properties are important, which are influenced by the way. Hypervisio Fabric in our case actually works for those more control inclined.

A

This protocol actually should be certainly explicit authorization and positive control. You may only pass if you get a personal signed sleep for having the authority to place pass, but in that case this would have been much less dramatic and for good, while certainly DLT will only just Shadow control in the safety critical space I'm, not sure how long this will remain the case. But the more general idea that I wanted to support is that we have truly important functions in our I.T systems. These we call critical systems. These can have many forms.

A

We have business, critical systems, Mission critical systems, safety, critical systems, and even if you are not talking about safety, critical systems, your dlt-based financial solution, a cbdc say, can be critical enough for this stuff to be actually important. So we need assurances and often quantitative ones. We have to assure Integrity reliability, availability, safety maintainability, and it's not just about crypto asset security.

A

Contrary to what you can see in the solidity world, critical system design actually has standards and quasi-standards requirement driven design processes, as well as techniques to avoid remove and tolerate faults, and my specific message is about applying this stuff for hyperledger fabric. As we see the three major key cross-domain concerns for hyperledger fabric are actually the following: one reasoning about potential fault and effect chains in a specific application context. This is what I've been covering with my simple and dramatized example.

A

The second major pain point, if you like, is or are the hidden common mode faults chain code. We all know from the solidity world that blockchains are all well and nice. Until someone has a bright decentralized. Finance idea goes on to implement. It leaves a bug in the code, puts it on domain net and hilarity ensues. If you have money stake, then not necessarily hilarity.

A

Smart contracts in general are a common mode fault, so you run it in 10 000 instances, but it's still the same code. If it's faulty, then it's faulty and we should have. We already are beginning to have approaches to detect and remove those bugs in development and to tolerate them during operation, which is something that ethereum doesn't really can do, but hyperledger fabric should be able to do and there's a third category. I do not want to talk about here today, performance and Ledger update timing guarantees.

A

It would be very nice to have the out of the box admission control for fabric to have some form of Equitable block space allocation. If I have iot devices, then to have the guarantee that every tense block I have at least one transaction that I can put in, and there are no so-called age of information guarantees for The Ledger, but that's a talk for another time for the propagating errors. We are doing combinatorial search based, modeling and analyzes called error. Propagation analysis, the current evolution of which is in answer set programming.

A

If you know prologue or constraint solving, that's the modern way to do almost the same on those in those domains, but that's not something that I really want to focus on here today. How can we create Dependable chain code, whether we should create it at all?

A

uh You may argue that well chain code, smart contracts are small species of software, so I can see it through. If I can read it through and audit it manually, then it will be okay. That argument May stand.

A

You can also argue that I don't have to do any special tricks during operation, because in design time I test it I perform formal validation, formal verification. It will be fine. That argument may also stand. We actually don't really know for chain code, because there's ether scan for solidity, smart contracts and people all around the world are doing doing stuff like getting all these smart contracts from etherscan analyzing.

A

Those building even AI models for correcting them fabric chain code is bespoke custom software, so we don't have etherscan, we don't have rec.newsense or any of the similar sites for fabric. So then some mishap happens due to chain code faults errors inside some large company. We don't really know about that. There are indications that chain code deploy to production is Not, always completely fault free, but we won't really have any public registry for that. We don't really have these for Corda or demo or any of the other permission platforms either.

A

It's just the way uh how these smart contracts are, but I submit to you that empirically desolate. The world itself is a counter argument. Solidity, smart contracts are not that long. They can't be that long, otherwise, you would pay an arm and elect to run them still. There are errors and arguably creating correct chain code is a harder problem for fabric, because it's not only about losing assets or avoiding the loss of assets because for solidity code, that's the Alpha and Omega of V and V. Because what do you do on ethereum Main?

A

Not you juggle virtual assets and that's it. If you want to use this DLT for something else: control, for instance, or command and control of cyber physical systems, then the properties that that I will be interested in will be actually a wider set, and it's still manually written bespoke software and faults may be sparse, but existing for IEEE does have standards for the number of faults to expect in 1000 lines of code and for code that's been developed properly, shall we say it's a low number I believe one in Thousand lines, but still not zero.

A

We did actually an experiment to show that it may be more than about losing assets back in 2020 with colleagues from coimbra in Portugal we did an experiment where we took solidity. Smart contracts injected various programming errors automatically into those smart contracts and try to run them on the bureau, evm interpreter on hyperledger fabric. It was a bit of an unnecessary, unnecessary complication, but still it demonstrates it will demonstrate my point. We had a bunch of faulty contracts, formal verification did detect quite a few of those.

A

So those programming errors, but by far not all of those and those were back, then the state-of-the-art tools, contract self checks. If we do, those so special asserts that we intelligently put into the smart contract most people don't do that still for solidity did catch quite a few errors again, so that's nice, but we still had quite a few that had been passed simply by what we called runtime platform checks. So the evm didn't throw an exception.

A

Everything went through and what follows here is already a huge problem because when we say output in very uncheck that only says that the client receiving back the transaction sees that something is fishy here and by that time the modifications are already on ledger. So the fact that uh so many faults were ineffective doesn't mean that it's not a problem. It just means that we should have had a test case set that activates these faults more easily more readily.

A

So what we began to put a put together is actually a rethinking of very classic patterns from Fall tolerance, software there's, a decayed old still developing library of patterns for architecting fault, tolerance, software and in the end, when we talk about photoler and chain code Dependable chain code, it's actually tolerating faults in the chain code execution service.

A

So it's based on actually the very nice tomb of Robert ashammer, and you can see that yes in development, we can do fault prevention and removal, and we are doing it by coding practices, testing, fuzzing, static analysis, formal verification, but certainly model based design will be also important, and we had a presentation on that here today. When I generate my token contracts, then, if somebody took the time to go through the generator logic.

A

Well, then, probably the quality of that code will be better than something that I just slept together and we are beginning to see reusable domain contracts too. If you are familiar with demo, they have implemented the common domain model of isda. So if there's something that's as uh reliable as open, Zeppelin In the solidity world, then maybe should just use that or extend that.

A

My message is that, with hyperledger fabric, we can do even more meaning that even in operation, we can do error, detection and fault tolerance for chain code, and possibly we should also have error recovery. So when we see that something is problematic, then we should be able to do some quarantining and checkpointing, but that's that's still in the works technologically. The reason we can do this is that for high in hypology of fabric, we do not pay for every execution step.

A

So I can do software diversity, meaning that if I have the same chain code, functionality implemented one two three four five ways. This means that if those different implementations don't have the same implementation faults, then they can cross check each other and I can do some voting mechanism to be able to do that on the technical level. Actually, the read write set model is something that's quite important, so I will try.

A

I will talk a bit about these and, at the end about the somewhat sorry state of chain code, static analysis, 2, which is arguably important. Just not the only piece of the puzzle, so M version programming is a very simple idea. Actually you do independent implementations of the same function and do a very reliable voting voting is easy, simple. You can implement it so that it it will work.

A

Hopefully, the hypothesis is that, after a point, it will become cheaper to do that at runtime than testing your chain code further and further and further and doing formal verification and validation even more to reach the same level of confidence. Now. The interesting thing is that for fabric, the application of M version programming is rather straightforward and there are multiple possible architectures to implement it. One way to do it is that you have your organizations, you have a set of diverse implementations of your chain code version.

A

One version, two version three, due to the way Fabrics. We can even say that one uh one team does it in Java one team: does it in Python one team: does it in node.js and you simply execute all of them and do some voting with three voters. You can you can actually even mask a single, a single fault in one of the containers, okay and organization. 2 does the same, and the whole thing is actually a transparent to fabric way too.

A

To do it is organization, one runs version, one organization, two runs version two and you can actually even ideologize it, because you can say that all organizations shall bring their own implementation for the common business because they have to understand it and shouldn't necessarily trust an implementation that somebody gave to them. But that's beside the point. In that case, you can say that the fabric consensus itself is a voting mechanism. Is a majority voting mechanism. So, whatever majority consensus, you specify in your in your endorsement policy that will be used for that purpose. Granted.

A

If you begin to think about this, this can interfere with your consensus guarantees in a major way, but it works runtime approach. 2 is what's called runtime verification. So the idea here is that whatever we compute as Ledger modification, there may be certain properties that must hold on all ledger modifications and actually from the solidity where there's a very simple, widely used property.

A

For that, when we are handling some kind of asset money, then the first thing that one tends to verify for solidity is that the cumulative sum of assets at the beginning of the smart contract call and the at the end of the smart contract call is the same.

A

No money is created and no money is destroyed unless, unless the function is explicitly about creating or destroying money uh and the pattern in full, tolerant Computing to do that, is that there's a system we wrap it in an instrumentation wrapper and there's a monitor component that observes what's happening in the system and gives feedback to the system, and when it decides that a internal execution path of the system does not comply to the properties. Something happens. What should happen is is a case specific.

A

You can say that we can wrap the chain code into instrumentation and see whether the changes it tries to make on The Ledger or are compliant with known properties, and a very good student from the University of Alberta bahariological elementorship project is looking at that.

A

It's useful: it's easy to implement technically, because the if you want to be really transparent, it's just a wrapper around your chain code implementation inside your chain code container, but it's limited in scope, because if you want to cross check with other data, that's on The Ledger for that you would have to read from The Ledger and that interferes with the read write set of the chain code execution. Although this may have been a bit too technical option.

A

Two is that you may want to look at The Ledger content, where you can do preventive measures, but that needs transaction validation in integration. When we do the mvcc there at least theoretically, we can do runtime verification, whether the modifications that we try to do are in line with a set of properties that have to be upheld for The, Ledger content or it can be reactive. So we can say that somebody monitors The, Ledger and looks for patterns that bring the Ledger content to a state which is out of line with respect to properties.

A

But then you have to be able to do coordinated drawbacks across the participants of the Consortium, which we believe is a function that would be needed, but not implemented and is technically somewhat challenging.

A

To be able to do this. There is technology uh if you want to delve a bit more into this, the Java modeling, the tool, support for the Java modeling language seems to be readily applicable to do chain code based runtime verification so um and, as a last closing constructive, constructive thought. I would like to go back a bit to the first phase, the only phase that's being done for solidity, for obvious reasons, when we are looking at our smart contract source code and want to make really sure that no fault remains in there.

A

So it's understandable that the solidity vert fixates on vulnerabilities, specific code constructs that have known attacks. If you like many of the papers looking at the problems of hyperledger fabric chain code chain codes, use the same terminal terminology and try to use the same approaches, and this may be actually a misnomer, so usable solid vulnerabilities are very specific and for chain code. The interesting effects are more diverse and we don't have a vulnerability registry and, arguably will never be so.

A

It may be a better approach to focus on detecting patterns of code weaknesses, such constructs that may lead to improperly behaving chain code and uh well, if we are at that inject them to evaluate effects and gauge the effectiveness of the few V and V2s that we already have for chain code using a common platform. The difference between weaknesses and vulnerabilities is easy to catch. If one opens up, the both Metra hosted common weakness, enumeration and common vulnerability.

A

Enumeration one speaks about specific exploitable problems in code and the other one gives you a taxonomy of ways. You can mess up creating software systems, so we have a early prototype for a system. When we have patterns of weaknesses, we created a visual syntax domain specific language to describes, though, describe those a whole bunch of things happens here. The key thing is here is that we are trying to reason about chain code in terms of actual structure, so no regular, Expressions, abstract, syntax, T3, annotated, abstract, syntax, 3 and do pattern, detection and pattern. River.

A

Writing on that now.

A

This is how what a pattern looks like as an example the weaknesses that we read Ledger data, but do not use it. This is a known weakness of change of chain code in chain code programming.

A

Could we do better? Possibly it's not that readable and those who may be versed in this stuff? This is a trick here because it refers to a this. Doesn't use that there's the not relation that we computed on the graph database actually, but that's we have and that's what we have and we don't know anything better.

A

Ideally, we should have a chain code, weakness registry, so the same day as the solidity world is maintaining the SVC registry, the solidity weakness, classification and test cases registry. It would be nice to have something like that for chain code weaknesses too, because we may not be able to create a registry for vulnerabilities for chain codes. But if we come, if the community comes comes together and discusses that what are the ways smelly code can look like to use a technical term, we may be able, we may just be able to create a registry.

A

We began to do that. We'll extend it in the coming months and hope to reach a point where we can publish it and open source it. So, in summary, fabric is a truly great platform, but to use it in critical contexts, critical settings, we need to create engineering Bridges to critical system design. We have to have a much firmer grasp of what constitutes good chain code, and we argue that we absolutely have to begin to implement runtime dependability mechanisms of which creating diverse chain code and checking what chain code actually does.

A

Quasi-Independently are important parts, Ledger audits that I'm not convinced that in the practice that would work, but that's also an option when we are constantly auditing the contents of The Ledger based on a data model and to able to rectify any errors that have been already committed to The Ledger.

A

We should begin to think we should begin to think about creating mechanisms, not one-off mechanisms, not saying that the Consortium should come together and everybody should roll back the state of The Ledger to a given point.

A

These things rarely but happen come up uh idea the ideally there should be mechanisms at the platform level to be able to to do that. For instance, Ledger roll forwards or rollbacks granted tied to a consensus of the participants.

A

Maintaining The Ledger I did steer clear of timeliness on purpose because it's a topic on its own I hope I could pick your interests and there were actually useful messages in these uh half finished academic results and uh truly uncharacteristically I left five minutes open. So if you have questions, then let's discuss.

A

Yes, yeah, so to be more precise when you say um like weak patterns or like patterns of weakness, yes, applying that the way to sell that is like um reachability analysis, or is that static analysis? Or is it yeah great question? So um weaknesses have multiple definitions. Originally the term was used just for security weaknesses, so such code structures and such Behavior or behavior in code that may lead to vulnerabilities and that refers to transitively to the definition of vulnerabilities. But the term has been generalized, so some weaknesses can be caught with static analysis.

A

Just by looking at the code. This actually, what I tried to show you Ledger data red but not used. This is pushing the limits of static analysis and there are such weaknesses that you can formulate informally well in closed text, but which you can't really attack with static approaches.

A

So you have to you have to look for dynamic verification approaches. You can do model checking on chain code. It's um there were a few guys incarceria who were playing with using years ago, who were playing with using a Java byte code model checker on Java chain code theoretically doable. But if you are involved with model checking, you know the pitfalls, but yes, there are weaknesses. There are kinds of problems in code that are weaknesses per se, but you can't really detect them with with static analysis.

A

But great question: there's there are I believe there are multiple phds to be done in in these particular niches.

B

So you're saying generally like anti-patterns that.

A

Yes, yes, one of them is ledger.red, but not used. Actually, um let's look at specifics, so initializing Global variables in init functions That's a classic that you can find it in N, plus one copies on medium that when you are writing chain code, you you shouldn't initialize Global variables in any functions.

A

Non-Deterministic chain code is a big No-No, then modifying a narrow, subset of keys, No input, type validation, no input, validation, no execution timer set uh unexpected unchecked call return, value, yeah and so on, and so on, and so on. So defining what we mean by weakness is a bit like nailing Jello to the wall. I know, but I hope that you that the general message is.

A

Yeah, it goes go through. Yes,.

B

B

Some asset was created in the chain called.

A

B

So, okay, okay,.

B

Whoa, okay, nice and I go back to him says, but it works. There are no I said. Okay, you have one endorse. Yes,.

B

A

Yeah yeah, so look, the technology is great, but exactly this for embedded software, this stuff has been. This is already behind us because it's it had to be dealt with. You have miserazi see you have coding practices at practically all companies doing uh doing serious development in C. You have a statical analysis tools for weaknesses. You have Dynamic analysis tools, that's orders of magnitude harder, but there are tools for that and so on and so on and so on.

A

All just because there are applications when we have to be where we have to be this serious. This is still missing for fabric I. Don't say that it has to be. There has to be available, but if we want to climb gradually up the criticality ladder, these will become necessary because I don't see any way to avoid creating recreating these techniques. If you like and reapplying them.

A

Okay, my time is up, thank you for being here.