Cloud Native Computing Foundation KubeCon + CloudNativeCon China 2018 (Shanghai), 21 Apr 2021

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Challenges and Solutions of Using Kubernetes for Blockchain Applications - Tong Li, IBM

Description

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io

Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects

Challenges and Solutions of Using Kubernetes for Blockchain Applications - Tong Li, IBM

Kubernetes easily allowed for users to scale up and down the resources used by their application, which makes it a good fit for both proof-of-concept and product ready usage. Blockchain applications require a lot of compute power and storage and introduce their own set of unique challenges when running in a containerized environment. In this talk, Tong will provide a brief introduction to the Blockchain and Hyperledger Fabric technologies and then dive into how they can be successfully deployed and managed on more K8S cloud providers (e.g. IKS, GKE and AKS). Highlighting the challenges Blockchain poses for networking, storage and CPU resources, he will then discuss the solutions he developed for each and how you can leverage them yourselves - finally, touching on what possible features might be needed in the future to make K8S even more robust for these compute intensive applications.

To learn more click here: https://sched.co/FuKt

A

I think here we can get started, so this talk is about the challenges when you trying to deploy blog to applications on to kubernetes. Specifically, are we are talking about the hyper laser fabric? How can you actually deploy and run your applications on communities and what kind of problem you might face, and how can you solve those problems.

A

Actually I can talk in Chinese, I, don't know, I assumed you wanted English.

A

Okay, so who else want once English.

A

I can continue in English. Okay, that's fine! Actually that's what I told the organiser I'm gonna be using English, okay.

A

So just a little bit of introduction by myself, my name is Tong Lee I work for IBM as senior software engineer have been worked for IBM IBM for a long time. A couple years ago, I was working on OpenStack.

A

The cloud now I work on typology of fabric I am one of the four fabrica cello commuters, so cello, actually I'm not gonna focus on cello in this talk, but I just want to give you a little bit introduction of a cello cello actually is an installer or deep lower for fabric for the for the products actually under hypo later umbrella.

A

So if you want to deploy fabric network unto or some kind of environment, for example, kubernetes or just bare metal machines- and you can use cello to do that- so you can find me on github.

A

So I'm thinking we have to talk about a little bit agenda, so you know what content we have for this talk. So first I'm gonna be just briefly talk about blockchain technology. Then what is unique about it, then the differences between different blockchain or lasers or applications.

A

Then I'm going to talk about high valachia fabric, because this is the product that open-source a couple years ago. It is very different compared to other a blockchain based applications.

A

And then I will talk about. What's the reason that we want use communities to run fabric after that, I'm gonna talk about the challenges in their solutions and some good news, and we gonna have a little bit of time for questions and answers.

A

Okay, a blockchain technology mmm actually after I sign up for this session, I was thinking. You know what is actually the best way to describe blockchain technologies. Then I find actually this description from ibm.com. If you go to bluemix, comm or IBM comm, you can find the Paluxy solutions. Then you'll find this quote. It says blockchain can do for business. What the internet did for communications actually I. Think that's very true. You know the invention of the internet actually changed. The world.

A

I may also add that the mobile devices right the cell phone, the mobile devices, actually really changed how people actually live their lives. Those are the big changes. Now people put actually blockchain technologies in that same category. So if we I mean human being use the technology the right way, I believe this will be a revolutionary technology.

A

So it also says every second of every day, business exchange of value with supplier partners, customers and others by value. What we really mean is the goods services, money and data in whatever actually has values. For example, you have a block right, you post it and then actually people go in. There read your blog says: yeah I like it. Actually that also has some values right. All those things you know eventually will have something that people probably feel you know it's valuable.

A

So mmm when people actually exchange those values, you actually creates a transaction right and you want to record those transaction transactions actually accurately so so that those transactions reflect the actual the real world right. So successful transactions need to be fast, precise, easily agreed on by all the parties right and they say you gonna, send the money to somebody- and you know all the parties probably include bank will have to agree that I the sender sent $10 or $100 right.

A

You cannot have a system that the sender says: I send a $100, the receiver says no, no I didn't receive $100 I received fifty right. That would be a problem right, so you want to be able to actually successfully record those transactions and accurately so I mean people do that already right right since the very beginning of a human history, people do transactions right. Blockchain is just the system or technology that it's a better way to record those transactions.

A

So what is what is the unique about blockchain right, if you look at the.

A

The blocked applications actually I should have asked how many people know the coin. P2P right, I think almost everybody. How many of you even know either.

A

Very good excuse me: okay,.

A

So how many people know proof-of-work.

A

That's about half, that's pretty cool, it's very good! So so, if you know blockchain systems, you pretty much know that all blocks and system supposed to be decentralized right decision was, it doesn't mean it's. You know distributed okay. There is a difference there right, the the leaders are shared. Anything recorded on the later is supposed to be temporary evident. So if there's some errors actually recorded, that kind of error should be very easy to find out.

A

Also, if you put a transaction on the ledger, you're not supposed to to change that, it will be there right, you can't you can't change it actually.

A

Before you put the transaction on the ledger, all the part is supposed to agree on right. So this is a thing actually happened. We're gonna record as such right.

A

So that's actually very different compared to the traditional way of you know. The digital lasers.

A

So you know a lot of people already know a Bitcoin and other cryptocurrencies.

A

Those things most of those applications or the coins, actually our public, so the transactions available on the network right, if you actually participating in the network, you you'll be able to see all the transactions, but you may not be able to find who owns that account right.

A

That takes some effort, but it's not to say you cannot really find who owns that right. So the data actually on your later is public open to everybody and then there's some other kind of a change that they want to make those transactions to be private or super mission based only the ones that.

A

Have the permission allowed to to see those transactions? Okay, so there are certainly different ways to see the data right. So I actually asked this question. If you look at all the blockchain technologies, the way that people put the transaction on the ledger, there's so many different consensus method you can use the most popular, probably is the proof of work. There's some other ways like approve a stake, proof of elapsed, time and the proof of authority.

A

You know each of those method. They have has their pros and cons. So you have to look at the application you're trying to develop to choose what kind of a consensus method you wanted to use right.

A

Of course, there's some other consensus methods. You know there's actually so many.

A

So hyper later fabric I.

A

Think in early 2015, IBM and other companies started develop this product. The we know that not all the business can be put there. Transaction can put their transaction on the public ledger right. Some of the transaction has to be private right, especially when you look at the banks. I. We don't want people to know who owns how much money you know who who will deal with who right you just can't can't make those things public.

A

So the think about you know: how can we secure those transactions? I? Think that's how actually happily to fabric get started so very quickly. The we put this product on the Linux Foundation open source the fabric.

A

So, as I said earlier, fabric actually is permission based. You have to be part of this consortium.

A

A

A

It's sort of okay, you have to be part of this big party right. You have to belong to two Agana organization, that's part of a consortium so to join that consortium. Actually you have to go through a prole processing. You know sign up and request, allow the existing members to actually prove your request, and eventually you can join the consortium.

A

For the consensus method, my understanding is currently it's some sort of like proof of authority, because we actually fabricate implement the the system. We call the Odin system that that system is like everybody. You know, trust this ordering system when the transaction comes in, and trans I'm gonna send it to the ordering system. The ordinance system actually line up. Every transaction eventually create a block.

A

So you sort of a trust, this alternate system, and so to me, that's, like you, know, proof of authority right to make that work. You really have to have all the parties participating in in a in a consortium that everybody is approved to be part of it right.

A

So in the early days, I think the ordering system.

A

Was using some other method now started at 1l I? Think the ordering system use the Kafka queueing system. We also support support a a solo system. It's like it. Just one note. You know.

A

I mean, if you want to know the details. Actually it will probably going to take a lot of time to go down to each of the component. I could just consider that the the consensus method, the currently fabric sport, is like a proof of authority. Okay, the other thing it's quite unique about hyper legend fabric is that when we actually build fabric, we deliver those component as containers right rather than the binary code.

A

So if you look at the darker hub actually can find all the different releases on docker hub for different component like appear order. You know the kafka container image all that kind of stuff, you'll find many different versions. So I don't think if we actually produce the binary and put have the boundary for they say for 0.04 others, we simply produce container images and put on docker hub.

A

If you're really interested interested in in development of fabric, you can certainly join the community a you can contribute a code in fix bugs help out and, of course, if you just the user, you want to use it. I also encourage you to join the the Rakhi chat fabric channel. You can ask a lot of questions. Normally.

A

Those commuters will respond to your questions fairly quickly.

A

I mentioned cello cello is one of the 10 hyper ledger projects.

A

The sole purpose of a cello is to help you deploy those products onto different environment right so I'm responsible for the instable agent that actually allow you to provision virtual machines from let's say AWS or OpenStack cloud or some other cloud producing virtual machines first and then deploy actually communities unto those nodes. Then you eventually put fabric unto the communities cluster.

A

Okay, so why use Cuban Aries for a fair pick, as I mentioned? Actually, when we design fabric it was designed to run as in container environment. So it's natural that you know communities cluster, provide providing this environment. Then you want to run those things in that kind of environment.

A

Affirmative network is decentralized like many other production networks, so you can stand up your own notes anywhere in the world. Then you can join existing fabric network. Of course you have to go through the process that how you join the network- it's not like you know, like a Bitcoin network right, you can stand up your minor E just just doing it right. It's it's quite different and you can also start very small right.

A

Okay have just one peer in one organization, then you can join a bigger fabric network. It's done it two transactions in you. You know what data might be on the ledger, so you can start small use some sick use. You know very little superior or you can grow your business that you can add more peers.

A

So you know when you look at many blockchain, consistent method. Like a proof-of-work, you know that's very resource-intensive, because a lot of actually money or power to actually produce the hash code, but fabrica actually is different, because the consistent method is more of proof of authority like so. The odorant system is responsible to line up all the transactions to produce the hash code. So it's not CPU or memory resource intensive right. So you know you can certainly run this. It's not like it. Gonna cost you a lot of money to just produce one hash code.

A

Okay, so that's pretty much all the things I want to talk about about fabric and blockchain technologies, because this is really a cuban eighties conference, so I cannot really dive into the details of block chain or a fabric. So next I'm gonna talk about that. The challenges and the solutions actually I know challenge that I faced while trying to deploy if I break down to humanity's environment, even even though those sort of challenges I'm not saying that communities is not good right. Probably communities is the best out there right now.

A

Okay, so don't take the wrong way.

A

Now you have fabric, or you have some other I, don't know blocks, and you know systems you want to deploy onto Cuban, ladies cluster. So what is the first thing you have to do right, get the Cuban in this cluster right right.

A

So basically, there are two ways you can do that one is okay: go to Google I'll go to some Microsoft or IBM cell okay, I want to Cuban a this cluster. Alright, that's one way right, so you just pay them money and say give me one right so well,.

B

A

Get one or you say and I forget about that so I'm gonna just get my hands dirty and get a few servers in I'm gonna set up Cuban eighties on the sparrow sure you can do that right.

B

A

I remember one actually keynote speaker as the audience.

A

Things that I've, given it is from there metal is easy.

A

No one right, it's not easy right, I mean you have to actually to know a lot of things to make that work right. So okay, I work on OpenStack for a long time and then contribute to to Neutron.

A

You know to actually make your humanities class to work. You really have to have very good understanding of networking technologies right. If you don't know what the overlay network is named, I, don't know how you can say that for working QA, this cluster right. So you know that that matter normally is you know if you're serious about things? Probably you don't want to do that.

A

You're, probably gonna, go to the cluster community cluster provider and say: okay, give me one even with that method, if it's still very difficult, yeah I, don't know how many of you have tried different communities providers.

A

How many have you tried at least one you?

A

My name is you have a try too.

A

How many of you have try three from streak providers? Okay? So what's your experience.

A

You I'll be very similar.

A

Okay, yeah, actually you know if you yeah I supposed to repeat the yoga, your answer, I guess so you think it. You think you have to use the CLI right, not.

C

Have to but I prefer because there's a easier to switch because.

B

C

Structure of the command are very similar, so if I.

A

C

To switch from this crowd, two-bit cow is as simple as I want to write a boat to come solos. You icing, okay,.

A

I I tried like four from four providers, I wouldn't say the names because it's kind of sensitive afraid you're not supposed to criticize those providers, but my experience is Verity from the firm one provider to the other, a you know in some of the cases it's just you have to know. You know the cloud first, you have to add those nodes. First, you have to provision those first, then you can add. Then gradually you do. You follow the steps. Then you have a cluster right. That's one way the other way it would be.

A

Okay, go in to download their the providers command, I to and now get certificate, and you know follow those steps. Of course, some other providers actually do a better job. You log into the UI say: okay now, I want us three nodes: I want to four nodes, find those. Now fifteen minutes later you probably have communities right. Some of the providers I take a long long time, okay hi. So so what I'm saying is you know even to get the cluster? Is not that easy?

A

And you know, sometimes you have to write some vendor specific code to actually get those things going. You.

A

So now let's say you have a communities cluster right, you you wanted to do things right normally, as a developer, you probably, of course you can use the dashboard right. But most time probably you want to use the cubic arrow to do stuff to be able to do that. You will need the cubic config file right to get a cubic configure file. It's not you think. Ok! Well! This is one file. What's the big deal right, but the problem is different providers. Give you different ways to get that some are provider.

A

You may say: ok now get this certificate. First then copy/paste that those things then edit those file. Now, after a lot of steps, you can you get the one cubic configure file right, it's very tedious in you can make a small mistake and then just kind of log in right.

A

So so that's actually a a challenge right just to get the very basic could become big file.

A

Now the I want to check the time.

A

So hum how many minutes we have left mmm I, guess I can't keep on going so the third one you know for the blockchain applications- probably that's also common for other applications, so you always want to save some data. You need some storage right. The issue is the storage. Is that Cuba ninis supports so many different, persistent storage I just have a list on the left right hand, side. This is another complete list. You can see the capabilities, the performance and everything is very, very different.

A

Okay, even though, when you create this persistent volume, you can use different mode, but the deep that the one actually when you create your cluster and in the storage class, you have may not support all those mode right now, when you want to monitor those persistent volume claim onto your your container. Actually in many cases it won't work right. If you don't, even if the mounting is successful right, you probably cannot write to it right.

A

So you you you, you actually have to try. Many different combinations just realize that oh okay, this actually won't work. So what I end up doing is that, because, because I cannot just create a program which works against IBM cluster I have to create the code actually works against many other clusters right. So so you have to make sure things actually work. So what I end up doing is to set up actually an FS system. Myself then created the this environment class right. So that way it will work consistently through all different clusters.

A

So so this one I mean if you have worked on kubernetes. You probably know that you know when you stand up something you know you may have a lot of pots.

A

Now you wonder your pulse actually communicate with each other, so you don't want to keep try and find that those IPAs right. You actually want to use the name right. So that's where the DNS really helps, but the problem is that you know to make those things work actually, for every part, you probably have to set up the service right, if you don't, as I, showed you the example here the pulse.

A

Actually, if your cluster actually enable pods also to have an entry in your DNS services, actually the the the name contains the actual local IP address, which makes that entry actually not usable right, because you don't know that before you deploy it right, that's after fact right so you know.

A

So the only way I see that working is you know for every component for a fabric like a peer. You know, older or or even they say we use Kafka right. Then you have to set up a Kafka cluster, then Kafka actually also uses zookeeper right, so you have a lot of actually component dependent on each other, and this cluster have to know each other to make it work right.

A

Normally, when you set up Kafka or a zookeeper, you want to have at least three notes right, so those windows that have to eat to know each other, that's how they communicate so DNS service actually is important, but you know you have to set up the services to make that work.

A

B

Oh hi, this there's kind of a work around with stateful sets that you can do this with where you set up a headless service with no IP bound to it, and you have a stateful set connect to it. You map them together and it'll. Give you like service zero service, one service and they'll be persistent and you have persistent storage and you don't have to set up a service for each node in your Kafka thing to get a guaranteed name. So.

A

That way you use sort of a like yeah. You have to know those names you.

B

Will its if it follows a pattern mm-hmm.

A

B

Start they start numbering at zero and it goes to n. So if you have five, you have zero one. Two three four I will be service name I forget the exact DNS parent ended and then it's like service name, one service name, two, three or four. So if you have Kafka, you could have a Kafka service and you would have all the pods exposed by. However many pods you created in your stateful set and then they're just you don't have to manage a one-to-one service mapping for like N Things mm-hm.

A

Okay, yeah, we probably can't talk that flying yeah. I got two minutes left. Okay, yeah appreciate that though, so the other challenge is really the because sometimes you do want to actually see things going and the components easily from dashboard. So cuban is dashboard. Access. Actor is also issue. Different providers actually have different ways to launch the dashboard. It's it's really a less important challenge, but nonetheless it's a it's a challenge.

A

Okay, doctor issue: how many of you know smart contract, good, good, okay, so fabric actually support smart contract, but we don't. We don't call it smart contract. Actually, we call the chain code.

C

A

It's it's sort of like a smart contract, but when you actually install the chain code onto the pier here, eventually creates another container instance. Okay, so in the communities environment, that's not really a part. It is a native docker container instance of running in the environment. You know to be able to do that. Actually, you have to have a darker service and the point right, but in the Cuban is cluster. You just don't have that right.

A

So so this actually is a really a bigger problem for fabric and also you want to run those containers side by side with the up here with your application. So what we actually a walk around with we use is to use the daemon set to actually create the top Korean docker container. For the end, the point that make sure that and the point exists on every node in your communities, cluster, okay,.

A

I think I have run out of time. So very last one. The the other challenge is the application deployment. I told you guys that I work on cello, which is a Installer. So when you do is like a fabric like a large system, and you always have many many different components. Those components will have to deep deployed in in in order that it makes sense. For example, if you want to set up the odorant system right, you you first have to set up your zookeeper I have to make sure your zookeeper class actually working.

A

Then you have the dick Africa working. So you have to go step by step, but when you use actually the deployment method of communities, you don't get that right you deploy, then you know you cannot have all those things deployed in one shot. Okay, so you have to have some kind of a sequence to make sure the stuff actually work, one component work, and then you go on to the next component. So actually what what we use is to use the instable to deploy the first, let's say cluster, then we move up.

A

Wait wait until everything comes back all the service in the point and the ports and everything mmmm want to the next one.

A

Okay, so the good news is that they're, the Cuban is: has this conformist working group they trying to make sure that all the providers actually conform to some certain standards and what I hope they can do? Is that not only to stop at the API level? They can also consider some usability. For example, some of you is easy to use. So this is really just a wish. Okay,.

B

A

Don't have to put this slides honest yesterday, my medicine Mia link says okay, wherever comfort, comfort, Zico. You have to add this to your deck. So this is the IBM thing you can sign up. If you're in North America, you can sign up to participate in the contest and if you win and you get it wrong right, we have a 1500 actually available. So if you're interested, you can go.

A

If you want to take pictures.

A

Okay, I think we run out of time. If you have a question, you want to talk to me, probably after these sessions over all right. Thank you very much.