Fluence Labs Hands-on p2p network choreography. Workshop, 5 Mar 2021

Previous Meeting

⏯

youtube image

►

From YouTube: How to make a multi-module project & write on disk

Description

Workshop, at Hack the System hackathon
Speaker: Alexey Pyshnenko
Engineer at Fluence Labs

----
Source https://youtu.be/om038Mnzj_w
Workshop flow https://www.notion.so/83beddecd2194a4789039406a85224df
Fluence challenges https://medium.com/encode-club/hack-the-system-announcing-fluence-87fa8dae59?source=friends_link&sk=18b7d34bdf4436d4193f69090aae9157
Get started with Fluence https://fluence.dev/docs
Protocol Paper https://github.com/fluencelabs/rfcs/blob/main/0-overview.md
Developer Hub https://dash.fluence.dev/
Github https://github.com/fluencelabs
Fluence Home https://fluence.network/

A

Now, as I promised, let's extend our service with ability to write files to the disk.

A

We can use uh familiar functions from rust, standard library from sdg fs the read and write functions. They just work. You don't need to have any external, binaries or stuff. You can just use standard library functions for that.

A

So now we have, we will have an application. That's called url downloader that uses curl to download uh something from internet some file, and we will use local storage to write that file to the disk all right, and that makes two web assembly modules, and we will have the second webassembly modules to provide the public api to glue those two modules together.

A

So, let's first, the the main focus here of the url downloader will be on local storage. That was because it's a new uh functionality that we extend the previous one, but you should also know that this is not a different name a little. This is just a convenience, never mind. It doesn't mean anything so.

A

Here is the local storage uh file, the main code of the local storage module? It does a simple thing: it takes a file name, it takes file content as a byte array. It uh sets, uh creates a path and it will always write all files to the same directory. This is for simplicity purposes, and then we just write content to the uh file system and that's it and we. If we have an error, we return it as a string.

A

You can obviously define your own structure with different fields to return error as a separate field, as you would do in rust. But here we return. It is a simple string and if everything is okay, then we return.

A

A

So get follows the same: we have a file name and we return by the ring.

A

We construct path from file name and we just use fs3 to read the contents of the file and if something goes wrong, we return error instead of file contacts, it's not the best way to approach error handling, but it's for simplicity for for simplicity purposes. It's really! Okay! We will see the error uh we will so it's gonna be okay for now, but you can approach that differently.

A

You can use your you, your favorite rust patterns here, except you can't use enough right now, but the fc macro doesn't support them, but everything is still expressible in terms of structures.

A

So that's the whole logic of the our new uh wonderful functionality of writing files to disk in a peer-to-peer network on a distributed, slash, decentralized, application.

A

So our up here it won't be much distributed. It will be a single app, but we will later see how to make it a distributed application and compose different modules as different services in the single application through airscript. But first, let's start with a simple approach where we have all this modules linked together.

A

Let's review the facade module I'll, go to examples, repository.

A

Then I go to url downloader folder, and here I have facade module, it's a simple module. It has simple dependencies from sdk log for local development and base 634 for well deserializing and serializing file file contents to stream.

A

It's again a single file here it could be it shouldn't, be just a single file. You could use any number of files you want. So this is a facade module. What's the first eight module?

A

um The thing that I didn't uh tell you yet about florence applications is that you can have only a single module to be publicly available. That's for security reasons, and for you know for designing reasons. When you design some service, you want to provide nice api to your users. You don't want to show all api uncontrollably right. You want to have some control over it and the facade module it gives a control over it. So only single module has its api publicly available.

A

If you need to have api from different modules publicly available, you can just repeat it here: we can just here define function and call the internal module here.

A

So we have two api functions. First, one is named, get and save uh and it basically returns. Okay, if ever, if the, if the file was saved so first it accepts url and file name, so it downloads contents of the file or site or whatever through the curl returns result and the result is saved to a file. There's a file.

A

So here you can see that we're using the same sd-out field. That was before that's because we are using external binder and if it's not successful, if download is not successful, we return error and we will log error. This log message will be seen through fc repo. If you develop your services on fc repo, you will see this work message.

A

So that's the first api first, a first function that we have in our api. The second function is a load file. You specify file name of the previously saved file and it returns it as a bait. Six base64 stream.

A

Pretty simple right: you download you store! uh If you query already downloaded file, you just get it from the file system and return its contents.

A

The crucial part here is how to link several webassembly modules together.

A

Here we have the curl adapter, it's a module around curl, it's almost the same as in the previous example, but it has a different, a little different signature. It has download function that returns result containing. We will look through it.

A

The difference is that it returns a result as bytes, not as string so okay and here, and we link the curl adapter to the facade module in almost the same way as we did with mounted binaries, but with this crucial difference that instead of specifying host here, we specify the webassembly module name here and the same with local storage. We specify local storage module name and we say that we want to use get function with the name file get we want to use, put function with the name file put and that's it.

A

So we can we're free at renaming the functions as we want. I didn't want to have, get and put functions here. I wanted to make it clear that this is a file put and file gate, and I did that this is through this link. Name attribute.

A

So far so good, so let's take a look at the curl adapter module. It's. It should be really familiar to us already, it's almost the same as the previous module.

A

The only difference is that, instead of previously, we had this function request and it was stringifying the results. So it was trying to convert std out uh or is to the error from byte array to string the download function. It doesn't do that for performance reasons. Obviously you don't want to to convert bytes to strings with no reason it's handy for returning to the front-end strings, but between modules, it's more makes more sense to to return bytes.

A

So the the difference is stringify and no string device. So in everything else is the same module as before it has. The curl mounted binary with it. So it has to have configuration file for that module, and we can see that we have this configuration file for curl here and it's the same as before. It specifies mounted binaries and let's take a look at the configuration file for a module local storage.

A

It's different, some funny reason it has mounted binaries here: it's it makes no sense, they don't break, it, doesn't break anything and it just uh it. Isn't it's unused so ignore that it's a configuration for a local storage module, so we need to have access to file system and here how it's done we have we need to specify broken files and we can only use t simple, temporary, slash, temporary folder directory and we say that we want to refer to the directory under name sites.

A

So here, if we take a look at the uh at the local storage again, we will see a constant defined here, the directory- and it's say it says sites. So you can kind of abstract here over the real path that will be used.

A

It can be handy in some cases, so, instead of linking sites to tmp, you can just. You can say that you link sites to a different folder.

A

Okay, I thought someone wrote me something about workshop, but now um so we have this uh project with three modules: cool, adapter, local storage and facade. Now, let's build it and use it in grapple.

A

Well done so, let's build.

A

It and let's do fc repo apple rebelconfig.tumble, and here we have it. Let's take a look at the comments prepared here. We can call get and save function to save uh logo to file system. Let's do that and we can see here and we can then uh load that file from the file system in base 64..

A

So as the all, this is happening since it's a rebel, it's a local thing. All this happens on my machine locally. That means that you that the file is stored somewhere on my machine. Let's try to find it, I'm not sure. Oh it's here, logo.

A

Svg, okay, well,.

A

It's a svg file, but let's open it in a more good version: yeah, it's flint's logo. Okay, oh well! I have nine.

A

Questions: okay, that's good! That's that's a good conversation! We have there. um So we what just happened. We have built the uh project into three wasm files. If I were to write here, I will see was impossible, it's more than free because I did build sh. It builds files like here and then copies them to artifacts, folder or well. It's more handy data.

A

Okay, now we build it and we used repo to to run the service locally, how it works. We have three modules described here. First, one is local storage that describes pre-opened files and marketers almost the same way as before, except it uses current directory instead of gmp.

A

It describes core adapter and specifies mount binaries as before, and it describes facade module with no uh access to file system or network whatsoever so facade on it. So it can't access file system, it can't access network. Only local storage can only call adapter.

A

Thanks well so uh we can also deploy the service. uh I will spare myself from doing that to save time, but anyway we can call it from cli the same way as we were calling the previous service. Let's review the script for the, so I just execute a download but air script here with service id specified to some previous build service.

A

I can deploy again and use this new service id and it will work so what's inside doing what it's this, it says that go to relay call this service call the function get unsafe, pass their url to logo. Here I short shorted it just for privity, save it as logo. Dot svg then go to the service and call call function, load file and read file from log svg, put it to variable name bytes as a base, 64 string and return to the client.

A

You can use you can define any service that converts from base 64 to base 58 or whatever you you can put any adapters on camera or converters services as separate services, and uh then you can send bytes there for some for, for example, for compression. If it would be in like a jpeg, you can compress it and return compressed files all in the network, all without returning responses to the client until the very end, if you need it, you can avoid returning bytes to the client whatsoever.

A

You can just take bytes from the file, compress it and upload to s3. If you are to write a s3 adapter conference, please do no one has done that yet. So that's a good idea to have a storage, s3 storage engine available to pull from fluence. That would be nice and it's totally possible to compose in that fashion.

A

Okay, everything of this- I already showed you in the code, so the interesting, the most interesting part as we are approaching through two hours already, and I still haven't- came to authorization so composition. Imagine that we! So what we did. We have linked. Three was modules together to make a single service, was it required? No, you could easily for your friend, develop a local storage service and deploy it and you deploy cool adapter service, develop and deploy, and then you can compose this to services here.

A

I have an example script of how to how to deploy this as separate services. So here we build was web assembly files. We don't use facade anymore because we deploy first local storage and the local storage. That was, it will be the first aid module. It will be the module that faces the public api, that's available to call from script and we deploy curl adapter in the same version. So this is the single module in the service before it was a service of three webassembly modules.

A

Now it will be two services, each of them with a single webassembly module.

A

The api will be different right because before it was get and save function, but now we will have to deal with the raw well with the api of the modules themselves. So in curl we have the function download in storage we have function put and we do the same thing. It didn't change much the discrete it's almost the same, but before we were using this api and then load file now you're using the api of specific modules, you call you call curl download to get contents as bytes and you pass those bytes to storage food.

A

The same thing that, let's take a look at, let's take a look at the facade module that it's here. No, no!

A

It's not in the picture! Sorry for scrolling around! So again. What I want to show you is that this air code does the same thing that we were doing in rust in the facade module. Let's take a look.

A

So here we have this, get and save function right. What it does it calls download on curl module. Then it calls foot on the local storage module we're here. We here doing same thing, download and put and then get the same as we were doing through load file before, but now anyone can reduce the curl service and storage service for their purposes right and we didn't they changed much. We split our kind of monolithic service into more in the smaller parts and provided them as a separate apis.

A

Isn't it awesome? You could do the same if you're working at the company and you have developed uh some api over s3 or over mini or or whatever, and you can give other people access to it through the uh webassembly adapter that will simply call it through curl or through locally mounted binaries. If it's whatever you can, uh you can give other people access to your api on the same product.

A

So, if a lot of imagine if a lot of people develops a lot of services and you can compose all of them, imagine if spotify gives you access to your listing history, and you have a service that some ai service that can consume that history and give you back some analysis.

A

You don't need to have api integrated into the service. You just have to know the format that ai service understands and how to convert spotify format to there. You could express that conversion of formats, for example, from json to xml or something as a webassembly service, and you can deploy it and others can use it, and you, you have deployed a small web assembly service that all it does is converts data from spotify to some analysis, preference, music analysis, service format and now not only you can enjoy the analysis of your musical preferences.

A

Anyone can isn't it just amazing. Imagine if the whole internet was working like.

A

That, okay, any questions whatsoever I mean so far.

A

Okay, yeah, you surely can ask any questions on currently you can I mean you can book a call with us with me. We will give our team and we will discuss your idea and help you architecture. It help you approach the solutions.