Fluence Labs Community, 8 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Developer community call #15 – Updates from the tech teams

Description

Timeline:

00:00 Intro
Updates from the business team
00:05 Hackathons to participate
Updates from the tech teams
01:53 Machines Department team
02:49 Marine book released
03:10 Marine architecture
05:57 Results from the summit in St.Petersburg
10:14 Addition to the upcoming events announcement
11:07 How Rust + WASI embedded a memory leak in Marine services
22:36 Network team updates
25:40 Service metrics
33:23 Q&A

Useful Resources:
*Community calls wiki
*Start hacking with Fluence https://www.notion.so/fluencenetwork/Hacking-On-Fluence-Primer-28a87754397048e1bec72e3bfc91fd9b
* Fluence Documentation https://doc.fluence.dev/docs/
* Aqua Book https://doc.fluence.dev/aqua-book/
* Fluence Quickstart https://github.com/fluencelabs/examples/tree/main/quickstart
* Fluence Examples https://github.com/fluencelabs/examples
* Discord https://fluence.chat

A

Yeah, hello, everyone um I can maybe share my screen. We have a few hackathons running at the moment, so I just want everyone to know that there is um first hackathon is polygon, built with hackathon started june 15 and it runs until july 25 you can still participate, they have prizes about 500, 000 prize pool. There are workshops, there are different talks, different sponsors and we with fluence we're sponsoring as well so just check this out. Then next one is sustainable. Blockchain hackathons, let's start at june 23rd and we'll be running until july.

A

18., sustainable blockchain hackathon runs through bitcoin platform. If you used it before. uh If you haven't just sign in with your github and uh your wallet and then you can participate, join the hackathon and see the the bounties um in the git coin. They also have the program of the hackathon. You can watch different workshops and talks and here are the prices and then the next hackathon we are uh participating, that's hack, fest that starts tomorrow and until the end of july. That's the page here.

A

So tomorrow is the opening of the hackathon, and we just had our workshop bernard. Okay. Sorry, it's it's just freezing and these are our bounties. We will publish all the links somewhere in the channels in discord. Okay,.

B

So lots going on anybody interested in hacking explore influence, sign up lots of them coming great prizes, and I think next we have uh updates from the tech team. I don't know mike, do you wanna? Kick it off.

C

Yep uh I had to share update from the machine teams, so I I want to share like our progress so internally, our team codes, machine department and the uh internally and like last uh months we established sorry, we established a process in our like uh influencer apps. That allows like everyone to share progress for what months and for every department in our women's labs. So this presentation from our internal uh all hands meeting and regarding the result of the last months so like the june, starts for for us with releasing the brain book.

C

So uh we presented it on the previous com call. So if it's fine and by this link, so marine books book includes, like almost everything that you should know about green architecture, sdk reference and and so on, then after releasing marine book, we uh like rethink the architecture of marine of marine itself, and you know like we came with the following idea with the following pictures that represent uh architecture of marine.

C

So here we could see several rectangles, so her rectangles on the top layer represents that, like top well api of marine, so they were read and they divided into rust and javascript. They have like javascript and rust aside. They have similar api, but like intended for different targets, different target language and the middle layer that consists of one rectangle called core.

C

It represents the same codes that share between these two targets, and you know like for us that compiles to rust directly and for javascript, it compiles the web assembly and is one of the feature of marine that we have core as a single source project single source, great, that use both for rust and for javascript, and on the like button, we have blue rectangles. They represent the web assembly runtimes that we reuse.

C

Apparently it's was mirror for rust and some like it really doesn't matter for javascript. We could run on every runtime on uh fun or spider, monkey and etc. Cetera and like we realized that such a scheme allows us to serve to feed different purposes, and we, like we factor a bit uh marine layer separation just to prevent abstraction linkage between these layers. We, like we, uh remove uh several entities from several things from the top layer called marine like a layer that uses actually marine.

C

It's called fluids observers, so the things encodes uh some internal things like call parameters, configuration files and stuff like that. So it's the scheme was partially implemented in code. We have some uh still some stuff to do so. I think we will continue uh doing that next month, but it's like it's natural separation, and now we could call marine as runtime without any reference to fluence and also marine could be used outside of fluids as a runtime itself, and also as like.

C

So I mean that you could use marine embedding in euros project or in your javascript project is like genuine purpose runtime that allows you to simultaneously load several modules, combine them together with a shared mustang linking scheme and use it like in several manner.

C

And next, uh so, in the middle of the previous months we have so-called summit, it takes place in san peterburg uh and took evic, and this time it was some form of global team synchronization.

C

So on the summit, we decided what to do next with marine run time, how we could approach uh last changes in the web assembly proposals.

C

Also, we stabilized road maps for marine and uh aquarium so like marine and microwave is the two main projects in the machine department, influence swaps and also we discussed several- uh was all well stuff, like the singaporean framework for aquarium improvements and marine cli uh developer experience service, configs and uh more and more that I didn't go to the slide after the summit.

C

So we decided to focus on stability and performance. uh We want to like to make this performance and stability as a first-class citizen in our team, so we want to focus uh on like on the next all next iterations and uh during this like work on stabilising performance, we found and fixed the problem with the memory leakage.

C

So, with help of larry and the valerie will present details later on the community goal, then we also dramatically improve the aquarium performance, uh so the fix was rather uh short but like it will allow aquarium to work uh much more uh performance than it was before, and we have like several uh about five more optimization and pipeline that we also improve performance in about one or two orders, so it will be in the next months.

C

I think also, as I said, that we want to focus on performance and that's why we wrote the first version of performance analyzer, so this performance analyzer allows you to see what's going on internally in aquarium, what's uh trigger resists, like uh memory usage or these uh some and no um long so far, is that or some function takes so long time to execute and so on and so on. So we want to be integrated to our pipeline and all uh next uh changes in equilibrium.

C

They would will be analyzed with this performance performance analyzer. So here I uh want to like credit to e1 like doing that, so it's really good job. Thank you, one and finally, finally, we implemented a mechanism for login anomaly. Data, so by normal data uh by normal data, we call data that, like executes quite too long, more than, for example, 20 milliseconds or, for example, it triggers a lot of memory, consumption inside aqua vm.

C

So we implemented this mechanism, it's already deployed on the crosslinker, and we have like a way to find anomaly data from a real network and analyze it with our performance analyzer.

C

So it's our main results so for machines department and later whether we will present uh like some internals about this bug. That was fixed. It's really interesting. So I think that's all for motions.

B

Very cool anybody have any questions for mike.

B

All right, I have some but I'll, save them uh uh well, larry. I think you're right next to lie. One more thing.

D

uh Before we uh before we proceed uh on the event side, I think what we forgot to announce is uh like, besides online habits upon running, we have two offline things uh going on next week. uh One is the uh ip first conference with protocol lives in iceland happening like next monday.

D

All week, we're gonna be there and then east and like this week in paris, a lot of people gonna, be there we're also sponsoring hackathon there and more events, but I think we don't yet have the exact schedule. So just uh where we're gonna be in parties uh happy to meet, uh please join our meetups, we'll be happy to see you in person. Yeah. Sorry, that's that's! It.

E

Okay, that's fine! So, as uh I might say, we found a memory lake, but it was actually in uh embedded in web assembly files and uh I need to tell some prerequisites for you, so you can understand what happened and so on.

E

First, uh in the last years, like one or two years, the assembly was a group was developing a new execution models, common model and reactor model. It's their models of execution of use, case or execution of velocity.

E

So command is a like a usual binary, usual executable on your computer. It has a single entry point, you execute it, then you drop and it's uh pass the state control back to the caller. And then all resources are freed. The memory state is no longer valid it can. It must be thrown to the press and reused so, and the reaction model is something new. It is compared to shade libraries and uh in case of web assembly. It uh looks like that.

E

It's a assembly that has an x and export function for initialization and has many expert functions that can be called after the initialization function, functional and they they will use the state they use the memory. The globals and so on and the command module can also have many export functions, but there is no uh only one can be called on. One instance, like your incentivity model, call one arbitrary function once and then you can use this web assembly instance. Okay and uh before there are these execution modules.

E

All modules were like comments, but there are no documentation or guarantees or also or something that say that you can't call an expel function twice or two export functions on one module, okay and what we do. In france. We do a marines, we create marine services from vessels and they are reactors by the definition by the use case.

E

um They are compiled as comments, but we owe many expert functions many times and we have some kind of initialization functions on each one and some kind of initialization function on each model, so they like reactors, but we compile compile them as commons, because there are no other way to compile wasn't.

E

But there was a future problem that someone can start distributing comments and records and start using the differences between them, and that happened uh c. 1.56 switched to lm13, which added support for common reactor execution models and started uh emitting the call forward assembly different by default. It's common and uh in that mode it wraps every expert function with a constructors on the striker's calls that, like wasn't called ciphers initializes the state and wasn't called deterrence they initialized the state by by design it's good for commands.

E

You need in every command and in every export function and installation and the initialization in our case wasn't called stores, uh initialized twice ellipsee and uh initialized.

A

E

Files initialization of azure fc, includes a memory allocation, copying the environment variables and it allocates some memory and stores it's somewhere in the washing, and uh this memory, like expected to be treated in the washing call deterrence but wasn't called earth, does nothing literally literally nothing and uh if michael also static and starters start to distract study destructors from other languages like super fast, but in rust it doesn't there. There are no static, constructors, static structures, so each expert function.

E

I, if we call uh several expert functions, then we have the initialization function called several times and uh nothing frees its resources. So it's a memory leak by definition and, moreover, our 18 interface types. Implementation uses several uh inter several expert functions to work, to pass values to webassembly and outside of the wasn't.

E

Our cage, release, objects and results related, and they all also wrapped with the tutors guitar skulls, and uh that's a huge memory and a huge performance degradation because, uh like if you send a vector of like 1 000 strings, you will have a thousand of those called more than one thousand times during one call and uh all them will leak some memory. So it's super bad and that's super hard to find like you have memory, but everything inside your last coat is fine, but there are two ways to mitigate it.

E

First is installed explicit version called stars, o in function named start or interlace, so llvm will detect it and will not um wrap functions with a star center.

E

Skulls just will do as it before or past the feature or pass the flag to the odm, so christ that we need to compile a reactor, not a command, but it works only in nitrous, and we don't want nitrous and the we resource uh what community says about it and we found that people have the same issue on their side because of their problem and people who work on these changes just think that they he's designed it and uh there there is no um workaround, except that I mentioned so.

E

What's next, we will use an explicit constructor's call. That's uh I described it, and uh the one problem is that you need to change.

A

E

A

E

The support of the reactors commons will be stabilized, but it will require nothing from users, only changing the sd card, sdk version, because everything will be inside of the key or we could use option from nightly. But we decided not to do it because we don't want to switch to nightly and still we will. We may have to change it when our interfaces will be stabilized.

E

So our plan, we will use that pixel constructors approach. We will bump minor sdk version of minor minor. My version will turn an error uh when loaded um a module to the new marine that is compiled to the old sdk sdk. Before the fix, and though we will release updates for everything for sdk marine nodes ripple examples documentation.

E

We will announce it everywhere and we will do it in the second half of july, because we have now many ongoing ongoing accounts and we will have. We want our systems stable at the time of the ipfs summit. We don't want that people will encounter some errors, but it would fall, couldn't be able to load modules to the network and so on.

E

We we just fixed our infrastructure services like registry.blog.

E

And that by bfs we just added a workaround to each module. It is possible because we only need to like invert a function and code.

E

So that's a story of one disaster we found and if you are using blossom, you may have to check your lessons too, because you may be affected by this problem and maybe you will need to work around it. So, thank you ask me what questions you want or everything.

B

All right, thanks for the larry, it was great anybody, questions for valeri memory, leaks and uh fixes. Okay and let's move along, we have uh folex aka, alex a with a whole slew of.

B

Updates on measurements, metrics and uh hello system.

F

Yeah great uh well, okay, um so yeah, I don't have any fancy presentations. This was awesome by the way um yeah and uh I guess I will start with the performance just to continue what guys were talking about what we've been doing on the network side. What I will we'll be talking about, like we have several like teams and stuff inside inside our influence, development teams, and uh one of them is the network team uh that I will be reporting on right now uh and what we do.

F

We work on like network uh rust, peer, devops, stuff, infrastructure, stuff, all that all the surroundings built in services just services. Everything like that. So when guys from marine from machines team deliver on performance, it's our job to integrate it and run it and deploy, and so anyone can everyone can experience it. That's exactly what we did like integrated and deployed fixes from machines team. I saw huge huge performance improvements now different aquascripts digest as fast as light, because everything is is improved.

F

Both services and uh aqua, vm, so particles, interpreted fast service calls are fast and everything just flies. That's awesome also. We started to to dig deeper on performance, so integrated uh anomaly, detection for particles, so if some particles are like executing too slow previously, this low was around 22 to 60 seconds and now too slow is uh around 500 milliseconds.

F

So you can already feel the difference in performance yeah. So now we're collecting those particles say them to disk, and that allows guys on machines team to like take the data and experiment and debug and profile locally. Previously it was like only a line in logs that said: hey this was slow, and that was it now. We can actually address this issues. That's awesome, yeah, so we're still investigating some dream outs on no starts.

F

That happens because some of the services fell to to be deployed, but that doesn't actually affect anyone who doesn't host a lot of nodes like we do so that that's great, uh we had a lot of work invested in metrics uh last several like months, um and we delivered a lot in that area. uh For example, like I can show you some some vlogs right now.

F

Let me close food delivery yeah. So uh what we've been working on? Maria was working on uh service metrics um and she implemented like aggregated uh service metrics that are stored in prometheus and they're collected, and now we can see like how many services each node has like how many services we have in total on the network. For example, we can like switch here and just to stage, and we will see a different numbers right, so we can see that on stage we have only this much services and every stage.

F

Node has this much services and it all looks a little bit better because uh less data uh yeah, so we can like understand, what's happening on the network uh services wise. We can see like how they use memory like we can see that total use memory grows constantly. That's because we don't have the allocation in like we, don't don't stream memory uh in web assembly services so every time uh a lot of memory used it will it just stop at that point yeah, and we can see like here in in this graph.

F

We can see that uh currently most of the services have only one module.

F

Roughly one-fifth of them has two modules, just a little small. Really small number has three modules and there are no services with like five modules in it. We can see it and once we have, we will have some more complex services. This number numbers will still start to grow as well yeah. Also, we can see like how many free memory uh services have, in average, like it's a histogram, so so yeah, that's all, really awesome and gave us a lot of insights.

F

Also. We have what we have also also we have I already presented several months ago, but now we can move to a new infrastructure and it's all green um yeah, we'll just scroll real fast. So this is node. Metric image looks like rust, peer metrics.

F

They show the overall health of the network and we can see that everything from the last six hours was was fine. If we switched to 24 hours, we will see that some nodes on test net diet a few times because they have too much services, for example, and they just ran out of memory, um and we can. We can see how many particles arrive, how many particles are arriving per second stuff, like that, we can see like what the current kind of bandwidth on the network we have like the average size of the particle.

F

We can see that most of the particles here are between one kilobyte and and 10 kilobytes, and just some of them are more than uh one megabyte yeah stuff like that. We can also see the interpretation time previously. It was like in in here like around some time. Sometimes interpretation time took up to 15 or even minutes uh and right now everything is below one second, so that's that's not some improvement. I guess we still have some.

F

You know, for example, here we have uh some particles that were have been interpreted in more than 15 seconds. That's why we have uh anomaly detection, because sometimes it still happens, yeah stuff like that, and what we actually have a plan to to pretty exciting thing to open this dashboard, uh not only for viewing but also give away for for people who host uh rust peers to send their metrics to us.

F

Like, say, hey, come to us and grab our metrics, and in that case uh those hosters, they will be able to see themselves in this dashboard. There will be like more ends here, not just r but uh other people's too, and they will be able to see them on public network and with all those awesome metrics. I think that that'll be great another thing that we have and I'm really excited, because it's really useful thing we have. Finally, uh uh we have finally implemented a log collection.

F

For example, um if I want to see uh the interpretation blogs on uh crescendo, I can see I can just uh write interpreted in and I will see uh which particle took, how many uh seconds- and just I can see just all the logs, for example, I want to to take a look, how this particle uh was like traveling through the network.

F

I can just enter its id and I will see that uh like it. It started to run. There were some failures. uh Some nodes couldn't reach other nodes because, like we don't sometimes we have nodes run by other people and they don't have like ports open stuff like that, so this particles, so they uh often spawn the whole network, so they can kind of spawn nodes that we don't control and they can be unreachable.

F

That's completely fine uh and we can see that uh once it gets to one wire nodes and starts executing we can see like it was executed, traveling between the nodes and stuff, like that, that's pretty awesome for debugging um that will speed up our development quite a bit. So that's awesome what else yeah uh and we we we got. uh Our team got bigger. We we got anatoly uh in devops position, you're going infrastructure, ops position uh to to name it correctly.

F

We finally moved uh to aws for uh continuous integration, uh because we need like powerful machines to build rust, uh binaries, publish, docker containers. I think at some point we will move our continuous integration to fluence, but while we didn't don't have that we're using uh github actions so and the runners are now on aws and builds uh now take like five minutes and instead of 30., that's awesome too, and overall we're moving to the to zero trust.

F

Infrastructure want to have some expertise in it and apply it gather it like learn how industry works in that sense and apply that knowledge to design influence yeah. That would be great. uh Also. We had uh quite a bit of different uh developer. Experience updates, like there were problems with creating services where you got an error and you couldn't like decipher it. You couldn't understand what truly tells you now you could anyone can, because it just tells what's wrong. That's awesome as well.

F

Also we had a lot of work done on built-in services, like registry trust, graph and stuff, like that.

F

They are now have all the fixes from machines team, so the performance rise up uh quite a bit and they work real fast and they work without bugs we started to cover them with uh aqua tests before it was only rust tests, but now that's kind of end-to-end integration tests that could span several instances of service, so we can like actually test cover with tests the distributed properties of the services we develop for the firm's network.

F

Yeah. I guess that's it for me.

B

All right great, it was a great update, uh obviously lots of improvements on that end. It's gonna benefit everybody, and uh I don't know, I think, that's about it. Anybody else.

B

F

Service on the on your local computer, you would have to write the whole script project with packages on and all that stuff uh you have to like. It was rather complex and right now you can just write a simple uh javascript file that has export function.

F

Plugins that returns an object where we've just functions like, for example, I want to define a service called ipfs client that has function upload that would go to remote, apfestnode and upload some file to it, and I want to use that function from uh from aqua like like this, like, for example, I I want to have this kind of service in aqua, and previously I would have to provide index.js or index ts and manage dependencies and all that stuff and right now uh things has changed.

F

All you have to do is just have an mgs file with your service definition and it should be returned from plugins function, and you can have a service that you can call in aqua and that's pretty awesome. I think I wouldn't, like showcase, wouldn't go into too much details. uh Anyone who is using aqua should be excited. uh If you don't know what I'm talking about, then you should read aqua book and come to our chats and ask, and I will teach you how to use the plugins and so on. I guess that's it.

F

That's a short description. It's really awesome feature changing the whole experience of aqua development, so yeah, maybe we'll have a dedicated call to it.

B

Okay way to bury the lead all right, I think any questions on that.

B

No all right! Well then, uh thanks everybody for being here and uh we'll see you in four weeks and we'll cheers cheers.

D

C