Fluence Labs FOSDEM, 6 Feb 2021

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Aquamarine under the hood.

Description

We've seen the theory behind Aquamarine, and how it can be used to build apps. In this section we'll review the AIR language that is used to program distributed backends running on Aquamarine. We'll dive into the possibilities of the language, and some of the patterns that emerged from our experience of writing AIR scripts.

A

Hello, my name is alexey and I'm a developer at fluence labs, and today I'm going to talk about the air, the aquamarine intermediate representation, language that underlies the apps you have seen built in previous videos.

A

That's the workhorse, or rather a control horse of the applications and uh influence.

A

So what is air air is uh is based on calculus, so it's our attempt to put by calculus or into practice and implementing to configure a distributed network.

A

So you can think about air as a configuration language for distributed network. uh It controls, topology and execution flows, uh saying which node should execute what at what point in time and where execution should move after that uh interpreter, uh it is a an interpreted. Language interpreter is written in rust, so the whole interpreter is distributed as a single uh web assembly module.

A

The language is s expressed, so it's kinda lisp, like it is a low level, so it's meant to be a compilation target rather than endless and language for the developers.

A

So it's not for humans, but for machines, and it is it's inspired by webassembly text format.

A

So what can you expect from air in the future? What kind of evolution could you expect from it.

A

We we're going and already developing a graphql like type safe builders in sdks, uh for example, in js sdk. What that means is that you will be able uh to avoid uh writing air scripts by hand, and uh instead of that, you will write a composition of javascript or rust or ill functions that will, at the end output a ready to run air script.

A

Another way to to avoid writing air scripts by hand is a high-level language. That's called aqua.

A

The this is a work name, so it might change that will be a type safe language that will help developers to avoid pit fails of distributed computing like deadlocks and leaked resources and such things you can expect uh flow visualization to appear, so you just put an air script to some web application and it will uh draw you a map of the execution, how it will go from one node to another and so on.

A

We are also working on ide support so on syntax, highlighting uh code completion and such and yeah we're constantly working on development tooling. So if you think you're missing something, please share your ideas with us.

A

So what's the essence of air execution, the the main, the the whole thing is about uh handling incoming messages that are called particles um and contain script and data uh interpret them in the webassembly interpreter modifying data inside particle along the way. So we receive one particle incoming to our peer.

A

We interpret the script, that's built in the particle and we output new data, so data is uh immutable in the moment, but mutable across the whole computation and then, uh according to the script that controls the execution flow and says where to uh send the particle later.

A

The pier sends that particle where the script says and along the execution. Some effect some effects can happen. For example, some calls, for example, calls of cli ups or calls to demons running on the pier or any anything that that is instructed by the air script.

A

So each script consists of uh of list of instructions, and each instruction has the following structure: first comes the instruction name in this case it is called the one of the main instructions that does all the work. I will talk about that later it.

A

The second argument is a address of the peer that execution should happen on. So in this case, we are saying that go to node with this peer id and execute the call on that. Node then comes uh service and function. Specifiers uh first says what kind of service or what service should be.

A

This function uh run on so in this case we're saying that please run function, put on service dht on the node this and pass the arguments, k and value to the function, put and write the output of the function to variable with name result. This variable variable will be you can access this variable later in script. So if there are any following instruct instructions after that, they can read and modify result.

A

So yeah, as I said previously, execution data is used along the execution it can be read from. In this case, the arguments, k and value are read from execution data, so it is assumed that this instruction executed after someone already calculated the key and value and put them to execution data and it can can be written so result here, will be written to execution data if we are to draw parallels with uh imperative languages that we all used to uh this.

A

This instruction can be seen like uh the following expression in the pseudocode, so we execute dht, put with arguments that are read from execution data and the result is put back written back to execution data.

A

Once we have uh one instruction we want to have more, and one way to achieve that is to execute one instruction after another, in a sequential fashion, so execute the first one. Then the second one in this case first execute ht, get then execute the sqlite put.

A

What it means is that we go to this node read something from dht here, put the result into variable with name value and then uh right, go to the you know to another node with the id storage and uh expect there to be a service called escolite that we instruct to put key and value into. So we write the result from ght to some escalate service. On that note,.

A

Sometimes there is no need for a second instruction to wait for a first one. In that case, we can use parallel. It does what what it says it combines instruction in a parallel fashion, so it doesn't so. The second instruction doesn't wait for the first one. Both are executed simultaneously, or rather concurrently,.

A

Sometimes instruction may fail, for example, if there is no chat service on a client a it will, the first instruction will fail. In that case, we can use xor to instruct air interpreter to execute the second instruction. If the first one fails, it's like a tri-cage, so the first instruction in a tribal block and the second executed only if the sec, the first one, fails.

A

Sometimes we operate not just on scalar data, but also on arrays and vectors of data. In that case, we will need an iteration concept in our language. For this we have fold instruction. It takes an array to iterate on sets and starts iterating on that array and on each iteration. It sets the second. In this case I uh variable to each consequential element of the array.

A

The third argument to fold is an instruction to execute uh with on each iteration.

A

Let's see how it looks in a more real code.

A

So here we went to node with this id executed user list get users there with no arguments and put a result of get users into variable with name address, then we instructed script to start iterating on the res.users.

A

So we can imagine that res has the following structure: it's an object that has a field users of type array inside and we say that, let's iterate on that array called users and on each iteration, let's assign each user to variable. U, then, on each iteration, we go to the peer view, id that's stored inside that user. So this array of users it contains objects that store peer id and name of which user, for example, in a chat or in a fluent path.

A

The editor, the shared editor that we have then on each user, for each user. We send a particle there that instructs that contains this whole script and each user executes the script up until this point, and then it sees that it should call chat service with function, display and argument message on it. So what it means is uh take all the users iterate through them and on each user display the following message: the instruction on the fourth line uh says that says, start next iteration.

A

So it's a pretty flexible construction fault uh expects uh next to appear somewhere uh to proceed with iteration. So next, u triggers next iteration.

A

And the last of all instructions is null, it does nothing. It's usefu, it's useful for prototyping or for code generation, where you need to insert seek in automatic fashion and don't want to care about. This number of instructions is even or odd.

A

So that's about all the instructions that we have uh in our language. um As I said, it's a more like a configuration language rather than programming language. So that's why it's so narrow in its functionality. It allows us to to implement different security guarantees and invariants with that language, so we can guarantee that nothing bad happens.

A

Another concept that we have in the language is built-ins, it's. uh Those are just regular services uh like you have seen before in the examples, but they are provided by each node. So you can expect this to appear on every uh non-client peer in the network.

A

There are four namespaces for built-ins. First one is dht, uh it provides you functions related to kadamlia network. Second, one is sroe, it means service service creation, inspection and, and such dist for distribution functions related to distributing webassembly, modulus and blueprints.

A

I will talk about that later and the final one is a peer namespace that contains some pure local operations.

A

Using built-ins is very straightforward, for example, in the namespace srv there is a function list and we can call it just like any other ordinary service, but instead of service id, we specify the namespace.

A

So there are the following possible.

A

Fun built-in functions in ght namespace. We have just a single function that gives you neighborhood of the key. I will talk about that in detail later in the srv namespace. We have functions for creating the service, listing existing services and getting interface of a service for inspire inspection and exploration purposes.

A

In the op operation namespace we have identity function that just returns an argument back, it's useful for doing nothing but still specifying to which node should execution flow, move.

A

The next namespace is pure namespace. It contains functions like subscribe, that can put subscriber on the node, so it can be retrieved later, for example, you can subscribe on some topic here and then you can list subscribers for a topic yeah and, for example, iterate through them and send message. I will show an example of that later.

A

Another function here is identify peer, identify function that get gets peer info. uh For now there are there's just external addresses and later there will be more information and the last function in pure namespace is timestamp. It just says peer. Give me your current timestamp.

A

There are functions uh related to distribution of vas, modulus modules and blueprints. I think those are pretty self-subscribing, you can add module module to the peer you can list. Existing was modules on the peer. You can get module interface once it's, it is added there and you can add blueprint for a service and list existing blueprints on appear.

A

From using instructions together with services there are, some patterns arise that we that we think are important to understand for each developer, that you who uses uh air the first one uh and the most most essential pattern is accessing peers through a relay uh for, for example, consider a browser peer that sits in mcdonald's on wi-fi and it it's not available from the outside network. It can send requests, it can open a connection with a server, but it can't listen for new connections, so it can be accessed from other nodes voluntarily like a server.

A

It is strictly a client so uh to solve that problem to receive uh messages, particles from peers. uh The client does know about uh the client delegates networking to a node and such node is called a relay.

A

So consider this example of a client being connected to relay and sending a message and basically sending nothing to relay and receiving the answer back here we trigger uh execution of op identity built in that does nothing. It returns an empty list back, so we send such a particle to relay it receives it executes hope, identity does nothing and then it sees that it should return the particle back to initpure id the p, the client that initiated the request. So it does just that it returns particle back to the client and everything works flawlessly.

A

But uh if we try to connect to do the same thing with the other node but relay so not with relay but other node, um we will fail. Let's see, why remember that client is connected directly to relay and it is not available from for any other node, because it's behind the nut or something like that.

A

So what clan does it say sends a particle with the following script: to the relay relay sees the first line. It sees that it should forward that particle to other node does just this just that forwards. The particle to the other node other node sees the next line understands that it should uh send message back to the client tries to do that, but fails why?

A

Because uh other node can't access the client, because client is behind not so to fix this, we must specify the relay that that's involved in a client topology manually, or rather explicitly so, for example, uh everything like before we're sending the following particle to the other node. It sees uh to the relay sorry. It sees this line forwards, particle to other node. Other node then sees this line that says, send message to relay relay receives this particle sees the last line that says, send message back to client and does just that.

A

It sends message back to the client.

A

So this is a pattern that you you better to understand uh in order to be able to communicate with browser based clients.

A

Another essential par pattern is creating a service for creating a service. You will first need webassembly modules that you want to compose. To create a service, uh you need to specify a trivial configuration for them with just module name and memory limit that can be just emitted and there is a service blueprint service blueprint is a recipe for creating a service. uh It contains required. List of required was modules in a specified order of loading. So, for example, if you have a blueprint that says, take sqlite module, then user list module and then fs8 module.

A

It will first sqlite, then user list and then facet module. That's very important, because a user list module can then import functions from a sqlite module and fosse8 module. Can import functions from user list and from scolite module.

A

So in airscript, the creation of a service looks like this. You first upload two modules to some node here you call built-in function on a namespace, dist and function called add module. You specify module, bytes and module config. You do the same for sqlite.

A

After that, you add the blueprint that you need to specify upon sending the particle. That's easy through jssdk, that's easy to do.

A

And finally, almost finally, you go to the same node and say execute this built-in function in namespace. Srv function named create with this blueprint. Id blueprint id is generated on a ad blueprint call so upon executing a survey create you receive output service id that service id. You can then pass to the client mind the relay here. So if client must be connected directly to node for this to work, so this service id, we pass to the client and return it to the js code, for example.

A

So that's pretty easy once you get use of that.

A

And the powerful pattern of sending some sending particles to list of users is that's used across apps that we developed with air, and we fluence is as follows. You hear a little bit more complex example that shows you that you can use uh variables and access their properties through jsonpath.

A

So, for example, here you have a user list variable in execution data. It contains the following fields: peer and service id, and you can use this object to send particles to the peer id specified by this field and call service id specified by this field. This allows you or to be more dry in a sense of your code, so yeah what's happening here is we're going to user list peer, the node that hosts a user list service.

A

We say that please call function, get users on the service id that's assigned to user list service and put the result into variable res.

A

Then we fold through users that are in the variable res and for each user we call chat display you will. We have already seen this pattern in the fold example. This is very similar.

A

So that's a fan out pattern. That's used mainly in multi-user applications.

A

And the final pattern that we have it's a pretty complex one to understand, but very easy to use so on each peer on each node peer. There is a simple key value: storage. The key for that keyword. Storage can be anything. You can imagine that this is some topic you're interested into. So, for example, you want to uh what the what's the problem that's solved by this pattern.

A

Imagine that you want to uh some topics that other users are interested in, for example like a twitter handle, or something like that, so uh people subscribe to the twitter handle through this pattern and later you can ask the network give me all users that are subscribers of that topic and some send each of them a message.

A

So here's how it works uh each. As I said, each node contains a key value. Storage where key can be used, is used as a topic and value is a subscriber. So what's subscriber subscriber is a peer. We have optional service id specified. So, for example, I can say that my service is subscribed on some twitter handle and later when you want to notify all the subscribers of some twitter handle. You can uh also.

A

You will also notify my twitter bot, for example, that's running on that peer under this service id or I can emit service id altogether and specify just peer id saying that the subscriber on that topic is me, uh and I will receive notification that I want so uh how it works under the hood. It uses kadamlya, kadamli topology and the algorithm. That's that underlines cadamilia to replicate subscribers of the topic.

A

Each node has this simple key value storage, but in order to be able to query that key value storage without know knowing the peer id of the node that stores the data, we need to calculate the set of nodes we want to replicate to, and cadently allows to do us just that. Okay, cadembly provides a set of key closed, uh closest nodes to any key. Usually, k is 20., so what it means is that we can say cademilia give me 20 nodes that are closest in the sense of cadamblia distance.

A

To that key that I want, and we will receive the same nodes for that it doesn't matter from what peers. uh From what peer we asked to get those closest notes, that's how this replication works.

A

So after we gathered a set of k, closest of k, closest nodes, we can go through all of them and store in their key value. Storage, store topic subscribers for a topic on them. Later we will. We can query that keyword, storage from the same nodes using the same technique.

A

So if we are to visualize what's happening there when we add the subscriber on the topic first, we say uh cademia give me neighborhood for a key topic, uh put the result into variable, neighbors and iterate through all neighbors and use internal key value. Storage, that's available under the built-in pure subscribe and the store, the subscriber that I have under the key topic and then iterate further.

A

So imagine we have this circle and the circle is a network.

A

We have some topic and we want kadamlia to give us closest notes to that key to that topic.

A

What we what's happening when we say ght, neighborhood topic, cademblia goes layer, goes to the network and finds all the nodes that are closest in a sense of academic distance to that to the hash of the topic, since the hash is always the same, and the academic invariant is that k closest are always the same, except they can change with time, but not much. um We can use that to our. We can use that to replicate the data and then we, in the same fashion, we can uh retrieve the data from the the same nodes.

A

We go to the neighborhood for a topic. Then we iterate and we say peer release subscribers to get subscribers from the keyword, storage and we continue with iteration. We gather results and send them through relay back to the client.

A

The important thing here is that, since uh the key value storage is replicated uh the results that we gather here in subscribers, they must be deduplicated before they can be used.

A

So after they duplicate it, we can use the spears in a fanout pattern or any way that we want and that's about it. Thank you.