Rust Programming Language RustFest Barcelona 2019, 27 Nov 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: RustFest Barcelona - David Bach & Malte Sandstede: Right inside the database

Description

Modern, stateful web applications are increasingly dependent on high-frequency updates from multiple database systems. Each application maintains its own dynamic view of shared and private data. What if querying and synchronizing this state consistently was as simple as writing a Datalog query?

Using a reactive query engine powered by differential computation built in Rust, we explore this question. We present a novel architecture that selectively and incrementally replicates state between server and application and thus allows you to develop as if you were sitting right inside of the database.

https://barcelona.rustfest.eu/sessions/right-inside-the-database

A

A

I want you to imagine a world where obtaining backend data is as simple as writing. A query,.

A

Where your apps are always up-to-date,, where offline-first is only a caching decision about.

A

What kind of your backend you replicate in your frontend, and where object relational.

A

Matters or manual graph, cure, resolvers or mappers in general, don't even exist.

A

We want you to imagine a world where you are right inside the database, and with that,.

A

Welcome. this is david., my name is malte.. We work at our own small software consultancy where we consult people on the stuff. We are.

A

Talking about today and a lot of the stuff originated from the eth zurich systems group.

A

And also frank mcsherry,, who is the creator of some of the systems we are going to introduce.

A

A

So what better way to start a rust talk than to show some javascript,? So, of course, if we.

A

Start at the frontend and want to make this utopia come true of writing them in a declarative.

A

Manner,, we could just imagine what this should look. Like.

A

At the bottom half of the slide you can see the frontend function,, so it's just a component,.

A

And we just describe what we want to display.

A

some senders. and at the top we see a declarative, query.

A

This is written in datalog, but it is the names of the senders of the messages that the current.

A

User has received,, so you can see on the message id slide, there, message id entity. There is.

A

The recipient and then it matches all the sender ids and gets the names from them,. So.

A

You could also write sql if you are more inclined to do so.

A

This gives us a very declarative manner of specifying our data needs in the frontend,.

A

And since we register this query,, as you can see down below, what we want to say with that.

A

Is that you also don't have to worry about at what time this data arrives in our system.

A

We always think that our data just will flow into the system and changes will propagate.

A

Through the system, as soon as they arrive., we in the frontend don't have to care about this at all.

A

So declarative and reactive frontends are something you can kind of have today. If you.

A

Look at libraries such as react or think about component queries,. But now, if we look at the.

A

Whole full stack side of things, things look very differently, so we are not really reactive.

A

Anymore, since we mostly resort to polling.

A

We ask time and again our backend:. Is there something new,? Is there something new, without.

A

Knowing if there is something new., there is also a translation step between the application, server and the source of truth.

A

Since, for example,, if we use an object, relational matter or resolvers, we always have to speak.

A

To our source of truth and translate the queries we formulate in the frontend so that our applications,.

A

We can actually understand them and the backend can understand. Them.

A

What we would rather like to see is something that looks like this and is slightly stolen.

A

From a great blog, post,, "the source of truth of tomorrow",, and the first we can use is.

A

By using a declarative query,, what type should be accessible,, so this is in the server.

A

We also want to formulate what of the data is currently relevant.

A

This is the query you just saw in the frontend javascript code thing where we specified what.

A

Data we are actually interested in, and, lastly, we want to push this data into the frontend.

A

Instead of asking for it and pulling it, to make things reactive.

A

So if we want to re-evaluate how we build this full stack architecture,, the first thing.

A

We have to look for is a push-based programming model, to avoid, or to become reactive.

A

If we look somewhere, we might stumble upon the dataflow programming model, and this is.

A

Really a nice way to model these kind of things, because in the dataflow programming model,.

A

You do it as graphs.

A

The nodes are operators and they do some work,. For example, if you have a map operator, it.

A

Takes some data and it will map over it and produce a result based on it.

A

The edges specify from which operator to which operator data can flow.

A

For example,, if you have a join operator, it probably has to input edges and produces a.

A

Single output, edge.

A

This offers some nice characteristics.

A

For example, there is no central co-ordination, needed.

A

Data co-ordinates, the execution of the data flow.

A

Secondly,- and this plays nicely with rust or if you want to implement this in rust,.

A

We don't really need to fight the borrow checker, since every operator owns its own part of.

A

The data, so there are never two separate entities owning the same part of data and fighting.

A

Over it. lastly,, it's quite easy to distribute, since we can just clone operators and chart the.

A

Data we input into our system and then just feed one operator,, one half of the data and.

A

Another one, the other, and this way parallel with ease.

A

And an implementation in rust, based on this dataflow model, is called timely data flow.

A

And it offers those that can even be cyclic, and this can be something like this, so this.

A

Is a rust program using timely dataflow and the first thing is you always write it from.

A

The perspective of a single worker, so, while constructing my dataflow, I don't have to worry.

A

About distribution or parallelisation., I can do all of that stuff later on.

A

Then comes the first stage of dataflow programming,, the creation of the actual dataflow. So this.

A

Is how we construct the dataflow from small operators that are chained together and to.

A

The right here you can see a visualisation of that,, so we are ingesting some data, exchanging.

A

It to possibly other workers and then inspecting it.

A

Then,, as the second stage of the dataflow programming, we are actually running it.

A

So now we are feeding after we have created the structure,. We are now feeding input into.

A

The dataflow to compute something.

A

So now that we know what dataflows are all about,, we can re-examine our introductory.

A

Problem of thinking about how we can build a full stack architecture using that, and.

A

If we squint a little, it could look something like this,, where we have the ingesting operators.

A

In our dataflow being the source of truth,, then we have the application server level.

A

Which is the actual dataflow computation and which will narrow our data down to relevant.

A

Data and apply access policies and, lastly, the exit nodes are our frontend, and, of course,.

A

We won't model our frontend as a dataflow operator,, so this is more of a conceptual.

A

Dataflow sync, but we can actually model all of the backend stuff as a dataflow, as a real.

A

Dataflow, or in this case a timely dataflow.

A

So what does this look? Like?.

A

The first thing we have to worry about is how we now model our data, because we are not.

A

Anymore in a static mindset where we have just a database sitting somewhere and we query.

A

It, but we are talking about dataflows and, as the name suggests, data is streamed through.

A

This system., what we propose is using a fully normalised attribute-oriented data model,, so something.

A

Like the six normal form and in that data model, the fundamental unit is the fact, a.

A

Triple of entity, attribute and value.

A

for example,, if you have an entity one, it might have an attribute name and the value.

A

Of that attribute is peter..

A

This allows us to freely compose very small units of data into higher level concepts,.

A

So, for example, if we want to model a person and her residence,, she might have a name,.

A

She might have an age and a residence, and the residence then again links to the entity.

A

Id of the resident facts we define below and this composes,, as is quite straightforward,.

A

To see, to a higher level concept, such as a person or a resident.

A

If we model this in a database setting, we would have a separate table,, since we are.

A

Completely normalised, for every attribute.

A

so, for example,, the pers/name has an entity column and a value column.

A

If we want to transfer this to the dataflow system, where we think about streams, we can.

A

Think about entity and value streams. So we introduce the notion of time because in a.

A

Streaming world we have to somehow define at what point this data arrived,, so alice,.

A

For example,, the name alice was introduced at t0 and we also want to make multiplicities.

A

Or differences explicit, because data can change in this stream.

A

Alice's name as of t0 is alice but later on, she changed her name to bob,. So in this case.

A

By using these multiplicities and additions and retractions,, we can say at t2 we retract.

A

The fact alice, or the first name is alice for entity 1, and we add the fact bob for.

A

A

this also allows us to become very reactive, since now we can observe or make explicit.

A

Every single data change in our system.

A

If we now want to arrive at this higher level concept of a whole person,, what we are going.

A

To do is we consolidate the streams,, that is, we add, together all these differences and.

A

Subtract, the ones that are negative so that we obtain a consistent view. As of a common.

A

T*, so, for example, if you have a common pers/name, stream, age, which will probably update more.

A

Frequently and residents,, we just sum up all these facts to obtain the person table as.

A

Of some consistent time, t*.

A

Cool. now we have a data model that can express everything we want from our full stack application.

A

In a dataflow setting, but what is still missing is making things declarative and formulating.

A

Queries on top of them,, so the first thing we want to do is register a query, again. So.

A

That every time something changes updates are pushed,, for example, via a websocket connection,.

A

To our frontend,- and we also want to make sure that we can even formulate very complex.

A

Queries such as access policies using that model.

A

So here, this is a reachability query,. So for whatever reason we might be interested.

A

In finding out all of alice's friends and friends of friends, and again, it's datalog.

A

But basically, what it does is it takes the first clause and matches it,. So this would.

A

Be direct friends, or it matches the second clause, which is a direct friend of one of.

A

The friends or friends of friends we have already discovered.

A

Even if you don't read datalog all day,, what you can see here just from syntax, is that.

A

We are still using these facts. We defined before, even now that we are in our query.

A

World., what is not so straightforward is that to match these clauses, we have to introduce additional.

A

Operators, so, for example, here matching on the hop is nothing more than a relational.

A

Join,, so we have to have some kind of means to express this in a dataflow setting and.

A

Iteratively calling the reachability query means we need some kind of iterate clause.

A

Especially if you know stream, processing and dataflows, this is often very hard to do and.

A

Fortunately, again, there is already a system, again written in rust, that solves this.

A

It's called differential dataflow and it provides complex, operators.

A

So to the left, you can see a differential dataflow having these iterate operators and.

A

This join rate, and it looks like any other rust programme, and the cool thing is that.

A

Also, it can incrementalise these operators,. So if we at some later point in time, change.

A

A fact, we don't have to recompute all of the friends of friends from scratch,, but can.

A

Just do work in the order of change.

A

So now that we've covered how we can actually model these complex query, matching clauses.

A

In our dataflow world,, what is still missing is kind of obvious:. How do we get from the.

A

Query representation to the rust representation?.

A

This is what david is going to talk: about., david:, okay,, cool.,.

B

All right, yes.

B

so malte just sort of explained the way we end up with these timing programmes written.

B

In rust- and we know how rust works, we write this programme,, we compile it and we run it,.

B

And the idea of these dataflow programmes that you start to run them and they run indefinitely.

B

Because we never know if we are done with the input., there could be potentially unbounded new input coming in and we always want results given.

B

These new inputs, but as a client talking to some sort of database, we always have this.

B

Interaction role that a client comes at some arbitrary point in time, asks the database.

B

To compute some query,, the database computes the query,, the client takes the result and.

B

Minds its own business., so this is what this match between aesthetically compiled rust programme on the right-hand.

B

Side doing this dataflow business and a dynamic query from client and what we build is something.

B

Called 3df,, which stands for declarative dataflow, and what you can see here is an excerpt of.

B

The plan extraction., we have this rust data structure.

B

That is the idea that it provides the server with a plan of how to implement these operators.

B

So the idea is that we enumerate all possible operations that clients could ask us for,.

B

For example, the join operation, and when we give the server or someone implementing this.

B

Library, this plan struct, he will just walk through it and implement all these plans.

B

What he will do when he implements this plan, is we basically take all these operators? We.

B

Saw earlier from malte,, we stick them together and, for example, a website with a stream of.

B

The resulting data.

B

Perhaps time for a recap.

B

The first thing was basically having a run time to run these in.

B

Then we talked: how do we model the data such that it makes sense in a streaming setting,.

B

Having all these normalised streams coming into our database or into our system.

B

I just told you how we can make this all dynamic by using this layer that, in a running rust,.

B

Program constructs these dataflows and hands us back the resulting data or the results.

B

Using this declarative layer, we can actually now query this system, and the system will.

B

Produce these dataflows and keeps us up-to-date with all the data needs we might have as a.

B

Client. but of course,, we need to now think of how this now fits into the frontend story,. So.

B

We are going to do this now.

B

So what we are supposed to do here with this talk is giving you the idea that a client.

B

Is actually also just a peer through the database, and in another sense the client? Basically.

B

Has the illusion that he sits just next to the database, or inside the database,, so he.

B

Has all the query power that a normal relational database has and the whole rest of this data.

B

Is extracted away.

B

when we start thinking of this,, just as a quick reminder,, maybe relational databases.

B

Are also created at modelling states for the frontend themselves,. So if you have some frontend.

B

Application that some web page, different things,, you can think of this state of this.

B

Web page as being the database and if different parts need different data, it will just query.

B

The database and ask for some data.

B

if it wants to do some transitions,. For example, someone puts in data into the text fields,.

B

It will transact this into the database, so there is already a good fit as using a relational.

B

Database as a means of modelling frontend state.

B

When we talk about databases, usually we say we have a query and the query will be evaluated.

B

And we get a result set,, but the whole story we are telling here is: we want to replicate.

B

The query for the client, so the client feels he is right inside the database,. So what we.

B

Want is this function f that takes the database and gives us back this selective database.

B

That only contains the data that is accessible,, relevant and important for the application.

B

B

So how do we go from queries to these replication queries,? What is this function f that makes.

B

It possible for the database to be replicated in this sense?.

B

The first thing to notice is that,: if we use the same data model, it will make it easier.

B

So we will use this same thing. That malte told us earlier, that our application database.

B

Is also this database of facts, making up our local database, and there is one implementation.

B

In javascript called data script that will do that, gives us a datalog interface over.

B

This local application database.

B

our job, is now. How do we populate this database with all the facts that this client is interested.

B

In seeing?, the idea is to just query for all the attributes that the client is interested in.

B

These attributes are in itself, binary relations,, so that when we query it, we get back some.

B

Triplets and attribute value that the client can put into its local database.

B

To make this a bit clearer, I have come up with a simple example.

B

On the right-hand side, again we see this query struct or the query in datalog, where we have.

B

Some message, sender, query: we ask, and the query body,, starting with the find and having.

B

All these clauses, and if we look at what the find says, it says, we want the message.

B

Id which corresponds to the entity we are wanting to find.

B

We have a value which is value of the thing, and, of course we give it a name, and this is.

B

Sort of the fact,, so the right-hand side, query, message: sender, query,. We are resolving a stream.

B

Of resolved entity attributes that we can put into our local database,, thus replicating.

B

It at our local database.

B

The cool thing is, we don't need to necessarily only replicate data;. We can potentially use.

B

The full power of our datalog or sql query language to do some aggregations, as malte.

B

Showed us earlier.

B

we have to describe all the data needs that we would like to have replicated in our local.

B

B

Some of these things may seem a bit effortful because we need to take all these different.

B

Attributes maybe, corresponding, for example, to the same entity,. But there is something.

B

Called a pull syntax where we can see, I want all the attributes corresponding to some one.

B

Entity., it just makes a few things: easier. all right,, now, let's step back again and have a look at what we've covered so far.

B

From the beginning, we started with this system that allows us to write these dataflow programs,.

B

Distribute dataflow programmes and gives us this push access, model.

B

We showed you 3df, which allows us to write these queries and the queries are our means.

B

Of describing the data interest, we want and also describes all the access policy that.

B

Is potentially necessary to enforce privacy.

B

Then now we covered, when these streams come in, or these entity attribute value streams.

B

Come in, we put them into our local database and the rest of the system of the frontend.

B

Can talk to this replicated database right now and render its messages, for example,.

B

From this local database, and does not have to care about all the rest of the stack being.

B

Kept in sync, so to say.

B

The cool thing about this is that all these errors we were showing earlier in all the.

B

Thing,, it's just one declarative query that describes this. One attribute message: sender.

B

And the rest is basically taken care of,, so one declarative query pulled them all,. So.

B

B

We are nearing the end of the talk, so we thought we would include a bit of a what sort of.

B

Open things are there currently occurring?? The first is we are using this data script thing for our frontend application,, but 3df.

B

Is written in rust and we can use literally the same code as is used in the big end. Machines.

B

And this is something we have been playing in a bit and there is a point if you are interested.

B

In this. of course, we love datalog, but other people might not,. So this plan struct, I showed earlier.

B

Is an easy target for sql or graphql, other languages,, so we could simply write a graphql.

B

Path, which then gives it to our database or to the system, and it will work just fine.

B

All right,, let's see what we did here.

B

So we started off with the way it used to be, arrows, pointing up, because people have.

B

To pull down all the data, they are interested, in,, transform it and potentially in different.

B

Languages, and the first thing we did was introduced a dataflow and push-based model.

B

Where all these arrows are push-based, but we still query languages up on the top to.

B

Talk about in database languages.

B

I hope what we've sort of conveyed now is that we pushed relational queries. All the.

B

Way down to the frontend itself, so it's all written as a single datalog query and the.

B

Frontend is also in a relational query, so in a sense we put the frontend right inside.

B

B

C

Thank you for that great talk.

C

Are there any questions in the room?? Please raise your hand.

C

D

D

So an association I immediately have, when I see attribute value, is rdf where you have.

D

Subject, predicate, object, and I was wondering the semantic web, which is based also on this.

D

Would benefit tremendously from having a client-side database.

D

Projects are already working on that, but do you think this might be an alternative to.

D

Work, for example, in the [inaudible]?.

B

David:, I would say, if you are interested in these sort of incrementalising, these updates,.

B

So if your database is really interested in getting push-based access to all the rest,.

B

Of the data that is potentially needed,, then of course, the stuff I showed you is simply.

B

Super early., so if there are some funny cursor movements and colours.

B

I wouldn't at all suggest you use this right now., but definitely, in the future, yes,. I would think so, yes.

C

Next question., thank you very much..

D

Great talk., I am totally on board with the idea of moving relational queries down to the client,. However,.

E

In many contexts, where you have this version,, the clients are it's running on someone.

E

Else's machine and now that it's providing them with a lot of additional power, they might.

E

Decide to run a query that is just going to make your backend slow down to a crawl.

E

Have you thought about how to deal with this?.

B

David: yes,, we did., we had a slide, but we thought maybe that would be too much.

B

totally fair.. Usually you wouldn't trust any client to just execute any query, of course.

B

There are two things to this: story, basically.

B

One is worst case,, which is like a research topic where you can do all these join operations.

B

In that, even if clients ask for weird join queries, we won't throw up the whole server.

B

Because we are smart in these worst case, cases.

B

If we restrict the aggregations to only count as long as we don't allow user-defined aggregations,.

B

It couldn't blow up too much in a sense..

B

Last part is basically access policy,, so the access policy must be enforced not by the.

B

Client itself, but of course, by our system, but yes, great question.,.

C

The last question?.

C

Anyone? thank you,, great talk.

F

Some time ago I wrote down an application with kafka streams for interactive queries.

F

I didn't do it properly as they suggested, and I was moving all the state to all the.

F

Nodes in the front, which gave us a lot of problems,, especially when a node goes down.

F

And needs to recover.. There is a lot of overhead of gigabytes of data being moved.

F

How do you deal with boot, strapping and fault re-runs?.

F

Malte: yes, so also a very good question and I think there are multiple projects that are.

A

Trying to replicate basically the whole database on the frontend side.

A

One thing to tackle this is, of course,, because we can write the specific queries we already.

A

Narrow down the data to the data that is actually relevant to the query.

A

The other thing is, for example,, if you want to recover from a node breaking down,, that.

A

You don't have to send all of this data over as data,, so not only the differences, but that.

A

You can, of course, think about creating snapshots, similar to what we showed about the table.

A

As of t*, so that you can send over an entity just as a single serialised entity and not.

A

Have to replay all the work you've done, before.

A

A

but of course it's a real concern,, especially in the frontend part.

F

Yes. thank you.

C

Thanks. give a great hand to david and malte.