Rust Programming Language Rust and Tell Berlin, 16 Dec 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Crash Course with Bastion— Mahmut Bulut

Description

From the Berlin Rust's November "Rust and Tell" Meetup (https://www.meetup.com/Rust-Berlin/)

Mahmut talks about current asynchronous execution patterns, fault-tolerance, and error handling in Rust and how they interconnected in modern systems. He explains how Bastion(https://bastion.rs) runtime's different approaches help to solve problems arisen from current implementations.

Mahmut Bulut: https://twitter.com/vertexclique

Hosted at Prisma: https://www.prisma.io/

🎥 Filmed by iStream: http://www.istream.pl/

A

So normally this talk is called crash course with bastion, but I want to call it. Let it crash course with Bastion and you might have known this. What is this wording coming from and we are going to talk about this voting and ongoing slides? So, let's talk about what I am Who I am, let me introduce myself so I'm doing data processing by day, I'm, writing rust.

A

Since 0.4 and I wrote a couple of the compiler stuff inside and currently I have a couple of open to requests to the miry, and we are unleashing the miry right now and some other rough projects I'm working on I'm member of the community, and that is all I can say, I think, and let's start, and by the way with this, you can see that my drawings are really not good and let's talk about how people communicate so we communicate or synchronously.

A

So what we are doing is basically exchanging messages and saying things to each other, and then there is no actually something that is blocking us right. I mean this is how our life and communication works and I think I believe that the software system should be like the how the nature in nature is.

A

So if you have it, if you don't see any blocking in the nature and everything is basically kind of working like a message passing and there's a vein blowing, there's everything is working like that and when it comes to come communication of shared memory, I think it is not nature, and so, let's think about a case that is like we don't write a paper on the desk and then we just don't read from there right to communicate with each other, with you example and I'm, telling some words you're hearing and you can reply back, and this is how it is working and like every single time.

A

Faders and Falls are natural, so we can, we can get sick and then basically, we need to recover from that right to continue our lives. So this is how people are communicating and how the nature works.

A

But let's talk about the rust right now. How rust does this error handling? We just I think, and it's purely subjective- we don't do much when we say error handling. We have some error typing, but not purely error handling.

A

We write a lot of code to determine the control flow in case of errors, and basically this is like, if this error kind happens, do that and this if this one happens, do yet another thing like a B and what you see in here is basically that this is like from trade like how other errors, trucks actually assembled and based on the error kinds, but we are defining. What are we going to do and we have some, as you can see, in the yellow parts that we don't have?

A

Actually things defined, or maybe we are ignoring these or maybe we don't want to handle these, and this is just a one use case and inside you can see that you are basically propagating everything one by one and your handling, and you need to write these things to actually recover from the her errors. Okay, so this is basically in this example.

A

This is basically a response error coming from the radius client and it's basically propagated to your error type with, let's say with the error chain create- and this is basically how you are doing the error handling when we are doing this error handling. Let's say we are sacrificing the availability and we we also sacrifice the synchrony and we don't take take action on errors.

A

We always use, try or question mark you know, and these crates does long arrow chains and beaches also creating long recovery codes. In my opinion, and it's hard to maintain it when the project gets bigger, it's hard to maintain I would say so.

A

This is the things that we need to fix actually this year, so you can, if you are actually inside the res community and if you are watching which kind of things are happening, probably you so the talks about error handling so but today, I'm going to talk about the nut error, handling that we are going to talk about. It's going to be more like how Ireland dance does so, it's kind of different than the rust rust zone, error handling code.

A

So, let's enter to the dependable systems, so obviously we need to talk about the dependable systems, because if you are familiar with the electronics and stuff and embedded systems, it depends on the systems. Probably you heard, and we need to know first how errors and faults and failures are propagating, so we need to talk about the dependable systems, and this is like how it is inside the embedded system.

A

So we should also know about a dependable systems to reflect this to normal services that we are writing like for serving to the customers and stuff like that. So let's come to the point. First error handling is not fault recovery, so we have this, which kind of problem that everybody is actually thinking about, and actually this is completely wrong and it's actually. The first thing is actually told in the classes and also in the books of the dependable systems. This is the thing and we need to know about the failures.

A

Then, if error handling is not a fault recovery, we need to know how folds and how? What is error and what is failure. So this is how fault error failure is actually propagated inside the all software systems, normally what we, the software engineering, part an upper layer of the abstraction. You don't see it like this. This is coming from the embedded systems.

A

Let's take a look, so in the lower level of the system, you'll see a fault, there's a fault happening, it's dormant, it's staying in there and then it transforms in transforms, in theory, an error when you detect an error. The first thing that what error handling is that gathering information about error? Okay, so that is, error, handling, detecting and gathering information. That is the older part of the error. Handling rest of the part, is that what you're doing is a fault recovery.

A

So then you have a failure and then failure propagates to the upper layer as a fault and then fall through it. Does the recursive thing unto the layers are actually at the surface of the program? Okay, when you are at the service of the program that failure the last failure that you see in that is basically become an hazard and then, but you are what happens- is that your program crashes?

A

So what is basically all these orchestration tools are there just because to prevent you from the hazards, so the old Azar's are basically the happening from this kind of chaining. So so we know right now. What what is the fault was the failure and the how is propagated. So what we need to do in the rust environment and maybe in the most of our software engineering, but so we we have. We need to separate the business logic and this error recovery when I say error. Recovery is false recovery, my my words.

A

So if you need to clearly separate to these two, the things, unfortunately my subjective opinion, we don't do this inside the rust. So what we have is that every single time you have some kind of result and you need to actually wrap these results, error kind and errors inside the results and the results are up inside the results again and then the armored up in the correct place or expect something right.

A

This is the problematic part of it and it's going to be programming problematic part of it, because you are writing these things inside of the I/o operations or the things that has the side effects and these side effects is are actually having some problems.

A

You are writing this around the code that your business logic is where your business logic is. So we need to first clearly separate the business logic and error recovery. This is the first part that we need to do. Second, yes, so when everybody comes here, this talk and these meet up and stuff like that, probably and every single time we are talking about a synchrony, there is a synchrony. Nowadays, the systems are completely concurrent and they should be concurrent right.

A

So this is the part that should be also separate in a mind that how the design it the design of them, so design of the synchrony should be in parallel to business logic, but also be logical and be separate, and now, let's separate the error, handling and fault recovery here, error handling, like I've, said gathering information and are creating this information to the place. That is going to do the fault recovery. So we have this four stages right so right now.

A

That is what we have done for last seven months to create this project, so the bastion is basically what it is to handle all these things for you to help you to actually create a synchronous ever had getting information about the errors, kind of a runtime and separate your business logic from that and the fault recovery is built in inside. So this is what of Bastion is so right now. This is what I can say about the bastion and first thing that I really want to talk about.

A

Is that it's not an actor framework everybody's asking you on the reddit: it's not an actor framework, no, no! Damn No and basically what it is that how I can say that you heard airlin, probably along, is has some kind of lightweight processes right, so we have replicating inside the back. Bastion is the lightweight processes and then, on top of it, we built this executor.

A

So the executor is basically taking this lightweight processes and applying their lifecycles at that level, and the upper layers I am going to show you about the callbacks and how these kind of helpful things in the bastion will help you. So everything is good right right, I mean you want to have it all. Yes, everybody wants to have it all I, think and I'm saying use Bastion, so how Bastian recovers errors through the supervisors, so I, don't know how many people heard the supervisors. Did you hear supervisors? Well, what are supervisors cool to people?

A

Okay, so, but no problem, no problem I'm going to talk about this, so supervisors are the things that that are actually income slating, your actual.

A

How can I managing that actual code that you are writing with the thing with rules that you are defining? So let's say you have a new heavy web server. Let's say tons of web servers opened up to ten ports and you have one supervisor that is actually saying that restart one by one.

A

So if one of the server's failed in that case, that supervisor is the managing manager of these worker processes and what it does is that yeah I'm going to restart that one server that isn't important, something and lead let's say, and then it just restarts that burglar process. This is, for example, one for run supervision strategy and there are some other supervision strategies that will help you that you can completely restart your application.

A

While your application is running when an error occurred and that causes your application will continue working, there will be no problems, so this very basic syntax of half the bastion is, and, as you can see, that you just need to initialize give a supervisor and you don't need to give a supervisor to there's a built-in supervisor that will do all the things for you to you. If you don't know, you can learn what I mean. What is the supervisors? What are the things called children and what are you? What are they used for?

A

We have written very extensive documentation about this, and this part Shapira is prizes and the restarting strategies are helping you to contain the business logic. Also, do the fault, a fault recovery for you, so one more thing that will help you do in case of faults. For example, this is a simple function that is called the SP, but it's basically creating a supervisor. So inside the supervisor you can have callbacks.

A

So through the callbacks, you can define the life cycles of the lightweight processes so who used akka qu, you know in akka, there are trace, start, /, stop and things like post, op, post start and old things like the octo life cycle right. So it is in here exactly the same. These callbacks are the life cycles, but in for all the individual children, but in addition to the archive order, any actor frameworks architecture. These life cycles are basically also be replicated across a span of children.

A

So we have a children group that can go up to thousand processes and you can still have these callbacks inside all these processes, so you can gracefully shut down after the stop and gracefully connect back to do some other database and stuff like that back again and you can generate children, you can give the strategy again and you can give the callbacks and you can create a supervision tree under it. So, basically, you are creating a tree of things that can be managed themselves, which has built-in fault, tolerance and error handling boom.

A

That is what I can say so I mean some some esteban wrote. This I will not do your I'm grabbing the production. Oh no, no I mean and then pascai wrote something like yeah I added restart. Always yes, systemd, oh come on, don't mean, don't need to do that. We are saying that dance on production with us, so just restart I mean it is going to restart anyway. The only the process that you need. It is going to be restarted. Your database.

A

If your database connection didn't fail, it is not going to restart the database connection. You have inside the synchronous code, so if there is no problem in here, so you can just use this and basically it so what Bastian brings is this super version hierarchy that I told you and this complete system of it, and this natural, a synchronous, communication and I will come to this natural part. That will also might hurt the minds like because it's like subjective opinion again so, and it comes with the resiliency.

A

So this this is like the part that is the faltering spring, so the it should be resilient and they might your workers or the things that is their working in the background, should can fail, but it should be. It should recover that. So this is the one of the things that I think. That is important, because everything is failing and we don't need to write some extra things if we can handle things by ourselves.

A

Okay and the thing that I show you with this cross of things, that a synchronous called business logic, the separation of concerns. You already know that, so we need to separate these concerns, so this bastion allows you to separate all these concerns and it eliminates the need of the wrapping your result type. So everybody is right now.

A

This error handling talks are happening in the Ross community and then everybody is talking about wrapping the result types and we need to eliminate this, and this actually animal is illuminators in a sense because you can just unwrap in that place if you have enough cold enough data to unwrap the information to unwrap. So you don't need to do this. You don't need to write the chains and stuff like that.

A

If you don't want to alter the error that is coming up to you and you don't need to carry these tiny Glaros this, you know if you, if you're in the rascal media quite at the boil. This is a pain in the PETA moment of everything eliminate the need of carrying these. So it's like dynamic dispatch and all these things. So if there are errors that can be shared between the threads and sand between among the threads and stuff like that, so this is the problematic part that we are having right now and I.

A

Think the this also allows us to fix this thing and, as you can see, this is completely different approach to the error handling because I we didn't actually use anything like the error chain. Kray-Kray, we don't use anything that is actually doing these kind of things you just write expect and if it fails the the information that you are expecting is it going to be inside so, and we are working right now to improve the information gathering part and- and we also think that the writing resilient systems shouldn't be hard.

A

So that is the thing and let's talk a little bit about the supervised supervision. Here are key, so we have a system at the top of everything. This is a process that is actually running and, let's, let's call it process and it's not a process in the STD or something like that. It's just a process.

A

It's like the replication of status quo routines like that. We have this in our second weight and we have two separate things inside the system. At the supervision hierarchy left side is a user. So what you are creating. What using the Bastion is that the left side is purple area that your is you and right side is like when you include the library or something like the system is going to do something about it.

A

So, in the next stages of the development we are going to write something very close to the ln2 observer that you can observe the processes, so the observer will be placed under the system. So it's basically the system's that user shouldn't see and also makes things continue to operate, is falling under the system and all the user code is falling under the left-hand side. So this is how it is and we have order of signalization.

A

So every signal that is randy's a signal like the shot, no messages and broadcasts, so the batching also broadcast to the whole tree. So if you give the three node, it is going to broadcast all the messages to all the children under that children, all the supervisors, all the children, supervisors and stuff like that, under that you can have a full communication, but there there's the order of shutdown and broadcast goes over.

A

There goes over the system, so actually system is kind of the manager of everything and then basically does most of the lifecycle stuff and you care you are writing the callbacks and stuff like that. But life cycle is managed by the system you are. We are just parsing these things, using these lifecycle methods as a dynamic functions, and then we are just executing them in a correct place. The callbacks are how projects are executed, is exactly like this.

A

So let's come to the unnatural, a synchrony, the natural and unnatural synchrony difference. So we have a problem in the rust. We shouldn't need the same return type for all features. For example, if you are going to, we call it race and I think that it is wrong, but it's not racing in a sense. It's just come yeah, it is racing, but it is like working in a different way.

A

I would say and we shouldn't need the same intern time for all the features, and we shouldn't need to fuse this at all to actually say that the signal back to the completion. So we are all day signaling back to the completion and hey, are you done? Yes, I am done because I am none. This is not how things should be working with fuse and it shouldn't be like that and to be honest streams. Api should have handed out elements and we should have asked for the elements, so we are pushing down the demand.

A

So this is how we are doing like pushing down the demand when we initialize something in rust and basically in streams and the futures can nowadays we are pushing down the demand from up to down, but it's how how it shouldn't be, and and obviously this makes the back pressure. If you know the burden is not natural, so it is like very unnatural, and there are problems like this. That is actually from my point makes life a little bit harder.

A

So this is just a stale copy of the futures, documentation and I think that this is a kind of wrong and it shouldn't be like that because in most of the other languages not like that, so we need to learn about a lot of things from the other languages. Maybe- and let's come to the naturalist, synchrony yeah, this girl call it so much indented. My friend told me that please I mean that this thing and I'm too lazy to do that. Sorry and but I need to show a couple of things here.

A

So this is just a worker instantiation. We can put everything inside for functions and you don't need to do these things inside and it can be flat, not nested like this, and it's basically, we create a redundancy. So I said that the we replicate the children and then we create a redundancy.

A

So what you are passing inside the closure is going to be redundant and it will all the rules of the work submitted. Workers are going to be replicated. So what does that mean? So it's like, if you have some callbacks for and if you execute a bit exact function. If you define a callback inside and I'm sorry with it callbacks, you define a callback since I'd had all the redundant children or children groups are going to have the same thing same kind of callbacks.

A

So basically we have here. We are instantiating 10 workers that can that they can take TCP streams from a simple TCP server compress down with the snappy are synchronously. There is no for loop inside. There is nothing inside. This loop is just for the message polling from the mailbox. This message back a message macro is basically pulling the message from them a mailbox and in the outer systems.

A

You might have heard that there's a mailbox, this kind of a mailbox and we are going to I, think have a different macro that will help to put a loop inside of in it and then basically have all these TCP streams compressed and right back are synchronously, and that is all- and if you don't know the message type, if you don't expect something you just right, yeah I, don't expect something or I want to pee fail immediately in here or just write unreachable, so it's like and then then there's a panic occurs in any place.

A

Do you see my arm? Reps, I'm, too lazy. So for every single day is a person in this room. This is chipper boot, I, just right, arm, wrap I, don't care! This is going to restart. This is going to restart. So when you unwrap and there's an error over there, it's going to restart it's not going to be like oh yeah, you know, I crashed, there's a panic over that. No it's going to restart for every single day.

A

If it's, it can't compress the thing if it can't write back to the port is going to unwrap that's how it is so. This is how the nature of the synchronous should be, but the current implementations, I will say one more thing to visualize how things are working stream porn, X direction of demand is upside down, then stream app. Then we mapped this thing then returned and is exhausted. So why should I return? None when it is exhausted, should I, say I, shouldn't I, say something like yeah I am done and just cancel the upstream.

A

This is how it should be right. These are the nature is like, if there's a stream of things, that is a stream of water that is coming down from the mountains. Let's say: if we cut this water from there isn't going to come late over there. So this how it should be come later with the bottom side of the mountain, for example. So this is how it should be. This is a problem in here that we don't ask for the demand we just put. It demand actually drives us in these implementations or outside right.

A

Now, in the rest, so what we are suggesting and what we are doing in the bastion and what we are currently working on is this. We have source sink. That might remind you the reaction streams, exactly that is reactive streams and the sink is going to demand the line back.

A

So a sink is going to ask for 10 elements if there are 10 element capability to process and then transform is going to transform these and as if there are things inside and then ask back if there are no nothing to transform to the map and the map back to the source, and basically you don't need to continuously pour from the source and/or the push any single push is going to make. You make your system harder to cope with the in memory problems.

A

So this is how the stream processing and the whole that future person should work, and you can see these kind of example inside the bastion the example, we're example is exactly like that and there's also parallel computation example. This is also doing the same thing. Basically, and these are very how can I say that very basic? You can do much more things like many more things that I mean meanwhile, I'm. Writing the examples I'm. Also too lazy I was a yeah. Let's write something!

A

That's actually quite a bit easy and I wrote that, like that, but you can do some magical things with this improving over the existing examples. We have also showcase a repository that we can talk about. We will talk about so when it comes to the interoperability, so this is a runtime, so it all. These are synchronous. Operations are basically running on executors, like icing a CD or the talkee itself. Ok, so how what we are? Basically what we can do, what we are doing in the Showcase reporter?

A

There is only one example what we can add more, so you can see a usage of Bastion with a sink a CD. So, for example in here we are writing to a log file in a very simple way that every single request- and we are just unloading- all the I/o operations without a single city because we are not made so the bastion is not made for the I/o operations. I didn't write a blocking pool or something like that and I actually step on is actually very good at that.

A

He just wrote it and everybody is using it. What should I, why should I write it? I just want to need to write the management part. So this is the management part on top of the thing that is actually doing the I/o operation, and then we are running side by side, but I think I see these executors and Bastion. They are running perfect. There is no problem and all basic examples.

A

Examples inside are like I, said out there all basic, but you can see these interoperation examples inside the Showcase repository and this usage and the management are complete up to you. So I'm finalizing it a little bit and if you want to how how if you are asking for you, can learn about these and getting mold check out our code, you can check joint or discord server and we have a discourse server. You can join as for questions.

A

There are many things that needs to be worked on, but we are very less of people and we have a very good aim to do this thing in a very. How can I say concrete way of doing it, and we are really willing to hurt here from the people that are using this and check out our examples check out or showcase repo.

A

If you are interested in what other things are, what other integrations are and one more thing that I want to add this project I started in June, 10, I, think and I report this to digitalocean and and basically what happened is that this thing was a discord server. It sorry.

A

It was a discord boat and it was actually serving to the 250 people, which are all young teenagers, that they are continuously removing their messages, and this is a test case and we were just highlighting them because they are just removing their messages and not sending back and they were I think sometimes very bad things. So we just deploy testing as it is running since then, until today and in there- and there were a lot of panics were happening in there, but it was Ronnie.

A

No single crash and I didn't actually I just wrote, no hop and project name is running under the in it, and it's just working and I didn't write anything like the systemd stuff, and this is basically how it is. Thank you for listening me.

A

A

A

A

Yes, the dermis is pressing meeting. huh He is asking for the mess. Are there any messages passing between the workers right, yeah and yeah? There are message passing between the workers and we are using channels, and these workers can also communicate with each other. We can push messages broadcast many messages to all workers under a specific node in the supervision hierarchy. That I showed just so sure, basically yeah.

A

Yeah I mean in the this is a good part, so this took a lot of time to make it like that. So we had put a small aesthetics in github in distributed part and everybody else asking this is not distributed right now we are working on something called artery artillery.

A

That is actually going to do this distributed part. So you know maybe the Byzantine generals I. Imagine the Byzantine generals can use the artillery toast message pass, so it is going to use. We are going to write all these distributed features inside of is yet another package that is going to do this for Bastion, so is right. Now is not to doing that, but just Aslam and RG our PC client over there that is going to work, I think.

A

A

Yeah, that is true, you might say that there will be information lost, but all the errors are causing from a point. That is actually your what you are using from the create ecosystem or from the rust itself. So all these features are all these flood functions and all these elements are basically containing some kind of information about it. If you want to add more like in the first slide, let's go to go back to this again, so here, oh yeah,.

A

So like in the my create DB error, for example, you can add more things inside so it's up to you, I mean if you want to add more things inside your error. Handling code is okay, but I. Don't think that you need more because all the panic information and all the panic hooks that is actually triggering is actually the venom panic hook triggers is already gathering information about it actually, at least in the normal city environment.

A

It's like that in the quarry environment is just doing the irq calls and then actually trapping- and this is not for the embedded to be honest. So if it is going to be embedded somebody you need to help me to make it no ACD and I'm. Okay, with that yeah I hope, I answered the question.

A

Cool everything I want to rewrite everything in Bastian. I really want to do that, but the most thing most things if you are, if you're, if you heard things about the air long and you probably all the Alexa developers out there, actually writing the first game, servers and stuff like that, you can. Somebody wants to get into the action.

A

They can write the game service, this camera, one of our developers in the Bastion site, one two core developers are actually using Bastion to write a streaming engine and we are also writing some kind of things like that to actually use this. What I've told you the nature or synchrony stuff? So basically, we are doing these kind of things and that for data processing, it's going to be cool, I, think, and there are plenty of things that can be done and it's up to your imagination, but I.

A

Think if you want to have a ground-up stuff, you can write a completely asynchronous web server. That is opening a mapping to the old port like the island does so I mean it is going to really cool. That is showing that how many things that you can spawn and actually the all the scheduler inside of this thing is actually the Ralph replication of the islands scheduler. So it is using this kind of high-level things that the airline has done. So it's going to be nice to see something like this.

A

For my point of view, buddy is up to you know.