OpenJS Foundation OpenJS Foundation Collaborator Summit, Berlin, May 2019, 5 Jun 2019

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: "Bob" Streams in Node js

Description

OpenJS Foundation Collaborator Summit, Berlin, 2019

More details: https://github.com/nodejs/summit/issues/172

A

So, okay, anyone who's used strings and note those that it's any ass.

A

A

Yeah, you know consumer participation, they asked the consumers right. There is a movement of the back pressure, the state model complicated, and we can't separate the statement from me. You know me implementation.

A

There are three versions of screens that are all kind of overlapping each other you've got a lot of from. It has come into this, so there's not only clean three versions: half deep Jonathan level, there's also at the native of the screen.

A

It works and the native low that goes back and forth right there there's the streams, plank method.

A

That doesn't work and the native layer a.

A

Medium height going well, we make it even more efficient, but it's just over the years just spend that the student is going from the earth right and harder to work with. So was it two years ago.

B

C

A

He was got together at lunch at at interactive two years ago, Vancouver, medicate and ronnie'll idea for this new streams. Api yeah somebody asked: what are we gonna call it? Oh it's this la comedia.

A

Jeremiah has been the one I'm really pushing it forward, because the rest of us that we're talking about it just got distracted. She was the only one.

A

That's good work, but reaching the point now where, in order to move it forward, you need to disband it and those listed were collected by the next week out and then, if anyone else wants to help out when I kind of give a basic idea how it works. So.

A

You're going to need to the difference is the bottom is then entirely.

A

You have a Sinkin source, the source is providing the data, but the see past you pull right, that's to say when it's ready and the state model is greatly simplified, that you only have the one read right and if you read there's no data available and say right away. If it's an error, reading or failure right away, there's very few state transitions, as you are actually reading the data another part of it.

A

Is it emphasizes being memory stable so rather than the rather than the source, allocating memory and then pushing it, and then this receiver having to copy? Did it right? We're notifying this. Of course, when it's done, the puller would allocate the source port and the source location ready to explore right. So the intent is to hopefully reduce the number of men copies and actually become I. Think.

C

Of the other micro store, so I guess I'm just going to go vertically. This is so this.

D

Is kind of just p.m.

C

This is flight today. Thank you. For the last time we go Rashmi. Are you Sheriff, I'm, honest I? Don't know you just like to disagree? Yes, we do moving. Where do you start?

C

You go through each issue number one, seven one one, seven, one on solid, good I think that's easiest thing.

E

C

C

Cool so yeah I, just kind of like one of the things love it because.

C

It's the very same thing still continue to discuss, regardless of that, so I'm gonna go over like you sliced it. I do have perhaps a little bit more quickly than my stocking I'm already discussed some things, written them so yeah we went over white skull that and future question or streams or stream like things so I'm just up here, there's kind of like why a little bit of a fool's API status discuss.

C

So when we have been screams a you may know why here, but the user experience is really kind. People run into problems all the time, all the time. I don't think this is a very.

C

New that do so I have some goals and.

C

Things that I'm calling a consumer saying is the API and the point where data users do and a producer or the source is the API and where the data comes from and the protocol is a combination.

C

So the goal of this effort is to make a protocol really its full base. That is binary, so you don't need to worry about object mode, where's, Jay! Thank you, the stainless as much as we do so any safer. Well, we got this plastic gene.

C

There is one to one as possible so that there's no advance bidders, because those also follows oil, money, collecting issues and very important.

C

There is also claiming agnostic as much as also as much as we can get out of it. We don't need to worry about it. Next extend the best like you currently do no buffering in the protocol, so we would like the underlying classes do not have buffering logic and any buffering that is done. She doesn't mean a component to where it actually needs to be done rather than all the time, and also the error should be able to be in line and sort of like ends or Enzo.

C

Files should also be able for you in line message, so I think what was doing and unfortunately is late in the day and I'm still going to show you code, which I regret, but that's what it is I'm. So the sing API of this essentially looks like this rather simple code, there's a method to kind of find things together, descent into singing to the source, so you get the protocol and then, on the same, which is the thing that is going to receiving data.

C

You have a next method, which is what the source is going to call.

C

They can be filled out a bit more, so this actually more or less kind of like works together and then in your next wave yep there you would be once you are done receiving and processing data. You are going to pull it again. These are pulleys, so you request data always and the source API is kind of like this. We go to methods that are the other side, equivalents classic and pull this cold and that maybe fill those two once you get, the data send that back to the sink next and quickest.

C

Obviously there's bit more filler, but that's really a lot to the protocol. So a very.

B

C

Can be cooked beyond constructive just as this, this works as a pass-through completely there's no extra code needed, and it fits on this pretty nice effect.

C

So imposing these things into something you can actually like stream data through is kind of like not you know perfectly Pleasant like now, a header would be appreciated or like as a rain cover is probably going to be the ideal to or to just change the planning mechanism. I love it rough like that. That is kind of it. It's got its most simplest and brought for discussion, and so, if you want to put a transform any community, basically now you're saying binds to sources.

C

So how do you pinch for inviting to your source and then your sequence then bind to that transform and doing the way most of this right now is so that kind of like one night, we start there's some stuff I basically broke like this very minimal thing on like what we may want to add to that or extensions I'm, not sure that's life, but in cases where you have the network sockets it's nice to be able to start that explicitly. You know how to privately start from.

C

So all this is system on my in my in repose, under fish or 23 /, thanks to everything, and if you give diagrams, you can blank these diagrams. So this shows you the entire flow on where there is no point or anything. But you you start down where the funny that's kind of thing is, you know, cut for yourself is and then you go and you pull up the stream.

C

So in this case you may be from a network socket or something molding compression transformer that is going to pull from as a final source and that once your final source has that theta it is going to send that back down and.

E

You know transform or something you made.

C

How they need to do some buffering, so it made times and then you're going to get down to the bottom, and it's going to repeat as many times it's necessary and when that's done, you will get a message. They're complying this everything or you go an error, in which case the error flow looks very similar. The error could be emitted anywhere, it bubbles up. So the.

C

Anything it was below and also the thing that admitted I, don't know if that makes sense that we go over it again discuss it.

C

We princesses, if I, have some various modules published they're published in nvm they're steep up sources for these things, I'm buck status at the source at the sink, do look transform, they do work and then there's something which kind of attempts to make a network socket. Oh, this requests with that sucker work with the server server looks.

D

Lease memories I'm not.

B

C

And I have limited time spend honest, there's some things on the API performance. If someone wants to build one that doesn't already put, this is much cheaper to do than using current streams and everything. This guess.

A

Any questions on any.

A

Questions on the benefits or concern.

D

What we have for streams? How do you want to support the new API in the existing live net or hinge to be and doesn't have any overlap with the promises friendly API for of engineering streams.

C

So I kind of left that on such a nice light, but the plan is to change all of the underside to no basically so. The streams internally would use this because streams is mapable on.

A

Top of including all the stars so leave the paprika juiciness is basically just an attic as a separate new streams, implementation right in Carlos. So what's there, the existing streams API would continue to exist untouched right, but there would be a map layer on top of this. That would give a little lazy stream mapping on top of all right, yeah.

C

The idea was to do this in a way that where streams could be implemented on top of it, so that we could then prototype is essentially also a node by switching out internals to use this and then once we were happy with how that works, then we could publicize it and then.

A

One of the other nice things about this is that the same protocol. This is essentially just a protocol and it works for that the C++ and JavaScript using the same model. So we don't have two completely different: Navy I interchange models operating those two layers.

B

I'd be a mystic because I just came in but aside, the question was: how does this relate to some existing their ongoing standardization, which looks great.

A

So what are the other intents of this was to give a the new little stream primitive upon wish something like whatever you shins could be built right now, the existing streams and note if we try to do that, what the new Jean implementation on top of that the performance will be rather bad and there's the model system quite thick. With this, we should be rather straightforward to build what W stream to the fish Mart. On top of this furniture.

B

A

B

So then one question would be.

B

B

A

C

The idea is that we.

B

Would potentially.

C

Eventually expose it because they would be much cheaper to do and it is much simpler than there and it is much more cheaper. Do it's much more performant than what will do speeds for ours? What we do streams can be built on top of your problem. What we do streams are first base when you are in the screening, but the actual like end points of consuming something or giving data somewhere or actually so just find this also.

A

C

A lot easier to build, put these streams on top of a streams. All you do is have something that immediately grabs the data and pushes it along rather rather than the opposite way. Try to both pulled a stream stay screams. You just have to buffer a ton of data all the time.

E

So this is probably just exposing my ignorance about how strings work, but if you don't have any buffering and it's full based entirely, what happens when the source is not.

A

So this is intended to reduce as much as possible. You know when we do have a source which isn't far host, then it will have to buffer what we're when we. What the goal is is right now on the streams, for course, offering at every layer like rewritable, hungry beautiful, has this buffering. We want to isolate the buffering where it only when it where it's absolutely needed.

A

There are some. Unfortunately, the lot of the underlying protocol now are giving us options for providing flow control in the protocol HP to wait, for instance, we can actually, if we're not reading, we can tell the sender, stop right. Some of the other, the other protocol just will have to buffer inside of that of that one.

C

Layer, so exactly something will always be stop buffering so market. You are further and as your options, but the goal is to try to avoid doing that, certainly at every step along the way.

C

Another thing that's not gonna give that I'd like to mention is kind of like a live, even layer of things, so in in my circuit work. This is also implemented in a sequel plus and should be also so along at the system level. A lot of calls you make the network one way or there being like a request from from the operating system, which kind of max lot better to replace things than most big streams. New and let me be currently exposed quickly streams, but would expose patients potentially in a way that made more sense.

C

C

A

One thing we haven't touched on, sir: the turn streams API has both the readable and writeable interfaces right. Another two very scenic things with this: there's only one protocol that works for both right, so maybe on the writable side. What your? What your and implementing is a it is a source right and then he will provide your brother than pushing data into it. You're you're, essentially just waiting for whatever your destination to pull data out, so there would be a slightly different model.

A

It's like very normally what we have now. This would be something, but we will be able to support that liable side as well. So.

A

Any any other questions in terms of the basic model terms of what's needed here is just more validation of the model you know this is this is something.

A

I've been working on this for two years and you know meets anybody interested health, pushing forward that I would mention. The next step is actually getting land. Nice significant stuff is actually landing. My support for.

C

This, the next up, is getting some sort of conversion between this and streams for every app that is required to do stuff that you just said.

B

Is it time or reasonable to move that project under the nodejs ordinate was a something.

A

That is more easily discoverable and having other people get involved.

C

I'm not really convinced how much that not apparently.

A

So so yeah the the streams, the existing stream jet guys for the complicated doing that map require somebody that you know that has a good understanding of those extremes.

A

But you know if you might get involved in that something that we can deal with. Let us see this section or earlier quickly is something we can introduce within. We should build issues in the future of the minor.

C

Yeah I mean so.

F

I just wanted to add one big thing: once we have the transform layer, one thing that can be really really nice to check is that this mechanism is.

F

F

C

F

C

I brought it across the barrier of meaning to talk to Joe certainty. Loss is pretty much the same. There's not some wildly different layer. That's been it's like needed to encase it speed based.

C

Yeah and a couple other things I mean dimensions point there. Please there's other things: I probably should listen more, a bit still use like prototyping like if you do want events with this like for some reason you want to see when they like, or something really putting like a transforming.

C

You just want to like this is the errors or something for some reason that way, another thing is okay. This is from a space. It's not only the apparently callback based here, it's kind of just like objects that talk to each other. They call each other than others, but consuming a foolish thing with a async iterator, also not very well in spacing good razors are essentially full length less and less this.

C

On that note, the reason why it isn't sort of like a sneak away base is because or facing aerator sort of base, because there's not really a way to pass up from the consumer to the thing that is giving you data and that is or memory.

C

A

The one thing that I will see the current API was just universe, your employer. Next there is one limitation there. You know that we have a quick, for instance, if they might also think we have to hold on to the buffer data until is explicitly acknowledged. So we have to write it again, so it's perceived I should sir, we might need to add one additional.

C

Then, and also don't mean to locate a Dennis student writing stuff recently, the inherently have is like seeking, although I guess like thing before all the other pipeline that doesn't like you continue until we get to this point.

C

Or you're supposed.

A

To just do that so yeah, so one of those lines- yeah, just you know, validation of me yeah, you know just you know, think through the model, how it works show how that ago and if there's, if for some reason, this this model just doesn't work for taste, because it was not easy, the great every boat that we can start discussions on.

A

Any other questions comments.

D

You mention had performance improvements. Do you have numbers on like how much pasta TCP gets with this implementation.

C

For like doing like file reads and stuff, even in five cases, you get like eight thirty percent you seen you're using and for good cases like so like a five cases like your is I believe with Cosima transforms, and it still takes you and just like doing a final copy screaming my hockey is, is about anybody.

A

Personality to this table and with the current stream based model.

A

Stream base went when you write data to the stream to the right of those which like and it passes that through to the C++, you can't write on another jump until I call back this right in because of the way I have to buffer the data for port quick.

A

The maximum throughput, by extension, to do a mem copy on every right, then call him back right away, so I, don't necessarily whether they need to be acknowledged. But you know which will treat.

A

F

Way, I, don't know where the olenka is.

F

It has been a fantastic experiences thing for everybody, and I will open an issue to receive to collect feedback on repo so that we can improve for next time, so things FM for our system didn't work and teaser. Do you mind if we might do better next time, I've seen that a bunch of people who like to a main tax, which is a fantastic thing, because it means that more people that doesn't know each other coming to these events? So.

F

Things we have to do for the next time named tax right.

F

F

Thank you for picking up a lot of work and also to many other, a lot of work that.

F

D

F

Was not along those big team working on this also proceed from this table helped a lot and as well.

F

Rooms and everything it's been great, so I am so to say. Thank you all of you for coming and that's it.