libp2p Team Meetings, 24 Feb 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: libp2p weekly sync - February 24, 2020

Description

https://github.com/libp2p/team-mgmt/issues/16

A

All right, we are recording hello, everybody welcome to the February 24th, the p2p sync. If you have any agenda items, please add them to the doc and we will take them. First, come first serve. If we don't have any of those, we will jump into weekly updates and with that there are no current agenda items. Also, please add your name to the attendees list and we will get started on weekly updates.

A

I will go first, yeah, so was out mostly last week at the ipfs team week working through some stuff, we started planning out content, routing improvements through I, think June, so I'm gonna be working on getting those posted in the public forum.

A

So we kind of better see the roadmap of where we're headed at for the next four months, other than that just minor bug fixes for jeaious and then working more heavily on Interop testing for the upcoming go and jeaious noise implementations and then other than that we'll be spending the rest of this week, focus on unblocking people on PR reviews and then the noise work. Anybody have any questions for me.

B

Just that, if, when we do the other hand, snake patterns for the interrupt testing, we just need to add options to the daemon and I might I. Think there's. Maybe maybe, if you want to pay me about that later, I think there's one thing missing from the noise code that we need to expose, but yeah, there's I think it's. The key pair needs to be serializable instance, but and graph for the static keys, yeah cool yeah, we'll see Coughlin.

A

A

C

Age last week, besides that this week, I'm gonna be so, we started an initiative where the residents with a missile in networks lab to look at ways to secure concepts of both for fun coin and dream. 2.0. You gotta be delving into that this week, along with bees or the beads, and yes and others, and why probably.

C

Document is like moving I'm gonna, be shipping tests around the zero point to test grant release. Probably today are we gonna start planning for test ground zero point three starting tomorrow, then I'm gonna be working closely with you set tool and going to be to be noise finally, and to design an and the potentially address provenances attribute in the ps4, which is an an item that I just added to to the agenda, because I would like to discuss this briefly with you guys.

C

This is basically an outstanding item to to merge to be able to merge and well-rounded version of sine periodic records, based on some feedback that we received from my BFS and and some some animal itself of textile as well, that we need to consider textiles architecture and other other users. There'll be reviewing well sorta, not PRS as well and arches, ps4, persistence, PR and the other PR for the routing table. Dissociations for the connections table.

C

Any questions for me.

A

No, and with that, let's make sure we get to those so we'll switch back to the agenda for now. Do you want to go over those items.

C

A

I want to make sure we leave.

C

A

For that, because the async updates are as important as using the same time to solve those problems.

C

Okay, so that means that we should okay, so okay, so the thing here is that do you want me, should I explain it, or would you like to explain it? You said.

B

Yeah explain real quick, so there's there's a few things like right now. The current where the routing records thing is set up is the existing interface which accepts unsigned addresses like it always has, and we have a new interface which executes records and the current behavior is. It will replace any unsigned addresses and from that point on, you will stop accepting anything but peer records, so like you're, all in on peer records from you get the first one. But what?

B

If you want to explicitly dial and address because the user typed it in at the command line? We didn't think about that exactly it because, like now that you have this peer record in it's good, the peer stores not going to accept it and then.

C

B

Dialer will not be able to access. We just basically need to be able to accept, addresses that were like locally, confident in and also like textile is having they're gonna be, or they already are, I think disseminating multi addresses in their own, so signed record format. So that's another trusted format that they could. They might want to track the provenance of those addresses separately and be able to say, like oh we're, gonna favor, the ones that came in through our special format, but we don't really have any provision for that now.

B

So the idea is like we will add this like little numeration I guess with the ability to add, like your own custom type and say, like the system, values would be like. Maybe came from. A peer record came from the existing unsigned interface. You know or came from some get whatever we can spell out all the places we think addresses are likely to come from I wanted. It occurred to me that I hadn't really thought of it. All was DNS link redirects, where we're like we're resolving a name from some other source.

B

Now we have a new address that wasn't signed, but we trusted the name, so we trust this sort of by association anyway, the point is we kind of needs to figure out the API. For this we can have defaults for all the current methods.

B

Like anything, that's going in to add a ders right now and have a default value and anything that comes in through a record will have that value, and then we need to be able to explicitly specify the value and then do something with like exposed those you know in an API, so you can tell. Where did this address come from? Then?

B

What do you without information and I? Guess? There's some another question: like? Does the dialer prioritize certain origins over others? We may eventually have to figure that out, but I think the point now is just me first making sure we don't break anything with this restriction on like we're, not gonna, take your addresses anymore. So we have a peer record and then, but once we've removed that restriction. Well, then, how do we tell what the good addresses are? I guess is the problem. I'll stop talking no.

C

That's thanks for the overview, so the way that we think in terms of API is the the component that we'll need to so by default, the diner will just use every single address that is found in the ps4, no matter what the provenance is. This is to present backwards compatibility, but then the subsystem a subsystem, that's open. The stream, although that is connecting specifically connecting to appear above according Network connect, will be able to provide options.

C

We need to add the support for options in dials, and one of those options would be that the kinds of addresses or the provenance of the addresses that you want considered for that does so, basically, by defining, as you have said, a innum type or basically a type in an in goal which, by definition, enum types are not sealed.

C

So this means that any any any application like, for example, test text on, can define custom types, custom, enup types and whenever they add addresses that they've that they've consumed from their customer record, for example, or a custom.

C

You know system or whatever they want to register the particular provenance for they can just pass in that enum type and whenever they want, they want to perform a dial, and they want to explicitly just use the addresses that have been authenticated by that record or whatever addresses that were inserted by virtue of consuming that record or in with that. With this particular attribute, they can just tell the diner.

C

So this is the way that we currently currently thinking about this, and this way the DHD is well whatever, whenever it tries to whenever it opens a stream which implies a connection, if one doesn't exist, we'll be able to pass in the criteria for first filtering. The addresses that secure version in this, the current version of the DHD that we building so the one that shipping in 0.5 we'll just use, sign pure backwards and that's where, as.

D

Ib LS yeah, so one way we can pass up. This propeller is through context options.

C

Yeah, that's that's another! That's another possibility I, particularly don't like this. This option, I think hacking I came like the context, could grow indefinitely and it could become very messy, but- and it is already the case- because we kind of like already took the path of least resistance to introduce dial options by hacking them into the context and I don't want to protect you like this this this approach, but that's my my current thinking, I, don't know what what I was think.

C

That's what I would propose in a PR to actually model and go ahead and model tile options as they just have to be modeled. There's going to be more down the down the line, oh yeah, any any other suggestions, any other input.

C

But things are gonna get more defined once you see them in code, so I mean.

E

Functional tile options should not have issues the way to do it.

A

Is there going to be like a design in an issue for this, or is there already a design, an issue for this? No.

B

I can write one up after this call. I've got yeah I think I have a decent understanding of like what we need to do.

B

A

We're also looking to do an overhaul of the peer store in jeaious very soon, so it would be good to just sink on this beforehand. Just in case we have things that we would like to do in both yeah.

C

And that I guess leads me to the second point: the second agenda, spinning another another functionality that we need to desperately in the Psy so yeah. So this is kind of like you know, I'm dancing around with some ideas of self that were proposed and floated 4:04. Sorry, ps4 version two as well but like these are urgent needs right now. Basically, the the phone in the functionality that we want to propose that arsh and I have been thinking about and Stephen as well has been in the discussion and well as well.

C

Basically, we need a way to to signal to the ps4, but a particular subsystem is it's needing to keep addresses alive for a particular peer because they are they're holding a reference to that bit to that pier and it's important for their operation right now.

C

What happens is that one appeared disconnects identify we'll just set the TTL force, the TTL for all all the dresses of that pier to ten minutes, which is also an arbitrary number and like we all agree that TTL czar, probably not shouldn't be modeled externally in the pier story, but but yeah like let's. Let's assume that you know any deeper changes in in terms of removing the TTL from from the interface and like making making the ps4 a lot more event-driven are gonna, be like punted to version two.

C

The question is: what can we do right now to make sure that addresses are not changing under our feet right and addresses are being removed on under from under our feet being like just disappearing, just disappearing and vanishing suddenly. So, basically, the idea here is to introduce a pin and unpin method on on the ps4. That would be that will ref count that would be driven by ref, counting such that subsystems can ask the Pierce drawer to in a set of addresses, until it's told otherwise, by an on pin method.

C

The question here it becomes hey what, if subsystems are badly behaved, subsystems are buggy and they just let pinned addresses, build up at one point that could you know that could build up a lot of garbage in the PS Rory. So a different way to go about this are to to basically make the pinning and unpinning make the pinning a best-effort thing.

C

So the PR story does not guarantee that those addresses are gonna, be there, but it's gonna, it's gonna, they're gonna remain there forever until the pin is lifted, but it just you know it's best effort at one point, so you know there are some like GCE considerations that we need to that. We need to take into account as well another possibility. Another design here is to is to is to Forrest subsistence to keep pins alive by making them refresh those pins.

C

So yeah there's different different approaches that we can that we can consider here, but basically the end functionality that we need is for addresses just subtly vanished when subsystems are keeping I keeping track of particular theories.

B

Jake, this kind of reminds me of something you were describing awhile when we were in person at the lab. We, the like user-defined topologies, that I remember you describing something really similar to this, where you would say, like my subsystem, cares about this peer or the set of peers, and then the peer store would make some decisions based on that yeah.

A

So we started: building in chess was the ability for subsystems like pub/sub in the DHT, to register with the connection manager and say I care about these protocols and whenever a identify was done for a particular peer that subsystem would then be notified of that peer. Like hey, you have a fear that you might care about. That peer. Could then do an analysis to see if it actually cares about that fear. So like, for example, like pub/sub might decide that you know it has all the peers that it cares about and it's mesh.

A

So it's not gonna do anything with it or it could choose to like say yes, I care about this peer. It would then inform the connection manager of that, and then the connection manager would prioritize that accordingly and then, as part of like topology registration, you could specify like what percentage of peers your subsystems care about, so you'd be like okay, the DHT gets 50% the pub/sub gets 25, and then we leave 25 open or whatever, and then the connection manager could then make more informed decisions about that.

A

That was kind of like the the playground that we were messing with for kind of a befo connection manager, Avicii to experiment, cool.

B

Okay, yeah, that that makes it so it's not not directly to do with the Pierre store, but I guess he restored per se- didn't exist in jas yet. But but it sounds like a related kind of concept. Where we're saying you know here's we went to register our interest in this particular peer or address and yeah give you an opportunity to just clean it. Yes, yeah.

C

So in in go, we already have that functionality, but we omit identify events on the event bus, so, whenever appear is, is properly identified. Subsystem can subscribe to receive those events, so we kind of like decouple.

C

You know the the actual, the actual knowledge that the Pierre is conversing, a specific protocol from the connection manager itself, but a connection manager is like super super slow below and go right now we haven't really evolved it since version version, one I, just posted I'm, gonna post a link to a very strong man proposal that defines some news ideas for what the connection manager version two could be going forward.

C

I open this like a lot of months ago and I've already landed anything more concrete than this, but yeah and I'm gonna post that post another link as well to some ideas for the next evolution of the ps4. But everybody in the scholars is aware and helpful context. You know potential grief, actors that are coming down. The pike.

A

Any other questions run app.

A

No all right, we had one more item on the agenda: pop-up I, think that is drop.

E

Using for exchanging STP each use a scrim, so I have I'm looking on that. Most specifically, the problem is with the swarm implementation. It will always take the first transport to actually or return a connection, so my transport, even if it will be faster than we like because it will require there- is a relay circuit in the first place it will never actually be used. So I want to know. Is there anything planned to get swarm.

E

Because I think we can just just.

E

Interface with which is missing the current transport interface, with just a function such as how good will quality quality, which return an integer at 0, is P. 1 million is quick and -1 minion is stuck with, let's say, and simply the transport manager. This one will try to get the highest quality assurance for possible. So even if the first transport is, let's say, struck wit form will be able to do that, which is a bad transport and was I try to get a better transport.

E

So even if struck with work first, then a neutral, a newer transport is working, he just checked with quality and he receives a quench is bigger. It updates, which is used for creating corn to connection to set peers and the fact with just simple things: we can actually making quite seamless in the brain, because if the transfer function, if we see the type is just a transport we can operate depending of the function, I forgot its proxy if proxy. If it's practice, we just give it a zero, because we have no more information.

E

We just let's say well, that's like TCP if-if-if bourtzi is a true. We like we can duplicate it, so it will actually be quite seamless and just so I'll order, a transport with server that we will be able to add for things like quake or my transport, which is not good at batteries and socket things so the quality function. Those free concerns.

C

Yes, yes, the difficulty there is. When do you stop dialing, and when do you stop attempting to contact? So when do you stop? So imagine you see yourself in the situation right you, you dial your dad appear by a relay it succeeded.

C

Then you have three alternative transports that you can try, all of which are better than the remain in terms of you know the quality or whatever score you assign to it, but none of them is actually gonna.

C

It's gonna is gonna need to a direct connection, because the peer is not reachable right, but still you spend how many, whatever seconds you have said, as your timer as the connection time off for those transports trying to connect to those to those addresses, even though you already had a good connection to begin with would appear, and the only connection that would have worked in that context right. So it's this trade-off between how much we willing to try, potentially better scoring transport versus return.

C

The first connection that we that we have immediately right, ideally the the ideally where you'll be able to do, is like transplant transplant, a connection so stride off, and this is like we've discussed this.

E

In the context of.

E

Quite easily, it's not that easy, but simply when new stream is created just for upstream and when a mystery.

C

Thing we lost, you did everybody, you lose it yeah.

C

We gonna can't hear your horrible okay, yeah I can hear now.

E

E

E

We can just, let's assume, we're just about what I was what I. First film was just when we do a connect. We try every transport possible and let's say we have a turnout of 32nd, so the first transport feature. So if we turn the corner its connection and if a new word transport, better transport, we turn after that new recollection will use the humor transport.

E

But other connections that were already made with the slow actions of the test circuit will actually stay on circuit, because, right now we doesn't have any easy way to migrate and for migration we can just when your new stream is created, send the kind of continued packet with an ID to the stream to replace and P&C. Also in could just reset the connection is a stream to replace and.

E

C

So, ultimately, this would be the direction that we want to pursue and the reason being that for hole punching as well. This is gonna be. This is gonna, be important. So when you so when you the way that we are conceiving hole punching, is you use every day as the signal er? So you we already have relays and use the relay as a singular? No, it's not actually in turn mode, so you don't actually pipe pipe data, but in order to okay.

C

So imagine that you have imagine that you have a canal, so you imagine that you managed to establish a real, a connection right. You could potentially start exchanging data without peer and in parallel start your hole punching procedure. If hole punching succeeds and you managed to establish a direct connection. You would ideally and transparently to the application you would want to. You would want to migrate all of the connection state and all of the streams onto the new connection right.

C

So this is kind of like transplanting the the state of the old connection into the new connection, and this, in my opinion, would be would be the way to go forward. It is, it is a complex thing, that's why it hasn't been implemented. Yet.

E

The ioan I just took a look at swarm and, oh my god, there's so many abstraction layers.

C

Is the swarm the pipeline I don't know if you've seen the pull request for the for.

B

C

Yeah so I think that would we need to learn that it's been open for a while and there's a few misses that there's a few callers that need to be implemented and it needs to be rebased because you know master has moved as well, but definitely I think that is going to make life a lot easier for these potential use cases right. But the question like it's not gonna ultimately solve the problem, which is we want to optimus Takai return a connection as soon as we have it.

C

While we keep doing while we keep dialing other transports in the background, hoping that we'll find a better one and then replacing the state underneath the hood under the hood right. That.

E

Would be the idea abstraction layer to actually switch connection I mean? Is that what I? What I think we can already do? That's that's real. We can do easily is instead of migrating connection, setting it's just newer stream to better transport and hopes, and older string will just die away because connection work, because the protocol would love it.

C

B

Like an affinity.

C

B

Connection, it's like streams prefer this new one.

D

Step as well in the context of all panting, so the the simplest way to move forward is to basically create all new streams to the the better connection and have a grace period for the older connection and eventually close it, because transplanting the state of the stream learning from from one connection to the other might be impossible, because this tree might be a quick connection and the other one went to a young connection. What are you gonna do yeah.

E

That's possible, but just what we say we need, and if we want to trust my we need another abstraction layer between applications, training, I, say.

D

I would say, don't even don't even bother with transplantation.

E

Because it's possible I can just I think its first time, because it's really a lot of we right and.

C

Yeah so I'm having considering solutions that are way simpler in. If we, if we consider not transplanting, then what I would say is that we need some form of call back, so that protocols can register to know when a second better connection has been open for peer to which they such that when they have an old stream. So such that when they have a new stream associated with the old connection, that would be getting a call back telling them hey. You know open a new stream again or transplant in your state.

C

So that way you move like all of the concern of transplanting onto the application and then they can find a checkpoint and move the move. The move, the state of the screens themselves right. So, if I'm, so, if I'm, the DHD and and I open a connection to appear it's by a relay which is not going to happen anyway, but we do that. Imagine imagine that for imagine it for particular example, we open a connection to appear.

C

We send a we open, a stream on the over the relayed connection and in parallels five seconds later, we managed to establish a direct connection as the DHD subsystem. I would like to receive a call back such that. I know that, when the request implied over the old stream is over, I should close that stream and open a new one versus keeping that stream alive, because then that new one would be open against the the new connection. Yes,.

E

Subsystem to know when they should recreate a stream, and so in fact, in reality, sir transplantation will not be made by libya rupiah, but as by its application of themselves, exactly.

C

The application were transplanted.

D

I would like to give my my blessing here because I think that's right.

D

C

Yeah, so maybe a call back like alternative connection open or something like that. We have the eventbus right. So now it's very easy to produce these kind of callbacks. These kind of events.

E

C

This with the pipeline er with with a pipeline PR, which already allows already dissociates or decouples the address, so the address restored. The address resolution for appear with the actual dialing and the execution of the dials and so on. I think this would make for a really really nice dynamic system, in the sense that we could keep trying things in the background and the version of this dialer or, let's say a configuration of nepeta P that uses the bright dial components in this pipeline.

C

A very trivial like could easily keep attempting addresses in the background, even having after having returned a particular, a particular connection. You keep attempting addresses and if you manage to find a better connection you and made the call, you omit the event and that's it add the connection to your swarm and made the event and that's it and then graph down to the number of streams on the old camp on the old connection to kill it when it reaches 0.

E

Or 1 and I'm not sure what it is yeah.

C

Also that just posted in the chat window, Thanks.

A

Great that is it for the agenda items. Then we are are at time a little bit over. So unless anybody has anything else, we'll probably skip the weekly updates. Murray goes offline.

A

No all right, nobody has anything so I will leave the window open for a little bit, so we can there's the chat. I'll make sure this gets out the crib pad as well cool all right. Everybody have a good week and we will see you all next week, bye E.