libp2p Team Meetings, 9 Sep 2019

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Libp2p Bi-Weekly Sync - September 09, 2019

Description

https://github.com/libp2p/team-mgmt/issues/16

A

B

Right, we are recording.

C

Okay, cool the crib pad is loading right now, so you know it takes a little while, but yeah I mean basically I got a bunch of things that were kind of on the house standing agenda finally to go through last week, so the visualizer projects are today we, the final security audits that will produce a report started. It starts tomorrow and yeah I've run through a bunch of candidates to try to onboard more people. Is it's pretty slow going to have to look look at every person's github and fig tree?

C

You know dig into it, but I think is a better stream screaming strategy than just talking to people so um yeah.

C

That's that's my report.

B

All right, Mike, next up a team.

D

Yeah so I merge. The effects pack, which just get stuff based on a path, is useful for making pub/sub persistent.

D

Also some with p2p discovery. Things to PR is one for back off and caching and one for basically discovering content. Routing are the same thing. We have an interface and implementation that takes one and turns into the other. We know the implementation that goes the other way, so.

D

Next I'm, probably just gonna, take all that and use it in the pub/sub discovery. Pr.

E

I'm just gonna interrupt to say: I am back via the magic of cell phones, so yeah, but let's continue on those updates. Thank you. The.

B

Mics up is me yeah, so we updated J s, multi adder, to use P DP by default, so that will get rolled out over the next month or so as we update the dependency. But that is marked as a breaking change, so it should slowly work its way out.

B

Then working a lot on the a single weight refactor for Libby 2p, to focus more on that this week. With that we're doing a couple, internal updates to the pdb horror as well like we're, also adding a upgrader that we're going to pass the transport similar to how it go would be to be, has passed an upper greater for the connection and so far just locally working on the refactor. That's cleaned up a lot of that logic for Jas, so I think that should be friendlier to the user downstream.

B

In this week, I'm just gonna work on getting a few of the blocking modules finished up and released, so we can move forward there.

F

E

Right yeah, the big shifts I've been noticing, was just generally towards working Harlan's getting us an orientation started. So that's going to start with the proposal document. I've.

C

E

Like the first drafts of that I'll circulate that today and the hope this week is generally to take flush it out, and it's like the ones deliverable I really want by the end of the week, is to have like the first action item, the first step towards realizing this finalized, so that I can actually start putting away at that.

E

Alongside that, I've been just kind of investigating security options, yeah for the message-oriented transports, the ones I'm looking at our DTLS and noise, primarily I, had a chance to see CLS and see how it could potentially play interplay with something like this. There was also like kind of promising and kind of frustrating, so there'll be more on that later and yeah. A lot of wonderful, a new contributor and the DHG and up next yeah I, just want to kind of finish, get that a surveying proposal finalized, get that first step going.

E

Do a little more code. Review I've got some stuff in flight for you, Dean and evermore PhD contributions to review, and that is me up. Next- have.

G

You so yeah so I think last call I was talking a big game about how he's gonna dig into the cat's back didn't happen. So much I did some more reading, but I didn't progress there and said I made some edits for the noise.

G

Spec did a few interviews and did if you read me edits and also dug into gossips, ugh which I I learned new things by rereading the spec had I hadn't read it for a while and I really want to understand it so that we can were in in a few weeks we're gonna be contracting out to build like a gossips of explainer, like it really kind of goes in detail about how it works, and so I want to be able to support that, and then my next thing that I would really like to focus on.

G

You need parking cab for another. A couple weeks is multi-select too, and try to firm that up, and you know really get something that we can something actually so but I. Unless somebody really wants to point me at something else, I feel like that would be a good thing to sink into.

G

For the next week with you, and hopefully we could get a working draft up and a couple like within that time range and then I've also noticed that I like pulling a read me off the stack every day or so, and just getting some edits in in place. Because it's a nice momentum builder, so I'm going to try and keep that up. But.

E

Yeah, that's great I. Have it also I would love to pick your brain a.

C

Little bit about noise.

E

C

E

So maybe yeah we could set up a concert tomorrow, something I'll hit you up out of hand. That sounds great up. Next, we have row yeah.

H

Incidentally, about noise I, just added a note on your update, because it definitely using inline keys with the noise cycle. Handshake could really go a long way and possibly even remove the need for DTLS at all. So so that is something.

C

That you induce an.

H

In-House expert on boys so yeah, so my my update here is that I've been pretty heads down on the ipfs Limpy to be test ground I got a test running with 100 DHT nodes, running locally and synchronizing Redis instance. So this is so.

H

This is going to be pretty so this is kinda might be the first implementation of the sync service, which is pretty instrumental to to the test ground itself, pretty happy about that and I'm gonna be working on automating, the docker pipeline and the shipping of the test jobs to a nomad cluster next working with the two guys they are this week in Canada around 70 of them they are locked up in a cabin running, interrupt tests against, like between different clients.

H

Have we already got like pretty good, pretty good feedback and positive signs emerging well from that last weekend, where two or three it's I, think we're now up to four clients are successfully interoperating by unity to be so so. This is awesome like I. Definitely expected a few more. You know words in the interoperability storing there, but for me it's been more or less unless they did have a glitch and an issue with the way that with multi-select sorry, what multi-stream we're.

H

Basically some implementations were expecting the multi stream preamble plus the actual protocol selection, and this is actually brass trouble and there weren't acting multi stream itself, whereas the other implementations were waiting for the app before in the actual protocol selection, so I mean just like really minor stuff. Quite frankly, even Berlin we had a pretty amazing successes there. If you were in the old hands or a few. A few days ago, you probably heard about that. We've got white detectors. The contributor that built those continues contributing.

H

So this is also he is continuing to evolve the y front effectors themselves. We he created a fork of Sakaya that has a key locking feature, so it dumps the keys the symmetric keys to a file which then the detector hooks up to and that way. Basically, it manages to dissect traffic. We are now turning that into a layered pipeline so that whatever cyclist X- and it sends it down to downstream to other wife to other secretary such that they can continue unpacking that payload iteratively.

H

So then there is a flex and there's gonna be all kinds of application protocols as well he's refactoring that into a multi-select multi-stream things such that the multiscreen Y shared a sector would be an entity of itself such that other applications can then match onto it and say: hey sorry, other dissectors can latch onto it and say: hey I'm able to handle this particular protocol. So the heuristics of selecting dissectors will become a lot more efficient.

H

But that's hit me up if you're interested in a story, messy and gossips are nice, handshake, gossip, profiling and mtns. These are things that we got out of. Berlin also worked with Mike on kicking off the final draft data first iteration, which is also into interviews and this week I'm going to continue focusing heads down on test round and Mike and I'm, probably going to start the PM refactor this week or the next week today are going to be working with that and just want to give a heads up next week.

H

I'm gonna be on holiday, so I'm really looking forward to and these what we create before I start DEFCON, which is gonna, be another stressful, crazy event. So yeah. It's all for me.

E

Awesome, that's amazing, with the DHT testing I'd love to take a look at that and I'll just get in touch at event. I! Think last up we have Steven. Are you on this call? Hi cool I.

F

Had a bunch of rest I would really love reviews on the CIDR FC I. Don't think this is all that controversial but yeah. Basically, if you are aware, the idea is to encode PR IDs as C, IDs and text. This makes it easy to tell if something is actually a PID and not just some other item, both a hash. It also it gives us multi basic encoding for free and it doesn't change anything about my formats. It's all just in text.

F

Second, one is a knod time out, I'm saying and go ahead. Defense or some of our users was singing goodness we're like we establish a connection and then we timeout negotiating the multi strain their technology on a string multiplexer. This is after doing the security handshake. The thing is the server isn't overloaded. So if the server overloaded I'd say well like maybe just taking a really really long time to negotiate the like to do the security handshake and then it's hang up my time, he gets the final hendrik, but yeah.

F

This just feels kind of weird. So if someone could just if they have time quickly look into why this might be happening, it'll be awesome, yeah, I.

F

Finally, there's a test failure on the stabilize branch and go confess, so something is wrong when stabilize branch it doesn't appear actually no I take that back and because it's also going wrong with the master branch as well, but this didn't wasn't happening before I updated with me to be so yeah. This may be related to do you see changes I'm, not entirely sure this actually edgington's back boo this one for now, because this could be a bit squishy I have to do more testing there.

E

Sounds good! uh Thank you very much and I think the last thing I don't know. Mike I just left a spot on the agenda. If you wanted to talk about the retrospective.

C

Yes, okay, sorry I took a little time to find the window. Yeah. Basically I put up a q3, retro doc. I know q3 is not over yet it's still like a month away, but I thought there was a desire or something from lab OS to basically do the retro like before the okrs and the first draft of the okay. Ours is due on Friday or something so I, don't know, I think just richer on.

C

What's happened so far this quarter or in recent memory I think the timing is less important than the content, but I hope everyone will take some time to put their thoughts in there and yeah. Thank you. Yeah.

E

Absolutely does anybody else have any agenda points they want to add.

E

All right going once going twice and I. Think then is it for today yeah thanks, everybody and I guess I'll be reaching out to.

H

Talk about is the huge mess that we have with gold and go on three one: thirty, the proxies and all that sorry for using profanity but I'm sick of this particular issue. The tweet that I put out received a lot of attention and I think this is gonna turn into a real issue for for the NGO community.

H

General Martin did a great job, that's where I say you know how fragile the whole of releases and the low career to actually make a release just by creating a tag and pushing it creates like a huge surface for accidents where which might not be so important if you're just pushing a minor or hot release. But if, for whatever reason you push a major release, then that is gonna alter it like it's gonna make all your downstreams alter their their employ pots, and that is super messy.

H

So I don't really know like how to proceed with this. But frankly, we do have an immediate problem and I came across this as soon as I agree to as soon as I upgraded my micro installation and I created a brand new project and I imported going to p2p and imported six point 0.23, because and it's what what go? What the proxy story like the centralized proxy story, so yeah I think so pretty bad yeah, I'd.

F

Like to take this point, your money to everyone, please delete those tags. If you have them Oakley. There are scripts that have published many times to do this. Please just do it, but yeah this.

A

Was a real disaster which we should advise our users to turn off their the GoPro GoPro thing.

H

Yeah, so these.

A

Fulfill their promise.

H

Exactly it's good for their privacy, I mean this is.

F

To argue it from their side, like we've, had a lot of problems where people will change the tags on us and we don't want that so, like the idea behind this is like, if user deletes, attack or changes something or does something sorry I like that, the code should still be available. That's why they did this, and that makes sense because like going pass, did or like or 420 doesn't know this right, but 421 doesn't build right now because of a change in Bajor like you just you can't rebuild it.

F

Yeah I unfortunate, of course, before 22 will build either because going event. Sorry, because I go released, a change, it's really really anal about numbers and about a timestamp since version numbers, but that's the different issue.

H

Yeah, so it's it's great that every time you build something it reports to Google servers, though yeah.

F

But like this, this is no different than like NPM or rest or any other programming language they're, like their system at least lets you choose a different cash or no cash yeah.

H

Except that NPM is like a difference for you, because it started being centralized and it is the package manager and it is in theory, a different tool that then got absorbed into the into the no echo system, whereas go there actually bundling like this is the only way to build.

F

This is the only matter but I again I, don't think there was a good alternative solution. I just don't think they had one and, like all the metadata they get is like. I am pulling these packages. They don't learn about your private packages, data. Sorry about public crash. Is you fetch through their system? So they do get extra stats, but it's not by default.

H

They do actually they do actually get like your I. Think your your go, build, will, oh, they go. The go to chain will actually try to fetch even private packages from them and then we'll just feel because they didn't have them. Yes,.

F

Yeah it'll try to fetch the private packages, but yeah they have no way of knowing that I guess yeah, fine. What they could do is like you could learn this separately. Somehow.

F

Yeah that's hard to do. Yeah.

H

Oh, you know I'm mostly concerned with what should be a word ongoing like what should be our next steps here to solve this issue for our users, things that that occurred to me is like basically recommend to disable this flag, sorry to disable proxy usage or to add the go.

H

Let me to be organization to the private reckless such that or they can go to direct I, think, there's a way that you can configure that to like only selectively avoid the proxy usage for a certain purpose: I'm, just not not comfortable at all with that, like it means that golden beetle means broken forever in the proxy, and that makes me you know very sad know.

F

So, like they're, going to add some way to like lockout versions, and we can do that. We can just like go through all these bad tags and like knock them.

C

F

Fortunately did how people happen for a while I mean this doesn't require shipping you go release just requires change in the foxy, so they could ship it immediately. Yeah.

H

Yeah another possibility is to basically change our import pots forever and move to something like go. Don't let me repeat: Ohio slash, you know that way.

F

I think we probably do this eventually anyways.

H

So that we create an entirely different namespace for us when we could start afresh. However, if somebody still has to pad text him by their name space again so I, don't really like lesser people to like a custom, get or get hot, yes together and that's projected protected tags or something like that. We won't be able to reject those tags on well.

F

I have no faith and give no getting this together and stuff like this. It's slightly it's getting better, but they yeah. This has been something it's no necessary for years and they haven't so I. One thing we can do like if we move to get lap which I think we change URLs, we could probably just do that and just near it get up. Then we get protected tags.

F

So you can like it's not, unfortunately, I can't say like like if a tag, if this tag exist, I'll have this tag, that's ideally what we'd have, but we can't just say no much. Basically, no one can create version tags and then we have a bot release version. You create an issue for the version decided to bot the bot face version mmm-hmm.

E

Yeah, that sounds.

H

Pretty compelling it's just like another thing: I, don't know like I, don't know when the proxy actually starts cashing a I guess on the first batch for that module.

H

But if we have a github action that reactively removes tags like if it does that very fast, are you know, there's gonna, be a timing, condition of course involved and there might be like in just the one bad case where you know that the hook is little whatever, and somebody tries to fetch that that module before the actually removes that tag is gonna break the namespace again. So it's definitely not a good solution, but I wanted to put it out there. I.

G

Feel obligated to point out the stupidest possible solution, mid p2p version 7 we're releasing it tomorrow, damn anyway, the.

F

Problem with that is that, like then, we'd have to upgrade all the import ass anyways.

F

So yeah yeah I think.

G

It's probably there are probably other reasons not to do that. I.

E

Say we just fork the project every time we need to cut a major release.

H

Another another possibility here is: you know this, probably like it's just for a consideration as that, let's, let's talk about other possibilities, probably not the not the best one, but for completeness sake, we could operate a world github and have github be the mirror of our github. This is what many organizations and cuz, literally it davido stopping. So whatever this.

F

H

F

Was never finished. Fourth gitlab like if I use get lemon, we would get our own custom subdomain there. We probably all see, is pretty important. So if you defeat io / something another free package know yeah that would like it's. The poem is like that's a lot of work to do that migration I think we should do it eventually anyways, just cuz like it extracts us from like it, helps hold and get up as before. It's late, not really delivering in terms of what users need.

I

I have a couple other points in one row. Is this the other one not related the one worded? This is a fun point. I will just say that this is all like after we feel your pain because, like in the NPM world, the problems are exactly the same but I. This is karma because we talked about the versions three years ago, like supposedly versions and go together like we are all playing, because so I know it's ironic.

I

The other point is so I'm I just wanted to kind of like catch up with everyone, and you know like podcast, like what the heck am I going to be doing now and essentially and I, don't think like it's wildly known in the organization. It's something I also discovered only in July, but there was an intent or desire back in June to create like a the simulator at some point, as a research group right now, a research lab within the PowerPoint project.

I

There was a proposal within the research team to create like something that can focus on networks. Research, so think about problems like HT, scalability, just privacy, preserving networks, nuclear channels and so on, and so just to give you like a little bit of like context from the last months is a key in June. Was this conversation, this research p.m.

I

summit I was sexual out there and then the the ball kept moving forward, and so in July, post ipfs or it's due in June and IP first team weekend at the fest camp me and Ian is what at were events and one shared a presentation about how to structure at the personally peer to peer research and in the balcony enrolling into what is now known as either the network lab or the resilient networks, and so I had a chance also to chat with the research team and they are like structuring what is like these research groups.

I

So instead I like creating identity themes such as like when research that is independent in for their own reasons. What piraka labs research is like 12, almost like multiple departments, as you Catalan, University and so kind of like a first instance of that is this.

I

This networks lab and so I spent today about the time just like articulating, like grabbing from all the conversations back in July and all the notes, Wheaton and all the discussions, it's like what that works out is and what it is going to focus and what it is not going to focus, and now it is going to support the project teams. Then we especially, of course, we peer-to-peer ipfs, you might have felt with the energies in the last few weeks, yet it has been doing an amazing work.

I

Oh I think all the topics, like all the open problems and all the topics for protocol design. There is this intense to organize a research intensive event still on the words I'll actually have to catch up with the any statistic like super, sometimes in space but yeah, and so there is this networks. Research group that is forming and intent right now is to have a team that eventually, in the future, has the capability, the resources to pick up on any open research problem and go from like finding collaborations putting out research grants.

I

It's finding the problem doing the first like vertical, like surveying the state-of-the-art testing, what is out there and so on and so kind of like helping both like you personally to appear and and I take some of that work from your guys's plate. Of course, the team is pretty much just me and yannis right now like doing that, so so like it's not just by having it seemed like the problems get solved, so this will have to be gradually like I think that we get, but also it is something that is kind of new.

I

We are still crystallizing, so I sent you an email Raoul just like to catch up and I, see how things are on the lips. I'm, just I think like one thing that is essential for having a solid research platform for networks is to have exactly the word that we presume is doing.

C

I

Test infrastructure, telemetry and so on, and so my intuition is that we actually should just like be investing our energy and time to make that work and then focus then on like articulating and all the open problems and reaching out to the research labs and start creating the the collaboration that will be needed to solve future problems, but but first I just just document the team, perhaps with a more like research angle perspective like all, can we make it for my great platform for researchers as well, but but I can support you in augment.

I

So this is where I'm at this is a good I know there is this repo I can send it to you and, like you, can just breathe the full request I made today, so that you have access to our the description and goals and mission and and so on, but yeah happy to answer any questions or if you have any thoughts, ideas, comments things or to share things that you have the chest discussed in the last few months, I.

H

Just want to give you a positive feedback on on what you're doing here. I think we definitely need to be a lot more Theory, first, with many of the changes that we decide to make, especially because we are many times playing about a year, which is which is great, because you know it's like there's a hyper.

C

H

We want we want to play with and so on, but at the end of the day, it is true that the size of the network and the respondent amount of projects that are depending on the work and I BFS is doing Libby to be is doing especially Libby to be now has become. You know, a top-level project that is being adopted by you know nourished platforms and networks out there. We definitely need to get a lot more like research first, with what many of assistance and subsystems that we develop.

H

So this is a great initiative in terms of of testing. I'm gonna. Send you the engineering document here that summarizes the design. I, don't know if you have it, but it basically it's to find out that we're planting here, I think that would be useful on the other hand of telemetry I, think it's gonna, it's gonna, be so very important as well.

H

I go as I was listening to a podcast a few a few weeks ago about a guy about how to make changes in in Facebook and how that will not changes and how they decide to go because Facebook doesn't test their changes at scale in general. They don't test the changes before they release them. They have a very controlled release, process and telemetry and like understanding how this change is actually rolling out how systems are being affected by them and how the change is being perceived by the users as well to decide.

H

You know whether to continue rolling it out or to like continue expanding the sample or not so I mean that is potentially another avenue for us to explore. You're, not burning out changes like the changes that we're making. Besides attempting to simulate doing huge scale simulation of you know, change sets. We can also have do canary deployments in a very controlled fashion or, like you notice, there's a lot of things that we can incorporate. All of that predicates on having a telemetry protocol such that we can actually sense.

H

You know and understand and gather intelligence about. What's going on mm-hmm.

I

Yeah, absolutely so what you're saying is that there is all of these also research, in some sense, around engineering, best practices, if flowing west systems, and not even by picking the example of peer-to-peer network peer-to-peer SATs from the past, like for at 5:00 and so on. All of those projects are like multiple contributors over multiple actually, multiple backgrounds. They.

H

I

Also learn a ton of things to that process. This is something that, for example, like this network's lab, could go in advance and kind of, like just gather all of the warnings, and also be like a team that can like the way that IGN is the like back in July. I, basically brought all of these experience and knowledge, and a workshop and complete believer to the team for the best training as a source for inspiration.

I

So, but well, but yeah. This is something we can do as well like if the.

E

I

You have in you or a potential Avenue that hasn't been explored because lack of time engineers have to hack and deploying right. This is something that this team can actually tackle and any like.

I

If you see the public right here here, sorry I think I forgot to add the Webber steam, but but now you should they have access it's just like a research pipeline image with two squares and, like the row left square, it's like where this lab should focus on edge like yes, certainly and even I, keep omitting some of the code, and then there should be like a transition like a support for the engineering team to then deploy and so on in in and I think long term.

I

This makes a little bit more sane approach between like product oriented teams like deployment teams versus like teams that just need to I, like this longer like like longer schedules to to like explore this research, and it's at which all these collaborations, but but then again, like I, expect also a lot of overlap between some people that will be across both both sides of the equation. Just like a more about a process like formalization and allocating proper resources to to all the spectrum.

J

To could two quick points thanks for somebody to wait yeah, we should progress from that. Just a quick thing that we had in in Berlin, we had a meeting between a few of us role was a Mike, was a George won and we kind of tried to structure and become a team in an efficient way, and we were discussing with role, for example, the fact that, depending on what is the problem at hand, you know it might take me a day to do something that someone else could take two hours and the other way around.

J

So for each one of those problem, depending on the experience, we should be flexible to assign they issue, at least as a first step to someone who has got prior experience and who is bringing background and then pass the ball to other. So this would be both from the research team to the engineering team and the other way around and fit both ways so that you know until they should get further. So this is going to be a thing yeah. It's also in the document that I listed all the opening problems.

J

The thinking behind was kind of like that, and it will have to be done together. Yeah and another side comment: I've had it's not only Facebook I've heard many actually other companies that say when they push something into production in order to save cost and time they. You know they just don't do any testing and the argument from an engineer, a friend of mine. You know, we've got this. You know user days of hundreds of thousands of people to test it for us. Why should we do it?

J

You know we just get, but these you know we get the measurements and we see what needs to be fixed and we fix it a day later, instead of spending our own day or our own week to do tests and fix before. But these these you need to have the telemetry protocol there, and not only that it should be not.