IPFS IPLD, 14 Aug 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: 🖧 IPLD Every-four-weeks Sync 🙌🏽 2023-08-14

Description

An every four weeks meeting to sync up on all IPLD (https://ipld.io) related topics. It's open for everyone and recorded. https://github.com/ipld/team-mgmt

A

Welcome everyone to the ipld community and thing call it's August, 14 2023 and as every four weeks we talk about I, believe things and stuff. We worked on and yeah then discuss any engine items, and today we even have one because I've added two to the list, all right, um I.

A

Heard from like this was already a long time ago: um oh no! This is the old notes, I'm, not sure if I've yeah anyway, I start with my update so um for uh people interested I was at a conference showing a webrtc direct browser to browser connections and give a talk about it. It should be online pretty soon, hopefully the recordings of anyone's interested, um and it basically was really about being.

A

You need to be in the same network, but you don't have internet connectivity and you can connect with two browser instances from your mobile phone to each other. I'm doing the talk that demo didn't work. I still have no clue what the problem is. It's some weird web RTC issue because sometimes it works. Sometimes it does not work and it's not reproducible and, interestingly enough, it works really across platforms. So I did from my Android phone with Firefox to an iPhone with Safari works to Windows machines. It works sometimes as obviously so.

A

It works with everyone. It works sometimes even to Chrome through Firefox works or then a Firefox on Mac OS works with the Firefox on android, so yeah. So it's really.

A

Those things I got basically all all possible ways working once, but not reliably am I still in need to clean up the code a bit, but hopefully um people will get excited and then someone will find the bug. But I really think it's some weird.

A

Debug so um yeah, um but I can also put a link to the project in the notes. Yeah.

A

Other than that I can't recall if it was before the last meeting or not, but I just repeated that we had a big rust, multi-hash release, um which was a big refactoring, um but it should easily still be easy to upgrade, but um we split it into several trades so that there might hopefully are less breaking changes all right, um I think yeah. Then, of course there's this big effort on the ietf stuff for multi formats, but I guess someone else will talk about it. So yeah.

B

uh I I give a report out, uh so the um the the plan I think this week like the week that starts today is the week they decide about formation of new working groups or like one of the two ways to make it working whatever.

B

um So that's why I was trying to hustle to get slightly updated versions of the drafts that they got a month ago over the email list and uh the more you know the more long term like if they say Yes to the Dress and a working group happens at ITF, um then it would have its only scope would be to make multi-base and multi-hash specs and corresponding registries.

B

um The idea would be to um sort of like because those two are relatively easy to specify and relatively easy to use on their own.

B

uh The idea would be to like test case it and if that goes well, maybe re-charter in a year to get the rest of multi-formats in there, which would give us a year to sort of clean up everything, make make any changes or start making any long-term changes.

B

uh If those other things um would also go over the fence- and so just to like summarize, all the different like back Channel one-on-one conversations and everything, the consensus seems to be that it makes to think of the ietf versions of those specs as like, the the less opinionated more General use versions that maybe completely non-ipfs or even non-decentralized use cases might find sort of like to find other audiences for them and make a sort of like stable, Normie version of all these things.

B

Even if ipfs keeps versioning things and starts using, you know, maybe like three years from now, there's the ipfs tag is unrecognizable. The ipld's tech is unrecognizable, but like this ITF version of each thing would be sort of stable um and be happily being used by. Other people. I think that's sort of the priority so.

B

I think it's, um the idea is that the the multi-formats repos would stay where they are and the community would still use it the same way and it would be sort of like the ietf ones, would only be the final and draft uh you know final and candidate, sorry, permanent and candidate uh Registries and everything experimental and reserved. Could you know there could be tons of things preserved on the multi-formats side of the fence that don't need?

B

You know people just using the ITF version, wouldn't need to know about or think about um so like it would be sort of a stable nightly distinction. uh Yeah! That's that's where my head is at so far, um but.

C

B

C

They is this just think this emergency for us to get those things merged, that you're working on.

B

Oh, no, uh there's no emergency um in this sense that um I, don't think the working group being accepted or not is influenced by a reviewer clicking through the links and not seeing those PR's merged yet or anything or whatever I think the the it's more just like.

B

Yeah, there's no urgency to to the those PR's. Those are just more like if you don't understand something in those PR's, the logic is probably something about um it gradually, aligning it more with like the ITF versions of things which are sort of like simplified and abstracted a little bit um so like I might be P pring into the readme's, or something paragraphs of explanation that don't make sense it's sort of like, because the the audience would be that there might be new people reading the specs or something or coming to the commute.

B

You know thinking of the visas, like the community version, that's more verbose and has more experimental, Registries and open, PRS and stuff so like this would be the the living version.

C

Okay, yeah um I, haven't been back in my I, haven't, put my head back in there recently and been trying to catch up on stuff after being away.

B

C

And look again this week.

B

Yeah there's none of it's urgent, it's more like just uh if, if the working group gets accepted this week, um there might just be a little bit of uh you know, also YouTube, Rod and Folker. You guys are the um subject matter Experts of the group, if it's accepted, so it might just be a little more email.

B

It might be a little more back and forth on GitHub on, like one more GitHub repo to follow kind of thing, um but mostly the idea is um that if, in terms of like new Registries I think the idea is all Registries would still start. Oh sorry, all registrations would still start on the multi-formats repo and then, when things get to final, they would also be registered over there.

B

You know so, like you can you can sort of point to the additional rigor if you want to say no to something or warn people like this will never get to final, because these are the requirements now for final, are you okay with experimental, and then you can just have a lot of experimental registry registrations that that's my I think that's the only thing that would change for these. These repos, if the ITF work went as fat, you know as fast as it possibly could.

B

C

Well, thanks for that, thanks for the work.

B

Thank you guys, you're doing more of it than I am.

C

Cool I'll give my little update um so I I aside from missing last week's meeting. um Unfortunately, both Volker and I were out. My absence was a um an unplanned absence. I've been away from a nearly a month, I think, um but uh anyway, um some things that have been done. uh There's a new one go up, go ipld, Prime release um this one's a the first tag since I. Think February was the last release, so it takes off a bunch of small fixes, but there's a couple of big things in there.

C

um One of them is the traversal preloader um and it uh it lets you attach on it's it's like a at the moment. The code that we have in there is fairly simple, but we have an extension to it that we're working on in the Lassie code base that may we may migrate more chunks of it. But right now it's it's like a Notifier. When the traverser gets to a block, it can do two passes in the first pass.

C

It can just list up all of the links that are going to be there and then the second pass it'll actually go through and do the traversal. So that gives you the ability to collect links and in a batch and do what you like with them really.

C

But the idea is that you could use that the list, the information about what links are coming up to to to precede a loader, rather than following the the single link by link by link walk that you get out of the um a traversal um so we're using that to really good effect in in Lassie for running traversals and loading via bit, swap because bit swap's a um bitsop is a protocol that doesn't really afford much intelligence um so being able to queue up um loads over bit.

C

Top is actually really um really effective, um so and I'm actually working on that stuff at the moment, as well, trying to figure out what the best way to do parallelism across a large number of fetches is so that's in ability Prime.

C

The next thing: that's in there, that's notable that has got a lot of work is the the match selector for particularly for matching particular slices, so the slice matcha will match a string or a bytes, and you can specify the two in the from um a lot of work has gone into getting that correct, making it work well and then adding in negative ranges too, and the negative ranges work roughly similar to how you can do range selectors on HTTP.

C

um So you can say from the end of the file. um You know you want to select this range from the end um and a lot of that work goes into supporting the trustless Gateway um protocol. That um I think. Maybe if we don't have another talk next week, maybe I'll do a little talk on the trustless Gateway protocol. I.

C

Think it's I think it's really interesting um use of ipld um and that's the other thing I've been doing is just lots of retrieval work in Lassie and also frisbee on the opposite end of that to serve the trust, the Skateway protocol over HTTP um tons of opld in there a couple of things to mention that are on my radar at least um there's two two things in um the JS multi um multi: it's not multi hash, Js, multicolic,.

C

um No JS multi-formats, but it's too early, okay, JS modifiers is the main library where all the IPL these stuff happens in JavaScript um and there's two issues that are active uh one is um Iraqi, has proposed a streaming multi-cash interface, it's just a typescript interface at the moment, um just for doing larger, hashing and also being able to reuse um byte arrays. So you don't have to be always producing binaries, that's a fairly simple proposal. There we're just really talking about naming now um what the method names are.

C

There's also an ongoing discussion about typescript in JS multi formats. um It's a bunch of people that would want it to be typescript and um it's JavaScript at the moment with typescript annotations um and then there's sort of two and two sides of this fence and um it's an ongoing discussion um and and if you scroll down to the bottom, you'll see summaries of pros and cons. um So not that I want more people in there.

C

But if you have something particularly valuable to add to that discussion, then you're welcome to to jump in that's in the JS multi formats, repo. um That's it for me.

A

Do you have any fancy ipld things from the past eight weeks.

D

Yeah, just a bit of an update on the pr I've been working on that, um for everybody else adds support for more efficiently updating ipld nodes. They have traditionally been mutable and um changing a node means mostly copying it over to a new node with updated information which um can lead to some of the inefficiencies. So the pr I've been working on tries to do that a little more efficiently on the posted, some more updates um recently. So anyone would like to chime in on the discussion in the pr or look at the code.

D

That'd be great.

C

um I've got a link in the notes. um It's the guy I believe primary by Paul request number four: four: five.

D

And um yeah I also pulled in a different or like code from a different PR I had that, in addition to adding some efficiencies for updating, it also adds map and list interfaces. So now you can use basic node types lists and maps with in a way that makes it easier for for maps and lists you can use the, for example, puts and gets and stuff like that for maps. That'll work like you would expect for a hash map.

D

um Yeah there are some open questions in the pr about how to deal with uh um contracts and immutability and mutations, and things like that. That would be a good kind of pass out before, uh like the final implementation. But the tests include a lot of examples of how to use the new interfaces.

C

Yeah I'll um I'll I'll jump in there this week, um I did say you've done some more work in.

D

There cool, thank you yep, that's it for me,.

A

Thanks um yeah, so I have to probably quick agenda items related to the standardization work and um yeah multi-format stuff, um so we were discussing about so um there is like multi-bases kind of special compared to the other, like the multi-hash multi, whatever we have M or CID um that the identifiers are really like characters and not like bytes, and there is still this zero byte thing which identifies that the next, the it's really not really a multi-base. It's more like the next thing that comes is then a binary data.

A

It is used in stake keyboard, but we truly just treat it as an artifact of the past and treated out like well. Just be sure that you prefix your CID with this zero button. You will be fine, and so the question is: should we remove the ability for multi-base to have basically the zero prefix, because I'm also not really aware of the use of it? And so basically, so the cover looks also going to be a question of like.

A

Do we take the chance when we move things to the iatf, logistically clean up the things then have bugged us for the past few years and finally take the chance to say well, we just move over the stuff that approved useful and where we think yeah. This is really like the things you should be using and get rid of the things we don't really care about, or have been there even before we were involved, which is the case, for example, for the Civil prefix in my case um yeah.

A

So the question is: should we remove such things, or should we just keep it and drag it over and not like, specifically to the zero byte prefix like brought to you, have any opinion on the like complicated.

C

Yeah I I think yeah. We've had this discussion in the in the other repo didn't we that um yeah I don't I'm not aware of any other use other than the historically accidental use in Derek sibo. um Now I we're going with the language that it's prefix by code point which I think works nicely. So it's not a character. It's a code point um is the null by a code point.

A

C

A

It is so it's the UT like the Unicode, zero, zero, zero, zero code, Point first.

B

Zero of code points.

B

I mean I I think it might there's a there's, a distinction, though, between saying that null is a multibase and having it reserved, which is saying no one can register this as a multi-base right so like in an Iona registry. If you write reserved and just have all the other like columns, blank you're saying this is not an entry in this registry, but it also can never be one so like I think if you say that null is reserved, we will not accept a registration for the null uh code point.

B

That is maybe good enough and you don't even really need to get document too well. Reservations,.

B

um So it might just be simpler to say, and- and you know in the in the current draft, which could still be changed, it's not written in stone or anything um I had like slash reserved because it might be inviting possible. It might be a foot good if people are using slash as a multi-base prefix, uh because it would just you know, like you can you can.

B

You can add a few more reserves if there are other foot guns where it would just be un ergonomic or just dangerous to let people register a character that would just cause confusion or mess up a lazy parser like basically, if any major implementation, if the go implementation or the JavaScript implementation, has some Quirk where it'll blow up. If tomorrow any character becomes a multi-base, you could just reserve it and say like we just don't want anyone to use this like it's.

B

It's fine um so like maybe that's a way out of the identity problem, because right now, identity is a multi-base it's so it like was described as one in the older versions of the spec I. Think the ITF should just refer to it as a non-multi base.

C

Yeah I, I, I I think I would be I'd, be fine with that saying it's reserved and not because the purpose of multi-bases rip is to represent um what do we say, the purposes to represent um or binary values in it um in a sort of a human readable or a non non-binary form like a form. That is the printable characters really um and identity get this sort of ruins that um but I do wonder about other prefixes once we get into this thing of reserving prefixes, because what about all the various special characters?

C

What about the hash? What about the question mark um there's a whole bunch of characters that become problematic where mult, where we tend to use multi-base um like a CID with a question mark at the beginning, might be a problem.

C

um Maybe maybe not I, don't know, I, don't know um so I, don't know how far we would go with reserving these things, and maybe.

A

I I think, like I I, think we shouldn't go too far with reserving, because, like I mean in the end, it's still like we like once it is in the um iena. Hopefully we will have like this. Like someone make the attack the you know what they call like the comedy that decides on those things can still say: no I mean like someone wants to reserve a funny character and we just say no like it's always the always possible.

C

Can they, if there's nothing that precludes it in the document, then does that? Does that cause an issue with these processes where you're saying you're, saying no but you're not saying no based on anything, that's in the spec you're saying no, because you just don't like it.

A

I mean if good point but I mean like, for example like if like. If, for example, like someone comes up and says you want a question mark, it's like oh well, it's often used as kind of like in the in URLs, for example, and then you have the problem with like the question mark I. Think that's like like I would hope. That's a good enough explanation of um for rejecting it. I guess.

C

You know, and and I guess to to the point earlier about how this thing can evolve outside of our implementations. We don't have to implement these things. um Like you, don't have you don't have to implement base 256 Emoji, that's not required, and not even that useful, so um that one's.

B

Not even going to ATF, okay I think that would stay behind as a permanent experimental until it ships as part of something in production, I'm, Gonna, Leave, It, experimental uh but like, but just the the question mark and the hash character are good examples of like um just it's. It's sort of a an exercise in hardening to just think a little antagonistically or like uh imagine some I, don't know: Gray beard, ITF troll, who.

C

B

Opens a bunch of registrations just uh just.

C

D

B

Us to publicly say no to five of them in one day, uh so it might be good just to reserve a few just because, like I, don't know like low hanging fruit, almost like things. That would that you can imagine tripping up an implementation today, like the I, do think like question mark and hash, make perfect sense, because there's an implicit um like accepting that registration would would sort of break the utility of uh multi-bases that end up in um uh you know like sub domain cids or like it just messes up the whole.

B

The whole reason you would multi-base binary is to put it in Uris and URLs, and uh it would just break things to have.

C

um But it opens up in this gray area where you're. So if you start reserving things for those reasons, then what about all the other ones that you have haven't thought to reserve? um Then then you will have those trolls showing up saying well, this one's tricky too, like well I've got a dollar sign here. How's.

A

C

Are you gonna accept that.

C

It's probably easier to go through the ASCII range, because there's not many special characters: Beyond ASCII that are concerned, so it's probably worth pulling up an ASCII table and just um highlighting outside of printable ASCII range is a concern um outside of printable.

C

um Unicode ranges are concerned too, so maybe the spec should say that, but it should be printable printable and then some run some reserved.

A

I I, like I'm I'm, not sure like I I, don't know how many, how many uh rows you would get but, like my hope, would be that, like, like my career thinking, isn't to not overthinking it and just like see how it goes. And then, if it turns out like it's like a huge problem, then but I don't know but perhaps like as I said it might be with the iitf, and it might be a problem that.

B

A

Do this anymore, like that's I I, don't know like that's I'm, not been, maybe nothing to that so.

C

um Yeah, it's it's the I I see. This is the difference between the way we manage things in multi-codic and multi-based versus going into IDF. um It's one of the reasons I sort of avoid those processes is because they're, so upfront heavy get it right up front. Take it up front, then it's locked in stone, that's a much more onerous process than the way we manage things. um Yeah yeah.

A

um But in regards to the um I have to say, like I, I I'm, not sure it was fully correct what I said about the null things to. Of course, the the null bite is a Unicode code point, but the problem is that we explicitly say, like all the other character or all the other multi-based prefixes are always um independent of the encoding.

A

So, like you have, for example, like a like I, don't know like a b, which is then a Duty code code point and then in udfa, it's encoded as this and advice, but we release as a character, but in case of the null byte. It's really like. It's really the null bite. So it's not. It is totally the code point I mean there is a code point with the same bytes, but in case of them all, but it's really the bytes.

A

So therefore, I think we should yeah just reserve it because, like it's just like yeah hard to explain in so much out of scope of the technical specs, so much more difficult, because it's the special case so I think yeah so I'm, also fine with um with reserving it um and also like the Bellion indicated that he would even be open to remove it. So I'm I guess he's also fine, with reserving it and I think that's yeah sounds like a good way to deal with it and.

B

Like if, if you just think about the semantics of what reserved means like reserved means something to the registry, but nothing to the implementation, like you're only obligated to implement the final entries in the registry and the you know, even the draft ones here like you, can be a fully conformed spec. That only does the final ones so, like I, think just calling it Reserve this like saying it will never be final either like it's. It's reserved to never be vital.

B

um So yeah I mean I, think uh I think that's just uh an Ayanna Quirk right like there's a lot of reserved entries in this Diana Registries and you just sort of ignore them. If you're, if you're, just implementing and not registering.

A

Yeah um and the other question is quite similar- you came up today yesterday whenever that about the base two. So basically, there was a. There was.

B

A

Open that we, based to like how we couldn't do page two is not like, doesn't follow the same structure as the other base encoding, so you always use the highest character as the prefix.

A

Another question came up with like do we even need base two because, like it's kind of like an technicality to have base two because, like what's the really useful base, two and the discussion ended up with like again stippling saying he would be also totally fine with just removing it from the spec um and I'll come over to the iitf, so I guess like we will move over only the like. What do we call them today, like the final ones? What do we call them today.

B

They're final and they're sorry, final they're called permanent uh in multi-format, multicodec and they're called final and Ayanna, and and that's a that's another subtle thing like the uh the ietf version of all the spec uses the Iona terminology instead of the one, no one here so like.

A

Oh, like in the current spec, it says it says default is the permanent one.

B

Yeah, uh it's the I, think it's oh.

A

Wow yeah, so it's default in the like I think permanent is the is the multicode one and multi-bases default anyway, but is the like it's currently the plan that we like that for the iitfs thing we only move over the default ones, except for identity.

B

uh uh Default and draft all.

C

Right we'll base two is candidate anyway, so okay, but I'm, fine with removing that too, or switching it to one or whatever the thing about the the base. Two one I find is: um there's educational value in it, but it's not the highly valuable, um that's yeah. It's.

D

C

With base 256, it's the little opposite ends of the spectrum that are useful for demonstrating a point, but not a very strong point. So yeah.

A

Okay, the other thing like then like as we go with like we don't move those to Ayana. If we get the chance, then I will probably make the base two SSA as for educational purpose, but then eventually switch the switch, the switch it to making it a one, prefix, perhaps yeah.

B

A

More consistent, so basically people see like how those things work and then yeah. That sounds good to me. um Okay, there.

C

Was a proposal before uh I think a few a couple of months ago um or where is it?

C

D

Anyway, did somebody somebody.

C

Count with a proposal for a new Emoji based one that actually did some compression using emojis, which was quite interesting um anyway. Let's discussion on one of the repos.

B

But they proposed it straight to a specific implementation.

C

uh I, don't know I think it was, uh it was um just listed in an issue. You should use this encoding and it's here's the way it works, and this is how we can save fights and it.

D

C

Interesting but I'm not compelling enough to to action unless they pushed it forward.

B

C

I, don't even know where it is now. Maybe we closed it out anyway. That's not important.

A

Okay: um okay, those were the two things that just there yeah came up just recently, but yeah so yeah I think we sorted those out. That's great.

B

um And Pro Plan proquint has to be knocked down to experimental because it doesn't have test vectors right.

A

Yes and I think it's also like it's like I, don't think it's even like implemented anyway, I I'm, not aware of augmentation, so I think yeah. It should be yeah, I, think also it should be too experimental or what you call it today. We call it candidate yeah, we should make it experimental. I. Think program makes sense um and I'm just like quickly going over like do. We have any other.

B

Yeah like if another another, just sort of like administrative detail, if it, if the ion, if the working group are accepted at ITF, uh would you guys mind like switching the terminology to be unified between the two, like.

C

That would be highly valuable, I think, regardless.

A

B

I, just didn't want to do it preemptively at that get uh told by ITF to come back next year and then look like a jerk but yeah I think we could do that, um uh because I actually really like the the terminology like um they're on their like third iteration of ionic terminology like they've, been doing this a long time and they've had a lot of controversial Registries and like I kind of like using their new as of 2021.

B

They have like new terms that are that are tidy and there's like an RFC that defines them all, and it's really nice.

A

That's really cool yeah um cool, yet so yeah after a quick look, I think it makes sense to like the ones that are default really are actually used and make sense, and the other ones.

C

um I have to go so yeah I.

A

Might I might say.

C

Goodbye and duck out.

A

C

A

Yeah, but also they actually think like. Does anyone have anything else or are we good, okay, um yeah, so I think we're good. So if anyone wants to stay or has time for the after party feel free to stay in case, there's anything else that shouldn't be discussed publicly, um all right then see you all in four weeks. uh Let me quickly look so the next meeting is then on um September 11th is the next.

C

If, if we don't get someone else um doing a presentation like I'll I'll present something on Preston, Skype, West, spec I think it's interesting for ipld anyway.

A

I gotta go all right, then goodbye everyone and see you, hopefully all in four weeks, cool thanks.