IPFS Helia (IPFS JS) Demo Days, 20 Mar 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: 2023-03-20 Helia (IPFS JS) Demo

Description

The March demo of Helia, a new IPFS implementation written in JS.
Information and source code for Helia: https://github.com/ipfs/helia
Sign up for future sessions: https://lu.ma/helia

A

We're recording.

B

Okay, hi everyone welcome to the Helia demo day for well. This is the delayed here. What's supposed to be the 13th of uh probably March 2023, uh it's now actually gonna be uh the 20th of March 2023. uh you're gonna have to oh, no forget it. I'm, not gonna fix it. I mean part of the fun of a demo day is doing rough edges and bugs and things so, there's a deliberate mistake in the slide I'm about to share.

B

Tada, you know you, can you don't have to tell me what the mistake is because I know what the mistake is. You have to tell everyone else, because you know that'll be boring, so you can just send me a DM anyway, who am I, say hi, so I'm, Alex, uh I'm, making brain on the internet. uh I'm part of the IP stewards team, I maintain just La Petite PHS, ipfs and I am leading the charge on this new version of uh ipfs in JS called Helia.

B

So here's a quick overview. What we're going to talk about um so healer itself, I'm going to just going to.

C

B

Know briefly, give a summation of that a couple of demos and then q a for any chat afterwards. So what is it like? What is it? What is Helia? Why do we need it who's it? For um so it's a replacement for JS ipfs. uh Why do we need to replace it? Because uh when Joseph was born, it was basically supposed to be a one-to-one port of go ipfs now Kubo, uh which turned out to be a horrific mistake, uh because it's enormous and they serve different use cases different audiences.

B

There are different strengths and different weaknesses, so why on Earth, would you try to do you know a one-to-one copy of these things? So what what Helia gives the opportunity to do is to kind of Leverage some strength of JavaScript, and uh you know, and the environments that it runs in and the tooling that it has available um and what we will end up with will hopefully be smaller, faster and it will make everybody more productive and who, who doesn't want that or literally no one. Everyone wants that. Okay. So what is it?

B

We've got a very simple API like if you distilled down what you need out of an ipfs like what is it so uh you need someone to put blocks, you need to be able to put them in it and get them out and then getting them out means like if you didn't have them to start with, you should be able to fetch them from the network.

B

So if you don't have a block in your local Block store, you should use bit swap to get the blocks from your peers.

B

um Blocked The, Block store is where you put the blocks. A lot of the uh things that are built on top of ipfs need so much of a data as well. So you have a data store which is basically a key value database. It is backed by level on node and it can be backed by indexeddb in the browser or you can be in memory uh all of these things. There are many implementations of block stores and data stores an accessible pinning.

B

So if you have a bunch of blocks, um you want to be able to to ensure that you have them at all times.

B

So you need to get a pin, a tag which may involve fetching the rest of the data that you don't have from the network, so you may want to say: I only want to pin this tag several levels deep or whatever, either way you need to be able to assert that you have uh blocks when you think that you have them and then let Peter P so LED P2P is a thing that underpins all of this.

B

uh Well, certainly, all the network operations and that deals with things like finding peers opening protocols, not traversal, all that good stuff um I want to be faster. So you know speed, slowness, fastness, there's it's multifaceted. There are lots of things. uh One one thing is just a sheer size of ill, because um you know loading things is slow, like bundle. Size is a problem in the browser, even in node like the uh climate takes to require or to import, things is non-trivial when you get very large packages, like Json investor, want to be small.

B

Very simple API, like only include the things that you want um and don't include things that you don't want. We want to be able to favor composition, so that means you would have. uh If you, you would have a way of extending Helia with the extra functionality that you need.

B

um So if you were dealing primarily with Unix FS things, you would use that you would use a module that extends Helia to add Unix. First capabilities. If you wanted a different file system like winifest, you would have another module that extends here to do winner. First things, then you might not have Unix FS things, which is great because then you're not including a bunch of redundant coding or bundle.

B

um Why don't talk about today? So uh the the kind of the big thing that's been been I've been working since last time is progress events, so what we really want to do is be able to make um Helia be observable uh from the outside and debuggable like in a way that JSI purpose isn't really. There are a lot of operations that we do in JS ipfs and you don't really know.

B

What's going on a lot of the times, things seem to hang um there's just not a lot of good feedback mechanisms, so you want to build these into Helio from the ground up.

B

um So what I want to talk about today, for the most part, is, is progress, events um and how to use them, how to figure out, let's have to see what's going on uh and and just how to how to make your life more pleasant and make things more predictable uh with by you by looking at the different kind of progress events that you get for different uh operations within helium, so demo time, I'm gonna, oh no I've got like a million things in here, okay right, so this is too big.

B

I will make it smaller.

B

So what we have here is uh it's just helium.

B

So I'm running running the main branch uh I just have a small change in here. So this is a unit test. All it does is as part of the interrupt test.

B

um We have two nodes running, we have a Cubo node and we have a Helia node, um we're going to put a block into the Kubo node and then we're going to try and get it from here.

B

So if I just run the test.

B

Very simple should be able to receive a block.

B

Great the test passes that's great, but what happened like who knows? I mean it's a mystery, so let's see what's actually happening. So if I add a progress, Handler.

B

So it's, this is all all typed, um so I know that my the type that I have is a progress event. So the progress events they have. They extend custom event, which is a standard, JavaScript event type um and there's a type field which is one of these strings and then a detail field, which is uh it depends on the kind of event that you have.

B

So you can disambiguate by doing event type and then let you choose which kind of type that which kind of event type that you you have and from that it can can uh work out what the detail type is.

B

So the detail here is a CID and just kind of run. The same test and all I'm doing is I'm logging, the pipe and then and then the detail, the context of the event. So if I run it again,.

B

I mean hopefully it will still pass. It should still pass.

B

Okay, so now I've got a whole bunch of output. I didn't have before so what's happened here. So if we just look at the test again and remember that we're adding we're adding some input, so this is actually because we're turning on Raw leaves. That means that this is going to end up uh being a raw note. So this is essentially one uh ipld block because it's smaller than the um than the maximum block size. So what we're doing basically is putting a single block into cubo's Block store and then pulling it from here.

B

So what actually happens so within uh The Block store, get we've gone to bit, swap to try and get the block because we don't have it in our in our block store bit swap has done a fine providers.

B

We've found a provider of the block, so this CID here sorry, this pair ID is going to be the pair ID of the Kubo node uh bits or dials that appear it sends the want list and then we receive a block back. So this block event is: is the remote player sending us the block back to us? So it's a block.

B

If the remote player didn't have it none of this, would we would basically stop here like stop at the send wonders um so we've received the block, we put it back in the Block store and now we're able to fetch all right. None of this is really visible before the JS ipfs. You would just get nothing, um it would be. You either count the block or you don't get the block and well you can either get a block or everything would time out. That's it so here we actually have some visibility into into bit.

B

Swap now we can go deeper and we will go deeper, so um find providers uh cause a method on the.

B

Content routing interface and there are different implementations of that interface, so one of them is the DHT. So then there will be DHC queries that are going on in the background and those will start to appear in here as well, but we need to roll these progress events out into the P2P stack at the moment. They just did the IP first level.

B

So that is, that is progress. Events in in The Block store, so we can see other kinds of progress events so, for example, if you're building a you're building a DAC, you can do the same thing. So here we have the Unix FS implementation for Helia. So if you come to any of the other demos, you will recognize this. um It is which is basically a module called.

B

Oh, it's not even written here. Is it uh it's a module called hidea unixfs um you exports a function called uh unixfs, which you would pass, in instance, of Helia to you get a you get an object back, which has has an interstate that you can use to do Unix of s operations, so one of them, you can do is add bytes.

B

So this is very similar to the operation that we just saw. um So if we're we're explicitly adding like a file that is that consists of this one uh one buffer, that's five bytes long!

B

um So if I I do the similar similar kind of thing, so you can do a event. You'll notice, it's the exact same API as you've, just seen with Helio, so I'm just going to do the same operation I'm going to do.

B

Just just log out the event.

B

Detail, I, don't need that and then I'm going to run run the test.

C

C

B

Do you get visibility, I'm.

C

Just looking at.

B

The chat: uh do you get? There's two questions at the end: q. A uh so what's happened here. So we've got some events, so the first event is read so we've read: we've read some input. We've read five bytes that end. There means this is a big end, um so we can handle absolutely enormous uh streams of data without overflowing numbers.

B

um Yes, talk to me about numbers in JavaScript, one time or don't because I get really angry um anyway, so we've read, we've read uh some bites and then we've written some bites. It's important to know that the the two numbers don't always add up. They add up here because we're using uh just a a block- that's small enough to fit into one node. But if we started doing things like adding Unix the best metadata, then the number of bytes to be right becomes bigger than the actual input.

B

Because there's a there's a you know, there's a unit, so that's metadata object, which then he's wrapping in a pro above and all this adds a bit of overhead. So the the written bytes are sometimes different to the red bike and then what happens so a layout event.

B

Something happened. We wrote like uh we wrote this uh block uh in a file layout.

B

um It's not massively obvious right now, because, through our our input is overly simplistic. We've just got a single buffer, um so we've written a single block and then our file at the end of it, is a single block. um This CID uh were you to put into the CID tool. It would tell you that it is a V1 or product CID.

B

um This is not terribly interesting like what we want to see is is like what else can we see from the from the layout? So if I change something if I make the chunker use a fixed size, chunker and I'm going to reduce the chunk size to one byte I.

C

Don't know what it.

B

Is all right, vs code always gets the Imports wrong anyway, so I've changed the chunker that I'm using it's a fixed size. Chunker, it's one byte so now I should get I should get a dag.

B

um What I'm going to do here is I'm going to print out the dag, so EVT I equals layout. Do the layout. Turning now I can get the block from The Block store cards block equals appreciate. So this is a I'm going to do something naughty promise dot resolve because the uh we only have we're on the progress events to be as quick as possible, and if you don't specify them, we don't want anything to slow the progress down process down.

B

So all the progress events are synchronous, um but I'm going to do some async operations, so I need to just pop out into an async context.

B

Equals hour, wait, Block store, get uh EVT detail, CID.

C

B

Obviously, I'm doing a Blog still get on Helia here, so I could pass in an on-progress event.

B

um The real recursion recursive demos I'm not going to do it uh I know what else I need I need to import Star from uh start as uh PB from.

B

Okay, PB Okay cool. So what else we got? We've got a block so I'm going to decode the block. I can't node equals decode block and then I'm just going to print it out.

B

So info node. So what have we got here? So we have changed the chunk size to one like this uh there's five byte block is going to get dunked up into five blocks of one byte and that's going to make us a dag, rather than just having a single a single block, I'm going to stick a little delay in because we've gone async and we don't want the test to end before we stop it.

B

Was the first failed because I am asserting on the CID and obviously we've changed the file layout, so the CID will have changed. But if we look at what we've got, we can now see. We've read uh We've read each individual bite because it's the chunk size that we've defined we've written our blocks for each chunk.

B

You can see that the you know the bytes written are incrementing as we're going along. These are all these are all going to be raw blocks again and then we have a layout event. The CID is different to the one that was the last time and so now, I'm now I'm in here in the layout, then and I've read the block out of the Block store, uh deserialize, the divepv node and just printed it out.

B

So we can see you can see the nodes that we've written there's the Unix of metadata, which tells us what the block sizes are of each link and then we have all the links. So the first link is there: it just sends an Adu and then the next one ends in the first one ends in Adu, FTI, FTI, etc, etc. So we have received a layout event and then we have managed to uh write out things um into the console based on on the event.

B

So now we can see how we're building Dives- and the same is true when we're fetching that.

B

uh Yes, um I've got one more demo, which is the ipns demo, but people want to see it because I do realize we're running over time now.

A

Hey thanks carry on Alex people have.

C

B

A

And taken yeah, but we'll record it and post it. Thank you absolutely.

B

Until you're going to carry on okay, so the final one is Helio ipn, sorry, this is uh similar again, so we have uh I'm an interrupt test. Suite of The Helio IPS package.

B

um uh We create two nodes, like openness is a bit. This is. This is even the DHT to revolve IPS names so just quickly. uh What is ipns right. Well, uh ipfs is all about our musical data right, so you put a block in and you get a CID out or down in you get a CID out and you're like that's great. What so you know what's a practical application of that is something like well I can host my website on ipfs.

B

So what you know what happens then right so I load, my my website to pinning service or whatever I, get a CID I, give it to my mates and they're like What's this, it's a CID. It's my website resolve it and you can see it and they're.

B

Like that's great I love it did you put that spelling mistake on the home page there on purpose and you're like oh, no, no I didn't hang on I'll fix that and then it's fixed try again and they're like no, it's still there and, of course, it's still there because you've, given them a CID which is an immutable appointed some immutable data.

B

So what you need is mutability to bring mutability to the permanent web um and that's where ipns comes in, so you publish use a public key to publish a record that other people can resolve and that that record results to the CID which you can change. So you can update the record with a new version, the CID. So when you fix your typo, the people that you've given instead of giving people a CID, you give them an ipns name instead which they can then resolve to a new CID.

B

When you update fantastic wonderful, how does that work? Well, there are different ways of resolving it. uh Pubs up is one and the DHT is another one, um and so this one this one is the DHT. So what's going on, so we have two nodes. We have a Cubo node and we have a helium node. They are connected um well, at least they know about each other on the network.

B

So we add a, we add, a block to the Kubo node. So this is like the contents of your website.

B

We get a CID back uh and then we use the the Cubase HTTP API to um to import a key, and then we publish uh we published the CID under a name, and so the key is the uh well. The pair ID is, is the ipns name and we use uh Helia to resolve it. So what's going to happen here so right, well and programs, EVT.

B

You know you'll be very familiar with this. By now.

C

C

B

So there we go, we have some events, so we're doing a day, still get uh there's nothing there. So we did a ght query and so you'll notice that we actually see the DHT query. um That has been run. So the query uh looks for the value for the given uh the given key. It's found that from this pair, which is the Kubo peer and here's the value, and so the value there is the ipns name.

B

um So it's the rbns record that's been published, which uh heliopians, 10, uh decodes and and users to fetch the CID, uh and so you can see the CID that's resolved from the record. uh It's asserted and that's the one that that's the one that we're after progress events, um you'll notice, that we actually see the DHT query in this, whereas we didn't see it when we were trying to fetch a block uh via a DNS content. Router. That's because um ipness actually uses a DHT API directly.

B

When you configure your ipns implementation, you can choose the different resolvers, and so they can. They can call directly into the the um the components that they use. So the difference is that when you're fetching, a block um from when bitstop is, is doing a fine providers, query it's actually using the content, routing API and not the DHT API, because there can be different implementations of content routing, one of which is the DHT another. One might be a delegate that kind of thing. So we have we have like we can.

B

Just uh you know we can interpret the the DHD query results directly in ipns, which is why it's trivial to then make those uh make the actual query progress available to to interested parties via the progress event.

B

That is that demo done.

B

Hopefully that kind of gives people a bit of a view into like how you would go about looking at the internals of of Helio like, and what I really hope is that, because all these, um these events are predictable, documented hyped.

B

We can start to see some like really interesting, like visualizations of how things work, because, when you're doing things like discovering other peers on the network and doing these queries and that kind of stuff, I love, it's very, very esoteric and very opaque.

B

um But now you can see everything that will possibly happen during a given operation, which hopefully will let people just be able to experiment more without having to dig into the actual source code.

B

B

B

What's next V1, maybe so you know, uh the idea here is that the apis can be very small, very, very small. um Wouldn't that be lovely, uh so we can actually have a V1 and everything else is experimental and built on top of it. So the API to Helio um is at this point since the uh progress event surrender it makes it pretty much complete um because it doesn't ship with a Unix surface.

B

For example, you would use your own well, you would use the the units, the best implementation, and if you wanted a different unixf implementation, you have the tools to implement. It yourself is what you have is you know a block store or a data store and a little P2P and go nuts. You know so maybe maybe we can ship it demo. Demo time, hey.

A

B

I'm just kind of waiting for this to run so the thing that I was waiting to to work before was uh uh who was the P2P shipping version 0.43, um which gives us like circuit, relay V2 and a bunch of other bug fixes, and that has happened. You know it's happened during during this demo, so this PR just updates uh Helia to use that version of the P2P, and when that is done, then my friends, we can merge this PR, which will release version.

B

One of Helia I was hoping to do in this meeting, but I'm not going to make you all stare at GitHub, CI binning away, that's my job! So uh if you check back in maybe I don't know 15 20 minutes or so we should hopefully be at version. One of Helio, maybe I'll, do a little screen recording of me doing it and we can all we can splice it into this, we'll download it from YouTube and we'll delete the original and we'll just like tack it. On the end, foreign.

B

Got any answers.

A

Well, thanks Alex I I have a couple, but others please feel free to jump in um just going back at the beginning, where you, with the Block store demo, where you were you, would we log out hey? We found this particular P. Sorry peer for Prime providers. Would we have visibility into if multiple peers were found.

B

Yes, absolutely so the idea is yeah, so you would have all the all the DHD queries, so you can actually see what's going on like which pairs you're connecting to like what addresses you're connected to them on like do they respond in time? Is it timing out that kind of thing? The DHC queries are very resilient um because you, you do a whole bunch of queries or different pairs in parallel.

B

um So if you see like a timeout message, it doesn't mean that's it. The query is over um because there are other. There are other queries that are going on at the same time to try and resolve the same bit of information, um so yeah yeah for sure, like making that stuff invisible, is very important. Okay,.

A

And is that there to like, if today, there had been multiple peers that had that CID would that have shown up in the progress events? Yeah, okay, got it cool? uh The other thing I was wondering: what's the API that you use in the interop suite for controlling a Kubo node.

B

uh Just the the HTTP client, so the keyboard HTTP client is using ipfsdctl to spin it up. Okay and then you get an instance of the um the HTTP client, but.

A

God, okay, so that's using the newer Kubo, RPC, client, package, okay and then I guess. I also had the question of uh apps, you demoed us being able to resolve an ipns record in Helia that was published by Kubo. Does the other direction work as well.

B

Of course, of course, it does it's, uh it's not dependent on Cuba; um it is dependent only on other compliant DHT implementations, cool.

B

If you just scroll up a little bit in the test, Suite, that's where the the test happens in the other direction.

D

I also have a question with regards to ipns, so remember, I think it may have been earlier in one of the first demo days. You mentioned um how the keys are managed. The ipn, the keys for ipns I, was wondering. Have you explored already? I know that, right now, if I remember correctly they're stored in the data store, have you explored it or using the crypto API and non-extractable browser keys.

B

No but I mean I'm sure that would be very interesting cool um at the moment because it used the data store so they're all encrypted at rest and and all that kind of thing.

D

How do you mean, oh so, the data store itself actually encrypts the keys when.

B

It's yeah, so there's a yes another data store itself, but the crypto, the Liberty uh crypto, like the Obesity Keys module.

B

um It takes a password which you know you use to derive a key pair from which is then used to encrypt the keys when they go into the data store.

D

So, okay, is that optional, or is that by default it already encrypts them.

B

uh It's it's by default. Everything is oh.

D

And and then the encryption key is also in like uh the index DB in the browser or level DB in the. uh So you.

B

By by default, the password is not secure, um but it's something that you, as the user will configure and ensure is stored somewhere, that isn't the data.

C

A

So I guess uh Alex. What's the recommended, how what's the recommended way for someone to be you passing this uh password around that is used for encryption securely.

B

um It's well I mean it's kind of it's absolutely easy to store it like somewhere. That isn't you know how do you, where do you store your password for your um yeah, exactly no secure in a locked room, um it's.

A

The same kind of problem.

B

A

B

Okay, everyone just ends up using an empty string for the password for their SSH key.

B

Are there any other questions.

A

Looks like you see, I run finished. Did it.

B

Yeah so now, when the main branch, when the main branch, this.

A

One here, oh I, see.

B

uh When the main branch runs, it will then update the pull request.

B

uh Here and then, then we will merge I gotcha.

A

Cool I think thanks Alex, um so do you want to just share a little bit of what's next? Obviously, ipfs thing is coming. Is there going to be another demo between now and itfs thing.

B

Yeah, so uh what's next, is there another demo day between now and IPS.

C

B

There is what I would like to see since we're going to shift V1 is I would like to see what other people are.

B

Building I would like to see some people using Helia and I would like the next demo data to be all about their demos rather than me, waffling on uh and running things in the console, um but since we like, since we won, will ship that means that the API docs will spring into life, um and it means that the the examples and uh or ipvs examples org slash, healer examples um and start using actual version numbers now, which will be very helpful in terms of making everything a bit more stable yeah.

B

But yeah I want to see people building.

C

A

Cool thanks Alex we'll stop the recording here, but uh thanks all for coming appreciate. It.

B

Yeah, thank you very much see you on the internet.