IPFS Launchpad: IPLD, 17 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: IPFS Camp 2019 - From Files to Merkle DAGs

Description

This section on Merkle DAGs from IPFS Camp 2019 goes over theory and live examples of what happens with files added to IPFS. Alan Shaw highlights different variations to the underlying data structure created with a file added to IPFS and how it affects the CID.

This content is created as a part of Protocol Labs' Launchpad program. You can get resources and written instructions in our learning platform at https://curriculum.pl-launchpad.io

Resources Slides as PDF & Course repo: https://github.com/ipfs/camp/#-core--elective-courses--youtube-playlist
Join Protocol Labs Launchpad: https://pl-launchpad.io/
For more information on IPFS: https://ipfs.io
Original video: https://youtu.be/Y_-TWTmF_1I
Sign up to get IPFS news, including releases, ecosystem updates, and community announcements in your inbox, each Tuesday: http://eepurl.com/gL2Pi5

Chapters:
0:00 Start
00:20 Importing Files to IPFS
00:45 DAGs
01:10 Calculating CIDs
02:15 Example: Building a DAG
03:50 Why Vary Chunk Size
06:31 Example: Deduplication
08:00 UnixFS
09:00 Example: UnixFS
12:09 Performance Characteristics of DAG Layouts
13:19 Example: DAG Layouts
15:20 Closing Q&A

A

Okay, let's get into some deep diving into how important files works and I give us.

A

There are a few people here, but I'm sure they'll Shuffle in all right, uh let's get started what actually happens when I use ipfs ad.

A

My file here is a tube of bits, and what actually happens is that those bits get divided up into chunks chunk one chunk two and chunk three uh I'll be referring to them later. So, just if you see C1 C2 C3 this, these are the bits of the file, the chunks of the file uh that get chunked up. So these chunks are arranged into a tree structure which is a dag, and so we've already talked about like items in the graph are called nodes. Not every node needs to contain a chunk of the file.

A

Some of these nodes are intermediate nodes. They could contain chunks of files, they don't always in this example. They don't so yeah. We've we've got. uh We've got the chunks in the these three chunks in the leaves of this dag that has been created.

A

When building the graph, we already talked a little bit about this, um but we calculate a CID for every node and that happens from the bottom up, and that happens because we can't calculate the CID for the parent until we notice the IDS for the children, because the CID is calculated from the data as well as any links it has so the links.

A

If we don't have the cids for the links, we can't calculate the CID for the node, and so we we have to start at the bottom here and and come back up and at the top. We can't do the top one because we don't have the one below it. So we have to Traverse all the way down to the to the leaf and then slowly make our way back up again uh and we finally get to the root CID, qm6 uh and so for a single file.

A

This is the CID that is returned to you and you can think of this CID uh as the CID for your file, but actually it's the CID of a node at the top of a tree which makes up your file.

A

Okay. So, let's see it in action, you can try this out in your browsers as well uh dag.ipfest.io all right. So here we go, you should end up. You should see something that looks like this. This is a tool that runs JS ipfs in your browser. If you drop a file onto it, it will um draw the dag that gets created.

A

um So what we can do is we can drop a file on.

A

Yeah right we can drop a fire alarm. This is the readme from the go ipfs repo and it will render my dag for me and I can hover over nodes and I can see that on the bottom here, we've got a bar which tells us information about each node, that's been created in the deck and each, and we can see that each node has a difference. The ID in there they're put here um that they're displayed here um and then I can change. I can change things like the chunk size.

A

I can at the moment we're we're. Creating 512 byte chunks, but I can I can change that to 1024 and I get a graph with far fewer nodes, because each chunk is able to have much more data in it.

A

I can push that up to get an even smaller graph and go all the way. This is the default chunk size in ivpfs I just get a single node for this small file.

A

There we go cool so back back to the slides. All right. We've already talked a little bit about this, but why? Why vary the chunk size? So the default in itfs at the moment is, whenever you add, a file just to chunk it up in fixed chunk sizes.

A

um We have a default for that and we've just been changing that a little bit um but like smaller chunks, uh potentially because I got fewer bytes in them. It we've got a better chance of deduping them, um but it like, like Michael's, already said uh it's more work up front to create all of those chunks you have to. You have to create hash the data and uh for the CID for every single chunk and the more chunks. You have that's more work up front.

A

uh If we have larger chunk files, then that's less work up front but uh easier to like this fewer nodes to Traverse when you're transferring the data for your notes to transfer um and uh yeah, but uh it's not so necessarily so great at. Did you pick um smart chunk sizes? On the other hand, now that's interesting one so, like maybe you have a chunker that chunks at keyframes.

A

So you can have better seeking in videos how cool would that be, or it might be, a chunker that does like, like very uh like cleverly chunks, the file, so that um so that, if you add data in the middle, it doesn't throw off all of your. So the problem with like fixed chunk sizes is that if you add a bit of data in the middle, then everything like it froze off the rest of the chunks and those.

A

So those chunks are now different, and so the deduping is not necessarily as good if you use a Smart chunker, which will uh take that into account. Maybe give you a chunk, that's a little bit bigger for that data. Then you can keep the same chunks that you had before for the rest of the file and so uh yeah. So we have a smart chunker already it's called Robin.

A

You can use it in uh in JS in the browser um which is brand new by the way, it's sort of like a little wasm thing, uh which is super cool, and it seems it's in that dag uh that dag Vision as well. So you can check that out um uh so yeah smart chunk size is really interesting, but again it's more work up front to do that. Work of kind of figuring out, where is the appropriate, bound chunk boundaries? So that's always content specific.

A

You have to know what type of data is well, so Robin doesn't know. Okay, uh Raven just takes a stream of bites and figures it out. It's very it's very clever. Okay, it is crazy, old arson. Magic, it's been around for a long time. It's not it's, not ipfs, specific.

A

Okay. So back to the example, let's look at de-tubing in the visualizer all right. What I've got here is a I've copied the readme and what I've done is I've just added some data to the end of it. So it's a little tiny bit bigger and what I can do is I can drag and drop that on on here and what the visualizer tool does is. It adds a directory for these two files. So these these two nodes, directly descending from the directory, are the two files that I've added the roots of the two files.

A

I've added basically I mean you can see from the lines in this graph. They actually share almost all of the nodes, apart from so right at the end here, so we can pick these up and drag them. We can see that there's two little chunks here and here, which are the only two chunks that are not shared by uh by these files, which is kind of cool, and you can see that this this chunk.

A

uh If you look at the um oh wow the bite size here, um it's this chunk has slightly more bytes than than this. One, and what's happened here, is that we haven't I, haven't managed to get it so it like it's it's on a complete boundary, so there will be some shared data in these two in these two chunks, but one of them has that extra bit of data that I've added to the file so yeah.

A

This is this is the duping, and this is how the two two very similar files on disk can uh don't actually take up. Two-Time storage space in iqfs.

A

Right, okay, so actually I've not told you the whole truth about this. The nodes in the graph are not just made up of the chunk of the file and file data. Every chunk is wrapped in this thing that we call the Unix FS wrapper the Unix FS wrappers are: they can either be files, they can be directories. There are other types, but these these two are the main main two and it allows to allows us to distinguish between what is file data and what is a directory.

A

uh So it's kind of a Dream Within a dream: uh it's not just that. It's not just the data, it's a rapper and that is that is becomes the data. uh So what that does? Is it actually adds like a few bites of overhead to every node, uh because we've got the Unix FS wrapper around each bit of data? So uh let's take a quick look at that in in here.

A

Okay, so we can see in the bottom left hand corner if you hover over a nose. The the Unix FS type that that node has and this and this one at the top as I said the.

A

What, if the what the tool does is it puts two files in a directory, so this is a Unix FS directory and these other nodes are all kind of Unix FS file uh nodes, and so we can see the total bites that make up this node and the bytes of data in this node are slightly different, and that's because there is this Unix FS wrapper, there's a few bytes of wrapping data that that is in every single node.

A

What we can do is we can change. uh We can change uh the leaves to using raw leaves and what that does is. Instead of adding the Unix FS wrapper to the to the leaves it will not do that and just uh it will just add a raw buffer of data as the leaf. You can see that the bytes total for these nodes are exactly the same as the bytes of the data.

A

So we don't have this wrapper anymore, and that means that we save a little bit of space on every node we create uh because we don't we're not wrapping it with with any information.

A

It also means that we get a V1 CID. So you can see here. This is this: is a v1c ID, so these ones are like start with qm, that's a v0. These ones start with well bath K in this. But in this case, why do they? Why is the CID different confusing? So CID is different because I know where the hash is different, but why is it a different format?

A

Cidv zeros, as we talked about earlier, they have implicit uh things about them, one of those things that is implicit that is not written. That means it's. A v0cid is the ipld format and the iPad format for these nodes is uh dag PB for the raw nodes. So the iprd format, if you remember, tells us how to decode the data. So we know that that what this data is it's it's a it's a protobuf encoded chunk of data decode it with using, like you know as it as if it were a proton.

A

So this this is just raw bites, and so this has the IPL D format, ipld Raw.

A

um So, like I said this is now. This is now a V1 CID, because it's using a different format and it we literally it cannot be a v zero CID, because it's not a DAC pbco um encoded data uh cool. So that's that.

A

All right, there's also different graph layouts that have different performance characteristics. Balance layout is the default.

A

You get when your ipfs add um it's really simple: to build um and really easy to Traverse, but it's kind of difficult to edit, if you were to change in the middle you'd, have to do a lot of rebalancing to um to kind of balance it out again, uh whereas like we've got trickle as well, is an option it's more difficult to build, but um it's really great for like streaming, for example, because the time to First byte is a lot less like you can imagine like if I, if I have my data initially then to get it's not streaming, I need to get hold of this node and then I need to get out of this.

A

Node and I can start streaming where it's like imagine. I've got a big file like a big movie file and it's made a big, balanced Dag, and if I want to start streaming, it then I'm going to have to get this now to get this node. Potentially this no distance in this node before it can start streaming. So it's time to First byte is going to be much better with a kind of trickle style, Dag cool all right.

A

So let's take a look at the different, the two different types of uh of dag layout that we have available.

A

A

This is the balance layout, and this is what we've been looking at so far. But if we can use this uh drop down here and we can switch to a trick or dag and you can see that the layout has completely changed and it's kind of it makes a really interesting structure. uh If you add like a bigger, bigger file on it and.

A

The one thing to notice is that this CID up here, if we check it out, it's a qm, it kind of enters CW ESU. If I, if I change the to balance tag, start securing doesn't start, doesn't end with the same same characters, and- and this is the same data if we've not changed the data we've, all we've done is change the options here. Actually, if you change any any of these options, then you'll get a different CID. So this there's something to note there uh or take take it off.

A

At least uh you do get different cids. If you change the different options, um because the the the the reason is that the it may have more or less children and those children will have different will have different cids because they have more or less children, so it basically propagates all the way up to the top, and you just get a different cids for the same data.

A

Now, if you've used the same chunking algorithm, then your leaves will probably still have the same same data that can be dedup, but it will just be the intermediate nodes that have different CI differences.

A

Question yeah I was you said that there was the same data, but technically all of the blocks include links which I think you mentioned, which causes the hash to change and that's the main reason. Yes, because exactly the link structures, yeah, yeah, so yeah so I say it's the same data in that it's the same file data that was added to the DAC, but the intermediate nodes. They are the things they are the things that change that are causing um the the the root CID to have changed, yeah you're right.

A

So, but if I understood it correctly, the actual chunks will stay the same. So you still have a lot of deduping options. Yes, yes, so you can imagine that not if you change the chunk size right or the channel, if you just change the diagram yeah, so you can imagine a service that takes data that was added using a balanced diet, restructuring it yeah sort of streaming purposes. Yes, that would be fairly cheap, yeah, yeah, exactly yeah yeah, that's interesting!

A

All right cool.

A

All right, foreign.