IPFS Launchpad: IPLD, 13 Jul 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: How IPFS Deals w Files (v2)

Description

This course from IPFS Camp 2019 is a deep exploration of the reasons behind immutable data, how we address immutable data, the data structures IPFS creates, and the different ways of interacting with files in IPFS.

Resources
Slides as PDF: https://github.com/ipfs/camp/blob/mas...
Course repo: https://github.com/ipfs/camp/tree/mas...
Join Protocol Labs Launchpad: https://pl-launchpad.io/
For more information on IPFS https://ipfs.io

Sign up to get IPFS news, including releases, ecosystem updates, and community announcements in your inbox, each Tuesday: http://eepurl.com/gL2Pi5

Chapters
0:00 Start
00:22 Why Immutablilty
05:22 Anatomy of a CID
07:00 Multihash
09:56 CID Version
11:29 Multibase
12:46 Example: Anatomy of a CID
16:12 Version 0
19:10 Importing Files to IPFS
19:51 DAGs
20:20 Calculating CIDs
21:22 Example: Building a DAG
22:56 Why Vary Chunk Size
25:37 Example: Deduplication
27:10 UnixFS
28:12 Example: UnixFS
31:17 Performance Characteristics of DAG Layouts
32:27 Example: DAG Layouts
34:26 Closing Q&A

A

This is the uh uh welcome it's very nice to see your faces, I hope you're, not too hungover or tired or whatever um I am alan. uh This is michael and there's stephen and back there and we're gonna talk to you about how ipfs deals with files part one of five. It's all right, they're exciting. I think.

A

Well, that's been the feedback so far, all right! First up! Okay, uh can anyone tell me what the meaning of the word immutable is?

A

Can't change all right, awesome unable to be changed as opposed to mutable, which is liable to be changed, so we're going to talk about why ipfs has a big focus on immutability and immutable data in a peer-to-peer network, the one on the right, uh there's, no central point of authority, no one to implicitly trust, uh but in the web of today we uh we implicitly trust kind of things like institutions, so certificate authorities, social media government, but that trust is easily subverted and we can't necessarily rely on the data we receive to be the data we ask for, even though we often do resource integrity.

A

Tekken is one way to ensure that data you ask for is the data you've got? uh The idea is to calculate an identifier for the data from the data itself. The identifier is called a hash uh and we use cryptography when computing the hash to ensure certain properties uh like uniqueness and determinism. uh But we're gonna talk a little bit more about that in the next section.

A

So once we've got this hash uh that we can share it with the rest of the world, these two people being the rest of the world, and when someone gets hold of that data they also calculate the hash and then they check that they match uh and if they do then, yes, you've got what the data you ask for winning.

A

If someone else gets hold of the data and the hashes don't match, then uh-oh someone's tempered with your data, so the guarantee of interrogating checking integrity checking is that content is immutable. If the data changes, the hash I generate no longer matches, so verifiability is one of the main reasons for using immutable data in the web of today I can put my poodle picture up at abc.com, poodle.js and 24 hours later. I can doodle on my poodle.

A

So let's say I add some glasses make him look a little bit more sophisticated, um so the the problem is that the content isn't inherently tied to the address. The content can change, but the url doesn't so depending on when I access this poodle, it could be completely different.

A

So what we have on the web today is location addressing it tells us where the data is stored, but not much else content and addressing, on the other hand, is where we use a hash to access the content and allows us to verify. The content we receive is the content that we asked for.

A

Okay, caching and deduping and mutable content completely solves the caching problem like that, since the data is never going to change, the cache rules are cache this forever.

A

So back to our example, if we were using content, addressing our poodle has a specific address derived from its content, the hash, and if I change that uh that poodle 24 hours later the address also changes, but that's okay, it could be verified, cached and fetched by anyone.

A

What if I wanted to keep both files? Well, uh because the content is immutable, ipfs only really really needs to store the changes, uh the differences, the two files that we see actually share many of the same bytes. So if I have one of those files, the amount of data I have to transfer to fetch, the other is really minimal.

A

But currently, if I want to store those two very similar pictures, I need to store both of them in full on my disk and if I want to share them with someone and they want both- and I want them to see both of those pictures, then I have to transfer them both in full um and we'll see more about how ipfs does that magic do duping stuff later, okay effect from anyone.

A

uh If the content is immutable and I can verify its integrity, I should just be able to get it from anyone, and this suits p2p really really well, but that's not true of the web today. uh So I have two sites on the on the net hosting the same content. How do I know which of these poodles is the correct poodle?

A

Well, okay, so the answer is that we can't trust that any of these poodles are correct. They are both adorable poodles, uh but uh we just can't get it from anyone. It needs to come from a just trusted source um because I can't verify its integrity all right, so we use content addressing in ipfs and we use a special hash called a cid. Cid stands for content identifier and we're going to take a closer look now in part two of five okay.

A

uh It all starts with cryptographic, hash, um a cryptographic, hash function, maps, input of arbitrary size to output of a fixed size, and we want a few properties from that hash. The same data should always produce the same hash, so it's deterministic.

A

It should be impossible to invert. We, as in we shouldn't, be able to reconstruct the data from the hash. It should also be unique, uh so no two different files should produce the same.

A

There are many different hashing algorithms that exist, uh ipfs uses sha-2256 by default, and that is just some of the some of them that are available in ipfs that we could use older algorithms like charwan, are broken, they're proven to not be collision free, and if and the problem is that, if algorithms can break we're going to want to switch that hash that we use by default in the future.

A

The problem with switching algorithms is that, given a hash, which is just a series of bits, what algorithm did we use to generate that hash?

A

We need a future-proof way of of identifying the hash function used to generate a hash, as well as the hash layer say hello to multihash it'll solve all your problems, uh so multihash is uh is the hash which is just at the end there, but it's also a prefix, and that prefix is one number algo algorithm and which identifies the hash algorithm that was used to generate the hash and another number, uh which is the hash length.

A

These two numbers are both varens and varin is just a compact encoding for integers. The algo number is a multi-codec, because a is a variant but b it's uh it's. It's value is a predefined value that we agree on it's in the in a table on the internet.

A

The multi-code identifier for sha256 is the number 18 and our hash length is 256.

A

in binary. It looks like this you'll see that the hash length is actually 2 bytes long and that's because the length is over 127 and in far in encoding numbers. Above one, two seven uh are encoded as two bytes, so you can kind of think of this as like two times. One two eight is two five six, but that's not really how variant works. uh That's just a coincidence, but yeah two bytes, because it's over one, two seven.

A

We want to support multiple encodings of data, so you've got a hash that um that addresses your data. When you get the data back, um how do you know how it's encoded like we might? It might be encoded in cbor, uh concise, binary, object, representation. It might be encoded in protocol buffers, it might be encoded as just plain old json. um Why do we want this? Well, it might be a particularly compact binary encoding, so really efficient for storage, um but it might be for convenience might be just easy.

A

It might be really quick, um but we have this same problem again like when we're looking at some encoded data. How do we know how to decode it same solution, more metadata prefix?

A

So this is an ipld, codec and ipfs at the moment, any any um any content that you add gets uh is the codec we use to encode. The data is tag pb um and it is uh pv, stands for protocol buffers, so it's just um pro call buffers encoded um yeah and it's like I said it's compact encoding for the data and we've added some uh metadata to the start of the the hash.

A

Are there other codecs correctly yeah? There are many ipld formats that you can use. um There are too many formats all right, so there are two different cid versions, so um that's mildly, frustrating, but uh so often nowaday or hopefully going into the future you'll see version one as the main version that we're using uh there's version zero and version one it's just another prefix and that we add on to the front.

A

So all of you, if you've ever used ipfs before you might be used to seeing cids like this um version. Zero is a this top one and the version one one is the bottom one um uh and the so the version, zero ones begin with kind of qm, usually version uh version, zero begins with qm version, one begins with baffy or fk, sometimes um all right. So um the thing is cids are binary.

A

Data um zeros and ones make for really long strings when they're printed out like it, doesn't even fit on the slot on the slide. So what we do is uh we use a higher number base encoding um to make them shorter and easier to like print out and recognize, even when in string form, we have another problem again. How do we interpret the the number base encoding that this string was crea created with um characters in one encoding are valid in another encoding?

A

So we can't just look at the string and be like that's that's a base 32 or that's a base 16 or whatever. um So how do we? How do we fix that? Well, you guessed it more meta prefix.

A

This is multi-base and multi-base helps us know the number base. The content is encoded with the first symbol is the identifier for the encoding b is base, 32, z or z is base. 58, btc and m is base 64.

A

version. 0 cids, like the qm one you saw in the last slide, are also base 58 encoded and I just told you that base 58 should be prefixed with z but they're. Not if you're wondering about that, then don't worry. We will talk about it in a minute, okay, so the rest of the string after the first first symbol is the rest of the cid uh encoded in that particular base.

A

So altogether for a cid. This is what it looks like in binary form. It's the cid version. It's the ipld format multi-codec and it's the multi-action. The multi-hash is split up in, as you know, because we discovered it into the hashing function. You use the hash length and the hash as as a string, we have the um the multi-base encoding and that encoding applied to the rest of the cid.

A

Cool all right, so you can try this as well. uh If you fire up your browser and go to cid.ipfs.io, and if you have a cid happen, they'll happen to have one lying around, then you can paste it in to that tool and it will pick out these properties that we've just been talking about of the cid and show you them there's a tool in ipfs. If you've got it installed on the command line called ipfscid and that'll. Allow you to change the number base or the version number of a cid.

A

uh If you type ipvs cid dash dash help, it will tell you how to use it, and so, if you want to, if you've got like a v zero c, I don't want to convert it to v one and put that in or whatever, but I'm gonna. So I'm gonna open this up now and show you a few pre-prepared some uh cds I prepared earlier.

A

All right, so this first cid I have.

A

Get back to the start of string. This is a baffy cid. This is a base. 32 encoded cid. uh Oh hang on. Let me make this a little bit bigger for you version one cid, it's encoded with the ipld format called dag pb uh we've got the multi-hash at the end, which is uh sharp the name of the multi-hash. The name of the hashing algorithm is shar2256, the length is 256, and that is the um the actual hash.

A

So multivase, as as I was saying, is its code is b start of the string is b. So that's good that matches up base 32 degree for the multi hash to cut like the code for char 2 2 5 6 is 18, as you saw on the slides so yeah. That's that's one. I've got a base. 64 encoded cib here. If I paste that in, if you remember, I said that basically before cid start with m and they do according to this- it's all good. So this is another cid v1 uh and yeah base64.

A

I've also got one here with a different ipl d format. Oh whoops, sorry.

A

So this is a v1 cid encoding with base 58 btc, and this is interesting because it's using a dag, cbo ipld format, not the dag pb, which is the double v, is the default for adding files in ipfs. But you can use the ipfs dag api to to create structures that have different encodings and this one's good for, like json data.

A

There we go okay, and so finally, the one you've probably all been waiting for what is going on here, all right qm. This is a v0cid, as you can see here, qm begins with qm, uh it's encoding, the base58.

A

What have we got here? Multi-Base code, implicit multi-codec code, implicit.

A

What should we should we talk about that?

A

Yeah right, uh cool, that's good, because I've got slides prepared for it all right. In the beginning there was only multi-hash and they were encoded as base 58 btc.

A

All version 0 cids have no multibase, and so that's why they don't start with z. Why do they start with qm, though? Well? They are just multi hashes uh and they're, just multi hashes encoded with base 58.

A

So if you remember uh back to the slides about multi, hashes, multi hashes are the hashing function, plus the hash length, plus the hash, and those two things at the start of the of the multi-hash are are kind of fixed, usually in ipfs, so, which is why you see a lot of these and and it's just what it is, these q, this q m is just those two numbers encoded in base 58 the multi base is always phase 58 btc, like I said, um and so what else do these not have?

A

If version zero cids are just multi-hashes? What do they not have other than not having a multi-base.

A

Anyone yeah, they don't have a format, they don't have a cid version.

A

So all together for v0s, this is way simpler. It's just they're just multi-hatches and in string form their base, 58 encoded, multi hashes, which is why they start with qm.

A

You can kind of think of them like this, so the multivase is always base. 58 btz ptc is zed. It's implicit, it's not written. Cid version is always zero. It's implicit, it's not written. The ipld format is always dag. Pb, it's implicit, it's not written. The hash function is always sha2256.

A

It's it's explicit, not implicit. It was a trick question it's in the multi-hash.

A

So the hash function is in the multi-hash it is in is, is there the the hash length is also explicit it's in the multi-hash and if any of those variables change, then you don't have a v0 cid anymore.

A

All right cool so now you know how itfs addresses content um using cids and hashes, and we can. We can now learn about the data structures that ipfs actually addresses.

A

Okay, let's get into some deep diving into how important files works in ips. There are fewer people here, but I'm sure they'll shuffle in all right. uh Let's get started. What actually happens when I use ipfs ad.

A

My file here is a tube of bits, and what actually happens is that those bits get divided up into chunks chunk one chunk two and chunk three uh I'll be referring to them later. So, just if you see c1 c2 c3 this, these are the bits of the file, the chunks of the file uh that get chunked up.

A

So these chunks are arranged into a tree structure uh which is a dag, and so we've already talked about like items in the graphical nodes. Not every node needs to contain a chunk of the file. Some of these nodes are intermediate nodes. They could contain chunks of files, they don't always in this example. They don't uh so yeah. We've we've got. uh We've got the chunks in the these three trunks in the leaves of this dag that has been created.

A

When building the graph, we already talked a little bit about this, but we calculate a cid for every node and that happens from the bottom up, and that happens because we can't calculate the cid for the parent until we know the cids for the children, because the cid is calculated from the data as well as any links it has.

A

So if the links, if we don't have the cids for the links, we can't calculate the cid for the node, and so we we have to start at the bottom here and and come back up and at the top. We can't do the top one because we don't have the one below it. So we have to traverse all the way down to the to the leaf and then slowly make our way back up again uh and we finally get to the root cid, qm, six uh and so for a single file.

A

This is the cid that is returned to you and you can think of this cid uh as the cid for your file, but actually it's the cid uh of a node at the top of a tree which makes up your file okay. So, let's see it in action, you can try this out in your browsers as well uh dag.ipfs.io all right. So here we go, you should end up. You should see something that looks like this. This is a tool that runs js ipfs in your browser.

A

If you drop a file onto it, it will um draw the dag that gets created.

A

So what we can do is we can drop a file on.

A

Right we can drop a file on this is the readme from the go ipfs repo and it will render my dag for me and I can hover over nodes, and I can see that on the bottom here, we've got a bar which tells us information about each node that has been created in the dag and each, and we can see that each node has a different cid in there. They're put here um they're displayed here um and then I can change.

A

I can change things like the chunk size at the moment, we're creating 512 byte chunks, but I can. I can change that to 1.024 and I get a graph with far fewer nodes, because each chunk is able to have much more data in it.

A

I can push that up to get an even smaller graph and go away. This is the default chunk size in ipfs. I just get a single node for this small file.

A

There we go cool so back back to the slides.

A

All right, we talked- we've already talked a little bit about this, but why? Why vary the chunk size? um So the default in ipfs at the moment is, whenever you add, a file just to chunk it up in fixed chunk sizes.

A

um We have a default for that and we've just been changing that a little bit um but like smaller chunks, potentially because I've got fewer bytes in them. We've got a better chance of duping them, um but it like, uh like michael's, already said uh it's more work up front to create all of those chunks you have to. You have to create hash the data and for the cid for every single chunk and the more chunks. You have that's more work up front.

A

uh If we have larger junk files, then that's less work up front but uh easier to like this fewer notes to traverse when you're transferring the data, fewer notes to transfer um and uh yeah, but uh it's not so necessarily so great at deduping, um smart chunk sizes, on the other hand, now that's interesting one so, like maybe you have a chunker that chunks at keyframes.

A

So you can have better seeking and videos how cool would that be, or it might be, a chunker that does like, like very uh like cleverly chunks, the file, so that um so that, if you add data in the middle, it doesn't throw off all of your. So the problem with like fixed chunk sizes is that if you add a bit of data in the middle, then everything like it throws off the rest of the chunks and those.

A

So those chunks are now different, uh and so the deduping is not necessarily as good if you use a smart chunker which will uh take that into account. Maybe give you a chunk, that's a little bit bigger for that data. Then you can keep the same chunk chunks that you had before for the rest of the file and so uh yeah. So we have a smart chunker already uh it's called robin. uh You can use it in uh in js in the browser um which is brand new by the way.

A

It's like a little wasm thing, uh which is super cool, and it seems it's in that dag uh that dag vision as well. So you can check that out um uh so yeah smart trunk size is really interesting, but again it's more work up front to do that. Work of kind of figuring out, where is the appropriate, bound chunk boundaries? So that's always content specific. You have to know what type the data is. Well, so robin doesn't know. Okay, uh robin just takes a stream of bytes and figures it out. It's very it's.

A

It's been around for a long time. It's not it's not ipfs specific.

A

Okay, so back to the example, let's look at the duping in the visualizer all right. What I've got here is a I've copied the readme, and what I've done is I've just added some data to the end of it. So it's a little tiny bit bigger and what I can do is I can drag and drop that on on here and what the visualizer tool does is. It adds a directory for these two files.

A

So these these two nodes, directly descending from uh the directory, are the two files that I've added the roots of the two files. I've added basically- and you can see from the lines in this graph- they actually share almost all of the nodes, apart from so right at the end here, so we can pick these up and drag them.

A

We can see that there's two little chunks here and here, which are the only two chunks that are not shared by uh by these files, which is kind of cool, and you can see that this this chunk. uh If you look at the um now the bite size here, um it's this trunk has slightly more bites than than this one.

A

And what's happened here is that we haven't, I haven't managed to get it so that it it like it's it's on a complete boundary, so there will be some shared data in these two in these two chunks, uh but one of them has that extra bit of data that I've added to the file so yeah. This is this is deduping, and this is how the two two very similar files on disk can don't actually take up two times: storage space in the ipfs.

A

Right, okay, so actually I've not told you the whole truth about this. The nodes in the graph are not just made up of the chunk of the file and file data. Every chunk is wrapped in this thing that we call the unix fs wrapper and unix fs wrappers uh are? They can either be files? There can be directories, there are other types, but these these two are the main main two and it allows to allows us to distinguish between what is file data and what is a directory.

A

uh So it's kind of a dream within a dream: uh it's not just that! It's not just the data, it's a wrapper, and that is that it becomes the data.

A

So what that does is actually adds like a few bytes of overhead to every node, uh because we've got the unix fs wrapper around each bit of data. So let's take a quick look at that in in here.

A

Okay, so we can see in the bottom left-hand corner if you hover over a node, the the unix fs type that that node has- and this this one at the top.

A

As I said the what it that what the tool does is, it puts two files in a directory, so this is a unix fs directory uh and these other nodes are all kind of unix fs file uh nodes, and so we can see the total bytes that make up this node and the bytes of data in this node are slightly different, and that's because there is this unix fs wrapper there's a few bytes of wrapping data that is in every single node.

A

What we can do is we can change. uh We can change the leaves to using raw leads and what that does is. Instead of adding the unix fs wrapper to the to the leaves it will not do that and just uh it will just add a raw buffer of data as the leaf. You can see that the bytes total for these nodes are exactly the same as the bytes of the data.

A

So we don't have this wrapper anymore, and that means that we save a little bit of space on every node we create because we don't we're not wrapping it with any information.

A

It also means that we get a v1 cid. So you can see here. This is this: is a v1 cid, so these ones are like start with qm, that's a v0. These ones start with well bath k in this play in this case.

A

Why do they? Why is the cid different confusing? So the cid is different, because I know where the hash is different, but why is it a different format?

A

Cidv0S, as we talked about earlier, they have implicit things about them, one of those things that is implicit that is not written. That means it's. A v0cid is the ipld format and the ipld format for these nodes is uh dag pb for the raw nodes.

A

So the iprd format, if you remember, tells us how to decode the data. So we know that that what this data is it's it's a it's a protobuf, encoded chunk of data and decode it um with uh using, like you know, as if it were a protocol. So this this is just raw bytes, and so this has the ipld format ipld raw.

A

um So, like I said this is now. This is now a v1 cid, because it's using a different format and it we literally it cannot be a v0 cid, because it's not a dac pbc encoded data uh cool. So that's that.

A

All right, there's also different graph layouts that have different performance characteristics.

A

Balance layout is the default you get when you ipfs add um it's really simple: to build um and really easy to traverse, but it's kind of difficult to edit, if you were to change in the middle you'd, have to do a lot of rebalancing to um to kind of balance it out again, whereas like we've got trickle as well, is an option it's more difficult to build, but um it's really great for like streaming, for example, because the time to first bite is a lot less like you can imagine like.

A

If I, if I have my data in this leaf, then to get to start streaming, I need to get hold of this node and then I need to get hold of this node and I can start streaming it's like imagine. I've got a big file like a big movie file and it's made a big, balanced, dag, and if I want to start streaming it then I'm going to have to get this node this node. Potentially this node is in this node.

A

Before I can start streaming says time to first bite is going to be much better with a kind of trickle style tag.

A

All right, so, let's take a look at the different. The two different types of.

A

Of dag layout that we have available.

A

A

This is the balance layout. This is what we've been looking at so far, but if we can use this uh drop down here and we can switch to a trickle, dag and you can see that the layout has completely changed, it's kind of it makes a really interesting structure. If you add like a bigger bigger file on it,.

A

But one thing to notice is that this cid up here, if we check it out, it's a qm, it kind of ends with cw esu. If I, if I change the to a balance, dag it starts with qm doesn't start doesn't end with the same same characters, um and this is the same data.

A

If we've not changed the data, all we've done is changed the options here. Actually, if you change any any of these options, then you'll get a different cid. So this there's something to know there uh or take takedo. At least uh you do get different cids.

A

If you change the different options, um because the the the reason is that the it may have more or less children and those children will have different will have different cods because they have more or less children, so it basically propagates all the way up to the top, and you just get a different cid for the same data. Now, if you've used the same chunking algorithm, then your leaves will probably still have the same same data that can be deduped, but it would just be the intermediate nodes that have different differences.

A

uh To this question yeah I was, you said that it was the same data but technically all of the blocks include links which I think you mentioned, which causes the hash to change and that's the main reason. Yes, because exactly the link structure, yeah yeah, so yeah. So I say it's the same data in that it's the same file data that was added to the dag, but the intermediate nodes.

A

They are the thing they are, the things that are changed, they're causing and the the the root cid to have changed. Yeah you're right so, but if I understood it correctly, the actual chunks will stay the same. So you still have a lot of deduping options. Yes, yes, so you could imagine that not if you change the chunk size right yeah, so you can imagine a service that takes data that was added using a balance. Tag restructuring it for streaming purposes. Yes, it would be fairly cheap, yeah, yeah, exactly yeah yeah. That's interesting.

A