IPFS Launchpad: IPLD, 17 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: IPFS Camp 2019 - Immutability & CIDs

Description

This section on Immutability and CIDs from IPFS Camp 2019 goes over the importance of immutability in IPFS. Alan Shaw gives an IPFS primer and explains variations of CIDs in the context of IPFS.

This content is created as a part of Protocol Labs' Launchpad program. You can get resources and written instructions in our learning platform at https://curriculum.pl-launchpad.io

Resources Slides as PDF & Course repo: https://github.com/ipfs/camp/#-core--elective-courses--youtube-playlist
Join Protocol Labs Launchpad: https://pl-launchpad.io/
For more information on IPFS: https://ipfs.io
Original video: https://youtu.be/Y_-TWTmF_1I

Sign up to get IPFS news, including releases, ecosystem updates, and community announcements in your inbox, each Tuesday: http://eepurl.com/gL2Pi5

Chapters:
0:00 Start
00:25 Why Immutablilty
05:25 Anatomy of a CID
07:03 Multihash
09:59 CID Version
11:32 Multibase
12:49 Example: Anatomy of a CID
16:15 Version 0

A

This is the uh Cocos a uh welcome, it's very nice to see your faces out, you're, not too hungover or tired or whatever um I am Alan. uh This is Michael and there's Steven in the back there and we're going to talk to you about how itfs deals with files part one of five. It's all right, they're, exciting I. Think so that's been the feedback so far, all right! First up! Okay, uh can anyone tell me what the meaning of the word immutable is.

A

All right awesome unable to be changed as opposed to mutable, which is liable to be changed.

A

So we're going to talk about why igfs has a big focus on immutability and immutable data in a peer-to-peer Network, the one on the right, uh there's, no central point of authority, no one to implicitly trust, uh but in the web of today we, uh we implicitly trust kind of things like institutions, so certificate authorities, social media government, um but that trust is easily subverted uh and we can't necessarily rely on the data we receive to be the data we asked for, even though we often do resource Integrity.

A

Tekken is one way to ensure that data you ask for is the data you've got? uh The idea is to calculate an identifier for the data from the data itself. The identifier is called a hash and we use cryptography when Computing the hash, to ensure certain properties like uniqueness and determinism, but we're going to talk a little bit more about that in the next section.

A

So once we've got this hash uh that we can share it with the rest of the world, these two people being nervous as well, and when someone gets hold of that data they also calculate the hash and then they check that they match and if they do then yes, you've got what the data you asked for winning.

A

If someone else gets hold of the data and the hashes don't match, then uh-oh someone's tampered with your data, so the guarantee of integrating checking Integrity checking is that content is immutable.

A

If the data changes, the hash I generate no longer matches, so verifiability is one of the main reasons for using immutable data in the web of today, I can put my poodle picture up at adc.com, poodle.jsa and 24 hours later, I can doodle on my poodle, so let's say I had some glasses make him look a little bit more sophisticated, um so the the problem is that the content isn't inherently tied to the address. The content can change, but the URL doesn't so depending on when I access this poodle, it could be completely different.

A

So what we have on the web today is location addressing it tells us where the data is stored, but not much else. Content addressing, on the other hand, is where we use a hash to access the content and allows us to verify. The content we receive is the content that we asked for.

A

Okay, caching and deduping: a musical content completely solves the caching problem like since the data is never going to change. The cash rules are cache this forever.

A

uh So back to our example, if we were using content, addressing uh our cute poodle has a specific address derived from its content, uh the hash and, if I, change, that uh that poodle 24 hours later the address also changes, uh but that's okay. It could be verified, cached and fetched by anyone.

A

What, if I, wanted to keep both files? Well, uh because the content is immutable, ipf has only really really needs to store the changes, uh the differences, the two files that we see actually share many of the same bytes. So if I have one of those files, the amount of data I have to transfer to fetch. The other is really minimal.

A

Now currently, if I want to store those, two very similar pictures I need to store both of them in full on my disk and if I want to share them with someone and they want both and I want them to see both of those pictures, then I have to transfer them both in full and we'll see more about how ipfs does that magic deooping stuff, later okay, fed from anyone.

A

uh If the content is immutable and I can verify its Integrity I should just be able to get it from anyone, um and this suits P2P really really well, but that's not true of the web today, uh so I have two sites on the on the net hosting the same content. How do I know which of these poodles is the correct poodle?

A

Well, okay, so the answer is that we can't trust that any of these poodles are correct. They are both adorable poodles, uh but uh we just can't get it from anyone where it needs to come from a trust trusted Source, um because I can't verify its integrity all right. uh So we use content addressing in ipfs and we use a special hash called a CID. Cid stands for Content identifier and we're going to take a closer look now in part two of five okay.

A

uh It all starts with cryptographic, hash, um a cryptographic, hash function, Maps input of arbitrary size to Output of a fixed size, and we want a few properties from that hash. The same data should always produce the same hash, so it's deterministic. uh It should be impossible to invert.

A

um We, as in we shouldn't, be able to reconstruct the data from the hash. It should also be unique, uh so no two different files should produce the same.

A

There are many different hashing algorithms that exist. uh Ipfs uses jar 2256 by default, and that is just some of the some of them that are available in ipfs that we could use older algorithms like xiaowan, are broken, they're proven to not be Collision free, and if and the problem is that, if algorithms can break we're going to want to switch that hash that we use by default in the future, the problem with switching algorithms is that, given a hash, which is just a series of bits, what algorithm did we use to generate that hash?

A

We need a future-proof way of of identifying the hash function used to generate a hash, as well as the hash. There say hello to multi-hash it'll solve all your problems, so multi-hash is uh is the hash which is just at the end there, but it's also a prefix, and that prefix is one number algo algorithm um and which identifies the hash algorithm that was used to generate the hash and another number, uh which is the hash length.

A

These two numbers are both variance and Varin is just a compact encoding for integers. The algo number is in multicode it because a is a variant but B it's uh it's. Its value is a predefined value that we agree on it's in the in a table on the internet.

A

The multicater identifier for char-2256 is the number 18 and our hash length is 256.

A

in binary. It looks like this uh you'll see that the hash length is actually two bytes long and that's because the length is over one, two, seven and in far in encoding uh numbers above 127 are encoded as two bytes. So you can kind of think of this as like two times. One two eight is two five six, but that's not really how far it works. uh That's just a coincidence, but yeah two bytes, because it's over one, two seven.

A

We want to support multiple encodations, so you've got a hash that um that addresses your data. When you get the data back, um how do you know how it's encoded like we might? It might be encoded in cboard, uh concise, binary, object, representation. It might be encoded in protocol buffers, it might be encoded as just plain old Json.

A

um Why do we want this? Well, it might be a particularly compact binary encoding, so really efficient for storage, but it might be for convenience might be just easy. It might be really quick, but we have this same problem again like when we're looking at some encoded data. How do we know how to decode it? Oh same solution, more metadata prefix.

A

So this is an ipld codec, uh an ipfs at the moment, any any um any content that you add gets uh is we the codec we use to encode the data is dag dagpv and it is uh PB stands for protocol buffers, so it's just protocol buffers, encoded um yeah and it's like I said it's a compact encoding for the data and we've had some uh metadata to the start of the the hash correctly yeah. There are many ipld formats that you can use.

A

um There are too many all right, uh so there are two different CID versions, so um that's mildly, frustrating, but uh so often now a day or hopefully going into the future you'll see version one as the main version that we're using uh there's version zero and version one. It's just another prefix that we add on to the front.

A

So all of you, if you've ever used ipfs before you might be used to seeing cids like this and version zero is a this top one and a version. One one is the bottom one um uh and the so the version, zero ones begin with kind of qm, usually version uh version, zero begins with qm version, one begins with Buffy or Baff K, sometimes um all right. So um the thing is cids are binary.

A

Data um zeros and ones make for really long strings when they're printed out like it, doesn't even fit on the side on the slide. So what we do is uh we use a higher number based encoding um to make them shorter and easier to like print out and recognize, even uh when in string form, we have another problem again. How do we interpret the the number based encoding that this string was create created with um characters in one encoding are valid in another encoding?

A

So we can't just look at the string and be like that's that's a base 32 or that's a base 16 or whatever. um So how do we? How do we fix that? Well, you guessed it more meta prefix. This is multi-base and multi-base helps us know the number base. The content is encoded with the first symbol is the identifier for the encoding B uh is base, 32, uh Z or Z is base 58, BTC and M is base64.

A

version. 0 cids, like the qm1 you saw in the last slide, are also base. 58, encoded and I just told you that base 58 should be prefixed with Z, but they're. Not if you're wondering about that, then don't worry. We will talk about it in a minute, okay, so the rest of the string after the first first symbol is the rest of the CID encoded in that particular base.

A

So all together for a CID. This is what it looks like in binary form. It's the CID version, It's, the IPL, D format, multicodec and it's the multi-action. The multi-hash is split up in, as you know, because we discovered it into the hashing function, use the hash length and the hash as a string. We have the um the multi-base encoding and that encoding applied to the rest of the CID.

A

Cool all right, so you can try this as well. uh If you fire up your browser and go to cid.ipfs.io, and if you have a CID happen, they'll happen to have one lying around, then you can paste it in to that tool and it will pick out these properties that we've just been talking about of the CID and show you them there's a tool in ipfs. If you've got it installed on the command line called ipfs CID, and that will allow you to change the number base or the version number of a CID.

A

If you type ipf CID help it'll tell you how to use it, and so, if you want to, if you've got like a v0, I see ID and want to convert it to V1 and put that in or whatever but I'm gonna. So I'm going to open this up now and show you a few pre-prepared some cids I prepared earlier.

A

All right, Sue.

A

This first CID I have.

A

Back to the start of the string, this is a baffy CID. This is a base. 32 encoded CID. uh Oh hang on. Let me make this a little bit bigger for you version one CID, it's encoded with the ipld format called dag PB uh we've got the multi-hash at the end, which is uh sharp. The name of the mountain hash name of the hashing algorithm is sha2256, the length is 256, and that is the um the actual hash. uh So multi-base, as as I was saying, is a uh it's coded.

A

B start of the string is B, so that's good that matches up phase 42 dagtv for the multi-hash to cut like the code for the Char 2256 is 18, as you saw on the slides uh so yeah. That's that's one. I've got a base. 64 encoded CID here, if I paste that in so remember, I said that basic before cids start with M, and they do according to this, it's all good. So this is another CID V1 uh and yeah. Base64 I've also got one here with a different IPL D format. Oh whoops, sorry.

A

hmm So this is a V1 CID encoded with base 58 BTC, uh and uh this is interesting because it's using the dag, sibo ipld format, not the dagpb, which is the dag3, is the default for adding files in ipff. But you can use the ipfs dag API to uh to create structures that have different encodings and this one's good for, uh like Json data, there we go okay and so finally, the one you've probably all been waiting for what is going on here, all right qm. This is a v0 CID.

A

As you can see here, qm begins with qm, uh it's encoded the space 58. But what have we got here? Multi-Base code, implicit monkey, codec code, implicit.

A

What should we should we talk about that? Yeah yeah, all right cool, that's good, because I've got slides prepared for it all right. In the beginning there was only multi-hash and they were encoded as base 58 BTC all version zero cids have no multi base, and so that's why they don't start with z. Why did I start with qm, though? Well? They are just multi-hashes uh and they're, just multi-hashes encoded with base 58.

A

So if you remember uh back to the slides about multi-hashes multi-hashes are the hashing function, plus the hash length, plus the hash and those two things at the start of the of the multi-hash are kind of fixed, usually in ipfs, so, which is why you see a lot of these and and but it's just the what it is this Q This qm is just those two numbers encoded in base 58..

A

The multibase is always base 58 BTC, like I, said, um and so what else do these not have? If version zero cids are just multi-hashes? What do they not have other than not having a multi-base.

A

Anyone right they don't have a format, they don't have a CID version.

A

So all together for V zeros, this is way simpler. It's just they're just multi-hatched and it's string form their base, 58 encoded multi-hashes, which is why they start with qm.

A

You can kind of think of them like this, so the Multiverse is always base. 58 btz BTC is z. It's implicit, it's not written CID version is always zero. It's implicit, it's not written. The IPL D format is always dagpv. It's implicit, it's not written. The hash function is always sha2256. It's it's explicit, not implicit. It was a trick question it's in the multi-hash, so the hash function is in the multi-fat hash it is in it is it is there?

A

uh The the hash length is also explicit it's in the multi-hash and if any of those variables change, then you don't have a v0 CID anymore.

A

All right cool so now you know how itfs addresses content um using cids and hashes, and we can. We can now learn about the data structures that ipfs actually addresses.

A