IPFS Learning Resources, 7 Apr 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: The Lifecycle of Data in DWeb - IPFS Camp Workshop

Description

This course from IPFS Camp 2019 covers how IPFS deals with data through a mix of demonstrations, slides and ...dancing! Follow along with ...
- Slides in Google Slides format: https://docs.google.com/presentation/d/1UOX-a1lCxi-LOyEOe9VEFRcxg_j93Xkywwa4Y95TJaw/edit?usp=sharing
- Slides in PDF: https://github.com/ipfs/camp/blob/master/CORE_AND_ELECTIVE_COURSES/CORE_COURSE_D/IPFS_Camp_Core_Course_D_Slides.pdf
- Course repo: https://github.com/ipfs/camp/tree/master/CORE_AND_ELECTIVE_COURSES/CORE_COURSE_D

COURSE TOPICS

Providing data via IPFS: We'll introduce how IPFS uses the Kademlia algorithm to find peers and announce the blocks they have in a way that others can find them.

Finding data on IPFS: We'll go over how we use Bitswap to ask connected peers for the CIDs you want, and to verify the data you receive.

Managing the data in your local node: By default, your IPFS node caches and provides every block it has. We'll cover how to use pinning and pin types to manage what blocks your node keeps locally, and how garbage collecting keeps your repo size under control.

Learn more about IPFS Camp: https://camp.ipfs.io/

For more information on IPFS
- visit the project website: https://ipfs.io
- or follow IPFS on Twitter: https://twitter.com/IPFS

Sign up to get IPFS news, including releases, ecosystem updates, and community announcements in your inbox, each Tuesday: http://eepurl.com/gL2Pi5

A

Great, so there was a lot of information yesterday for the purposes of this course. All you need to remember is a file you add to ipfs is stored as blocks in your repo, so I am in charge of the ugly first web UI I try and make it look better. Every day my usual approach to problem-solving is let's build a web app.

A

So when I was tasked with this course, I just wanted to get like a visual understanding of what adding a file and it becoming blocks in my repo looked like so this is a simple web app that just you drop a file on it. You probably saw a similar one in Cork or say so that that one was showing you the dag structure. This one is showing you what's going on in the block store, so this is a almost empty ipfs block store. There are some metadata in there.

A

This is the pin set metadata we'll cover that later, but that just so that we're all on the same page visually if I was to drop a file on this.

B

A

If I was to drop an image, our preferred cat image, you'll note he's also here and his there's a connection we'll come back to him later. If I drop my cool cat pitch it into my repo, it gets added as blocks its chunked up. You can you can change the chunk size which would affect how many blocks that image became, and you can see, there's some green ones or some red ones, it sort of meaning, but just quickly. This is your root CID for that file and there's a bunch of other blocks.

A

We're gonna go into this more later. I just want to get so on the same page books in a repo.

A

So we're going to cover, providing or how you tell the IP office network I, have these blocks and how we use Godot Melea to tell peers about which blocks that we can provide we're gonna cover getting so I profess, get or how you say in I. Profess I want those blocks, we'll talk about, want lists, we'll talk about bits. What we'll talk about verified data and bit swap sessions and we're gonna cover pinning.

A

So that's how you tell I confess which blocks to keep on your machine, and that means they're going to talk about, pin types and the pin set and garbage collection and we're also gonna cover. Do not what we're gonna cover deleting blocks, because sometimes you just don't want all on so how to remove blocks from your machine and then also some thinking about in the future. How we might share lists of bad bad bits that we want to make communal and well known so providing or how you tell I, confess I have these blocks?

A

How do you inform the network of what you have so first up? This should all be understood now, but to make sure that we're on the same page, we need to agree on how we name things in ipfs. Each block has a CID.

A

Is the content address of the data? The name of the block is derived from the bytes. It contains.

A

Every unique block has a unique CID, so in this way see IDs are giving us a scheme for distributed, naming the steps for creating a CID, allow multiple users to agree on names for blocks without asking a central authority, so peers, who've, never met can come up with the same name for an identical chunk of data without being told or without having to ask each other.

A

This is very useful, which means we can move away from URLs like content, addressing, let's separate what we want from where we get it, and if we can remove an explicit location and instead we can use the address to verify the data that we got was what we expected well. This allows us to get the date from anywhere which which unlocks things like distributed co-hosting of dicta, which is like that's for me, like the most exciting thing about idea, fest, like being able to say like I care about this data, I'm gonna participate in hosting.

A

It is an interesting radical improvement to the internet, but the day are still stored somewhere, and so how do we? Let the network know what content we're providing.

A

So the challenge is: how do we locate content address data yeah in the short? This is how ipfs does it, and particularly this is the Lib p2p flavor of the demo, which has some mild changes, but nothing that we need to worry about for the core. The consequences of this course.

A

So, if C IDs give us a scheme, participating naming Kadeem layer is providing a scheme for distributed discovery for finding content address data, so the ademma algorithm defines who to tell and who to ask about where to find data over a changing network of peers.

A

So ipfs users could emiliya to store records like this one about which peers are storing, which blocks. So, in a in a trivial centralized system, we can perhaps store these as like one big table that would let us look up for a given CID.

A

Tell me all the pair IDs that are providing it, but it wouldn't scale to have every pair store information about every block that everyone has so could Emily teaches us how to distribute that table so just to get us all on the same page with definitions, but Emily is a dish to be in hash table, so a table where the keys are hashes or in our case C IDs is known as a hash table. A distributed. Hash table is or DHT gives us a way to split that table across all the peers.

A

So the mental image is lots of small tables instead of one big table, so that brings us to the Kadima code. Emily is one weird trick: if we're going to split the table up, we need to know which peers to tell and which to ask a subset of them. Instead of all of them and condemned Leah says tell the peers with peer IDs that are most similar to the CID of the block that you have, that you can provide it. So you don't they don't store it for you. They will store a record that says.

A

I can provide this, so you might tell Brian that you can store bread. These are obviously fictionalized C IDs. You might tell Chandra that you can store cheese. This is obviously a grotesque simplification for anyone. Who's looked into the HTS and has implemented them, but it's a useful mental model to have if you haven't, and also to deal with peers coming and going from the network. The Kadima new algorithm suggest you you tell more than one by default.

A

We tell 20 and in ipfs we're looking at maybe increasing that number, because the size of the DHT has exploded pretty massively recently. So there are lots of options for configuration, but the the high level principle is. You tell peers whose parodies are most similar to the cid that you have that you can provide it. Also, if content gets popular in the network, then multiple peers will be able to provide that same block. So you might tell q and Chandra that you can provide um cheese by sending them.

A

This ad provide a message and then another pair QM trillion also has that block, so they to tell QM Chandra I can provide this block. So now we have more than one peer telling another that they can provide it. So with this one weird trick, both parties know who to tell and who to ask about where to find provider records. You know who you can provide this without asking a central registry. It's a it's, a mechanism of like finding agreement in the unknown.

A

Absolutely yeah so so, with a dispute hash table its key to valine and the value can be anything and so for when you're telling a distributed hash table about providers. In our case, it's key to a list of P roadies yeah, but I saw a peer ID doesn't help. You discover how to connect to them. That's the first mapping, then in live p2p, there's also a peer store that has the addresses the multi others.

A

So if I have, if I know a peer, ID I can go to the peer store and say: oh I'm, gonna need to tell you about how to connect to this person, so I. Send you the multi out of this fact.

A

Also of note, this is a big simplification, like things like how similar a peer ID is to a CID is defined mathematically as the exclusive-or distance between the bytes that make up the peer ID and the bytes that make up the CID. We don't need to go into that too much now, and hopefully you covered it a little more in some of the poster sessions and some of the other courses. But it's a useful mental model to have, and it's good to note it for our purposes.

A

It's useful to know that Kadena is a mechanism that ipfs users to enable peer to peer discovery of content.

A

If I go and tell someone else on, the network I can provide this. This cid, then that state is now outside of my control. Like eyes, a Ford told Chandra I can provide cheese, but then I close, my laptop and I drop off the network.

A

There's no way for me to have told Chandra I can't provide it anymore. So Chandra now has some stale information about a provider. So we have this notion of reprieve IDing that every 12 hours, at least you have to re-announce, see IDs that you can provide so that the people hosting these records these provider records for you can tell the information is still fresh. So if they don't hear back from you 12 hours later, they're going to purge that record.

A

Also, you can control which see IDs that you announced, as we saw in the quick visualization at the start. There are loads of locks in your repo, some of them pins. Some of them are unpinned. You don't have to announce every single block. You have and there's a thing called the provider strategy that lets you change, which blocks in your repo you can provide. So all is the default option and then there are other options like routes and pins. Thanks going to keep to that.

A

But you can see these if you want to take this back to something concrete that you can look at yourself. If you have a look in your config file in your IP FS repo and look under the Reaper Vidor section you'll see an interval, so that's to do with how often I'll reap revise. How often will I announce the network.

A

I still have this thing, so keeping the provider records up to date more frequently that twelve hours is the default and there's no great reason to change that and the strategy is set to all by default, and you can customize that to say just just announce things that I pin and we're gonna go into pinning later so that'll make more sense when we get to that.

A

Also, you can see yeah question.

A

Yeah, you would, if you wanted yeah. If you wanted this to be fully private, you would need that and that doesn't exist yet.

C

A

Yeah yeah yeah right, so exactly so. The strategy right now of the only available strategies are fairly simple in just terms of scoping which blocks you will announce, but there's no reason why you couldn't have a smart strategy that was like, please encrypt every every block you announce or only an I, deem these to be public. Only announce those.

D

A

Any other questions Brad so to see this in action. It's all a bit theoretical at this point. You can run the command IP, FS DHT find profs, you pass it a CID and it will output a list of peer IDs that can provide that block. So under the hood, what has happened? Is you have looked for in the closest peers that you know about?

A

So you, the CID you provided you, you have a list of connected peers and a list appears that your DHT implementation knows are, and it's going to try and find the one that it deems most similar to the cid and ask them. Do you know anyone closer to this? Or do you have a provider record for this? This block this cid and so you'll either get a response.

A

You leave it say: you'll either get a response that is I can provide, can tell you who can provide this or you'll get a response that says: I don't know who can provide this, but I do know about this other provider. This other peer, whose I deem their peer ID to be more similar to the cid than mine.

A

So you can, you could go on a multi hop journey to find the closest peer in the network and, ideally, if things are working and someone can provide that you converge on the same the same set of parodies as where to look for provider records, so this command isn't actually getting the blocks. It's just doing a DHT walk to find a list of providers for a given cid, which gets us to getting IVFs again. What happens when you run ipfs get.

A

So this is how you say to a BFS I want those blocks, so we just learned how you discover which peers are providing. But how do you actually get the data?

A

It's bit swap bits off. It's how you get the data, so this is the trading box with friends. Module in open fest world bit swap has two jobs to request blocks. You need from connected peers and to send blocks. You have from peers who want them. What so one important thing is like the the live. P2P implementation of Kedah million is a separate system from bit swap, which is what trades blocks. So academically is like peer discovery. Who has the things bit swap is for your connected currently connected swarm I can trade data with them.

A

Right so we've all sat down. Listen to me, talk for a little bit and so now it's time for some dance, a small dance that will demonstrate how it's what works I'm going to need. Two volunteers.

A

Thank you very much. Actually, I'm enforcement.

A

What's gonna happen now, Akshay is gonna, add a file, a cat picture to his ipfs repo and this gentlemen plate is gonna request it. So actually, could you add this to your octopus repo, please through the medium of unwrapping the file.

A

That's incredible so it's lucky its kind.

E

Of lucky that this.

A

Cat picture was chunked into a single block that was very convenient for the purposes of this demo, but you know chunking algorithm. We can set the block block size large and it turns out that when you added it very curious, but the hash came out as QM cap. Yes, that is a complete coincidence. Okay, so.

D

You now have a block.

A

In your block store, we can say, and it's it's got a CID very exciting. You guys it's you, okay, you guys appears connected.

A

A

Want my picture, do you want my picture all right? Can you see this little thing? This is a manifestation of his desire. This is a. This is a worthless.

F

Message, that's kind of underwhelming.

A

Actually I know because.

F

I'm, pretty small.

A

But we should do. Is you should throw that action? Amazing? Actually, what does it say on it? He wants a okay. Do you have?

A

Can you throw em captain yep now? Obviously you still find camp cap because of computers.

A

Job done so we just saw a blob get traded. There was a lot less than blocking back a demo, almost so trivial.

A

Details so you received a block from the network it, but it didn't come with a CID. Because would you would you trust, a random stranger on the Internet to tell you what the thing was called? Yes,.

F

But do you think we should write.

A

Software that just trusts data that comes in no say what you got from from. Actually it was a block, didn't have this big old CID on there you'll know it's the attention to detail here. This CID is on a large card. Well, you got well actually actually sent to you as rewind.

F

A

You have a block in your block, store with a CID, very cool when you make you a copy of it and send it as a bit swap blocks message what you put on. It is not the CID, but the recipe by which you can construct the CID whoo now notice that the recipe is smaller than the CID as well so age. If I send you a CID, you shouldn't trust it a B. If you shouldn't trust it, why send it at all? So what you're gonna send is.

A

They did with the recipe. Let's go I'm saying you want this, we can assume that and you caught it now. What are you gonna do got a recipe and a block. You can t-man.

F

A

The recipe which includes the multi hash header, to tell you, which hashing function to use and what actually function length, irrespective I, prefer blink. You sure you know sure that's not what you got and it isn't what you asked for, because you express CID, which included the inspected. Hash function have been used. So sorry, but that's plate to you next time, yeah, so you're, gonna hash, the incoming data, you're gonna discover by giving the recipe that it isn't fact you and cat and then there's one last step that you did, which is okay.

A

I, have independently verified that the data that has arrived I've to write the CID I, didn't trust your city, no offense just got a check and then there's one was that you had a one list you sent to air this.

F

A

Sent you received, it I mean the message is dead, but in your head.

A

You just hatched this book and it said it's Q and can you like I was waiting for queuing cat? That's turned me around and then you stick it in your.

A

Should that have come through with a CID that you weren't expecting, you might have dropped it on the floor. So as an example, you've got this.

A

Can I have Q and cat what.

B

Do you guys do.

A

Thanks guys so I take the recipe I hash it. Let's go rescue cat, I, look at my waist I say: yes, I'm cute cat, and then this one comes in and like oh, my god recipe. Oh this kid! It's cute, okay, so index what line? That's essentially, what happens if you get a block that you've already received right? Basically, it's no longer on your want list drop on the floor. That way, people can't just send the blocks and have to store them.

A

What, if I, send out my want this, but it's it's the QM dog.

A

In the situation that I throw out my one list, all and sundry I don't get response pretty quickly.

A

Okay, I know about QM collar bones, be like do you know who could provide this and Cosby like no, but I know about Kathy. She might know, give Kathy who can provide this, and she see me like exact got this. That's got this. That's not here. Sorry, Theo Theo's got this and then Theo I connect to you. I. Send you my want list.

A

Then you send me the book dude, it's okay! Let's we can imagine this happening. Yes, Thank You, Theo, random over.

A

Sorry, all right that was the bit swap dance, hope, you're, feeling a little fresher feel free to ask questions.

G

A

So that, in terms of bit swap you're only ever going to like the co DS, we have to match perfectly. So in the event, so it would be a strange situation where you sent out. One is for a CID, and someone gave you back a differently hashed version of the same thing, but in the event of that happen, you just you're.

A

The only thing your node could do is assume that was bad data and drop it in as a human, with a good brain you're like aa potential optimization, you could have actually found out that that was the same thing, but it would be strange for nodes to be like you asked for C, DX and I gave you c dy by happen to know that they're the same.

G

A

Yeah yeah I didn't didn't, go too deeply into the the prefixes, but we can see that there's well. I can see it. There's CID v-0, there's dag PB, which is the multi codec, and then there's the multi hash function and hash tags.

A

Very good question that information is already there, so the question is: if the CID is self describing, why do we need to send the recipe and not the CID? So essentially is because you shouldn't trust the CID. The CID includes the digests, the hash, whereas the recipe doesn't so in practical terms. The CID is more bites on the wire. Then then the recipe is, and also the recipient should never trust someone else's output of a hashing function.

C

A

Because the message, so it's not request response, there's no session, you you send out it's just message. Passing so I have sent out a message saying I want these things and then at some point later, I will start receiving blocks of data. So I'm not they're, not connected. So then I've got a block of data just with a recipe attached.

A

So then the first thing I do is use those parameters to have to tell me how to hash this data so that I end up with the CID that I shouldn't know with exactly the same CID that you think you've sent me and that's the ID, hopefully, is exactly the same CID. That's on my want list.

F

A

So if the speck- because of course this is like this multiple language implementations and if the speck was like bits, what messages include the CID? It leaves the room for developers to get like complacent and be like okay, I'll trusses and like way back when, when I started, JSI profess had a bug that it wasn't checking CID s against the want list.

A

So 99% of the time that was fine, you only ever received blocks that you asked for, but a malicious actor could have started just sending blocks to people and then hashing them and been like cool cool block. I'll store it. So you saw in the demo I received two blocks that were identical. I have stored the first one and then took in my head. I took the CID off my want list.

A

So when the same block came in milliseconds later, I was like why it's not my one list and it won't I, don't have to know at that point. It's not my want this, because I just received it. As far as that subsystem is concerned, it's like you, you're, not waiting for this, so dropped on the floor. So the.

G

A

So good question like if I announced to a bunch of peers, I want a thing and then I get it and peers some pairs of much higher latency err, still sending me things or I've asked for, like hundreds of see, IDs and I start getting them. So there's two messages in bit.

A

Swap one is one list and the other is data, and but a want list message also has a flag in it that if you set it like true, is basically like a cancel, so you could send a want list or basically I do not want list. So.

B

C

F

C

C

A

It's it, and that is a tricky problem, because it's one of those things, that's like. Quite it's quite simple like once. You understand those primitives you're like wait, isn't like what, if I have the same content and I've used different hashing function, but that's a difficult problem for ipfs to solve, because because we would have to like normalize, every hash or we'd have to do a lot of extra work like it. We can, it's all them all.

A

The tractable problem yeah, but it's more like the system is optimized for, like you're speaking the same so you're speaking the same set of hashes, and if people have got specific reasons to change it. The what's interesting is there's a lot of flexibility in the CID gives us a huge amount of leverage to upgrade things in the future, but it doesn't mean that if we do upgrade things all the old stuff is still discoverable by the new CID, which is a problem.

A

But there is a general assumption that you would have to have a good reason to want to start messing with the hashing function and that you are probably doing something. Something app app specific you're kind of creating your own little island of data. That's using the non-standard hashing function.

C

A

Totally but it's doable, but so then it's a performance hit because we would have to be essentially normalizing everything, maybe hashing them twice.

A

So and so remembering that this is just a wire protocol, that's designed to be influenced by lots of people. So if we were to wait, if we were to kind of lump I peeled B in with that that would be cut, it would become orders of magnitude more complex than it is now right. Now it's super simple, so just for everyone else, yeah, oh yeah, of course,.

H

A

Yeah I have a slide on that in the minute. Currently we don't prevent bleaches, but we could, and with your help, we will but right now, there's no logic in there to prevent that.

G

A

And and implemented the the bits what Ledger's exist and we track send and receive blocks between every pair, but we don't use it for anything other than stats.

A

So for everyone else, let's just recap: we saw a lovely dance. Thank you to our volunteers. The cast and crew of that performance were the want list, which is this staple trapped list in memory of a running node that, where each pier stores the list of C IDs that they want to get, then you've got a want list message. This is a snapshot in time of a current, like state, fully managed in memory list. So the one. This message is a snapshot. What this message is a snapshot. One. This message is a snapshot.

A

What this message is: it's not sure you get it. My in-memory representation changes, but the snapshot goes five zaps and network and is listen to you by connected peers and then hopefully you get some blocks messages back. So the blocks message look good shot blocks. Message is a message that contains one or more blocks of data, each one prefixed with the recipe for how to calculate their C IDs.

H

A

No cuz it come and we will get to that in one minute so just quickly, just to recap, on the verification start I think we covered it really detailed, but you send the recipe you don't send the CID. The recipe includes the CID version, the multi codec and the multi hash. These are all the things you need to calculate the CID and if you use all the same ones as the person on the other end, did you will end up with the same CID, assuming that the data is correct so on receiving the blocks?

A

The first thing you do is use the hashing function to hash the day you got, then you prefix it with the right codec and the right version. Cid will match something in your want list. If everything is working correctly, if it's not working correctly and you've, either already received it or someone is trying to send you bad data, you drop that you drop that data on the floor.

G

Where matters is.

A

The multinational binary comparison, I, don't know, I, don't know what it's been. It's yeah, it's not a string comparison, it is the it's. The the bytes array format so in the implementation see IDs are represented like this daughter's bytes and then you project them out as the string version so to your question. I've got a root. Cid and I want a whole file. I, don't just want to do so.

A

The general flow is I, want this CID and eventually get a block back when you verify it and you store it in your repo done all that and then you finally get around to looking at it and you're like holy crap Ola. This thing has so many links in it. This isn't the file. This is just like a thousand links. So then you take so what's interesting. Is you send it out?

A

I want this with one Co T on it, then you're like okay, a bit of a bigger one now is I want this with a thousand C IDs on it, and then the network will give you back whatever subsets of that it can, if you say, get a root. Cid it but you're, basically selling get get recursively like get all the way down the tree, so the general flow is i1. This ID, oh thanks, another layer of the graph I want these D IDs and you repeat, depending on how deep the graph is. Yes, sir.

A

D

That work, when you write.

A

So in terms of bits, what it just means.

D

A

The DHT thing right so that so there's two two subsystems at work bits what is to connect space, so the first thing I do is ask my connected peers because they're going to be the fastest responders, maybe I'm connected someone who already has this data and I can send out a one list of any arbitrary size. Great that bit you understand. So then, if I, don't, if maybe some people can respond with some of the C IDs, but maybe some of them I don't get back at that point. I will in parallel.

A

I can do many separate, DHT queries, so I can do a DHT query for every single CID.

A

Well in it, you send one out proactively as soon as you do a get and then yeah you will periodically refire it. If a new pic joins your swarm, you'll send if you've got an open one list, you'll send it to them, because there's an important like whether these two systems join up, so the DHT you finally tells you are here- is a set of multi adders and a peer ID. I know. This person has told me that they have that CID you receive the multi others at the DHT levels.

A

It could Emily receives them mmm and then the first thing it does ago pay lib p2p, we've just discovered. Another peer. Add them to your swarm. So you connect apart your swarm you've. Just a Danny person to your swarm and you've got an active one list that hasn't been fulfilled. Yet.

F

A

The first thing you do is be like: hey new peer got any of these, and, of course, so there is this slight kind of brain disconnect there we're like the DHT, has told us this person has it, but it just kind of by quirk of like how you, like, oh new person, connected I'm gonna. Send them I want list, it's not quite as directors like, oh and go and get the thing from them. It's like getting the key you get on the get as a connected peer sender, one list your into bits, what land bits?

A

What does its job and then because they're now part of your swarm you've sent out the first cid you've got back a block that now links to hundreds but you're still connected to the person who gave you the first one. So he's going to receive your let your bigger next layer of the graph thousand cid one list chances are if they gave you the the root, they probably have the whole tree, so that likely to be the first responder with all the rest.

A

Do you not believe they do currently? No they could they don't whoosh.

A

Also, some of the feelings that that kind of like broadcasting of want list starts to evoke is sounds a little bit spammy to ask everybody about everything all the time. So what's interesting is like bit swap as the definition on the wire. The protocol definition is very simple. Just broadcast want lists received, looks, but that then freeze the application layer up to be as intelligent or as dumb about how it chooses which peers to tell there's no mandate. That says you must help every peer, like you, don't have to send you one list to everyone.

A

That would be a simple, naive implementation, which was exactly what we had like a year ago and now now we have a thing called bit: swap sessions certainly and go ipfs Josiah vests is still catching up, but bits up sessions are just a way of like the application layer tracking some state about which peers it's been speaking to which peers have been responding well to previous requests. So it starts to optimize.

A

So, in the case where you get a root CID from peer a it means, the bits off is gonna target the next round of want lists for that graph directly to that peer. That responded well instead of sending out to everyone. But if peer a happened to have just the root block and not the the second layer of the graph for some reason, then it would eventually announce two more peers. It was like it's very stateful, very kind of like trying to do a lot of heuristics. So it's like what wait wait. A few milliseconds!

A

No, no, not responding, start to start to fan those requests out to more peers and more peers and more peers until eventually get response, and it also optimizes that list so optimizes by peers. Who've responded successfully, and it then adds another layer.

A

Optimization, which is I will prefer peers who are responding quickly so trying to speed things up very big note and an invitation to all of you like if you want to make ipfs faster, as you can probably tell by now, if we can make bits walk more intelligent, if we can make bits what work faster, you will literally improve everyone's experience of this peer-to-peer network. Get involved like it's active, endeavor Hannah is leading the charge on this. She's got she's working on go bit, swap so ipfs slash go bit swap on github.

A

If you can test out some of the open PR. That would be amazing if you can propose new, optimizations and kind of a unpack. How you think that will improve things with tests would be amazing. People always laugh, but with tests, it's a shame but like if you're, if you're looking to help make our business faster. This would be a really like high leverage spot to help out any questions about bits. What.

C

A poultice of the time frame that would be necessary to at times down is.

F

C

That, even though it isn't here that hasn't come to mind,.

C

A

Not as far as I'm aware we're pretty pathological that we just keep walking the DHT, the timeout is, if it, if it exists, it's incredibly high a.

D

D

The and should be at most the number.

D

C

But is that so many months like the.

F

C

Bhaiyaji, if that happens, then that soul.

C

A

Stable pears would definitely help.

D

C

Then someone else in the DHD for the second session.

C

C

Small wind, the clients that do you have to provision.

D

D

D

A

A huge cheating and performance.

C

C

A

Let's, let's pause, because this is good chat and there will be time at the end.

A

So if bit swap is thing, it's quite context lists in that you send us Co DS, you get one back and you're like oh I. Definitely want all the things that this is linked to, if only I could have told you that I want this CID and and everything it links to right in bits. What you can't do that you just you're, just sending C IDs and get in blocks back graph.

A

Sync is experimental, it's not something that you can really turn on in your ipfs today, but you will hear people talk about it if you listen to the ipfs IRC channels and get involved. So it's worth touching on it very briefly, where graphs Inc hopes to improve on bits. What is to be able to tell that? So when you send out a 1 list, instead of just sending a co d, you can send a CID plus some context like you can tell the recipient I want this CID and everything it links to.

A

So you might pass what we're gonna call an IPL, D selector. Basically, a way of saying like can I get CID star like give me everything from here, so that then the remote has more context. It's like okay, I won't just give you the block, you're, probably going to want all of them, so I'm going to start sending you all of them, but it's early days. If, but if that kind of optimization excites, you then graph think is the place to go, go and lend your muscle.

D

A

Pinning pinning is how you tell ipfs: please keep these blocks. So, to quote, why are you sleeping ipfs makes it feel like every block is local? There is no command to say, go and retrieve this file from a remote server, so pinning is how you would tell I profess to always keep a given block locally and right at the start. We said when you add a file, it's stored as blocks in your repo, but it is also worth noting that every block you add all you get from other Pitters is cached in your repo.

A

So if you, if you've got a cache, then you're going to want to manage it. So you can control the cache size by just setting a max max size limit in your ipfs config file, and then you can enable garbage collection to ensure that our vest does its best to keep the amount of storage under that max size limit. So if you store more blocks than your size limit, then the GC process will fire garbage question and it will delete all unpinned blocks from your local repo.

A

So we can see that in action in the web demo garbage collection I mean it's not the most exciting thing in the world, but so I added previously the cat picture I got these blocks and I can. So what these currently mean is red is the root block. So it's pinned recursively we'll go into that in a minute. The green ones are pinned indirectly, because the red one is pinned recursively there, transitively pinned and the gray ones are unpinned now what's interesting, is these grey ones? You can see right now.

A

These are metadata they're, not quite the same as unpinned blocks. These are this. Is the pin set it's a quirk of this demo because this demo is visualizing the output of ipfs refs local, which is like every single block in your repo, including secret metadata, that we don't really talk about? It's the pin set it's how it tracks what is pinned, and it's also the output of ipfs pins LS, which tells us every block that is pinned, so we can unpin.

A

So this is the cat pic if we were to go and open this one, this one on the gateway, if we're gonna open this CID on the gateway, we'd see the cat pick, but this is about pinning. So these things are pinned. So if I run DC now, nothing much happens because there's no unpinned blocks that it can remove caveat like the gray ones or unpinned, but they're the pin set so can't remove them.

A

But if we own pin everything the grayness of cash okay, so now they are all on, pin bike and run GC, and as this is what you might, this is a visual model like one pin things for an away, I'm gonna put put and then just for the take a bit miss. Let's see the cat again.

D

A

The block that's still the pin set, so we can, we can see like way more blocks.

A

Like DC bites, it's gonna be so many. The terrible idea never do a thing that you've not done before. Alright we'll come back to him later: mmm-hmm anywho, quick, visualization of pinning and unpinning. So let's talk about it, the red one that we saw in that previous slide and previous web demo. It's pinned recursively. So if you run ipfs pin.

A

So you can probably have a guess of what that means. Now, so they're recursively pinned red one is how you say: please pin this block and all the blocks it links to and all the blocks they link to and all the blocks a link to until you run out of font sizes to put any offenses like so the and this hopefully, you've got a bit of a mental model of the structure of DAGs and graphs from core course a so you get the whole like directed acyclic graph thing.

A

So this is, you have a route block for any given file, recursively pinned means, pin the whole tree, and so that be able to pin something. Recursively is a useful labor saving endeavor, because it means you didn't have to manually, pin every single transitively reference block. Just you know from your point of view. It's once the ID gets me. A file also of note, like the green ones, are pinned indirectly, so they're, the ones that were referenced also worth noting ipfs is going to pin files that you add to your repo.

A

So it's a way of like we assume that you care, like you, want to keep files. You add there, your files, but files you get from others are not penned, so they make it into your repo and they are cached and they're.

A

Reprimanding discrepancy, they're, like stuff I, add, is pinned stuff, I get is not pinned, but I can elect to pin it. So why have? Why have cached that not pins in pinned blocks in your repo at all? Why bother reciprocal co-hosting as I like to call it or, as came up from a question like don't be a leech? So if you get blocks from the network, you store it in your repo so that you can be a provider, but it's useful to have this kind of two level things.

A

So, as with torrents like you've downloaded some bits, you use them become someone who can reap revise those bits.

A

So then, what we use pinning for is just a way of saying. Ok, it's really like a mechanism for managing. What's a garbage collector and what not do so things you get so imagine, you're, running ipfs, companion and you're, visiting lots of web pages and you're loading them through the ipfs network. You're then becoming a co-host of those websites and you, you know ipfs kind of opts you into that by default, which is interesting and I'm. I applaud that choice.

A

I think it's good, but eventually that can you might start to discover that you've like visited all of Wikipedia and you've? Maybe don't want to post all of Wikipedia. So then you can either trigger garbage collection manually or you can just set a max size and let it slowly like clear out blocks that you don't want to keep around, but it and then pinning is a way of saying proactively.

A

Actually I bloody love, Wikipedia, I'm, gonna, pin it install that I'm gonna get a bigger, hard drive and I will also like proactively, say to IQ fest. Don't ever garbage collect Wikipedia because I care about it. So much I'm gonna, preserve a local copy and offer that up to the network and become another provider.

A

Exactly so, we don't tell the DHT that level of context. We just tell the DHT we're providing things which is interesting because, obviously, if everyone enables the AGC and you're telling the HTV about stuff that you just happen to her, that could lead to more DHT churn might be interesting context. There's an intention improvement there, but right now depending- and it also depends on your provider strategy.

A

So by default you provide things you have pinned and things that you have not been, but you could, if you know that you're going to have a like very journey cache that stuff, you don't really want to provide. You could limit your provider strategy to only announced blocks. You have pinned so an important point. The word pinning in ipfs land has become overloaded. It's a slightly confusing term now, because of the way that I profess is transparent about whether blocks are locally stored or remotely stored.

A

So, if you, if I, want I, can use the pinning command as a way of saying I'm gonna grab a copy of this remote thing andrey host it or it can I can mean I've added this file and I'm pinning it locally. It's the same thing either way it's a local operation, but some people use it to refer to, like I, will become a co-host of a remote data set. Also it's overloaded because there are pinning services, and these are people who provide an API that let you to ask them.

A

Could you pin this for me on my behalf on your local machine, but for the purposes of us you understand my PFS pinning is local, like I, pin to my machine and to help frame that understanding it's the pin set right. You, you saw the gray blocks in the web demo that I kept pointing to is like metadata. These ones won't go away like pinning is local, because all you're doing is adding see IDs to this pin set, which is just a list of unique C IDs that you have added to the you have said.

A

I put this pin this C.

A

There is an elective course with Hector about cluster and IP. This cluster is amazing. At some point, you're gonna write like just as a like infrastructural pattern of using IP FS. If you're doing more than just like personal experiments, if you're hosting data for other people, you probably don't want to just keep having to get a bigger and bigger hard drive and a bigger and bigger server, you probably want to scale out horizontally I professed cluster lets.

A

You do that, let's use shard up in set split up into across multiple servers and the most so unless you set a replication factor. So if you've got 10, 10 ipfs nodes in your cluster, you can say, give me a replication factor of 3 and then you just add your files to one cluster node and then cluster under the end of the hood, will spread those blocks out across three machines. So there's three copies at least across ten, and then it will start to load balanced, they'll sort of balance. There's data storage.

A

So, as you add more and more files, it's going to shuffle around the blocks so that there's always three copies, but that each machine is storing roughly the same amount of data. The other cool thing about cluster that I'm probably going to rob Hector of telling you, but you can make a cluster of clusters. So that's super cool I'm, just gonna leave her out there.

A

Cluster is so good. It's a bathroom, almost 1.0 yeah, there's, there's there's people with inside the ipfs team who are like idea that should be ugly. That's an cluster all the times faster is great. All right. The home stretch deleting or do not want and do not want this block. I don't want this block. No deleting is local, most important point we're all friends of Peter player world. This probably won't come as a surprise to you.

A

You can run ipfs blocks, RM a cid and remove it from your repo, but you cannot remove blocks from other people's repos because they are individuals with agency and as they can't delete things from your repo, you cannot delete things from their repo and you'd. Be surprised how many people come on the office house calls and be like how do I? How would I remove something from ipfs and it's like well? Have you ever sent an email to your boss that you shouldn't have right?

A

It's once you sent it. It's gone, I mean not me standing in like corporate Microsoft Exchange servers, but once you send it, it's gone. There's no like bring that block back. So, if you add a block to your local repo, you don't share it with anyone and you delete it. There's a good chance. That's gone normally standing other people adding it independently.

A

There's no there's! No there's no backsies.

C

A

Are paying attention so that the next line in my speaker notes is the difference between ipfs and the current model? Is, if you, if you, if some website publishes something illegal and you ask them to take an errand, they can take it down. There's no way to know if some other website hasn't replicated it mirrored it just cannot know, and in ipfs you can at least get a subjective view of the network. You can see and say, DHD, fine probs. Does anyone still got this?

A

Is anyone providing this thing that I'm very embarrassed about, and you can use that to then frame how you approach the world? Maybe it's gone and you can just not talk about it, but still there and you're, like probably gonna, have to own up to this now but yeah interest. That's a really interesting difference and I think that's an improvement that ipfs office here.

A

It's important to be honest about UX blunders. If you try and do an IP FS clock RM on a cid that is pinned instead of just doing what you meant currently I profess will give you an error. It'll say: yeah can unpin that I can't I can't delete that, because it's pinned and as someone who cares about UX I think that's just a thing. We have to fix.

A

I think we should I, think I understand what you meant in that case and I'll unpin it for you and then delete it, but so good to know right now, if you're gonna try and delete something, you may have to run, ipfs pin RM. First, you see that error, don't panic! You can delete it. Also. What I discovered with the the web demo is don't delete your pin set metadata because your IP FS gets very unhappy very quickly. The ipfs block remove command will do whatever you tell it to do so.

A

There's a couple of pointy edges in that one, because I was trying to simulate TC and bad things happened. I've deleted a lot of local index DBS. Getting that demo to link do not replicate so lots of people talk about sentient resistance and that kind of thing in our devices is trying to make that possible. But it's also worth noting that, like sometimes there's data that you can't replicate for legal reasons or won't, don't wanna replicate for ethical reasons.

A

So there are situations where you don't want to participate in the replication of some blocks. This is something that I believe s could do way more to support. So, like there's an idea that we could allow individuals, organizations, nations to publish lists of see IDs that that do not there shouldn't be replicated list of cities that you shouldn't attempt to host if in order to comply with local laws, beliefs or social norms.

A

That way, instead of it being a situation whereby say you try and access YouTube from the UK, and you try and see some video that has like restrictions or it hasn't been released in the UK, and it just says, like YouTube, just makes a guess of where I am accessing the internet from based on a you know, a really reasonable estimate of my IP address of access and then I often see various videos. It's like you.

A

You cannot watch this, so what it's ridiculous, stupid laws, but more more importantly, like everyone should be empowered to kind of decide what content is good and bad for them, and what we want to get to is a feature where you're not denied access from where you happen to be accessing the internet from but more based on, like electing to subscribe to some lists of co DS where you're, then choosing to say I won't do this and it's more based on informed consent rather than centralized like I'm gonna babysit.

A

You you don't to look at this and if you're interested in that topic, we're going to run a deep dive on it later today. There's lots of challenges there like imagine, we convince a nation-state to start publishing the list of see IDs. They don't want their inhabitants to look at. Those lists could become very long or, if you have a small child like I. Do imagine that people start publishing naughty lists of terrible, see IDs.

A

They probably shouldn't be stored in plain text where anyone can just go and start like looking at all the see IDs that are on them, because, instead of instead of not hosting them you're giving like people like children, I'm pretty I'm, just sort of reaching like if I was a kid and I found a naughty list of cities. The first thing I'd do is not apply. It I would look at each CID intern until I went mad, so it's challenged, but that's the deep dive.

A

This has been the life cycle of data in the d-- web. Did you have a nice time.