IPFS IPFS Camp 2022, 3 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Workshop: Indexed content routing - Ivan Schasny

Description

Exploring Kubo delegated routing, and using it to both publish and query network indexers. See the tutorial at https://github.com/ischasny/ipfs-camp-routing

A

My name is Yvonne I work in the indexing team with Will and recently we started experimenting with indexing, not just file coin content, but also they got it from the apfs and I would like to demonstrate to you hopefully a easy way to do that so I'll show you a couple of slides and then we'll do like a super, quick demo of how to quickly set up ipfs indexing on your local machine.

A

So after this few presentations he already probably familiar with the what reframe is so it's a high level protocol designed to uh to to be able to to delegate routing from ipfs node to uh to other systems. It's simple request response is extensible and stress, but agnostic. So what can we do with it? So in the indexing context, so by utilizing reframe kubernodes, they can advertise their cads to other external systems instead of alongside the DHT.

A

So you can configure multiple routers to walk alongside each other, so you don't have to choose between either DHT or indexing or some other system you can use, publish to all of them as a version. 016 could be natively suppository frame and you can configure multiple routers to work seamlessly alongside each other and uh reframe can already be used in your application by using go delegated routing Library. So how do we use reframe to integrate ipfs with indexing so kubernetes?

A

um The way it advertises its contents, its publishes, the snapshot of all of its cads into the configured routers every X hours say x is 12. By default uh we have a small sidecar process which is called index provider, and you can run this index provider alongside your kubernetes and index provider can receive snapshots from Kubo can chunk them up.

A

I can translate them into the index advertisements that are understood by the indexes and again so, just to reiterate, so you don't have to choose between either DHT or like reframe or routine, or some you can use both of them in the same time, by utilizing so called parallel helper, which I'm going to show you in a minute- um and this is just illustrates how you uh all things work on a couple. So, on the left hand side you have uh the kubernet, uh the apfs node.

A

That is publishing the cad snapshots, every 12 hours to the index provider that is running on the same box so and then index provider can translate those CD snapshots into index advertisements and advertise them to configure indexes which, uh for example, CAG dot contacts. So you can, if you put this URL into your browser, you can see like a simple UI for finding index content.

A

Okay, um this, if you would like to follow along so you please, please do that. So if you scan this QR code, it's going to take you to that GitHub repo.

A

There is a script there of what you need to, or you can run on your local computer to get things working, but I also be going uh executing against the script and explaining what I'm doing so. Please, if you would like to follow along just scan this QR code, and hopefully it's going to take you to the page foreign.

A

Let me make it a bit bigger.

A

So um obviously, to uh this is the page that QR code takes us to. um It assumes that you've got go set up on your environment. So, let's uh uh start with setting up the local indexes. So internet connection is not very good here. So I'm not that brave, to try to publish to online indexes so probably going to be using the local one, so I'm gonna, so I already have it installed. So I'm not going to be installed again, so instead it will just go straight to initializing it. So let me make.

A

A

A

Okay, so a story index is index is developed by uh by the team I'm walking in so I'm going to initialize it. So that's going to generate the default configuration by just pasting this command. So basically it generates a key pair uh generated the default config. So we don't need to change anything. We can just like start it and and just let it run.

A

So that's what I'm going to do so here you go so on the in the top left window, we've got now index arriving it's exactly the same index as if you go to seed.contact and that's exactly the same process. So next I'm going to start the kubernet, so I already have Kubo installed. So it's important to. If you would like to practice experiment with the delegated routing, you have to have it version 16 installed. So after that it started supporting the configurable routers, so I'm going to just uh generate default, configuration.

A

So done so next thing that I need to do is configure routing, and so there are a few things happening here. So uh let me show you so uh you we need to add a special configuration into the notes. Config, the routing configuration block so routing supports multiple different methods, such as find peers to look a piece on the network, find providers APNs, provide and Etc.

A

So what we're interested in we're interested, particularly in the provide method, so provide method, is the one that that is used when Snapchat get published and what I've done here is I've set up the parallel helper to be used when provide methods calls. So what is spiral helper? So if you see a parallel, helper is defined as two routers say: I'm, going to use a router called seed contact and router code, one DHT. In the same time so- and here you can see that seat context is my local indexer.

A

So it's genuinely so if you would like to use, for example, sid.contacts instead of my local indexer, you just change the endpoint and that's about it. So you know, other things need to be done and one DHT is the DHT client. So.

A

What I'm going to do now is I'm going to add this configuration block to my device config, so just copied paste. It enter okay done so one more thing that needs to be captured before we start ipfs node, so I need to take a note of its pure ID. So if I go to VFS config so and then I just need to find the prid and take a knotted note of it. So here it is I'm going to just paste it here.

A

Okay, so now we are ready to go with demon, so I'm, just starting kubernetes.

A

Cool so the last but not least, we need to set up the index provider, the sidecar that is going to translate between the kubernetes and the indexer.

A

So I also have it installed already, but you can just copy and paste the this command and install it then I'm going to initialize the default config so done, and then I need to edit the configuration file and paste the configuration with this block of config into it. So what it effectively does it says that I would like to be listening for the reframe protocol on the port number 50 617, and it has some a few configuration bits. So you can learn more about them on the index provider in the index provider, GitHub repo.

A

It also defines what provider we're expecting to receive. The reframe calls from that's why we captured the ipfs node pod and defines the multi-addressers where the ipfs nodes is available at so I'm going to go to the index provider.

A

um So I'm going to delete the default configuration.

A

And going past paste, the configuration block, oops.

A

A

Okay, so the only thing I need to change here is the kubernetes be ready so I'm going to delete it and paste the one from the file.

A

Yes, okay, so uh the final bits I need to just start: the demon for the index provider.

A

um Yeah, so uh it's basically started. Okay, now we've got all the pieces running so I can open the web UI for my locally running, apfs, node and I can add a file, so I think that's from my previous round. Let me refresh it Okay cool. So then we're going to input the file so just tiny file, so once it has imported. So you see that the two processes at the bottom is index provider. The top is indexed so they started chatting.

A

So now we can try to look up the file, so we can look it up in two ways we can like fast way is. We can look it up directly in the indexer, so I'm copying this command paste and I need to get a node of the take a note of the seeds so just copy it.

A

So, uh basically, when I run it, so it shows that this file is available at what providers is available at about the multi-head, SSI and Etc. And if you use a service like sid.contacts it, you could basically really quickly find where each seed is available and could then just use this multi-head Restless to reach that file. So original seats content with a seat and the other way to look up content, we can do it through the directly through kubernet, so I just copy that command and I'm going to be using the same seeds.

A

So what it does under the hood, it would send query to the uh to the locally running indexer and the indexer would return that seat as the uh that's basically where, where it's available as that's the ID provider that we captured uh that are captured here.

A

Okay, that's uh basically it thanks a lot for listening and, as you see like it's, uh it's hopefully easy to set up to index your ipfs node. So thank you.