Filecoin Ecosystem, 30 Mar 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Ecosystem-WG March 2022 Deep Dive: Working with Estuary Collections [ARG]

Description

The Ecosystem WG is a collection of 80+ people across 5+ organizations that are passionate about the long-term adoption of Web3 and dedicated to growing the Filecoin, IPFS, and libp2p ecosystems.

We regularly host all-hands meetings to check in on progress and showcase the growth of various developers, applications, storage providers, clients, and communities around the world.

This deep dive features Gabe Cruz on the ARG team explaining what organizing content on Estuary using collections you'll learn what estuary collections are and how to interact with them.

Check out more about Estuary here:
https://estuary.tech/
https://arg.protocol.ai/
https://twitter.com/aresearchgroup

If you, your team, or organization want to be a part of our working groups or would like to present and be featured in our next public All Hands, please reach out to:
ateam@protocol.ai.

A

Hello, everyone, my name is gabriel. I go by gabe and today I'm going to be talking about estuary collections and how to work with them, uh what they are, what purpose they serve and most of all, how to actually use them in a practical context right. So I only have one slide, which is this one, hello, I'm gabe.

A

uh You can find me everywhere at gmail today and agility melody crews, twitter, I don't know instagram whatever um and uh that's basically all the slides they have, because I want to jump right into um answer reflections, and so what azure is is the easiest way to upload data to filecoin? um It's a simple web server that you can just drag and drop files in it, and you can also use the apis if you're programming, something and that's what we're going to be using today.

A

um But we're going to be talking mostly about collections, because the way actually worked in the past is. It was basically a huge bucket um that you threw all your data in and there was actually no way to organize it. All and collections create a type of directory structure where you can organize your files and list files only on one folder um and that's basically it so, let's get started.

A

um The first thing I'm gonna do is I am here at this uh directory and, as you can see, there's there are three files here. uh If I uh get the content of all those five five one means this. I am five one five two here, sir hey there, you can see there's a little title there on file three. um So now the first thing I'm gonna do is: I have um an estuary um note running locally just for the sake of this presentation, but it's going to be just the same.

A

If you're doing the actual uh production, that's reserved or not so the first thing I'm going to do is I'm going to add file 1 to it. So what this command will do is it will create a pulse request to this endpoint uh content. Slash, add um and then I'm gonna pass it.

A

My api key, which is in this environment, variable right here um and then um I'm gonna there's a lot of headers here that are also important and then finally, in the data section, I'm gonna send it this file right and then I'm typing this, uh like the response of the request to jq, which is a nice little um command line thing that makes colorful stuff I'm going to see in a second.

A

So here we go, um and so you can see like this is nicely well parsed and colored because of jq, and then they respond, we got so. Basically we got a pid.

A

So what it's doing is it's getting that high one attacks and it's creating the dag it's giving it a cid, which is this one? It's giving you the necessary id and it's telling us what the providers are um and now I'm gonna basically do the same thing with files.

A

uh Two and three, because we're gonna need them to be an so. This is, this is just putting the files on actually not we're, not messing around with collections so far, just putting the files on s3 at this point right, um and so all the files are now on axiory, and now we can use this combine right here, um which is hitting again hitting another endpoint.

A

This time, we're hitting slash, content, slash stats and we're doing a get request. So here I'm not in the past, it was specified posts as a post request and because I'm not specifying the default is get so you're doing a get request in this endpoint hit enter and you can see the three files are there: fire 3, 502 501 with their respective cids and all that um but they're, not exactly on the selection they're just there and that's um how we did in the past. The things were just there.

A

Things were just like in this giant bucket of data that your user owns. That's it if you had millions thousands or like only a couple files, it worked the same, but as you can, as you can see, this does not play well in terms of user experience, because if you're using you have a bunch of files, then it gets hard to like track down those files. That's the whole reason why directories exist in the first place right, um not just, but it's not just the necessary thing.

A

So now we're going to use collections, but before we do that, we need to create the collection right in order to create a collection, we're going to hit again another endpoint and I'm going to give all the commands here, I'm going to put it in um a cheap heat um that you can just query later um and so this time, uh so don't worry about like copying the comments, it's going to be a link somewhere soon, so uh we're doing a post request now on collections, slash, create and now we're providing a json.

A

So this is json data with the name field, so the name of my collection is a new collection. The description of my collection is a new collection test uh and then again usual headers, with my api key piping to jq hit enter and there we go. You can see it was created at this time uh it gets a uuid which is like a unique id. Every collection has- and this is very important, because this is what we're going to be using to identify our collection.

A

um Every time we talk to api's like oh, do this get files on my collection put files in my collection um all what the way we specify our collections is by providing this uuid, so we're gonna, save it to an environment variable all I don't know, uh let's see collection lowercase, uh I don't actually need the wall, so we like to do uppercase I like each other, and then we paste that and now we can do excel collection and there we go. We've got this on our memory variable just just in case.

A

We um we scroll super down later and we don't get to see it all right. So this is um this: is our collection created it's an empty collection um right? Let's, let's list the content of this collection really quickly.

A

Let's, let's try that so for that I'm going to hit there's another endpoint, that's collections, fs lists, and then we specify call as a query argument and we put in our collection, uuid here and again type into jq uh all kinds of stuff, and we can see uh because my background is black, but this is really mill- let's not type it to jq, I'm not typing it. It will not give it like the black color and we're going to be able to see it there.

A

We go it's it's it's returning normal, because there's nothing on that collection right, I'm missing an empty collection, that's what we would expect to get um now. What we're gonna do is we're gonna. Add those files for questions. Remember we have five one two and three they're already on last three um but they're, not really in the collection. So let's put them on our collection and of course I have another one for that uh on my beautiful notes here um again uh we're doing a post request.

A

You see here, I identified the get request um I didn't have to because of because carl defaults forget, but I did earlier on. I didn't but uh yeah, but for pulse requirements. I do have to specify that so that's poster pass and we're doing add. Ipfs content addict events, and this is just another endpoint. You can use to add content directory, um and then I give my my file a name uh calling it file1.

A

I need to give it a root which is the the root cid of the file, so I'm gonna scroll up and for file one. This is it's the id, so I'm gonna copy that and I'm gonna move back here and I'm gonna paste the cid here and then on my collection. I actually need my collection unique id there. We go. That's our uuid!

A

So now we're gonna add this file file one and I'm gonna, give you the name of i1 just for the sake of like uh it could be another name, but it's just so. It's simpler, yeah same file, same name, uh but now I'm getting, I'm gonna. Add it to the collection you just created um and then here we go.

A

You can see the that is huge. um Here's, the cid, uh the name of the file and that's it. Now we can list the collection uh again and now, when we list the collection, you can see that the file is there, so you can actually list the file um or with the contents of the collection and the file. Is there um perfect again collections really are buckets. It's there's still kind of buckets of data at this point because, although like now, I can kind of like granularize a little bit more of my data.

A

You know, um but I I don't actually have a directory structure like I cannot have a collection inside a collection so like. If I want to have sub folders uh before calling a collection, a folder. uh I can't do that at least not with the information I have so far, because again, there's no way to put a collection inside of your collection. So the best I can do with the information you have so far is create a bunch of collections.

A

Let's say like for my document: I create a documents, collection and then I put all my documents there, but I cannot subdivide further these documents in between like medical documents or, like um I don't know, financial document. I cannot do that again with the information I have so far. So that's why collections can have paths inside them, so you can actually create a path inside of a collection. So, let's add file two.

A

So now we have five one, let's add title two to a collection. So here I'm doing the same thing posting to flash content, flash, add ipfs, but now I'm adding file 2 to it, and now I'm going to have to scroll up, get file to cid and here on root, I'm going to put it cid.

A

The reason I used um the collection with dollar sign here is because of quotes um because there's only one code, but here it's surrounded by quotes this.

A

This part of the request is surrounded by quotes so like putting a dollar sign um and then the environment variable name wouldn't work um because of the shell being weird, um actually being consistent, we're trying to be weird here but anyways, so basically I'm putting 502, which is this id here, I'm naming it file 2., I'm putting it inside our only one and only collection so far and I'm giving it now collection path.

A

So this is an optional um uh field, you can have an idfs and then you can actually add a path to the file inside of a question. So in this case I'm creating a folder foo and then, instead of the folder, I'm creating another folder called bar and then inside of that second folder, I'm putting my file there.

A

That's that's what this means. So let me go ahead and hit enter and this should work again. Let's list the contents of our collection, and now we can see file. One is still there, but now we can see. Foo is a directory, so a directory was created inside of that collection. And then, if you come back to this list, collection uh request what we're doing where we're specifying the collection here and what we can do is we can also specify oops um a directory with the their um field.

A

So I'm gonna do deer, equals slash food and then, if I hit enter it's gonna list only the things that are inside food, which is the directory bar and then I can go back and do list 2, slash bar. Then it's going to. Finally let 502.

A

So I have boo and then you have bar, then you have files and then you can list things and we can mix and match directories and files. The only thing we can do is create empty directories, because directories are basically created as we need. uh So, if you have like again as as we did, if we have uh foo slash bar, then we have a file, then esther is gonna kind of create food and then inside of that it's going to create bar and then inside of that it's going to put file or file too.

A

In this case um and yeah, that's uh pretty much all I had to say um this is not really a super um technical, deep dive, it's just a bunch of like uh end points and how things work um a little bit under the hood. I hope you liked it and um that's it um keep posted for the links in our channel. um I'm gonna release all all the comments I just used on a nice little guest cheat sheet or just um and that's pretty much it. Thank you.