Filecoin Retrieval Market Builders Demo Day, 5 Jan 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Retrieval Market Builders Demo Day - Ken Labs - January 5, 2022

Description

Every other week, the Retrieval Market Builders get together to share progress in their projects in a demo format. We want to sincerely thank all of our collaborators for demoing their developments and helping to improve the FIL Retrieval Markets one demo at a time.

January 5, 2022, Demo Day: In this video, you can find quick demos regarding the following -
• Ken Labs demos Pando, a program that takes data streams from the FIL network and aggregates them into CAR archives. It then backs them up to FIL, and provides a lightweight query interface that consumers can use to read specific points of metadata.
__________________________________________________________________________________________
If you would like more information about the Demo Days please follow the link below to the Demo Days planning document.
https://docs.google.com/document/d/1IrNbBm2_P79Xi2fXFj9YUfQ0PjWjreGdPYFjFntD1QM/edit#heading=h.akao8aspby4

A

Okay, welcome to the first retrieval market demo day of 2022.. The date is wednesday, the 5th of january. Today's schedule we have only one presenter talking um william from ken labs, is talking about pando. um Please put any questions. You've got in the in the slack channel the retrieval market slack channel and over to you, william, thanks patrick.

B

Okay, let me share my screen.

B

Okay, it's my screen: yes, cool! Okay! Let me start so. First of all, let me still uh happy new year to everyone. It's the beginning of the new year, so hope every team and everybody had a great break during the christmas and the new year season.

B

Okay, today, I'm going to give you another introduction about the pando metadata service that can labs developed in past few weeks.

B

So I think I have done a couple of demos in the different features of the service today today, I'm going to go through again and we'll also introduce you, the provider, integration and our full api spec and the documentation stuff.

B

So this is the agenda, so I'm going to give you a quick introduction and I'll show you, the provider, integration process, and I will also talk about the consumer api and in terms of the service monitor. This is currently used internally.

B

So it's to reveal the status of the pando service, but in the future we hope to release more features to the providers or the customers, so that and each provider can track their own the status of their own data and at last I will also give a brief introduction about the future plan and our contact information so introduction the panda service.

B

So it keeps the ipod format, metadata, included and constantly available by the consumer api. So we support two protocols, I can say the http api and p2p. So it's over rough sync, the consumers can carry the ipod data over graphsync.

B

I will talk a bit later, so in general, consumers still need to integrate the golex, which is the the module developed by the. I think, folks and uh pando also provides a lightweight, unbiased access to the metadata stored and the discouraged. Historical revisionism- and I say panda also persists to the archives on firecoin, which means they provide us metadata for the providers. Data will be stored and archived on firecall network, so the consumers can optionally also access directly from there.

B

This is a general architecture of the service, so there will be three parties provider consumer on the far core network providers can notify metadata update following the integration process, so notify the meta update to pando service. The panel service will synchronize the updated ipod data from provider and all these data will be periodically archived and stored on far core network as a consumer.

B

They should be able to carry the metadata by graphql and can also clearly provider metadata or provider data via graph sync.

B

So we have already put all the documentation api referenced together, so the github repo will be the root of the documentation. We also swagger api reference, the documentation, so any user can access. One of the link here to to see what kind of service provides.

B

We also deployed a panel service on one of our hosts, so I call it just a managed panel service. So by hitting the pando slash info url, you should be able to see the required information for, for example, pando sorry, the provider integration I'll show you quickly and for provider integration. uh There are two steps needed. First of all, each data provider must initialize a golex instance uh and publish data to a pando pop soft topic which pando subscribes.

B

These two steps are also listed on our documentation, so anybody who is interested can look into the steps so the second step, the data provider, must register itself from the instructions. So that's a simple command line execution.

B

After that the data provider can start using the panel service.

B

We are planning a sdk for providers in the future, so we will release it shortly for providers for now. If you look into the example we provide in the github repo, the providers may still need to do a bit co-change to inter grade or initialize the golex.

B

We are trying to ease the integration process as much as possible. So that's a that's the role of the sdk and the consumer api. uh So for a data consumer it would be easier for them. They. They still need to initialize a golex instance in order to carry data via graph sync for the metadata, the pando service metadata data consumer can simply just make a http api call for the metadata api.

B

It's also listed on the swagger documentation. I will also show you shortly uh next thing: is a service monitor?

B

I think I showed this dashboard before and in general, there's no more matrix metric introduced since the previous demo demo. So this pretty much we what we have now it's also. As I mentioned this is used internally by panda service for now, so we just use it to track the status of the handle service in the future. We are hoping to expose more uh metrics to individual providers.

B

They will also they will have a portal, a web web app to allow them to manage the their registration and their their data storage.

B

And this is a future plan, so we start serving real data providers in q1. I think now we are serving data providers. uh We are looking for any providers that are interested and we are happy to help them integrate into our panda service and next step. The other thing is a continuous feature: development.

B

For now we have done the main features. We still have a list of features to be developed, so it's part of a road map, so we will show more features in the future demos. uh This last thing will be the customer support. Any providers will be treated seriously so to make sure data can be properly synchronized and safely stored and archived and served to their consumers.

B

Okay, so we are happy to to welcome anybody interested in storing ipod data in pando or, if you have any problem in using pandora service or even address any bug in our code base, feel free to pin us on slack or email to info at kencloud.com.

B

Even if you are interested in joining us, you are more than welcome so feel free to reach out to us on slack.

B

So this is our contact information, uh so it's for the better future of cloud, 2.0 a slogan or can labs, and this is our team. I I'd like to introduce our call that team who have been working on the panda service for several weeks. So in the middle you you can see this. This is a cool team in the chandel office they celebrate the new year.

B

We have thousands uh xiaohua bin and ken in the middle so and able uh they are also the core members of the the whole team, and you can see this is me and jason on the right thanks to to the great effort from everyone.

B

Okay. So this is the introduction and, uh as usual, I'm going to show you uh what we have done in the documentation and the api set. So this is the official repo.

B

So we put everything here for or useful here as a provider as a consumer, so we are still working on the documentation and this is the developer guide. If you are interested so how you can set up the dev environment and that's this is a configuration specification so for each option you will know, what's uh what is used for and what you should pass to and the last part is api set, and this is the api so pando.

B

This is a tool to carry the bundle status and this is a provider api for now, there's just one endpoint. This allows the provider to resist itself to pando and the metadata. This is to show you the status or the snapshot information about the pandora service.

B

So this is the api and uh yeah. This is a github repo, and this is api. This is the graphna dashboard and this is just installed and deployed earlier today. So there's very few data.

B

I can show you how.

B

Before that, let me show you one thing is the example: uh this is uh what you can do if you're interested in populating data to kindle service. We have a example folder here and we developed, but this is what also what we I have been using in previous demo. This is how we uh popular data to handle service as a provider.

B

You pretty much need to do the similar thing, so this is a great reference for you to understand how to integrate the golex, and this is an example, and next step is I'll. Show you how to do registration.

B

uh So this is the the root of the the project, so we have a make file. We can build the the client and the server once it's built, it'll be listed in the bin folder. There are two binary files, so kindle server is a the core of the conduct service.

B

The pando client gives you the registration function so that you can reach yourself to a panel service. I just ran the command before, and so this is how you can register and you give the panda service url and the provide register means that you want to reduce the provider and give the peer id private key. So this is a p2p priority and then your address. This is multi-address, it's optional.

B

So it's not a mandatory option argument and in this way, so you can register yourself once you see, register success or 200 code 300. It means you have successfully registered and I did another subscribe subscription step before in in previous demo, but now it's not needed any richest user course provider will be automatically subscribed.

B

So now you can start populating the data to handle service.

B

Now, if I go to the mock provider dj, if I use this command so you you can see.

B

uh This is to populate some ipod data to your handle service. uh You just need to provide your private key and the the number of dag data and then the pandora pure id model address. Then you need to give it some time to wait until the data transfer completes and now let me try it.

B

Okay, so it will send many ten yeah ten pieces of dhg to the bundle service. One two three four: there should be ten set these here and it will wait 20 seconds and let me go back to the browser, so here will show you this api. So, first of all, this is the api. You can use to query uh the status of a panel service.

B

It give you the peer id under the api address, so you can use this information to populate the data, as I just show you here, so you need to provide a pando peer id than the multi address.

B

So this is this is the api where you can find those information?

B

Okay, so the next step is the metadata.

B

So the metadata list api this this guy will give you uh actually a snapshot list.

B

This is the most recent one which contains the dh information what I just produced. So if I use this cid in the api, this one, so it's a get- a snapshot detail can be carried by cd or height. Now, I'm using the steady.

B

So this is how you can see the the snapshot chart information.

B

Let me see how many are there one? Two, three, four five. So this is the older one.

B

This is the height. We can also carry it by height five.

B

Four yeah, so this is the most recent one, so we have five snapshots. So this is most recent one with height four and you can see there are one two, three, four, five, six, seven, eight nine ten ten cities here. I think they are just the ones I sent here. If I search one of them, yeah it's the kjx and the last one wsqh it's here.

B

So the order is slightly different as what I just see here, but it's okay, I think 10 all of them are listed in the list. So this is how we manage the the snapshot and the dags sent by the provider.

B

Okay, if you want to know more about a parameter supported, you can go to pando.api.comcar.com slash swagger stock, so this is the root of the documentation. I also put it in the slides here: okay and that's pretty much it so for consumers you can carry the the the gp api and.

B

The actually we also have the rock qr api. This part will be updated to the document shortly and yeah. That's pretty much it. I think the question will go. Go to slack thanks. Everyone.

A

Great thanks so much william um yeah any clarifications. Anyone wants to add.

B

Darjen, do you have anything to add.

A

Oh very good, now.

B

Okay, probably I can add a couple words to you to the pandora service, so to clarify the pandora service will provide the solution to any third parties that that are interested to get their ipod data persisted.

B

So it's it's. Sometimes it's not necessary to put everything on on the chain, but it's still nice to have them persist and serve to different consumers to reveal the the data or the entity status, the the state of the whole network. So that's the purpose of the the service now yeah, that's it! Oh.

A

Yeah thanks, it's always great, to hear for, like a very, very high level, what we're diving into because people are coming in cold. So, thanks for that clarification, um but yeah and a great presentation thanks guys I'll wrap up the recording there.