IPFS Virtual Meetups, 16 Oct 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Introducing dWeather: Weather Datasets on IPFS with Ben Andre

Description

Arbol, another featured IPFS Case Study, was the subject of CTO Ben Andre’s talk on how IPFS helps them sell parametric weather protection agreements.

For more information on IPFS
- visit the project website: https://ipfs.io
- or follow IPFS on Twitter: https://twitter.com/IPFS

Sign up to get IPFS news, including releases, ecosystem updates, and community announcements in your inbox, each Tuesday: http://eepurl.com/gL2Pi5

A

Yeah I'll just go through some slides here and then, if I have time I'll show you some terminal stuff, um let me talk about arable first kind of what we're doing and and why we're uh doing this work with with weather data yeah so arbol is um we do these parametric agreements for various different kinds of businesses, so think, like you know, in agriculture, which is a very classic use case for this think, a beet farmer wants to get paid if they're, uh if it didn't rain enough in their in their farm area, during the growing season, um and- and this is a this- is a very small but existent space in the u.s, and we've actually captured a lot of that business so far in in the us, agriculture, but we're also expanding out into a ton of other, very interesting markets, uh shipping and logistics.

A

So you know if uh if, if there's a storm, um a shipping company might want to get paid? If that happens, uh you have construction, you have energy for, uh for you know wind and solar, and even um uh electrical utilities, um vacation rentals anyway.

A

The list goes on, uh but the point is that we're doing these weather agreements and it's a very, very, very underserved space, even in the us and and very much globally um here, let me flip slides uh yeah, so this kind of talks about uh the existing kind of problems with the industry.

A

um It's just very old school right lack of access, millions of farms, businesses are excluded because current minimum policy sizes are too high. It's quite expensive. um Pricing is opaque, high transaction fees- and it's all summed up here really. Our business is crafting and distributing parametric weather contracts uh that payout based on global weather data sets um it's important to know that this is similar to insurance, but it's not quite insurance. You can think of it as uh being like insurance, except you know it's a little more. um It's based on data.

A

There's not you know on the ground assessment, that's the main difference! Okay! So if we're going to be doing these uh agreements that pay out based on weather data, um we need to get the good weather data. So there are a number of very good projects happening uh and that that have been happening for the past decade or so run by scientists and academics and governments. These are just some screenshots of a few of the different projects.

A

uh Here we have noah on the left. Second, from the left is chirps, which is uh uh climate hazards. Infrared interpret uh interpolated, uh precipitation stations, yeah it's a funny acronym, but that data set is generated by interpolating weather station data with infrared satellite readings, uh and then they they use a whole bunch of very advanced methods that are great, and then they post it to ftp. This is a screenshot of where you can get this right, along with the good old ds underscore store.

A

uh You know, uh the point of this slide is sort of to show that some of these, like hosting infrastructures, are not great and getting to ipfs as kind of the answer to that, but I don't want to in any way uh you know really depreciate the quality of these projects. They're great and like we email with these guys, all the time and they're you know, but they don't they aren't great at hosting. So you know here's uh era5.

A

uh This is a great project. They actually have a lot of stuff here, but their api is like super slow. um This is prism uh another uh another interpolated data set which you can get over uh over ftp as well. So these are some of the problems with the current uh weather data. That's out there and by the way this applied. You know these. These issues are kind of focused toward this parametric use case, but they really apply to everyone. I mean data scientists who are crafting government policy um scientists who are studying climate change.

A

I mean these. These weather data sets uh are are global and they are a global resource and they, in my view, like the just importance of them, is not really matched by the you know, attention that is paid to uh how they're made available so I'll just go through this list here, but with some of the problems um the formats are idiosyncratic. So you know all four of these different projects and also the other ones that exist have different formats. You have to kind of learn what they're doing um through their documentation.

A

Endpoints can be slower on available, and that makes it hard for us to uh to build production systems. You know we can't really hook directly into these sites because often they're just down so old data is revised without a log of changes. Yes, so uh you know one thing with these interpolated data sets is that you get your satellite data right away. You know and then station data kind of trickles in over time and what these folks do. Is they just edit the old data?

A

So you know I have a post from a month ago, I'll just update it, as the news agent comes in and prism actually does six revisions once per month uh for every data point, so that just become- and it's not tracked so that just becomes a huge problem for us and really anyone who's trying to get. You know exact numbers there. um Data is not securely hosted, so I mean you know these guys. I mean this. Isn't that they're not trying to like lock this down from attack, because why would anyone attack it?

A

But you know I mean if we're going to be building up this, uh this parametric space. uh Sorry guys, but you know that might that might end up happening um and it should be secure anyway um and finally, chain of custody for interpolated data sets is not well documented, so I mentioned that you know these interpolated sets have inputs and where the inputs come from. Well, you know, let's that would be great to have more information about that deweather. This is our answer.

A

This is a this is a project that uh we've been working on for a month or two or three or perhaps four it's hard to keep track of time. Frankly, these days uh on that scale, but um this is our answer to weather data so very succinctly.

A

What it is is, it's you know, decentralized weather, so uh take all the all the really good weather data and put it on ipfs, and uh we have a client library that I just published to uh pie, pie last night that can be downloaded and installed and if we have time I'll show that um pip install de-weather, underscore client and that's also on our github, so that's exciting, uh so I'll just kind of run through how this kind of solves those uh those problems.

A

I I showed in the previous slide, so there's all this ingestion code and that stuff standardizes things, so you can just say give me a prism. Give me chirps give me uh you know: noah cpc whatever and it'll um it'll return, all those in the same format, and it will return them in data structures that make sense for your language. So in python this de-weather library does.

A

uh You know dictionaries that we also a lot of our data science team use.

A

uh Numpy pandas, so this data frame object, um use an available and reliable hosting uh hosting infrastructure. I think that was supposed to be so. You know, let's not have downtime, let's not have really long weights.

A

Let's also have a uniform api for accessing everything. Right, not you know a different endpoint for every data set track. All data set revisions, so you know with prism with those six revisions.

A

We just take every revision and we just have six different data sets and you can just get whichever one you want, and that makes it possible to balance timeliness with accuracy. So you know with our use case. It's like if we're doing a prism contract, we'll usually go with a payout at the 30-day revision, and that gives us you know certain amount of accuracy and a certain amount of timeless using immutable data store data is secure once posted. So this is a real ipfs thing here.

A

um Having you know, uh uh weather data um be addressed with the cid system. uh Is a big deal because it it really makes it much more feasible to to do these. You know automated payouts where you're just pulling a cid and and that's it, there's no disputes, there's nothing. It just kind of cuts out that that link in the chain um create a mechanism for tracking data input. Data sets, so this is kind of getting. Actually. This is past the weather. This is getting into uh d climate, which is uh you heard it here.

A

First, that's going to be a very ambitious project which is going to use ipfs as its backbone. um More on that later. That's another talk, I suppose, and make it open source. We want to bring the scientific community on board. We want to give grants to these entities to uh to to incorporate this into their pipeline, so they can post directly and- and just everyone can have a great time.

A

You know I think ipfs really is a great match for uh these collaborative, like you know, scientific and academic projects, because you know everyone is contributing um and why why shouldn't the hosting be uh be distributed as well all right. So this is my info. uh We are hiring uh if this is interesting to you uh shoot me a resume uh if you want to come, write, ipfs code. If you want to do some, you know if you want to help help me manage this open source project.

A

We have about 15 people right now, uh working at arbol and and we're rapidly expanding.