IPFS IPFS Camp 2022 - Lightning Talks Track 1, 2 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Data for Degens - Techie Teee

Description

This talk was given at IPFS Camp 2022 in Lisbon, Portugal.

A

Hey I'm, techy, T and today, I'll be talking about the revolution of decentralized data, are also data free for degens and how we can bring more decentralization to data Pipelines IDC global data sphere estimates that there is currently 175 zetabytes of data. This is enough data to wrap around the world multiple times. That's a lot of data, and this data is only getting even more robust when we consider the web 3 data that's being produced with every single transaction across protocols.

A

Foreign data also has a tremendous value economically and when it also goes wrong, it has a tremendous impact. According to MIT Sloan Management in 2017, they estimate that nearly two trillion dollars is lost every year. An economic impact through poor data infrastructure and poor data quality.

A

Data ownership and data quality matters it matters, because when we don't manage data infrastructure properly, it increases the risk exposure, it decreases, Regulatory Compliance, it impacts the accessibility, traceability and also the Integrity of the data that is present foreign.

A

This is especially true in high stake sectors such as legal finance and medicine. This is where it's even more important that we get it right and we make sure that we have the property infrastructure in place to give the best outcomes possible for the business decisions being made.

A

In my perspective, centralized data is equal to vulnerable data. We only have a few major options available for cloud providers and even for a lot of the the damps that currently exist in the ecosystem, they still rely on web 2, tooling.

A

If you remember uh earlier this year, when AWS had a major outage that took out a bunch of damps with it right and I think that that's not sustainable long term. When we think about having the most reliable and robust in infrastructure available, we need to continue to think about. How can we make more decentralized data Pipelines and create more robust data infrastructure.

A

So we have to redefine the data life cycle. Here are three main areas of opportunity where I think we have the chance to do this, the hosting the storage, the data and also the query and transformation of the data.

A

So here's a side-by-side comparison of the tooling for web 2 versus web 3., so we think about hosting. We have our major Cloud providers like gcp, Azure, AWS, Oracle and then also hosting. You know your code for smaller products on GitHub, then in the web 3 side we have ipfs nextcloud textile diode and radical, and the difference between these two groups of I'm hosting options is dramatic, with gcp ads and Azure and the other hosting providers you have.

A

A tier gated features based off how much you're able to pay that also impacts how redundant your data can be, how accessible can be, and it makes it limited to the users who can pay the largest bills with web3 hosting that changes, your democratizing more robust accessibility and also traceability for data.

A

Then you also have the consideration of the databases themselves, so gcp, Azure, AWS and Oracle all have their different sets of databases available within them, but there's a growing group of decentralized databases that are becoming available like big chain DB, ceramic protocol, textile diode, Covenant, SQL and Noms.

A

These tools use a combination of traditional SQL approaches, with decentralized databases to make the data more readily accessible and available. To me, I think that this is an under utilize opportunity for a lot of dapps. A lot of daps still can use traditional databases and, as I said, it creates a major thought opportunity for the devs to fail. If there are ever any outages in our you know, one of our top four Cloud providers are currently available.

A

But I think the biggest opportunity for making a more decentralized data pipelines exists in the transformation environments and how you can do the queries. So some of the biggest uh private providers are databricks, gcp, bigquery and Azure synapse positional web 2 side and then on the web 3 side. You have some companies, like Flipside data, analytics chain, link and graph, so these providers use a combination of apis, oracles and traditional SQL queries to be able to it's, not a search. You know what's happening in the blockchain data, but there's also a big part.

A

That's being left out of this. When you go to your more mature web 2 environments for manipulating data, you can do like large batch processing, manipulation, transformation, the data machine learning all in one spot, whether it's you know gcp Azure AWS. Currently, nothing really like that exists for web 3.. The closest thing we have to it are.

A

You know some protocols like ceramic that are continuing to make progress and making extremely available and being able to segment the data, but we still have a long ways to go, but just because we have a long ways to go. Doesn't mean that we shouldn't use some of the tools available to make our pipelines more decentralized, so thinking about hey can I. Instead of Hosting my software on centralized, you know cloud provider can I use a decentralized server, you know can I.

A

Instead, you know think about you, know uh decentralized database and from start to finish thinking about what pieces can we progressively put in they're, more decentralized that can make the data more reliable, accessible and increase overall data quality and the ETL pipelines for blockchain?

A

We still have a long ways to go, but maybe the next iteration of how this will look for web 3 is mixing in some of the good pieces from web 2 and the Innovative pieces we have from web3 and having more decentralized data pipelines available.

A

So I'm excited about that, especially like Apache, Big, Data tools and how we can combine that with some decentralized tools and make some really Innovative stuff for decentralized to Pipelines.

A

So yeah we've still got more work to do in this area. It was exciting to me. I think this is like the infrastructure that supports the next few million people coming into web 3 like I love, our nft projects and metaverse plays, but we also think about what are the bones and the structure that's going to help to support the web 3 ecosystem at scale and I think this is like a really important topic, and this is why I wanted to share briefly about it. So I love to answer any questions. Anybody has.

A