Everything EOS Short Clips and Rants, 30 Jun 2023

Previous Meeting

Next Meeting

⏯

youtube image

►

From YouTube: Substreams for Antelope: Mastering Substreams for High Throughput Blockchain Data Extraction

Description

Substreams on Antelope is a high throughput blockchain extraction tool that enables the extraction, transformation, and loading of blockchain data into various mediums. It supports multiple languages, filtering options, and aims to bridge multi-chain and multi-platform ecosystems.

The EOS Network Foundation coordinates financial and non-financial support to encourage the growth and development of the EOS Network. We’re harnessing the power of decentralization to chart a coordinated future for the EOS Network as a force for positive global change.

// EOS Network Foundation //
https://www.eosnetwork.com
https://twitter.com/EosNFoundation
https://www.linkedin.com/company/eos-network-foundation
https://t.me/EOSNetworkFoundation
https://discord.gg/eos-network
info@eosnetwork.com
#EOS #Web3

A

A

S on Antelope we're using a technology called substream, it's been developed by stringfast team and it's under the umbrella of the graph and it's a high throughput, blockchain extraction tool. So the tools of what it does is that it extracts transforms and load typical data processing. Where, if you have data, you need to get it out of a database and you need to either manipulate it transform it. And then you need to load it into whatever medium that you want. It could be a CSV file.

A

It could be a database, it can be a SQL database. So those usually like the three main core pillars is extract, transform and load, and that's what substreams can do and there's all I'm going to explain all the tooling around it and how it gets across.

A

So I can start so for the streaming data it uses grpc and it also connects to connect web, which is a sort of a proxy for grpc and grpc, is built by Google.

A

So it's really reliable and it's also language agnostic, which means that grp, when communicating with grpc, you can use Python node.js, go rust, pick your language and most likely there's going to be grpc support, so you can stream in any language most of the team we use node.js, but the streaming fast team builds everything in go and we also have a tool in Python.

A

So you can see that it doesn't really matter what language you're building this with you can pick your language and they'll, probably support for it and when you're coming to extracting data, that's where the sub stream kind of like a smart contract, you would build the code to extract the blockchain data that you want like building an index, that's using rust. So all of the extraction tools are using rust.

A

So there's a little bit of learning curve, learning rust, but if anybody's build smart contracts on EOS and you get a learn, C plus plus that's also a learning curve as well. The type of data that you would extract is block data, so the whole block. That's the first primitive data set that you're going to get from some stream. It's the block information and that's like the block number, the clock called the Merkle Roots. You have actions transactions, you have a state changes.

A

So whenever there's an action that impacts a table on the smart contract, it will show you the the difference so the before and after values, for example, when you're doing a token transfer, you're going to user a transfers to user B minus one plus one, and that would be a state change and that in substream is capturing all of those State changes, whether you want to keep the before and after or only care about the after, or maybe you only care about the Delta right the in between what are the changes you have access to that type of information and you also have access to the what time it happened, but which block number that all these events happened.

A

This you can store all that information or you can ignore it. It really depends on your use case, but you have access to all of those. Primitive data sets action, events dates and all the block, meta information, and the other thing too is after that you've got the data that you care about. You might want to filter it, because you don't want everything. You don't want the whole block with every single transaction, and this is where you get into filtering. You can filter by, let's say account or you can filter by a token symbol.

A

You can say: I only want EOS tokens or I only want the account. That's my account for whatever or my smart contract you can filter and only fetch the information that you want to get. You can also put parameters. So maybe you don't know what you want to look for, but you want to put a parameter says if from equals, I'm going to tell you later, which is like an input parameter.

A

You can also provide that which makes the extraction and the sort of filtering very flexible, because you can put a parameter that in the future, you will tell it what the account name is down the road, so you can filter it. So there's a filtering and the once. It's all filtered and extracted and transformed to your liking. Then you need to load it into your database of choice, let's say so. That is what we call a sinks, so it'll be like some stream syncs they're, usually programmed in either node.js or in go.

A

So those will be the two languages, so you don't have to use rust anymore. You can either just skip to the I already have my data I just need to load it into my postgres or SQL, and that's where you get to see the data for the first time because you've extracted it now you're loading it into. Perhaps it's a CSV or perhaps it's a web server that you're just posting data to a web hook or maybe it's Google sheet or you want to load up your Google sheet.

A

So Google sheet is like a database might not be the most elaborate database, but it's a database so if you're using also postgres MySQL elasticsearch. All of that you can then load the data that you extracted from sub streams into your data source.

A

So that kind of sums up all the mechanics and you can see with that type of flexibility, with sort of an extract, transform and load you can build, you can, let's say, rebuild public API endpoints such as the atomic API, so you can rebuild some of these on-chain apis using all of these mechanics, the two big things that most all blockchains use or depend on tokens and nfts- that's, let's say the 95 percent of the use cases, everything that is D5 kind of revolves around tokens, whether it's the assets that you swap, whether it's the lp tokens that you perceive by providing liquidity.

A

So everything kind of revolves around tokens for the most partner aiming or is go and build a token API that would be leveraging substreams. That would be the the Bluetooth standard example that people can grab to get started. That way. It's simple mechanics! You have simple things like issue transfer all of the sort of token mechanics that you would expect. Then you have an example that you can then build. On top of that, maybe you have this very specific use case.

A

You have a game or you have a Marketplace, and then you can just modify the token some stream API example with your own and we're also going to be doing one with with nfts as well, starting with atomic assets. Atomic assets is a lot more complicated than the standard EOS IO token. So there's like collections, schemas templates, but it's all data that is a table or an action, and we're also going to be looking into doing the same for Ev chains as well. So we were going to have erc20 and erc721.

A

So that way you have something to compare with an evm environment, but using the same stack that way, regardless if you're on Antelope or regardless, if you're on evm, it's the same stack and it's the same mechanics using substreams and that's what we want to bridge. We want to bridge that it's multi-chain and multi-platform multi-ecosystem. Essentially it's! If, if you want to be able to use substream, you can be on evm or Antelope or any other supported chain that has sub streams on.