Ethereum Foundation DevCon 6 - Cryptoeconomics Track, 15 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Reinforcement Learning for Query Pricing in The Graph by Tomasz Kornuta | Devcon Bogotá

Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/reinforcement-learning-for-query-pricing-in-the-graph/

Indexers in The Graph protocol use a DSL called Agora to map query shapes to prices. However, manually populating and updating Agora models for each query is a tedious task, and, as a consequence, most indexers default to a flat pricing model. We have created and deployed reinforcement learning agents for Indexers to automatically compete on pricing. This talk will focus on our development process and our study of the market effects of multiple competing pricing agents.

Speaker(s): Tomasz Kornuta
Skill level: Intermediate
Track: Cryptoeconomics
Keywords: reinforcement learning,dynamic pricing,automation

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers.
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

A

A

Good morning, everyone, how are you so? Let's start I'm Tom I work for simiotic labs and today I'll have the pleasure to present you our work on reinforcement, learning in color pricing in the graph. So first, let me start with the outline. I will tell a few words about semiatric. What we are doing, then I'll introduce what what's really we are doing in this area. Automated price Discovery I will formulate the problem. I will add a few words up when it comes to modeling how we model that problem.

A

Then I'll briefly went through reinforcement, learning Basics. Let's say it's just a refresher for those ones of you who know or maybe do not know, enforcement learning then agent-based modeling is the the core of the talk and I will show how we're using that for testing system properties, then, and for testing well different behaviors when it comes to using single and multi-agent setups, and then I will show that actually we deploy that that solution in production. What are the results and finish with a small summary all right?

A

So, let's start the first semiotic symmetric Labs was founded in 2020 by Ai and cryptography researchers. uh We are focusing on applied research and we are one of the core depths of the of the graph protocol. Also, we are developers of others, the optimal text aggregator. uh Actually, the the lead of that product is giving a talk at the very moment on a different stage. So bad luck, okay, so our expertise.

A

We we combine the cryptography with AI That's, our core expertise and, of course there are software engineering involved and we focus mostly on building infrastructure. We focus on crypto economics. Today's talks actually focusing on this on crypto economics all right. So what was the scenario? So the scenario in here is that I don't know which one of you know the graph. Actually, who knows the graph protocol all right: okay, 50, 50. good.

A

So, basically, in this scenario, we just focus on on on one part of the graph system right of the graph protocol, but imagine that you've got some customers that are sending queries right and you've got indexers that are indexing, blockchain and our mechanical serving serving those queries. So, between those guys there is an entity called the gateway which is doing we're.

A

Just gonna go acting as a query market right so depending on the price, the price, depending on the quality of service, a given query sent by the customers when we would be distributed on this or or that indexer and indexers earn money by serving course right.

A

So they can control the prices of the service queries and the core idea that that I'm going to talk about in today's basically wanted to have Dynamic pricing based on query volume received by an indexer, and we call that auto Agra and then I will explain the second why we are calling that Auto augura.

A

So this is how we model the the scenario right. It's it's simplified from one point of view because, as you can see, ah as you can see, there are no no, let's say agents in here we get rid of the customers for the purpose of that flooding of those simulations. Instead, we've got a traffic generator right, something that just says okay at given timestamp. This is the query volume that you've that needs to be served right.

A

We also got query distributor so something that basically look at at the bits of the agents, so the agents are putting their bids and then, depending on those bits, the queries will be distributed across those different agents and those bits are expressed in a domain specific language called Agora. That's why the I the the the product we call it altag all right, because it automates that that building otherwise the users will have to the index cells have to basically create those those price models manually which is kind of tricky right uh all right.

A

So some assumptions selected assumptions what we're discovering in here. So this is only about how we generate the dot query volume, so we've got in here. So this is basically the shape that that we we model so the x-axis there is this price query price slash. This is a budget of of a customer that customers are saying. Okay. This is how much I can pay. For that query. This budget is unknown right, that's important! Then it's not sharing that with the agents. It's not sharing that with the indexers.

A

It's something that that that is setting on its own. So this is something that we want to discover right and the budget can move right from left to right, which means that if the, if the, if the indexer was set its price, its price bit higher than this well, he won't be picked right. Also, there is some kind of like a noise. We we added a noise to that volume.

A

To that query, volume and important is to know that there's there's a game happening right in here and the game is suraj's goal is to maximize the revenue, but gateway's goal, and basically protocol goal is actually quality of service right. We want all the queries to be served. You know uh five nines and so on right, so there's a game happening all right.

A

So if you know how model- let's not now switch gears to reinforcement, learning, just one-on-one, most classical thing so down there in reinforcement, learning, we've got two main entities, agents and environments agents interacts with environment. By executing an action, then agents actions, change the state of the environment. I just got reward in the observations after after oh okay and I just can up their its policy based on the received reward.

A

In short, that's reinforcement learning, so we're using different types of agents in our simulations uh in here, I put two most important: criterions one are trainable, reinforcement, learning, online learning and the other ones are like rule based.

A

So there are some predefined behaviors and also we've got this second Criterion Solutions can be stochastic or deterministic in here, I put some some simplified classification of of the reinforcement, learning algorithms and I highlighted the two that basically, we will be using in our simulation, so we'll be using vanilla policy gradients and mostly most of our agents, will be using PPL important.

A

There is one important type of ideas that actually we we are mostly focusing, which are we call them gaussian Bandits. So the idea that this is sustainable, uh stochastic agent, with policy that is represented with a gaussian and then when an agent is, is supposed to perform an action in something from that gaussian right. So imagine that there is this distribution and when that, when the digest is supposed to make an action, it's just sampling right and in this case Okay the agent sample, is 0.8 and from that blue distribution.

A

Second agent, sampled, python 24 right and why Bandit Bandit is because the agent is not building any an internal model, an internal representation of the environment, all right so going to the most important part, so testing based testing properties with edges-based modeling, so in here so already kind of explain this, but in the first set of experiments we focus on single agent simulation, just testing the properties of an agent, and this is a single gaussian Bandit, with a modified PPO policy. Update Rule and the query distributor is like a super naive.

A

You know, let's say version of that. What could happen if here is that we've got access okay, depending on the price. Business is just inverse inversely proportional and in the first step, we've got a fixed customer budget with some noise. Of course right, and what we are checking in here is the following: we've got market conditions. We've got two criterias for every experimental custom criteria and a property that we want to test whether the Bandit is fulfilling that that Criterium is is operating the way we want or not.

A

In this case the Criterion is customer budget discovery, as I've highlighted before, and the Agents do not know the budgets right, but so we want to discover that so first, so that's there will be lots of things happening right, because we are tracking many things there are in simulations. So let me try to let's say unfold that and go step by step, explaining what you will be seeing in a moment in in those animations in those videos that we captured during the experiments. So first there are five plots right in a single video.

A

First one is the most important. So what we've got in here is the is that budget right? The query, Volume Plus Plus customer budget. The solid red line is the agent's current policy. The dashed line is the initial policy that that agent is starting from second in the second plot from from the top. This is basically query volume server by the agent at that given time step. So what happened in here is Agent actually didn't serve.

A

A query right didn't get any queries, because it's he sampled somewhere in here right like few steps ago, he sampled, some like the the the price, was too high right, so the agent didn't get any queries so which which also is shown in here. So this is the aggregated query. Volume served by a given agent in red and the blue ones shows basically how many queries were dropped. Weren't served at all right, so you can see in here there's a tiny, tiny step which is which actually is associated with this.

A

Then, in the next plot we've got agents, revenue and aggregated agents Revenue over time, all right. So let's run our first experiment right. So once again, what we are testing in here. What we are trying to discover is word up that, where that that budget is right, so as agencies is learning right, so first, the gaussian is was super wide right like it was like much much wider and once the agent, as you can see it's something from time to time somewhere in here right above the budget.

A

So it's getting more and more sure that okay, this is the max that I can get right, that's the highest price! That's that's the highest bid that I can make. So, as you can see, the agent is dropping some of the queries right. uh What is revenues is nicely nicely growing, all right, so first proper property test right, that was the static environment. So how about Dynamic environment right by by Dynamic I mean that the customer budget can change, can vary over time right.

A

So in this case we are also testing the same property, whether the customer budget can be discovered by the agent, but in more difficult market conditions. Let's say right so once again, we are starting. Agent is kind of a converging capturing the the right, the price right, which is kind of optimal and then the price changes. So what happens is the agent is, is not getting getting any queries right. So it's saying: okay, I, don't know where am I, so it's press, the gaussian and once the queries are once again captured it.

A

It moves to the left right. So it's like okay I was above the budget. Let me let me go back right and that happens. That is happening over time right. So there are plenty of queries, queries dropped, but still we are. We are making some nice Revenue right and we are reacting to the to the dynamic, dynamic, Market cool second property check.

A

So in further experiment we said okay, so there are some subgraphs that actually you have no queries at all right. There is a moment where nobody's is asking for queries. So what will happen then? So that's a different scenario. In this case. We call that the marking conditions. Basically there is no demand for your services right.

A

So we wonder how we should add, so it will happen in a second right, so there will be like three iterations we're recording few of those those videos, but, as you can see already, there's a hint in the right hand corner that something is not right, all right, gaussian spread. This is great and now time set 400, because every 200 steps were like you know to write it. Okay. So what just happened? There are no queries.

A

So, according to the design, the agent just said, okay, let me spread right, so I can sample from wider distribution, but actually spread all across the the the the price Biz domain. So we said okay. Actually, this is not a good behavior right we. If this is happening, then the agent can can sample at the same probability like super some super small prices, price bits and at the same super high.

A

Moreover, recovery from that will take a lot of time, so we said okay, can we improve that, so we implemented something that we called graceful initial pull, which basically uh the idea is that okay, there's this initial distribution, that the agency is starting from.

A

This is something that the index service will parameterize the agent when, when, when the agent is deployed and where, when there are no queries, it just slowly is pulled towards that initial distribution right and when the queries will appear once again, it will get from there and once again start there's something I'm losing from time to time.

A

The voice I think um Okay cool, so we pass the property. So now, let's move into multi-aging simulation, so this was great at that point you know we aren't. We aren't competing with anyone up till that moment. Right now, we are starting with with some we're competing with some deterministic agents. The idea is that, right now, all the all the indexers have that the Agra models that is fixed right. They are not changing the models, so it can be seen as a rule base right.

A

So there's like okay, that's my bid and I'm sticking with it, no matter what right. So this is what we are trying to model here: competition with deterministic agents and the property that we are testing is called discover of prized bits of competitive agents right. So we want to make sure that we we also will, uh we can, let's say, compete with them and discover what are what were their bits without actually knowing them without having That explicit information. So those three lines in here.

A

Actually those are the three three competitive agents with deterministic policies that they're just set, and the blue agent in this case is the one is the Bandit right and, as you can see it after a while, it's set up like a price just below the the the the cheapest agents, which means that it's trying to once again snatch the whole Market. If you will see a number of total queries right, you can see clearly see a difference right.

A

That agent is dominating right, so I'll say: okay, this is nice check cool, so how about stochastic similar experiments? We also want to discover the the price bits of other regions, but their policies are stochastic, so we model them in the singular way as gaussians those gaussian distributions right, but they do not change right, so we observe similar Behavior. This is great right, so the agent our agent right, our band it moves somewhere in here and, of course, it's sampling.

A

You know that the red, the red, the red agent, uh stochastic heuristic agent, is also something from that distribution. So it's changing, but as we we can see direct moved slightly below and shrank once again to capture the market right. All right, right, I think it's. We check that property awesome. So how about it? Will right now compete with like make a competition where we've got plenty of those bands deployed, and we hope that actually, this is the most inter.

A

That's the most interesting case right, because we want Auto agara to be run by all the indexers in the graph ecosystem, so same property slightly different market conditions, all right. So what is happening in here? Sadly, it's not good, it's not good, so that I just are acting actually as designed, but as you can see, they're like fighting fighting fighting going moving moving to the left hand side, which means that they're lowering their prices all the time right and, as you can see in here, oh, you can see the labels, but this is this.

A

Is the revenue that they're getting right at a given time step? Basically, it's going it's converging to zero, which means that after a while, the digest will be basically serving for free right, and so that's not a good situation for the indexers right, because in here we are not even modeling the we are modeling the revenue, not the not the. We are not modeling the costs right, so at that point agents will be well indexes will be actually uh well paying for that, instead of making money all right.

A

So we call this this this uh phenomenon or this outcome raised to the bottom and we run many simulations. We discussed that a lot, and actually we realized that. Well, this is the expected Behavior right in this setup, so we've got some images that are basically trying to snatch the market right and they're, purely driven by by the by the query volume right. So if this is the case and the environment just natively distributes, you know the queries based on the the price bit rest of the bottom is the expected.

A

Is the expected outcome right, so it can be addressed in many ways, but we look at the graph protocols. What are the features right and down there and what are the assumptions? So one assumption is that all the elixirs should have freedom with their pricing right. There should be any limitations so that, if they can, if they want to serve queries, superb that are super expensive or super cheap. It's okay and all the indexers should be able to make any profits.

A

So the conclusion was that actually is the Gateway right that the thing that controls the market should implement this kind of anti-dominant policy and so happened it does. It already does so in our next step. What we did we wrapped the existing isi right, the Isis index selection algorithm, is one of the components of the of the of the Gateway right. So we just wrapped it in here and using the simulation. We are once again testing the same conditions right competing. You know many versions. Many government is competing with each other and the property. Okay.

A

Can we discover each other prices and maximize our profits? So the outcome is totally different right. We ready by changing changing the way the the queries are distributed. Actually you reach the like a consensus right so that all the agents are discovering the markets right. What was the market at the same? Gateway is looking. The ISA is looking at their their quality of service uh at their prices and trying to like feed everyone right, not just okay, you're, the winner, you're you're, searching all the queries all right, so this is really nice.

A

So please note that the gate will also leave some kind of budget right. That is the this design, such a way that actually it's it's not enabling the agents to to consume all the budgets right. So the budget is, there is a portion that is left to the to the let's say to the customer: okay, so we're on some more simulations. Of course it's like okay free can we run that with 10 20 agents right? Will that still work?

A

It seems yes, it does so we are testing the same properties right, which are just a little bit changing the market conditions right. This is looking really good. If you look at look at the the query serve and the revenue it's like, all the agents are, you know, are: are making money, which is the desired, the desired uh Behavior right and they are rediscovering the market cool. So what we also did uh we we started. We wanted to see whether okay, so how about different initial conditions right, so we're assuming that different different in Xmas.

A

That, okay, that's my mean price. That's my variance. I want to maybe move a bit more expensive for, like less expensive other being wanted to see whether the the the the the system will converge to that equilibrium, and it seems that this is really happening right, so disregard the initial conditions. This is good. We also run some additional experiments with with frequent a stronger policy updates. So this one is our modified PPO agent, as you can see it's much faster than okay.

A

The red agent is the vanilla policy gradients, like parallel mental policy, Elementary update rule and the other one is just pure PPO right, so our agent is kind of like a dominating, and it's kind of interesting to observe that it's serving less queries right. The the blue or CN line is clearly below, and this is the number of queries that are served, but but it's making bigger Revenue right so, in short, is making less work for more money. So policy really matters right, all right.

A

So having those results, we are quite quite happy with with those we said. Okay, so can we deploy that in production and actually that's our solution? You know on a Battlefield, so once again, there's there are many. There are many things to unfold in here, so let me try to focus on one thing at a time right, so first, this is the mean. So this is gonna, go where the gaussian is on that on that x-axis right.

A

So what happens is that the initial value was too high, so the mean was going down first, but then insteadly. It was going up up good, so it was moving to the right. At the same time, variance was steady going down, which was which means that the gyrogen was more and more narrower, which means that I just were.

B

A

Okay, I'm I think I'm there and I'm good right, I'm, more confident that I should sample from that our smaller distribution. So the outcome is that the reward goes up. Maybe you cannot see it in here like super nicely, but that's the revenue right, that's the total revenue and you can clearly see that it's it's going up. There is the tendency right, so we deploy that the results were great, so I think we are well. This is the amount where I should start telling.

A

Okay, if you're an indexer, please go ahead, there's a repository to download play with it and we there are some additional materials, okay, but you know actually in a second okay. So let me first summarize so agent based modeling for crypto economics. That was the core of that talk and in here is focus. We focus on the diamond Dynamic pricing for the applied to the graph protocol to to automated price Discovery, and we use reinforcement, learning and and a multiple multi-agent simulation multi-agent modeling for for Revenue maximization.

A

So we can show how to use that for testing the properties of the protocol right. We came with that framework that, okay, those are the market conditions. Those are the properties and systematically start the test right. uh Our our agents are solutions against against those, and finally, we have deployed out our in in in production and have shown that actually, it makes sense so Future Works, of course, better policy, better updates, better policies, agents of multiple rewards of taking quality of service into account.

A

We are not doing that right now, just we are just looking at the at the query, volume, uh modeling and putting consumer agents into play right so that right hand side that we simply fight actually putting that back in the simulation. It will give us some more insight in what is happening right, so that's kind of that will give a simulator with higher Fidelity and uncomfortable redesigning the game right. So in this case there was no information. The agents were like acting totally independently the questions it will redesign the game redesign a little bit.

A

The graph protocol would that kind of perfect information will help right. There are some here are some additional resources, so my friend Alex is he there is he wrote this blog post, which describes more the technical side of deploying our Target in production. He also gave a talk around this about this during the last track. Episode in June, I think so, there's a recording on YouTube and, of course this is open source. So I encourage you go ahead, play with it write all those visualizations. Are there basic?

A

You can reproduce that and try to deploy your your, your altarger in production right. uh Finally, I wanted to highlight some of the of the other words that samiotic is doing so their stock. Actually, that's happened right now that that our friend Matt is giving his the product and research data for us he's giving it right now. So that's uh that's.

A

That said and Savvy is giving a talk today about our work and or let's say our our our our our our our trip in the to the snarks world, and we are focusing on on this- is more like cryptography. uh We are focusing on verifiable payments for the graph protocol as well. So, if you're interested in those those topics please come to us, we've got plenty of positions open. We, if you're interested, please go ahead. Thank you for your information.

A

Excuse me there's a gentleman who wants to.

B

A

B

Hello and thank you for the talk I was wondering if the customer has a very, very low budget, will you try to match it and make a loss or will you shut down? That's.

A

A great question actually I wasn't showing this in here right because it's shaky like I, said the goal of graph is to have five nines. So right now we are working with with the guys that are, you know, implementing the guide when updating that a little bit, because for now, if the customer will say, okay, I'm I want that for basically free, the Gateway will still try to distribute that right.

A

It's not the best outcome, I would say for the indexers right, but we are changing that we are. We've got a tool right now right, so you can think about. I was showing you that no we're working in this work. We are focusing on the agents right, but we developed a tool that enables us to test the properties and the outcome of the whole thing right when there are different players in the game, including Gateway, so great question, it's not tricky right, the tricky thing. What do we want to achieve?

A

We want the indexer to make money, or we want five nights right. The quality of service to to be super high.

B

I have another question: sorry, uh maybe more technical. When you show the experiments whenever the budget of the client was going down, the spread of the ocean was was widening, but the mean was things the same, which means that you are still sampling to the right side. Where you know it's not the right price. Would it be more effective to move the gaussian to the left as you spread? It.

A

Yeah, uh so what what's the question? Actually because you just repeated that the design Behavior right, if we overshoot, if we overshoot with the the the the price, then we don't know whether the reason was there were no queries or we were too expensive right. So we spread because we want to sample from wider distribution right all right I've if there are any other. You know additional questions. I'm I'm here I'm I'll be delighted to answer your questions, but it seems that my time is up. So thank you for your attention.