Ethereum Foundation DevCon 6 - Governance and Coordination Track, 3 Nov 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Parameter Optimization and Emergent Behavior in DeFi

Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on IPFS and more.
https://archive.devcon.org/archive/watch/6/parameter-optimization-and-emergent-behavior-in-defi-agent-based-simulations-and-reinforcement-learning/index

As in any complex system there may be emergent behaviors in DeFi protocols. In this workshop we will show the basics of how agent based simulations combined with reinforcement learning can be used to explore these and also for optimization of various protocol parameter values.

This will be based on https://github.com/vegaprotocol/vega-market-sim and https://github.com/msabvid/cpm_agent_based_sim . Ideally, try to follow setup in Vega-market-sim beforehand, but we'll go through it together.

Speaker(s): David Siska, Marc, Tom McLean
Track: Governance & Coordination
Keywords: Protocol development,agent based models,validation

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers.
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

A

A

This is a workshop will be talking about using agent-based models for analyzing D5 protocol and the examples will be we'll be using is through the Vega Market simulator.

A

We do a quick introduction and then I'm going to jump to a slide which will show you where to go on GitHub so that you can hack along I, really assuming it's a workshop right. So I'm expecting everyone to have a computer in front of them and start hacking. But if not then hey, you know you can just stay along for the chit chat, so my name is David I've been with Vega for the last four years.

A

I mainly look after the kind of Financial Risk and economic design, game theory, analysis and, together with me, is Tom who is coming from traditional Finance he's a Quant with Vega help working on the working on the research team and one of the speakers. Mark isn't isn't here, but he's done a lot of the a lot of the RL stuff. So he's the main brain behind that so I'm going to jump to.

A

B

So yeah we'll go back to introducing what Vega is in a minute, but given I don't know how fast the Wi-Fi is going to be in here um just running through a few firsts that set up steps that we can get going with, while we're presenting uh for anyone who hasn't noticed, there is Wi-Fi specifically for the workshops uh called Defcon workshop and the password is: was it built, build it 22., so give that a go if the Wi-Fi's being slow? uh So this uh repo at the top here, Vega Market Sim, should be open.

B

If you go over there, clone that, that's your your first step, you're also going to need, go uh 19, I, think it's necessary python. Ideally 310 39 might work and poetry just because it's probably the better uh package manager, probably there's a requirements.txt. If you just need that as well, we can go through actually setting it up in a bit.

B

uh Then, if you want to be able to view our sort of front-end console, you'll also need some some UI framework, yarn and NVM for node management, and if you want to try some RL at the end, then Pi torch is a good starting point to download. So broadly. If everyone can make a note of that URL before we switch back and.

C

B

We will start on actually presenting.

B

Is everyone good.

B

Cool we'll have an actual break for setting everything up in a bit, but.

A

Yeah, so we're back to the introduction, but it's it's sort of a workshop and if you need help, the person who isn't speaking can actually come and help you with your. You know whatever it is you're trying to do within this framework and if you're not having luck, then we can see if one of us will have better luck and there are not so many of us and we've got I think two hours. So we've got plenty of time to actually get everyone going and hacking.

A

So in terms of what we'll cover we'll quickly introduce Vega protocol, then I will try to make sense for agent-based simulations in in the context of analyzing D5 protocols.

A

And then you know we will introduce what you already started. Setting up, which is the Vega markets in which is relying on something called vegan all chain. Will um we'll explain what that is and then once we have that we will sort of explain how to use the market same, how it's set up. You know what what environment is there?

A

uh What agents are there and what you can do and at the end, we're gonna show you how you can you know, train reinforcement, learning agents on top of this and why you might possibly want to do that. So that's that's the plan.

A

So we start with what is Vega protocol. So Vega protocol is a layer, one blockchain for derivatives trading. It's uh set up from the ground up for trading margins products and the price Discovery is either through limit order, books or through auctions. Where do the auctions come in? Typically, when you're setting up a new market, you don't know what the price is. So initially there it's running in an auction, the auction determines what the opening price should be and then, if everything goes well, it's it's running as a limit order book.

A

If there are big price moves, according to the risk model, it might switch to a protective auction to make sure that the big price move is really. You know, what's happening economically, it's not someone with a fat finger. You know for getting a zero on an order and or adding a zero on an order and causing causing Mayhem. uh Anyone can create a market. What it means is you submit a market proposal to the blockchain and then it goes through the governance and it's and it's voted on.

A

So that's what's meant by permissionless here and what kind of markets at the moment it's it's a supports cash settled futures. There is roadmap for supporting any kind of derivative through a derivative payoff language which, which will be probably basm based, but that's that's just on the roadmap. For now it's it's cash settled Futures, and so basically, if you can find an oracle for the market, then you can. Then you can propose the market and it will run.

A

um It's got a bespoke liquidity provision mechanism because it's it's not based on now ever so popular constant product markets, but it's based on a limit order book. So you have to do something slightly different to bond liquidity to a limit order book. So it does that and at the moment, the the assets that have value that you would use a settlement asset on Vega they're bridged from ethereum, though in the future that there will be other Bridges but uh ethereum bridge is there you can on Vega before proposing a market.

A

If you wanted to settle in some other asset, that's not there. Yet you can propose the asset. The asset will then be bridged from ethereum, so you can imagine that there will be usdc tether. Ethereum, whatever people want to do is that if you're designing something like a trading platform, some of the goals that you might have in the design really sort of run at Cross purposes between a general purpose.

A

Blockchain- and uh you know what do you- what you would ideally like for a trading platform, so the things that I picked out are you know a general purpose blockchain you. You need something like a gas mechanism for people to pay for the cost that whatever transaction the chain is processing for them.

A

You know the cost has to be covered by the gas and, if you're trading, this is really you know really sub-optimal in in the sense that there are transactions that you want to include, and you don't want to penalize them by charging for them. You know if someone wants to post a limit order, that's good they're, providing liquidity. You really don't want to discourage them. If someone cancels an order, that's good they're, providing price information. You probably don't want to discourage that either. So you know: that's in products they're effectively promises right.

A

If someone doesn't look like they're going to keep their promise, you want to close them out immediately. Ideally, you don't want it farmed out to some Bots that are running somewhere, because then you're opening yourself up to Mev and on a whole whole host of other problems. So you can have Atomic closeouts again for margin. You want to do potentially quiet, Maps, heavy risk of computations, and you don't want to be paying for gas for those on a general purpose.

A

Blockchain you can try to do Fair ordering or as Fair as your definition of fairness allows Fair ordering of orders, and you know you, you have one specific application in mind, so you can optimize for latency, for instance, rather than optimizing, for you know, data throughput, so you're not fighting at Cross purposes with other applications that would be running on top of a general purpose, blockchain, which would have other design goals in mind and one one thing that I didn't put on the slide.

A

But that's also, a very good reason is that sharding is actually much easier in the context of trading, because something happening on this market and something happening on that market can be completely separate apart from Talking together through a treasury chart.

A

So you have a very simple use case in the sense of what needs to know about what you can design charting relatively easily okay, so that was that was Vega, Now sort of more General case for doing just trying to get rid of the mouse more General case for doing agent-based simulations in in the context of D5.

A

So you know, we've all seen various D5 protocols and uh we've seen you know when things go well, it's great and then, when things don't go well, it's called a hack or an exploit, but quite often it's not an exploit of a code. Bug, though that can happen, it's it's simply an exploit of of the mechanism design. You know you, you go.

A

You want to design a protocol. You start you, you think what incentives should I create so that I've got the desired behaviors and you do it in isolation. You really try to think it through right. You you take the smartest people you can find there are. You know dozens hundreds of incredibly smart people in defy they do their best. They do what makes sense.

A

They think it through as much as they can, but uh it's difficult, it's really really difficult to think through all the consequences, especially if you have one incentive mechanism here and other Hair Design, maybe in isolation and now they're interacting right, yeah.

A

Oh vendor is a name of Claus's fairness, pre-protocol for transaction ordering. He can tell you all about that. He's there. The the.

A

You can just go and show him no.

A

Yeah yeah, so you you put two incentives mechanisms together. If they were designed in isolation, it's it's not. You know completely clear that the combined effect is actually what you've desired and then, of course, D5 is all about composability. They love that, but once you introduce composability you're introducing another disproof stuff, you set up a hypothesis which is testable. You will never prove that it's true, but you can set it up so that you run the experiments that should disprove it.

A

If you try their best and you've not disproved it then you'll, then you let the hypothesis stand still they'll prove an otherwise, and if you're wondering what this is you've, probably most of you have probably seen Game of Life. It's it's the idea that you know you. You can take a bunch of very, very simple rules and they can lead to very, very complex, behavior and sort of agent-based models are one thing that will allow you to analyze this complex Behavior.

A

So yeah uh definition of what an agent-based model is from from Wikipedia and everyone's favorite agent, and quite often you want a lot of Agents or a lot of similar agents. So uh agent Smith is good. I've used this I've used this picture actually twice. I I was when you talk about civil attacks, I like to talk I like to use that picture, but here it's not about civil attacks here, they're good agents who are stressing your protocol.

A

So, in terms of you know what type of Agents you you would find in agent-based modeling, there are the the simplest ones, are what I call zero intelligence, which is a bit insulting, but uh what it? What it means is that they're, not learning and they're not optimizing. You just say this is the state of the world, and if this is the state, then they're gonna do this, and if the state is something different they're going to do that. So you know they're agents that have hard-coded actions and that's what they do.

A

They're the simplest to set up they're the simplest to understand, and, of course, if you hard code, the optimal actions, then they're perfectly good.

A

If you don't, then they're only as useful as the actions that you've that you've hard-coded and so the next level you you can think, okay, you know each agent I'm not going to hard code. The actions a priori but I'm gonna tell the agent exactly what the environment, what the rules of the environment that they're in are exactly what the rewards are for interacting with this environment, according to the rules and I'm going to optimize, in other words, I'm going to solve for the optimal actions and that's something that comes under.

A

You know the name either mark of decision processes, if you're doing discrete time, steps or control theory, stochastic control, theory. It's it's a reasonably well established field, and you know there are a bunch of algorithms. I could spend hours talking about algorithms that they have for solving problems. There is there are a few problems that you can actually solve by hand. One of the most popular is the linear, quadratic regulator, which you know um together with the Kalman filter, was one of the great successes of of signs of the 60s.

A

You know Blended on the moon, and all that. So you know good stuff and then sort of the the fancy new stuff is. Is the reinforcement learning where you actually don't have to tell the agents what their environment is? They just repeatedly interact with it, observe the outcomes of their actions, observe the rewards and update their behaviors. So these are, these are the type of Agents you might want to run.

A

You know, on top of your environment, to see what is happening and, of course, what is what is appropriate and what you need will very much depend on what you're trying to simulate what you're trying to test. But basically this is your zoo and then the typical setup.

A

For for these things, sort of you, you understand you, you have the agent agent takes action, it changes the environment in the sense that it moves it from the current state to the next state, so sort of new state comes out, a reward comes out and the agent takes in the reward and the state and again chooses the next action, and- and you repeat- and of course you can have you know these setups- that you repeat: sort of Forever Until Infinity, maybe with some discounting or maybe you assume- that there is some steady state or there will be a terminal time when the Game Stops or whatever problem finishes.

A

Then you evaluate how everything then, whether it will end well or badly, and then one thing I wanted to point out here is because it will be relevant if you're, looking at any of the code that we're providing is that the agent as it learning it, sort of it's in a state it takes an action, collects the reward, sees the new state and decides on the new action, and if you record this, that that's sort of all the data that you need to train the agent sort of offline separate from the environment.

A

If you collect these sarsa sequences, then you can train the agent separate from your from your environment.

A

Okay, so that's that's not so important. Now it will be more useful information later right and I think I'll hand over to Tom. Now, who will evoke you through the you know the Vega Market Sim, um it's a workshop. So if you have questions, I think the gentleman there already said, yeah you're setting a perfect example, keep asking questions.

A

Oh, do we need a microphone.

D

You think, thank you, hello,.

D

Hello, um do state space approaches, have any role to play in parameter, optimization.

D

Like a control, control theory based approach for optimizing, a ffected.

A

Yeah, so so you would set up. You know your environment, which in you know in in the case which we will show would be. You know the whole Market, how the trades happen, how the orders are matched and then you run that whole thing through the simulation see what happens, and then you want my want to do the optimization on top of that right.

D

But that's a agent-based approach right! There's you could you could use a uh like a analytical solution. So that's, let's call it an agent-based approach, a numerical approach, but a control theory approach. Yes,.

A

If you could, if you could, if you have an environment which is sort of sufficiently simple that you can really write down, you know all that state transition equations. Then you can go and solve it through. You know either pen and paper, if you're, supremely lucky or from through something like I.

D

Mean you can policy.

A

Iteration value iteration, something.

D

Like Matlab rather well, that's actually numerical, but you could use um software obviously to yes.

A

Yeah- and there is software out there, yeah.

C

uh Does this work for creating bots trading Bots yeah I mean like the whole framework you have here.

A

Yeah, so we're I mean Tom will talk through that, so so I'll jump a bit, but basically you know the environment on its own. If it's just the market, you know it's an empty Market. It hasn't even gone out of the opening auction. If no one proposes anything. So so you automatically need bots. So we will talk a bit about the Bots that are provided as part of the environment and, of course you can write your own Bots. uh That can do you know well whatever they want to do. We've got.

A

uh We've got some basic Market. Making Bots we've got some basic. You know price taking Bots we've got some basic um momentum Traders, but Don will talk more about that so I'll hand over to him.

E

Just a quick question: so if this type of systems do you take the code from what you are testing, for instance, for the Vega protocol, I just quickly check that it's, it's all go code right! Okay! So do you take this code and run it inside your simulation, or do you create a model of that? Okay?

E

And do you think that that's that scalable, if you want to, if you want to simulate thousands of millions of Agents, because perhaps you need you need that or do you think it it should be more in some cases, you need a model of what you are actually testing.

A

The microphone so so we're running the whole Vega stack again. Tom will show that but you're absolutely right and in some situations, especially, we want to do reinforcement, learning, having a simplified model on which you can train before you actually go on to the full stack, which will be inevitably slower than you would like, because things are never as fast as you want would be useful, but we don't have that at the moment. We're learning on the whole thing.

A

That's why it's important that we use an algorithm which collects the sarsa sequences, and so then I can learn offline, but we don't have model. There are reinforcement, learning techniques which sort of build up model from observations, and you can do a lot of clever stuff, but but I'm far from being an expert on that.

E

Okay, thank you.

A

So so one thing that I didn't say is one reason why D5 is so good for agent-based modeling is you know you can try and people have been talking about. Agent-Based modeling in the in the real economy, and economists would love to do that, but they don't have the environment on which to execute it and in D5 we do have that because things are typically open source. You know, I can go and take the smart contract and run things through the smart contract. I don't have to create the environment.

A

I just say: okay I care about how this smart contract interacts with this one I take whatever ganache hard head, I put it there, I set it up as my environment and I pump. Thousands and thousands of simulations through that I can't do it in the real economy. But here we can so it's awesome. Okay, more questions.

F

Okay, are you really looking at things like protocol solvency or profitability.

G

Of certain agents.

F

That are performing trades or kind of what are the reward metrics so I'm assuming they're, not like bugs, but more financial kind of outcomes.

A

So so for us, it's very problem problem specific. So, for example, if we want to test you know what some of the network parameters should be, we set up an environment and, and then we look at things like you know, if if I've got a rationally behaving Market maker and the parameter is X, what is their return on Capital and I'm thinking? Well, you know. Maybe this is letting them earn too much money.

A

It doesn't make sense, but in terms of the reinforcement learning guys you, you would typically look at Financial rewards and then yeah you can either run it thinking of it. You know maybe you're thinking, they're trading perpetuals, so it's just forever or you're. Thinking that they're trading, a settlement market and so the reward comes at settlement. But it's it's very much something that you have to design for the specific purpose that you have in mind. This is sort of more the toolbox and I'm gonna.

A

You know wrap up the questions hand over to Tom, and then you can ask afterwards, because I have a field at with the questions. We're sort of jumping ahead a bit so I I want to hand over thank.

F

B

Okay, so yeah discussing the market, simulator will get to doing so ourselves in a minute, but first I think it's useful to go through and yeah to cover again slightly more technically some of the some of the questions, how we are, how we run these simulations um and how how we get interesting results out, so the vegan market, simulator broadly the way that Vega itself is set up. We run with a tenement consensus layer for those who don't know tenement is it's a cosmos based consensus.

B

The problem with that is that it's generally tied to our block time, which is about a second. So if you run the whole thing, you can watch the market going second by second and you can interact with it, but you can't run as the question in the back said. Thousands of scenarios run it through a year. Do all these things. So what we do we strip that out? We run what we call a null chain entirely on on one PC, and this just broadly accepts any transactions. It's sent it will it still checks they're valid.

B

It still runs through the whole Vega core logic, but um just adds into a block allows you to control time forwarding, so you can it'll just sit and do nothing until you tell it to run forwards or fill up a block, and so you can inspect at any point in time.

B

For, however long you want what's going on, then, on top of this, once we've got that running, we have in Python an API layer which lets you express your trading actions, your agents behaviors in much more Market Primitives, without having to worry about the fact that you're running on a blockchain, occasionally it still Creeps in because that's the way well, but you can broadly talk about an agent trading on a market without having to worry about what Block it's in whether it's been confirmed, and this kind of thing once we've got that we then build out our scenarios and our tests so I'll cover possibly next slide exactly what a scenario is, but we can, with a range of composable Agents composable setups, build out scenarios to investigate and test either certain parameters test that the market doesn't explode test.

B

This, the core itself doesn't explode, or if you just wanted to build an agent, you can try and have a realistic market and throw yours in there.

B

How this all works is your python layer manages. We've got two different kinds of interfaces at the moment: a Vega service null which spins up this as I mentioned a service entirely on your machine or what we've recently created, which is Vegas Network, and that allows you to connect to a running external network and run the same agents that you've just been running locally, but on the real Network that's going on. So that can be useful in some cases too. It allows us, for example, on our test net.

B

We can run agents and simulations that other real people can interact with and see how that's working um we'll demo all that in a bit um so yeah back to setting things up and viewing I'll go through the viewing in a second but yeah yeah I was gonna gonna break here anyway, so yeah, so um we're back here. I, don't know how many people have managed to get to the end of this, but I think we'll probably take a sort of 10 minute break to catch up. Anyone who has them.

B

We can wander around and see whether everyone will manage and then 10 15 minutes. We can come back and continue with doing something more interesting with it. If you think you've got it, set up, try and run these uh scripts at the bottom. We can also tell you something more interesting to run if, if you're stuck um I think I had a question. Yes,.

H

So I can then more uh in a more real way, basically test. My uh my ideas.

B

It's actually it's something we're pretty much currently working on, yeah is being able to better snapshot an existing running chain and be able to totally restore from that weird. There is there's the capability currently to you can run your simulation and save that and rerun the same thing. So you can have a an example that you run and then run your things after we don't currently have a way to pull from an existing Network.

H

Okay, thank you very much.

G

That people should be using in their protocol and I, tend to tell them that they can more easily solve for tokenomics problems by including a market mechanism to optimize. And let you know the market do those kinds of things. Like example, uniswap V3 solves the hard problem of how to set the curve by allowing concentrated liquidity. So my question is: do you guys have an intuition for which kind of simulation like which kind of systems require a simulation and which kind don't?

G

B

Good question I I guess I'd go back to as David was saying earlier. It depends on how well you can, if you can solve it. Mathematically, that's probably always going to be optimal, but when it reaches a certain point of complexity, um yeah I, don't know how you would do it without either an agent-based simulation or casting modeling.

G

Or that kind of thing is there an upper limit to the kind of complexity that you can model in this, because I'd imagine like in this example of using a market mechanism that adds a lot of additional complexity? Is that like? Would that be an issue in the Vega simulation.

B

In terms of complexity, well,.

G

For the simulation yeah I mean it multiplies the number of agents in the in the system um by like orders of magnitude. If you have to use a market to um Source lots more participants, ladies I guess is there: is there an upper limit to the kind of complex system that the Vega Market simulation can handle.

B

uh Well, practically I guess there is enough limit. We are the the system itself, we're hoping we'll be able to scale to many thousands of people, so that should also scale to to the number of Agents. So I guess it's it's a a goal of both the Vega protocol itself and the market. Sim, thankfully, have similar things, because, ultimately, what we want to do is build a realistic market. So if you can build as long as we can run a realistic Market on Vega protocol itself, which should be able to do in the simulation cool.

G

Thank you guys.

A

Something they're exactly going back to the going back to the points that we were making. I absolutely agree that if you, uh if you're, designing, tokenomics or protocol novel, then either you can prove that it works or you can use this agent-based simulation to sort of not prove that it works, but show that it doesn't and go back to the drawing board. Yeah.

B

A

In terms of training, though, there are two questions: I I can have the environment sort of I can have some agents that are better. You know the agents, take their actions quickly and get on with it. But of course, the moment you have more than one optimizing and learning agent, then it's the whole. You know Game, Theory and- and you know that they're, probably and and then it's also very difficult to set up or very difficult. You have to think carefully about how you set it up.

A

You know because is the is your is Agent a also trying to learn the optimal response of Agent B so that it can anticipate Etc and then you know they can end up chasing themselves in circles and all sorts of weird and wonderful things can happen. You know it's uh game, game theory and NRL comes in and uh it's it's very problem specific.

A

But you can you can end up with situations that they don't learn where you can end up in situations, then they take a lot longer to learn, because there is more than one learning.

H

A

I have a show of hands of the people who are actually trying to set it up on their computer, okay, okay, okay, so so we're we're sort of now is the time to take the break and we're going to have a have a look around Okay.

B

So I think a few people have it running at least uh we're gonna I'm gonna crack on and then we'll have another break in a minute for doing the next bit. So we can keep going around for anyone who's who's stuck on this bit, but I want to give more interesting things to do. On top of that. So a quick aside once once the the null chain is up and running the micro SIM is up and running. You see.

B

Anyone who got it working would have seen something like on the right here, which it is it's a printed output, log of some stuff, that's going on inside markets. Him exposes various apis for this, but often you don't want to just have to manually call python apis for every single thing that you might want to experience.

B

So we've got a couple of useful portals portals on your data. You can, if you've set up the UI components as well change a flag. uh This run with console equals true in generally, wherever you happen to be running your Vega uh null chain that will bring up in a web browser. The nice console that you see at.

C

The front there.

B

That is the same console that people use on Vega protocol itself, so that has theoretically pretty much everything you need. You can even trade manually through that as well. If you want you can log in as your agents that you've set up uh we'll, probably demo it a bit in a bit, you also can is actually always launched.

B

If you take the logs, there will be this graphql port, and if you go there and you know graphql, you can query the entire system with a fairly hopefully clear, graphql queries there's on um what is it docs.vega dot XYZ, there's a link in a bit the entire sort of structure of your graphql queries. You want as well, so for anyone who has good at running for more interesting scenario, that's something you can try out running I'll leave it up. um Actually, let me skip to the next thing.

B

This is what I'm going to talk through with this app, so anyone can type it up, so the components that Vega itself runs on a on a validated node that we're also going to be running locally, that the null service spins up. There are a few different ones: we've got Vega, which is the core process. That's the um go, blockchain client that does all of your transaction processing and outputs your States.

B

That is obviously it's a point in time proceeding thing. So we also want the data node, which is listens to external events, output by the note core node itself and saves those into a a postgres database and allows you to query all of that historic data and that's generally, what you'll be using when you're querying any data from Vega.

B

Storage layer, additionally there's a couple of extra things, there's a console as I mentioned previously, that isn't necessary, but it's very useful front. End, there's also the Vega wallet.

B

The wallet is basically size transactions and ensures that any sort of custom signing we have some proof of work to stop spam attacks and stuff, and that handles all that it's optional here, because for the null chain, what we actually do is Skip all of our validation, so that it will blindly accept. If you send a transaction and say it's from whoever it believes it's from whoever.

B

So you don't necessarily need that what it will spin up is if you spin up a console, the two interact and a lot together, so we use a full wallet for that.

B

So once we've got our our Market Sim set up, how do we actually run our scenarios there set up as ideally Plug and Play components? The overall thing we set up is we call a scenario that consists of an environment which is your background environment and your Vegas setup. So that will include things like how many steps you want to run your simulation for uh how your Vega is set up and sort of what kind of logging you want that kind of background detail.

B

We then plug in the various agents, some of whom will be setting prices. They could be acting randomly and each of them will have something they do you'll see when you build one yourself in a bit, something to do with startup often they'll force it themselves, some tokens that kind of thing and then they've just got a function and each step call that function and it does whatever it wants to do. It pulls in data from the market. It trades they do whatever so scenario.

B

Environment you'll see that when we go in a minute to the agents themselves, so yeah, what's an agent, it's it's fairly simple. Ultimately, it's an initialized step where they generally force it. Sometimes they will um do things so we'll have an agent that sets up all the assets, because you start with a blank blockchain. You don't have any assets, any markets, that kind of thing so there's you may see. If you look through a market manager who does all this setup for you, then they'll have a function. That's called every scenario step.

B

However many steps you do generally will sort of randomly shuffle them so that you're not always getting the first agent called First in any step and then a finalized method that most mostly you can ignore, but uh often you'll want one to sort of settle the market and things. So that's how you you want to set up your agent there's one more thing, I mentioned earlier that sometimes the blockchain does have to bleed back in you'll, see scattered throughout occasionally, you'll wait for total catch up function.

B

Basically, what this ensures is that you're getting a good view of the market. Currently, you haven't um thrown way too many uh transactions at it, everything's in sync, that kind of thing, so it just ensures that your more real world modeling, rather than trying to ensure that you can get everything.

F

B

One nanosecond through so um I'm gonna now talk through building one of these agents, a really simple one. It's just gonna do a bit of training, I, guess and then we'll break again, and hopefully you can put together your own a bit.

B

Where you're thank you, Simmons.

B

Okay, so for anyone who has it set up, we've got this run simple agent in exam reinforcement, run symbol, agent, I'll, put up on the screen after this to pass to these files, and then in agents. We have our simple agent. So what run simple agent does? Is it sets up a scenario?

B

um I can discuss the actual arguments with people but which we call curve Market maker, and that a.

B

H jump into marketing, basically that contains an agent who ensures that there's a nice shape of the market and a few who just sort of randomly buy and sell so they're, very dumb. Theoretically, you should be able to agent, which is what hopefully, we're going to build.

B

That runs this simple agent that we've got together here. So what I've put together is a few nice template functions that is without having to worry too much about any of this works. You should be able to cut and paste these into our step function, which is down here.

B

We've got the simple agent and you can see it's got the structure as described earlier. We've got our initialize, where it's finding the market, finding the asset and minting itself an asset and then say, submit the market order.

B

B

Check her positions.

A

Is it almost gonna buy.

B

It is currently always going to buy so 50 50 chance of making money.

A

Okay so uh they're down to the relay just the hardcore Workshop participants, you know you're the ones who prevailed with us, despite Python and its foibles and the slow internet so I'm going to crack on if you're, if you're coding keep coding, uh you know. If you have questions there will be time to ask questions and what we went to. What we want to get to now is just to show you a little bit more of you know what the agents that that you have there.

A

You know the market maker Etc that you're interacting with a little bit the ideas that they're based on and then move on to the reinforcement, learning part. So.

A

You know as part of the environment as Tom was saying: if you just set it up, there is nothing it's it's the empty Vega blockchain as it would as it would be when it launches live. You know no assets, no markets. Everything has to be created through governance, and you know once the asset is created once the market is voted through, which you know the python sort of expedites for you, you will still have just an empty market right, no liquidity on it, nothing, no activity so effectively.

A

The code provides various versions of of market makers that are based on some stochastic control, problems that are pre-solved coded up and basically running, and then some sort of hacks, where you know the optimal solution of the stochastic control problem, is used as a heuristic, and then you put something on top of that, and you know there are you: can you can have liquidity Trader? You can have liquidity takers. It just means they Place random, Market, buy and sell orders.

A

You can have informed Traders. What does it mean? Well, you just tell them what the price will be at the next step. Right, you know, maybe they they have inside information or whatever, and um there are the momentum Traders through a library. So you you, can uh you can trade on on all the patterns that these chartist people have? You know there are these vomiting, camels and I. Don't know what else you know you. You can code up agents that trade on that.