Ethereum Foundation DevCon 6 - Developer Infrastructure Track, 15 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: EVM Tracing in Geth by Sina Mahmoodi | Devcon Bogotá

Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/evm-tracing-in-geth/

Geth comes with a variety of ways to analyze the execution of past and constructed transactions. In addition to the collection of built-in tracers, users can submit scripts in JavaScript or Go which hook to the EVM, receiving minute events such as "opcode executed" or "entered new call frame". In the workshop, the audience can follow along a summary of basic tracing, commonly faced problems, as well as an introduction to the more recently shipped features and how to write efficient tracers.

Speaker(s): Sina Mahmoodi
Track: Developer Infrastructure
Keywords: evm,tracing,geth

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers.
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

A

A

This is going to be a workshop, so I will be working with the terminal. I will show you some code and show you how to use gas like I, guess tracing feature. That's why we specifically needed my laptop so yeah, sorry for for having to wait um also yeah, because I will be showing code. It would be great if you guys could come forward.

A

If you want to see what's happening because I imagine, the phones will be small, so yeah I'm going to give you a bit of time to come forward and uh while you're doing that, I want to talk a bit something a bit personal, uh so I come from Iran I, don't know if you heard, but there's some stuff happening there.

A

uh I just wanted to give a quick shout out to the people who are going to the streets, they're they're purchasing for their civil liberties for freedom of speech, um so yeah there's been three weeks of protests.

A

Now internet has been disconnected um and it all started when the girl on the left, who is 22 years old, uh was in custody because of showing a bit of hair and she somehow happened to die in custody, and this outraged people who went on the streets and during this protest uh close to 200 people, have died among them, Nica who was 16 years old and by like when running away from the police yeah.

A

She also died, so yeah just want to give a quick shout out to them they're, risking their lives asking for their rights and for for freedom. But this is a technical topic and I don't want to waste your time anymore. So, let's get straight to it. So what is uh tracing? What would you need? Tracing um I just want to show you an example of this is etherscan uh you, everybody knows etherscan right and the internal transaction section of etherscan is basically a trace of that transaction. It shows you what happened within that transaction.

A

Every call that happened there, uh who called, who and and so on- and this was a this- was actually probably not from a guest mode but I'm guessing from open, ethereum or Aragon, but we have a very. uh We have a similar feature to this, and this is not the only use case, um but there's many of them just to name a few like I, just just before coming here, I was talking to somebody who's developing a zkevm who are also heavily dependent on the tracing feature of gas.

A

I've, seen talks about from me, researchers who are using tracing features and many more yeah. This is also another tab in the etherscan, uh the details where it shows you the the state, like the state difference. uh We also have, as we will see later. This is also something that you can get from get so before we get to in. Actually we get into tracing.

A

um You know that you can use guests to execute a call on top of a given state, so you probably know East call you give it a set of parameters like as if it would be as if you want to send a transaction to the network, but this is actually not sent to the network. It's just executed locally and you see the result. uh It's a simulation uh I. Actually you have to give me a bit of time because I had to restart my laptop and I lost all my setup.

A

I have to bring up the terminal because I wanted to show you how this actually goes so yeah I have a I. Have a girly note on my machine. That is a kind of thing to uh to the network. uh Okay, yeah! Is it behind me yeah? Maybe I move it here all right. So um let's do a call, wait.

A

I have them all stored in a file, so I don't have to type it out because anyway, I'm losing too much on time on this, so I'm just gonna go I already lost a lot of time and you all know how eight call goes. Okay, let's, let's move on to the next one, um I'm, just gonna, explain to you. So basically uh this is the I'm sending um I'm sending like I'm assimilating a call to the wrapped eth contract on Gurley to get the balance of of an account of account zero right.

A

uh This will only give you the result like in this case. The result is as zero like it shouldn't be: I have to figure this out, like I. Think I I have the wrong address here, um like this address. The the zero address should have some balance, but anyway, what I wanted to show is that when you, when you uh do a scroll, you only get the execution result. What is returned from the contract back, but this is oftentimes, not enough information. We want usually more information.

A

We want to see what happened inside the transaction right and that you can do by doing the same thing, but using a debug method, Trace call which will simulate the call for you and trace it at the same time. I would give you more information. So if I were to run this now,.

A

I think this is the mainnet contract I was using. um I was going to present with mainnet node, but then I found out. The Wi-Fi here is not so good, so I can't SSH to my server here. We can already see that there is a lot more information, as it basically shows you all the steps that happened during the execution of this transaction yeah. There is going to be a lot of steps, I'm not going to go to show you all of them, but basically you you will see.

A

Okay like what is the first op code. What is the programming Contour at that point? Which opcode was executed? How much gas do we have at this point like what is the storage at this point and so on? So it will give you almost the the full information that the evm has uh when it runs uh through the transaction.

A

You can ask questions sure this stack is before the instruction yeah. So all of these information are from before we execute the opcode here. I showed you. In this example, I showed you like tracing a call of my own craft. Like basically I said, Okay I want to like do from to input and so on, but of course, uh guest supports tracing existing transactions, the one that were already mined on the Chain, um so historical transactions, so to say uh those you can do via these endpoints.

A

You have a debug, Trace transaction that traces, a historical transaction. You just give it a TX hash. You can also do it at the Block level. You can trace a Blog by using twice block by number or hash, or you can do even a whole range of blocks by using a trace chain but Trace chain, mind. You is a bit different. It's not a simple like it's, not using the simple Json RPC request. You have to it's a bit based on the subscription API, so you have to use websocket for it.

A

So the usage of it is a bit different. I'm not going to show you here, but yeah, it's cool for when you need to run over a range of blocks.

A

What what we saw um in the previous Trace was the default Tracer. It's basically the opcode Tracer like it shows you every step, but sometimes you need uh information at a different abstraction level like, for example, as we saw in the etheroscan screenshot, you want to see what calls happen or like you, want to get different information and get has a bunch of built-in tracers that you can just call them by name and and get the information you want.

A

So here like in this section, I want to show you uh three of them, the first one being the one that we saw I'm just gonna like here in this table. You will see all the information that you will get from the opcode Tracer, um which is yeah I'm gonna. Give you a second. If you want to see what's in there but yeah. Basically, you get the optical information.

A

All the gas related information like how much this up code cost to execute or will cost to execute how much gas there's left the the whole memory snapshot uh of that transaction. The the stock return data of the last call like when a call finishes in The Next Step. You will get the return data for it, uh storages for the storage slots for the contract, the depth of the execution refund, and if there was any error- but please note that if you want to use this Tracer, there are some things to note.

A

As I said memory stuck as storage and return data, these are Dynamic values and they can grow large, especially uh memory. You have to watch out for memory because uh mainnet transaction these days are very heavy uh so like. If you have memory enabled it can kill the node. Basically like yeah somebody for one of our users reported that they try to trace a Blog with uh on a server with a 64 gigabytes of memory and it crashed basically, but because of this, um the the Tracer accepts options to disable all these features.

A

So all of the ones that you will see in this list, they can be disabled. Some of them are enabled by default. Some are not but they're, all like you can toggle all of them.

A

Also, if you, if you really need to trace uh uh you, you need a memory, let's say and Json RPC is not sufficient for it and you control the node. Then there's an endpoint, a standard Trace block to file which will trace the block and save it on a file. Alongside the node that you can uh yeah, you can easily access. So it's not like a Json RPC request. The next one is the call Tracer. This is arguably one of the more like most interesting tracers.

A

uh It will basically give you all the call information like all the internal calls that happened during the execution I'm going to show you in in the previous example like, like.

A

Basically when we call wrapped ether balance of uh what information we will get in return, so you can see. um That's like balance off is a simple call. It's not going to call anything else and there's there's one call. It shows you who is the sender? uh Who is the recipient who's the contract? What is it call or is it create? uh Was there any value transfer and what was the call data? What was the input but yeah? This is a more like boring example.

A

We can um I hope these transaction hashes are okay, I feel like the file wasn't saved, yeah, okay, it works um so I'm just going to run. It show you guys so here, I'm I'm, trying like I'm, not doing Trace call but I'm using a historical transaction that was mined on Gurley, so I'm using Trace transaction endpoint, and this is the transaction hash, and here, like as an option you can specify you can specify what which Tracer you want to use.

A

So, if you don't provide anything, then it will default to the opcode Tracer, but you can say I want to use the call Tracer as we did here and you will get a bunch of information uh yeah. So this is. This is going to be the main call and notice that we can already see like this call uh errored like there. It reverted we can also see from since recently we can see the revert reason which is system time outdated.

A

This is the solidity revert error that was returned, the input output and then, here from here we have the nest that calls. So basically, the the the main contract did some uh called another one that we can see here, which which also reverted and like up there. You can see like there is um there's more nested calls, but that it's not being displayed here, but you can. You can get that information.

A

Also, so yeah call Tracer, you would use when you need information about the internal calls uh it also, since recently it accepts a Nifty option, so I'm gonna provide here like a you. Can you can basically configure the tracers you can say like I want only top call.

A

um This is. This is just useful if you need only the top call information like only if you want to know okay. Why did this transaction revert? Then I don't need to see all the nested call information I just get the top call and I check the revert, reason and I know. Okay, there was a time outdated or something this one is the pre-state Tracer.

A

It has two modes of execution by default when you, just you just say, I want to like trace this with the preset Tracer. Then it will give you all the accounts that are needed to execute uh the call or the transaction.

A

You can think of it, uh something similar to access list or Witnesses.

A

So if you somehow need to simulate a transaction and don't store the whole state, you can get the state via the Free State tracer, and then you can execute your transaction locally and in in the second mode. You can see all the state state modifications that happen during during the transaction. Basically, you can say you can see. Okay, the balance of this account increased to such value. The the storage slots changed such and such. If, if we were to see it in see it live it's this one.

A

So here, I'm um I need to change the address.

A

I'm gonna execute the same wrapped ether, balance of call and pass the to see the pre-state and yeah, as you can see like the zero Atlas has like, shows the balance of the zero address. This is the the contract itself. Has a balance the code uh nouns and like the all the storage slots that were needed to execute these transactions? These are not all the storage laws of this contract, but only those that were required here and let's say you wanted to see the the div you just pass: the parameter, div mode and yeah.

A

Basically, we can see like the biggest change here. Is that the non? So you have like a pre and post State and you you have to compare them. So you can see here like we have the zero address. The nonce is emitted because it's zero, so nouns was zero and now it's one. So this is the only uh the only change that happened in the state during this transaction.

A

But of course, if we were to run it with a more complicated transaction, you would see yeah you see like there's going to be much more stuff. This is uh yeah, and this is what parts of the state changed during this particular transaction like here. We had also balance updates changed.

A

And balance of zero account change and yeah. Here we have like um the format, works as follows. Basically, whenever something is created, let's say a contract is created. It will not show up in the pre-object, it will only show in the post object and whenever something is deleted, it will only show up in the pre-object, but not in the post object. So by by comparing these two objects, you can you can see um how the how the state was modified.

A

uh So now, I want to talk a bit about how gas stores stage, because it is very relevant when you're doing tracing like if you've been been tracing. You've probably saw this error required historical State unavailable specifically when you want to trace a historical transaction, one that was mined uh now now. Why is this so? First of all, how do we prepare the state for simulating a transaction or re-running a transaction? It's basically like you have to.

A

We have to find okay, the transaction in Block n, so we need. We will fetch the state for Block n minus one, the pro state of block n minus one and start executing all of the transactions within the target block until we reach the transaction that we want. At this point we have the transaction. We have the state, uh the pre-state, basically for our transaction, and we can we can execute it. But what happens if the state for this block n minus 1 is not available in the database that, basically that's hint?

A

That's what, when you get this error.

A

So what uh so now comes the question the state would be which blocks are actually persisted.

A

um Let's get the archive node out of the picture, because it's the easiest so for archive node, you have all the states. Next, one is full sync.

A

By forcing I specifically mean that you start executing all of the blocks from Genesis up until they had here. um It's worth to note that always guest stores, the latest 128 blocks the state for them in memory, so you always have like. While your node is thinking, you always have the state for the last 128 blocks. Now anything beyond that um is stored on, like it's persisted to this only periodically, and that is roughly every two hours.

A

So every two hours, um the state for a block will be persisted to disk and because we think from Genesis that means we have the state of a block roughly every two hours from Genesis on until the more recent ones and the difference between full sync and snapsync is that in synopsis you won't actually start. You won't actually execute the transaction. You want to actually execute the blocks from the beginning, but from some pivot point.

A

So let's say here is the point when you um start like you, fetch the state from the from the network and you start executing transactions. So that's why we only have the checkpoints from this point on and like so we. Basically, we cannot execute any transaction that that happened before we. We did the snap sync.

A

So it's important to know like based on your use case, um which think mode you should go for and now we can complete the picture again. So how do we prepare the state for for executing a transaction?

A

We fetch the state of the parent block from the database if it's not available, what do we do?

A

Death will go back in the blocks for for a number of logs and checks if it has the state available for any of them uh in on disk, and if so, then it will basically re-execute all of those blocks and prepare the state for your transaction. So here you can see in the error who we have. We have the error and we have some more information like re-exec equals 128.. This means that guests tried, the 100, went, 128 blocks.

A

Backwards to find some State on disk that it could use to to go forward, but it didn't find any- and this is a this is a um this- is a parameter that you can give.

A

So basically, when, when you do Trace transaction, you can provide this re-exact parameter and say: okay, I'm willing to go back two thousand or two million blocks to find a state that I can use to uh to execute uh this this transaction, but, of course, the more you're willing to go the more you go back the longer it's going to take, because all of those blocks have to be re-executed to to compute this state.

A

We have uh there's a there's, a method that you can use to see, States for which blocks are actually stored on disk and that's debug gets accessible state. So, let's say like I want to see.

A

I want to see like five five thousand blocks behind the head. Do I have any any state there. Computing no State found so I, don't have anything between the last five thousand and four thousand blocks. Okay, I I can try again or I can enlarge in this the search parameters.

A

It's probably also going to say: no, oh actually, no, it found it so here it says, like I, have there's this block that I have the state for so I know that uh block number yeah, so I know that I have the state for like 8 000 blocks ago, basically so yeah. This is.

A

This is the Nifty method for finding um which which states you have on your disk and then when, when you, when you know this, then you can set the re-exec parameter accordingly, and sometimes there is a range of blocks that you care about a lot like you. You need a lot, so you want to store the whole state for that. What you can do is basically stop Geth.

A

Add this parameter, GC mode archive and and run for as long as you need and then stop it again and take it off, because when this flag is there, then guests will store every state like the state for every block on disk.

A

We went over the some of the tracers that are that, come as like in in stock gas, but that's not your only option. You can also uh you can you can write a custom Tracer like you can basically uh get like. There is some hooks that you can use to.

A

Collect the relevant information for your use case, and only that you don't have to rely on the the tracers that we provide, and here I want to show you an example. I um doing devconnect in Amsterdam, I I met this guy, who was an Emmy researcher, and he was telling me that he found the tracing feature of Geth extremely useful and later on. I saw his talk and he was talking about this.

A

So uh apparently there is, there are some poisonous tokens out there or there used to be at some points that that wrecked, basically uh Searchers like the automatic Bots and one of one of the those tokens kind of looks like this. This I think this is a simple one: it's not a real real one, but basically it just checks uh to see. If you're in a development environment, then it does the the actual transfer of value.

A

But once you so, let's say you stimulate that uh in on hard hat and you see like yeah I got some money, but when you run it on mainnet, then you actually don't get that money and you probably lose some. So that's that's the gist of the token and like he, he wrote a like a simple Tracer that can detect this uh I'm gonna show you that this is how a custom, JavaScript Tracer looks like it's a it's a very simple one.

A

You have a few methods that you can Implement in a JavaScript object and you will pass this object to get when you, when you invoke the API.

A

In this case, we implement the step method, which is run for every step of the execution and we Implement also the result function which is executed once at the end of the transaction. So basically we want to the the idea of of the detection algorithm.

A

Is that token transfer shouldn't use any of the following up codes? Coinbase number difficulty origin: these are all block metadata right. Why, when you're doing your erc20 transfer, why would you need any of these op codes? So this is like a very simple detection thing. So what what I'm doing is basically like I, have a global variable called fishy and as I'm going through the op code as I'm, going through the steps I check that if any of these op codes appear then I'm gonna, say: okay, there's small coming out of this I!

A

Don't want to do this trade, so I just set that variable to true and return it, and now like here is my I. You can't see that, but basically the result is true. So what so, I uh I I run the Tracer over a contract uh and the result was true, but let's, let's dive. This is. This is probably the most part interesting part of the.

A

There was a few things here right, so I'm I'm, stimulating I'm, simulating a call uh from and to to an address, but note that this address is actually not um is empty. There's no, like there's no contract deployed on Gurley, for this address, uh so I'm going to introduce to you in a very Nifty feature, and that's called state overrides right.

A

um Basically, when, when you, when you simulate a call using IF dot, call or debug.trace.com, you can give a bunch of State modifications to to get like temporary and you say: okay apply. These State overrides temporarily for stimulating this transaction.

A

So here what I did was I I have the byte code of of the contract the poisonous token. This is the bytecode, and here I, say: okay, State override in this address. Please put this contract code, so this country code will not be deployed anywhere, but only when I'm simulating this transaction.

A

It will be as if this address has this code. So this is very useful, like if you're simulating, stuff and and the Tracer object, I'm passing also separately here. So what happens is basically yeah. There's a call to transfer to this contract, and my custom Tracer detects like says that it's a it's official call because it uses the coinbase opcode now the JavaScript Tracer is, um is what most of people know about. It's not the only option for writing a custom tracer since a while ago.

A

We also have another option, namely you can write your tracers in go.

A

This is specially relevant if performance is, is a concern to you. The the go JavaScript overhead is a lot so when, when you want to do tracing over a long range of blocks- and it just takes a long time, but now you can actually write your tracers in go and it's pretty simple how it works. I have I have one here that I'm gonna like walk through a bit, but I just want to show you how you can first I want to show you how you can write one. So basically, you write your tracer.

A

You add this. This is a very important part. This is how it, how guests will know that there's a tracer here to be found and you drop this file in a directory in yes, so we have this directory is tracers native. You simply have to drop your goal file here.

A

Do the registration and you're good to go, and, of course, your your struct has to adhere to an interface, and this is the interface that you will see. So the methods are capture start.

A

This is at the at the beginning of of the execution uh you have captured state, which is run for every every step of execution you have enter. We didn't see this before enter is basically a hook. That is, that is called every time we enter a new call frame. So when there's a call or a create, we enter a new code frame.

A

This uh this one is called and exit is when we leave the call frame- and you have fault and and the result, so the result is when basically, what you return and what the Tracer will return when you call the method uh yeah, it's a bit tight time wise, but I'm. Just going to give a short overview so.

A

What this Tracer does is that it will tell me the method, signature for all of the all of the functions that were called so as you, as you probably know, uh solidity functions like when you hash their signatures, the four bytes, that's how you actually invoke them at the evm level right so like when you, when you um and I want to just collect uh the this. This four bytes, basically like the signature of the methods that were called so this is the gist of the logic.

A

uh uh I checked that we're entering a new call frame, I check. That is this. uh It has to be a call or a delegate core statical, some kind of call. Otherwise just return, that's not important!

A

Also, we don't want to collect pre-compiled information, so they're also written if in otherwise take the four first four bytes of the call data- and this will be my signature- and this these are stored in some object here. uh Yeah I don't have more time to go through this. So that's how you write a tracer and I want to quickly mention uh what's coming next on this front, um so you saw the trace, call method. I'm planning to work on the trace multi-call. This is when you will have a list of call Objects.

A

They will be executed, sequentially, maintaining the intermediate State as they go to the next call, and the other thing is the trace, namespace or, as other people refer to it as parity tracing. So these are the things on the roadmap that have been worked on right now and on the last seconds there was a that was it basically. Thank you guys.

B

Hi, most of these will probably be available in the docs, but um does trace to file exist for transactions only rather than just blocks. uh No.

A

It's I think what we have is only four blocks.

B

Now got it uh is riazek re-exec available via Json RPC yeah.

A

So I didn't show it, but basically, when you do Trace transaction in the same object that you provide the Tracer name use you specify re-exec.

B

Got it and does get accessible, State return, only the highest block number that has state or all it.

A

Will return like in the range that you specify it will return the first state that it finds.

B

A

The first one, so if you want to get all of them, then you have to iterate. You have to narrow your range as you go and.

B

So if I did Genesis to now, if.

A

You need Genesis to know, then it will give you the first one like let's say it's block 10 and then you do another call from block 10 until now and so on.

C

Hey um quick question on the um the state diff tracer, so does it provide a kind of you know if there are multiple State changes in a transaction like at at the call level, do you provide all of those State changes, or is it just from the beginning to the end of the transaction? What does the state diff I may have missed this.

A

Oh, it's yeah, it's only from like, basically throughout the whole transaction. Okay, not you mean for code level. Yes, uh you want the site to focus yeah; no, no! uh It's all for all.

C

Translate, but this is something a custom Tracer.

A

But yeah you can you can definitely do that. Yeah yeah! You can just check the code for the prescription. It's in the same directory that I showed yeah uh I. Don't think it will be very hard to do yeah, but thanks so much. Thank you.