Ethereum Foundation DevCon 6 - Security Track, 14 Oct 2022

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Nosy Neighbor - Automated Fuzz Harness Generation for Golang Projects by David Theodore

Description

Visit the https://archive.devcon.org/ to gain access to the entire library of Devcon talks with the ease of filtering, playlists, personalized suggestions, decentralized access on Swarm, IPFS and more.
https://archive.devcon.org/archive/watch/6/nosy-neighbor-automated-fuzz-harness-generation-for-golang-projects/

Nosy Neighbor was developed as a breadth-first fuzzing tool for the critical golang clients in the ethereum network - Prysm, Go-Ethereum, and Mev-Boost. Nosy is a very annoying (to the devs) tool that aims to find bugs the moment they are introduced. Leveraging the go/types and go/parser libraries used by the Go compiler, Nosy analyzes the AST of a repo and generates fuzz harnesses for continuous fuzzing

Come learn about Nosy's novel approach to go-fuzzing and the issues it has uncovered!

Speaker(s): David Theodore
Skill level: Intermediate
Track: Security
Keywords: testing,fuzzing,layer-1

Follow us: https://twitter.com/efdevcon, https://twitter.com/ethereum
Learn more about devcon: https://www.devcon.org/
Learn more about ethereum: https://ethereum.org/

Devcon is the Ethereum conference for developers, researchers, thinkers, and makers.
Devcon 6 was held in Bogotá, Colombia on Oct 11 - 14, 2022.
Devcon is organized and presented by the Ethereum Foundation, with the support of our sponsors. To find out more, please visit https://ethereum.foundation/

A

A

Last time I saw you guys. I was in better shape. I'm surprised, I made it here um more on that later I'm going to talk about nosy, neighbor uh nosy neighbor is a tool. I've been working on open sourcing for everyone to use, hoping it makes some waves in the golang community um talk a little bit about the challenge: finding bugs in open source go projects and a little bit of the motivation, so ethereum really loves. Go uh some stats here.

A

Geth accounts for 82 percent of execution, clients, 42 percent of the consensus layer, clients are running prism, and this is as of like last week, these metrics actually used to be higher, like 90 Plus, on the execution side and like 60 70 plus on the the CL uh Mev boost is also written in, go it's the only production, open source, uh Mev client for flashbots right now it as of like I, think last week, 48 of blocks on mainnet were flash blocks, so I won't go into Mev and all that fun stuff.

A

But what I'm trying to kind of hit home here is that go is like critical in the ethereum stack so somewhere. I would say, like maybe three quarters of nodes or more are using go at some point, so it's super important. Let's see lots of code got to run two chains, the execution chain, the consensus chain. uh We've got the entire evm all kinds of stuff. So there's like a a huge amount of code here. How big is it exactly the pure go?

A

Ethereum stack is 583 000 lines of code, so as a security researcher like a deer in headlights, um I'm, not gonna, manually view. All of that right. We do do manual reviews here. All of these things have been reviewed at least once most likely, if not twice or more times, but they're moving targets. We've got uh every six months or so we've got hard Forks on both the El and the CL. So it's just kind of something that if we can automate anything we should. We should automate finding issues here.

A

My task there's a few of us at the ethereum foundation, working on the consensus layer, security, research team. My task is to kind of like look right now and focus on the ghost stuff. uh This is kind of a daunting thing. So, a few months ago, I guess about a year ago, started looking at this and trying to understand the problems that we have so talk a little bit about it. First of all, uh go is memory safe, which is awesome.

A

We don't have remote code execution issues stuff like that very often, the memory safety stuff helps also a lot in the I'll talk more about in this talk, but making it possible for me to make a tool like this a lot of common mistakes and go though there are some that are queryable right. There's a lot of these like little issues like the colon equals sign.

A

This, like quick variable assignment it, makes writing, go really easy and human friendly, but it also means like you, can have these shadow variables where, if you like declare a variable in a loop, you override it and you might reference it later. Thinking you're talking about the first version of the variable, if you call it Go, routine and- and you kick off like which is basically like for non-go people, it's like a it's like a piece I'd create under the hood.

A

The variables that are passed to that are not the same variable that you might think they are. So if you have like racy type things later, you don't really know and go doesn't complain. So we can query for certain things like this. This is a good example of like one query, gosec this just says like hey, you need to manual review all these uses of unsafe, probably not the best example, but we have code ql, Sim, grep and gosecker, like just major tools. You can look up for like automating querying static analysis right.

A

Another big one is race conditions, so goes I, think the the reason it was designed and the design decisions they made was to make parallelization very easy. So they have like this concept of channels and they have this concept of go routine. So it's very simple to be like hey just go on another thread: go, do something and then come back and Google made it to be Memory, safe, but also efficient and human readable, but to process massively large data sets, and so we can do a lot of parallel parallelization very easily.

A

But the problem here is that it's so easy that it's really easy to introduce race conditions. So, like one thing we have been doing is running these thread. Sanitizers this little terminal prompt right here is an example of actually a mainnet Geth bug a race condition that was causing some memory, corruption, which, if you see memory, corruption and go it's usually something race, related or you're, using like sigo or some native library, because go is relatively like almost 99 I would say: percent safe if you're, not in one of those conditions.

A

As far as memory goes, there's also some other sanitizers, there's asan msan ubisan things like this we're running notes on Robson sepolia, uh the Prater girly test net, and also on mainnet that have all these sanitizers running so we're like we've kind of automated. You know this querying we've automated, uh like sanitizers running stuff, like using Dynamic analysis, so we've kind of checked both of those boxes, but uh you know what else can we do, and that is where nosy neighbor comes in.

A

um So how else can we cover these 583 000 lines of code? The solution? Let's talk a little bit about the problem: I'll rewind, just a bit. um Obviously I already mentioned the huge attack surface. One thing I do want to point out here is the denial services are like critical for us, um not just like for other go repos necessarily, but a blockchain cannot have a denial of service, so usually a denial of service on the common vulnerability. Severity scale is like a three: it's not like a nine.

A

It's not considered critical, um there's, no information, disclosure, there's no remote code execution and thus there's not usually privilege escalation. So people aren't like lifting keys. But if you have you know more than 35 percent of the network running some kind of go under the hood and there's denial of services in those goes go routines or or these repos. Then you end up in a problem where the ethereum network could like be brought to its knees.

A

We obviously have a multi-client architecture, so these other clients would be kind of like carrying the network during that time. But it's not something we want. We wouldn't have finality for those of you that are like really familiar with proof of stake. It would be pretty pretty big deal, so we kind of have this like weird issue, where we, the worst kind of bugs we don't really see, but like we care a lot about these smaller bugs um the good I mentioned. Rc is rare. We have the source. This is really great.

A

I've been a security researcher for a long time and I've not had this Source very often in my career. So this is like a whole new ball game goes strongly tight. All of the panics and stack traces and failure reporting is like excellent. So if you write a fuzzer and it finds a crash, it doesn't bring down your fuzzer and your fuzzer doesn't commit suicide. So that's really helpful and I'm going to talk a little bit about the tooling.

A

This is a big deal, so go 1.18, I guess, like maybe six months ago, they released native fuzzing support. So this is an example of like in the testing. Library I can fuzz this function, so the function on a test on the right side and I lifted this straight from like the go fuzzing like native fuzzing design, dock Foo is the function under tests here, so we can fuzzfu, we can say: hey F dot, fuzz, give it a function.

A

Interface, I, say: I want an INT and I want to string, which are the two argument types to Foo. We can add a test Corpus of like right here. It says five and hello anytime. You want to like prevent like regression. You can add like previous bugs into your test corporate this way, and it will automatically tell you if the bug is reintroduced. So that's really cool.

A

um It's automatically coverage guided all the test cases when there is coverage they get added to the test corporate and they get mutated on errors are super descriptive, don't need health Checkers, because all this stuff is built in natively. It's awesome. This is kind of the next piece of the puzzle, uh discovering this kind of like opened my eyes to the possibilities of what we could do here. The AST is exposed the parser library and the go types Library expose everything about the source.

A

So when the compiler reads your code and then compiles it, everything that it sees you can see right here so like this is an example of the AST. It kind of is pretty printed. You can see like there's a variable you can see at the top. It says like AST, if statement, that's saying, there's an if statement here. uh The first directive is X. The value is two there's all this information here, there's more information than you'll ever want here.

A

um So that's really cool. So what can we do here? We can parse all the go code in a repo. We can basically collect all the dependencies for the packages and the types we can collect all the function, declarations, the package declarations, the type declarations, all the interfaces we can see what every function looks like like does it does it take like you know, arrays slices, bytes, strings, complex structs, also uh if it takes a complex struct.

A

What is that struck made of and recursively down the whole thing, so you can basically go all the way down to all the built-in types and go, and you can see all the information you want about something. So using this we can generate valid fuzz harnesses for all these functions, that we have typed and we can fuzz them. We can round robin them. We can fuzz them. uh While you know the target's running and I can talk a little bit about that more later, but then yep find their bugs profit.

A

I know this isn't like a security specific conference. So this is a reference to uh an old Frack magazine, uh smashing the stack for Fun and Profit I'm, not profiting off these I promise. I want to clean up all of these bugs um that's what I'm paid to do. That's how I profit! uh Oh and then you can repeat on every commit which is kind of why nosy got its name. It's the nosy neighbor! uh You can integrate it into your CI.

A

Hopefully, people will do this and then, when bugs are introduced, like the moment they're introduced, we can just like automatically fuzz a function before even the tests are written for the function and we can find these bugs. So it's like really annoying and really nosy to developers, because there's like this old granny across the street, that's like always looking in your business. That's where the name comes from I, just call it nosy, nosy, neighbors way too much so I'll refer to it as nosy from now on so nosy in action.

A

uh It basically has three main stages: there's an initialization there's the harness generation and the fuzzing. You can put this all into like one seamless. You know action if you're integrating into your CI, but for the purposes of like the tool as I made it you might want to debug stuff in the harness generation. You might want to add test corpora, there's all kinds of like interesting things you can do in between these steps. So I have it broken up into these three steps.

A

Just uh for like kind of Sanity reasons, this is the input every time you run nosy, no matter which of the three like actions, you tell it to do you're going to give it this yaml file and this yaml file contains a bunch of stuff. The most important thing, though, is the URL to the repo. So it'll pull down this repo, you can say I want this particular Branch.

A

You can specify different Go versions uh like prism, for instance, is like one of the big repos that I always look at and it won't build with go Knight 1.19 right now, so this is really easy as long as it's like 1.18 and above because that's what supports native fuzzing, you can kind of use older versions and that sort of stuff there's. Also these like ignore declarations, I put in there like. Maybe you have a bunch of test functions?

A

Maybe you have things that use networking stuff that writes in the file system that you don't want to fuzz, because you don't want to pulverize your file system them. You can declare at the package the function and the object level to ignore these kinds of things. It also has substitutions what this will do if you're familiar with go. If you put a substitution in here, it's you just put both packages in there and it will put a little replace directive in the go.mod file, and this is really nice.

A

If you like, want to knop out all of your like signature checks, you get a lot more coverage. This way, like, obviously your fuzzer's not going to be like signing ecdsa signatures correctly. That would be a whole other talk and and we'd be having bigger problems. If that was the case, the initialization it uses Docker, it basically makes this little fuzzing environment and building environment all in Docker and there's kind of a lot of reasons. For that, one is that we don't want to pulverize our host file system.

A

We have a lot of ease with dependency. If you're like, looking through and you're trying to dynamically write code, writing all these fuzzing harnesses, then it's a lot easier to not use your own go root. So this makes a valid go root. Inside of a Docker container.

A

It makes everything it adds all the dependencies for nosy for doing the The, Source parsing and the harness generation and all the fuzzing and all this stuff, those that, in this little Docker container and initializes it inside this repo or initializes the repo inside this container, it uh has a shared like host file that you'll see like in a little bit it's an asset directory that keeps everything that the host needs to get.

A

So you do all your fuzzing inside this like protect protected environment if you're a security researcher and you're familiar with like jailing, uh a Target. This is just like a true root under the hood, except that we get to stand on the shoulders of Docker, and you know uh we get to like potentially neuter all of the the networking we can like control things in like really jail stuff.

A

It also makes it where you could like run this on your host computer and like only give it a few cores, and you could still like work- and you know maybe you're you're kind of like Dual Purpose using a fuzzer. That's also like your desktop for research.

A

um Let's see generate hardness, generate harness will copy all the Assets in and then it will spit out this one-liner and if you double click this one liner and run it it'll start generating all of the harnesses, uh and so you can kind of see like the lower half of this terminal. It's spitting out all of these fuzz nosytest.go files. All of those files are placed into the respective package under test directory three.

A

The reason I do that, instead of having them all in one is that one bug doesn't prevent the whole thing from compiling and having go complain. The other thing is that we can fuzz internal functions. You might not want to fuzz internal functions, so nosy has a flag for that. But if you do want to fuzz everything and get like this serious breadth, first coverage of your target. This is the best way to do it that I found.

A

So this gets to the fuzzing. You same thing spits out a one-liner. It creates it adds all the assets to this asset directory, and then you start fuzzing. So you can see on the right side here it started fuzzing. It found a crash like right off the rip. It minimizes the test case that produces the crash, and then it spits out the like Panic output, so kind of funny that one found one like right off the rip.

A

This is an example, repo that I have that comes with nosy, so that you guys can all test this um I, don't provide yaml files uh for all the targets. I'm testing, because there's I don't want to give you guys free, bugs and uh knows he's still kind of like a work in progress.

A

So I will release uh other stuff further down the line as I've like hammered out all the bugs that are there, um I think what I'll probably do is kind of like have this like private repo, that's maybe like three months ahead and then, as as we've kind of like shaken out all the bugs that this fuzzer can find I'll open source, the other parts of it.

A

But you can copy the example yaml file for this target repo that I've made, and you can point it at the Go standard, Library I haven't had time to do that. I'm sure, there's tons of bugs out there like I, have one thing nosy does do. Is it causes like a decent amount of false positives, but it does find bugs so I have like a ton of crashes to look through before I'll release. All this other stuff, um oh, that was the this- is an example of like the round robin.

A

So when there is not a test when there's not a crash found. This was what the output will look like so it'll fuzz for like 10 seconds on each thing, the yaml file has a little variable there, where you can say like how long, how many seconds do you want to buzz for, as you um generate like larger test corpora that are getting better coverage you might want to bump this up to, like you, know, six minutes per function or something like that.

A

uh This is just a little shot of like what the script looked like like. What does the round robin? This is a shell script. That's just like kicked out into the asset file. That's run on the target, so there's some reasons that this is not the best way to do it. So I, don't think I'll be doing this way forever. But if you can see right here, it basically calls the go test fuzz on the function you want, if there is uh test data fuzz inside of that package.

A

Now that means that we found a crash copy that in the asset directory, so that it's available on the host. If the fuzzers either commit suicide or if you're done fuzzing, you don't lose. This a lot of people like know that you, a stalker your container, might not be persistent if you don't have some like asset directory where you save stuff off to so this prevents you from kind of losing work that you've done all right. Example.

A

Findings um I made this little repo so that everybody can kind of like see nosy in action and have an example, so they can point it at their own own repos. These are all the root cause of all. Of these bugs are copied from real bugs that nosy did find and I'll talk a little bit about like the type of bugs it finds, because it doesn't find everything and it's really good at finding a few things and and I'll talk about that. uh Looking at these here.

A

So um this is just showing like the Panic line like this is identifying the type of issues that we have. uh It looks like there's two index outer range. The second one actually has another bug. There's two bugs in that function, so I included the wrong screenshot. It should be like a divide by zero, so these four functions on the right show the vulnerable functions. So these are the kind of things that nosy like.

A

If you say like, spend three seconds on a function does he will find these things like immediately and if you notice what these are they're panics they're not like remote code executions, their panics, where things stop. So, if you have some uh blockchain software, it's highly social. It's listening to all these peers. Every time it receives a packet if you have like a grpc Handler, and it's meant to like Panic gracefully and it just panics in that go routine. You're! Fine!

A

But if you have this in, like core code, this Panic can like make the Panic go all the way up the stack and just completely bring down your notes. So these, like true like packet of deaths, they're a big deal, and this is kind of the thing that like keeps me up at night, because panics are not always handled gracefully, especially in uh like these huge systems.

A

So, like the evm, for example, if you found a panic in it, uh you might actually crash that part of the process and then Geth would just be like completely worthless right. So, let's talk a little bit. I've only got five more minutes left so I'm going to talk really quickly about like what these functions look like. So this is the most basic right. So uh if you look at this, this is the same native, supported testing.

A

The input to the top is: is it it's actually a test function it takes in the testing object from. Go uh you hand, it a function, interface right, so this function here all I'm saying is hey I want to fuzz this log validator web auth right I just pick some random function. It takes a string, a string and a string right. So you know this is the kind of thing I need to tell Gophers. Hey I want strings when you mutate, like I, need a valid type string. Super simple: this is something that go.

A

Testing does not support, complex structures. So if you see the second line there, what it accepts is actually a byte array. What this does is I use right now: nosy ships with the open source version we'll ship with go files utils from Trailer bits, there's some reasons that and I won't go through them unless I have time at the end. Why that's? When it ships with the biggest one, though, is that you see all of these fill errors, it will return if you don't have enough data.

A

So if I have like you know a bunch of nested structures and like you can imagine uh my function or test like needs, basically like 2 000 bytes to fill all the data correctly, it'll it'll say: hey uh return, there's no issue or anything give me something larger. It doesn't really say give me something larger, but it keeps letting the fuzzer mutate until it finds you know further pass, and this is coverage guided, so the fuzzer will very quickly make it all the way to that last line.

A

So what happens here um that last line is ACM import? It takes the context. Variable I need a valid context. Variable I want to basically test import, but I need ACM. This is a. This is an actual like object. So it doesn't just fuzz functions. It fuzzes meth, like methods on receivers, which is Ghost version of an object, so methods on objects. We need a valid object created. We need the argument there, that kind of stuff yet more complicated. Here we actually have a Constructor.

A

So why make like an object and fill it with random data when there's custom Constructors made so like uh you know these, these large blobs for the evm, um maybe like different peer structures, maybe like Beacon blocks things like this. We've already got Constructors for them so like.

A

Why make them ourselves we're going to get a bunch of like no pointer, D, refs and false positives, so what nosy will do- and this is hugely inspired and barred from fcgen, even like probably like 60 of the code for this type of interface- is uh borrowed from that. But what we do here is we go, look and say: hey.

A

Is there a function that returns this object and only this object that doesn't take that object as an input, and you know it can either just be that object or that object in an error, and if that's the case, we say. Oh, that is a Constructor. Sometimes you see false pauses for this, but in reality, they're actually really good at still generating valid objects. So in this case I need a new key manager. I didn't have to write new key manager, some developer that made the new key man.

A

The the key manager object, wrote this Constructor, that's what he uses it takes. You know this configuration deal and then whatever C1 is a context variable. So what happens here is nosy recognizes. This builds not only its own version, but a second function defines this that relies on the Constructor and, whichever one gets you know, more coverage can find the bugs so right here we we basically find the Constructor and we know how to see a function interface.

A

So we build everything we need for that Constructor and then we we make the Constructor hand us the object, and then we also provide everything for the function under test which in this case is fetch, validate something priv key I can't really read from here, but this is just like a random example. There's plenty to choose from one thing: I forgot to mention out of the 55 583 000 lines of code.

A

There's over 15 000 functions the in the in those five repos, which is basically all of the dependencies for MAV boost, getth and and um prism that are supported.

A

So that's fifteen thousand functions that I can get coverage in uh that's coverage, guided fuzzing that I don't have to write the the harness for that's kind of like the value out of this tool here um and and yeah notice here that, like no nosy didn't only like create the valid arguments to fuzz here, it created the object and by doing that, um the way that it did that was it created the valid Arguments for the Constructor.

A

So everything that needs to happen here to try to get like, as close to like a real test case as possible. We have mistakes, learning, I'm, gonna, go really fast, I got a minute and a half um version, one actually there's like a version. 0.5 uh shout out to Tyler Holmes uh one of my teammates. uh He did a big code, ql query for various things and we found like a bunch of stuff that just accepted byte arrays, so we wrote some python. It would like basically generate harnesses to fuzz those that was version.

A

0.5, then version one um was in Python. This is just like to show you my pain, I'm grepping, for regex here in Python. All that gibberish I felt like Jr token, like basically like writing, Elvis or something this was no good um trying to do this for for complex structures and stuff was like a total pain, so moved on to the AST objects that you get from the go parser library- but you guys have seen this. This is kind of cool, looks sort of pretty printed. It still sucks.

A

uh All that stuff is like you, don't really know like this AST ident on the fourth letter, Fifth Line, that X is you gotta, like do a type check on all of those, so you end up with like this massive parsing thing. That's got like you know a gazillion nested case statements still better than python, but still really ugly, then I ran into FC gin uses the go type libraries to write that, like more complex function, interface that I showed um I. It blew my mind.

A

I can't believe I wasted all my time for six months on these other things, so I basically grabbed all the code that works from there threw it in here it was a minimal rewrite. For me, um cool I talked about why we use Docker. These fuzzers will find the go binary and delete it. They will write all kinds of crazy stuff to your file system. If that happens, you can just restart nosy and your host isn't screwed.

A

um Let's see various fill. Libraries I talked a little bit about trailer bits. I have a proprietary version that I'll talk a little bit more about here. Last slide that really matters I know I'm out of time uh things that we want to do Auto corporate bootstrap. So you can imagine here. We already know how to dynamically write code. We can dynamically rewrite code.

A

So if I point it towards the repo and I say, hey I want to run, go ethereum, uh you know, let's say I support, 7 500 functions in it, then what I can do here is I can say all of those 7 500 functions instrument them.

B

A

Go normally run it on mainnet. If you want save off every valid call to all of those functions and and receivers and then mutate on those, so I can I can bootstrap a corporate that way. I can automatically fuzz in a separate, go routine in a Docker image or something in real time. You can be like continuously fuzzing mutating on real valid test cases.

A

um Let's see auto object, fuzzing, you can find race conditions. This way uh you could say: hey. I, know, I support this Constructor. It's got 10 methods on it, write a fuzzing function that will kind of round robin those there's. Some work like this in FC gen that I'd like to copy as well I. Think if you run that, with the thread, sanitization you'll find a ton of race conditions that way, lockdown networking you can do AST walk to say, like hey I, want to like look at all the reachability from this function.

A

If it writes the file system exclude it because I'm tired of something something's pulverizing my file system and it's destroying my fuzzer- that kind of stuff final task test case minimization at the end of a run and all the coverage analysis. This would be really great. If I had this done today, because then I could say, hey look. You know, prisms testing library. Has this much coverage and I added this much coverage automatically with nosy neighbor. That would be really cool.

A

Maybe I'll have that in six months, for you guys, all right, uh I will open sources in the next 24 hours. That's my promise to you. My creative excuse, as of all procrastinating years, has a creative excuse. I got bit by one of these snakes like four or five days ago. I wouldn't be here. If my wife didn't like do so much to get me here. I spent a lot of time in the hospital I've been elevating this foot.

A

Hence why I came up on crutches, I'm starting to be able to put weight on it, but yeah I actually have like a real excuse. This time the dog didn't eat. My homework one of those guys like legit bit me it was a whole thing, follow infosecule on GitHub or Twitter. I'll drop the repo links probably later tonight, we'll latest like this time tomorrow, depending on how the rest of the day goes.

A

uh Real quick I do want to thank FC, gen trailer bits for that: uh Phil, repo, uh zinchata and and Justin traglia for various things that they added to this uh repo into this project, uh the Gophers folks and then everyone in the Gopher slack that's been like super helpful. uh Any questions.

B

Yeah with that David I am going to say, you can go ahead and take questions over to the side, but thank you so much for a making it here through all of those different hurdles and for giving us your great presentation. So thank you so much if you do have any questions for David, please feel free.