GitLab Fuzzing 101, 4 Aug 2020

Previous Meeting Next Meeting

⏯

youtube image

►

From YouTube: Adding coverage guided fuzz testing to a project

Description

Principal PM Sam Kerr walks you through how to add fuzz testing to a basic Go application!

Fuzzing documentation: https://docs.gitlab.com/ee/user/application_security/coverage_fuzzing/#coverage-guided-fuzz-testing-ultimate

Fuzz testing direction page: https://about.gitlab.com/direction/secure/fuzz-testing/fuzz-testing/

A

Hi, I'm sam kerr, I'm a principal product manager here at gitlab in this video, I'm going to walk you through a basic example of how you would add, gitlab's new coverage, guided fuzz testing to a very basic go application, and so with that, let me share my screen and we will jump right into this. So the screen that I'm showing right now this is a repository hosted in gitlab for our basic go application.

A

The application is just a single function. It's called parts complex and all it's doing is taking an input called data and it's reading that input to see if it's six characters long and then it's doing a character by character. Comparison to see if the input is the word fuzzing.

A

So if you'll notice it's checking for if the length is six, but it's actually looking for seven letters total. So this is going to be what's commonly known as an off by one error. The program when it is run in that case is going to look at more data than it has available and that's going to cause an error and that's what we're hoping to find as part of fuzz testing.

A

So now that we've talked a little bit about the application, let's go ahead and talk about how to actually add fuzz testing to hopefully find this bug.

A

So if we go back to our repository, let's open the web ide to start looking at some of the other files that are in this repo, so one of the first things you want to do as part of adding coverage guide. Fuzz testing to any project is figuring out the function that you want to fuzz. In our case, this parts complex function and then writing, what's called a fuzz test, harness to tell the fuzzer how to actually fuzz it, and so what I've got open right now is we're calling parse complex fuzz.go you can see.

A

This is again a very simple function. It's only got one function in it named fuzz. It takes an input data and this is going to be coming from the fuzzing engine itself, and this is going to be random data. It could have all sorts of contents, it could be various links, and this is what we want to pass to our application to hopefully find all those bugs and security vulnerabilities that live inside of it.

A

So inside this fuzz function to tell the fuzzer to run our parts complex function. All we're going to do is invoke that parse complex function, using the data that the fuzz tester provides, and that's all there is to it. This is how the fuzzer will run and test our application in your own apps. This is really going to be the part where you would configure the fuzzer to load files from disk pass them to your different functions.

A

Do any sorts of transformations that need to be done specific to your application, and so now that we've got our fuzz test harness written. Let's look at how we would actually add this into our gitlab ci as a fuzz testing job.

A

So you can see here. I've got our gitlab ci file open for this repository and we're doing two things here to add coverage guide, fuzz testing to the pipeline so step. One is that we're always going to have to include the get lab coverage. Fuzz testing template file into our pipeline, and so the way that we would do that is just include it as you would any other template in get lab. This is going to be coverage as dash fuzzing.gitlabci.yaml.

A

Then what this template file is doing. It's doing some definitions behind the scenes.

A

It's doing some configurations and going to be getting our pipeline environment set up correctly, so we can run fuzzing jobs step two is we have to tell gitlab how to actually build that fuzz target that we just wrote the parse, complex, fuzz, dot go file, and so the way that we're doing that we're defining a new job inside of git lab and we're extending the dot fuzz base, and so what this is dot fuzz base is a hidden job, that's being defined by that coverage.

A

Fuzzing template file that we included earlier, and it already has all the necessary setup and configuration done inside the template. All we have to do is configure how we would run our specific app through that template and so to do that. What we'll do is define the docker image that should it should be run in, and this is going to be specific to the language that we're running, because we wrote our application in go we're using the go image you can see there.

A

It says: go lang, we're using the go image to run our fuzz testing, depending on when you're watching the video. This image name may have changed so please reference the get live, documentation to always know what the latest image and the latest version that you should be including here is.

A

Additionally, we are defining a few script steps in our job, which do a few different things. The first of these is going to be building our go application so that it's ready to be fuzzed, we're going to be including this lib fuzzer here and compiling all of those application files into my fuzz target.a.

A

After that, we're going to be compiling the a file that we just created with clang we're going to be adding this sanitize equals fuzzer flag and outputting the file, my fuzz target.

A

And finally, what we're going to be doing is taking that my fuzz target and passing it as an input to the get lab cover fuzz tool, uh we're going to be saying that we want to run the fuzz testing and that we want to potentially use the regression flag. This could be set by a variable as part of our pipeline. If we wished I'm not going to go into too much more about the regression flag.

A

If you want to know more, please look at the product documentation and then again at the end, we're going to be supplying that my fuzz target, and so this is tone gitlab fuzz, that myfast target binary that we just built.

A

So now that we've added all of that to our pipeline file, we are ready to actually run the coverage fuzzing job. So if we leave the web ide, let's go back to our repo.

A

So now that we're back at our repo, what we'll do is go to our pipeline tab and using all of the different things that we added into our ci file. What we're going to do is select our correct branch, add fuzz testing and we're just going to run the pipeline. That's all we're going to do, and one thing I did want to point out as part of this pipeline. We are running fuzz testing, but we're also running sas.

A

One of the big benefits of coverage guide, fuzz testing, is that it will find bugs and vulnerabilities that traditional security and traditional qa methods will not find so if you're running things like sas in your programs, they're going to be looking for specific sorts of vulnerabilities, but they will most likely not find the sort of off by one error that is in this application, and the fuzzing will find and just to double check that we're going to run sas and see what it finds, and so now that our pipeline has completed, we can see a few things immediately.

A

All of our jobs have finished some green and some yellow, notably both of our sas jobs, one for running gosek, as well as secret detection. Those have both passed. They found no vulnerabilities, however, our fuzz job, my fuzz target, is marked orange, which means it failed. Let's take a look and see what it found.

A

So you'll notice immediately as part of this log that we can see a lot of different information here, but what's really eye-catching here. Is this error as part of running the job? The fuzzer found an application crash, and if we look further through the output, we see a stack trace which tells us more about where the crash happens and also we'll see the input specifically that crashed it. This fuzzing strain and exactly as we predicted that is exercising that off by one error that lives in the app, which is what got us that crash.

A

So this job output has a lot of great information in it. You can go view the artifacts to see the sorts of inputs that cause crashes, more information about how the runs were done. But if we go back to our pipeline view, I want to highlight a different way that you would look at this, and that is as part of our security tab after the fuzz test is run all of the individual crashes and the results that the fuzzer found are going to be reported in our security tab.

A

So you can easily view and triage your most critical faults that happened and get more information about them. If we click on one, we can see again that that stack trace that I highlighted it's directly here in the vulnerability, so you can immediately read it. We have some additional metadata about what type of crash it was and some more information about what type of fuzzer specifically was used to find it.

A

And so, if we go back to our repo and we look at our ci file again just to recap, we went through a few different things. As part of today's video. We talked about how you would add coverage, guided fuzz testing to a job to a repo in get lab, the first step being that you would create a fuzz test harness which tells the fuzzer how to run your application lens testing it.

A

You configure your ci yaml file to do two things, one which would include the coverage, guided, fuzz testing, template and then secondarily defining how to build that fuzz test, harness that you created previously, and that is a very quick guide on how you would add coverage guide. Fuzz testing to your own applications hosted in gitlab.

A

If you want more information, our gitlab product documentation is always available. We'd love to hear from you on an issue. If you want to create an issue, you can ping the fuzzing team, or you can ping myself again, I'm sam kerr, my handle is at st curve and we would love to hear about your experience with fuzz testing, whether it's good, bad or ugly, we'd love to hear more hear what you're doing with it. Thank you very much for your time appreciate it.